DYNAMIC CONVOLUTIONS TO REFINE IMAGES WITH VARIATIONAL DEGRADATION

A system stores parameters of a feature extraction network and a refinement network. The system receives an input including a degraded image concatenated with a degradation estimation of the degraded image; performs operations of the feature extraction network to apply pre-trained weights to the input to generate feature maps; and performs operations of the refinement network including a sequence of dynamic blocks. One or more of the dynamic blocks dynamically generates per-grid kernels to be applied to corresponding grids of an intermediate image output from a prior dynamic block in the sequence. Each per-grid kernel is generated based on the intermediate image and the feature maps.

TECHNICAL FIELD

Embodiments of the invention relate to neural network operations for image quality enhancement.

BACKGROUND

Deep Convolutional Neural Networks (CNNs) have been widely adopted for image processing such as image refinement and super-resolution. The CNNs have been used to restore an image degraded by blur, noise, low resolution, and the like. The CNNs have been shown to be effective in solving single image super-resolution (SISR) problems, where a high-resolution (HR) image is reconstructed from a low-resolution (LR) image.

Some CNN-based methods have the assumption that a degraded image is subject to one fixed combination of degrading effects, e.g., blurring and bicubic down-sampling. These methods have limited capability in handling images where the degrading effects vary from one image to another. These methods also cannot handle an image that has one combination of degrading effects in one region and another combination of degrading effects in another region of the same image.

Another approach is to train an individual network for each combination of degrading effects. For example, if an image is degraded by three different combinations of degrading effects: bicubic down-sampling, bicubic down-sampling and noise, and direct down-sampling and blurring, three networks are trained to handle these degradations.

Therefore, there is a need for improving the existing methods for refining an image that is subject to variational degradation effects.

SUMMARY

In one embodiment, a method is provided for image refinement. The method includes the steps of: receiving an input including a degraded image concatenated with a degradation estimation of the degraded image; performing feature extraction operations to apply pre-trained weights to the input to generate feature maps; and performing operations of a refinement network that includes a sequence of dynamic blocks. One or more of the dynamic blocks dynamically generates per-grid kernels to be applied to corresponding grids of an intermediate image output from a prior dynamic block in the sequence. Each per-grid kernel is generated based on the intermediate image and the feature maps.

In another embodiment, a system includes memory to store parameters of a feature extraction network and a refinement network. The system further includes processing hardware coupled to the memory. The processing hardware is operative to: receive an input including a degraded image concatenated with a degradation estimation of the degraded image; perform operations of the feature extraction network to apply pre-trained weights to the input to generate feature maps; and perform operations of the refinement network that includes a sequence of dynamic blocks. One or more of the dynamic blocks dynamically generates per-grid kernels to be applied to corresponding grids of an intermediate image output from a prior dynamic block in the sequence. Each per-grid kernel is generated based on the intermediate image and the feature maps.

Other aspects and features will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments in conjunction with the accompanying figures.

DETAILED DESCRIPTION

Embodiments of the invention provide a framework of a Unified Dynamic Convolutional Network for Variational Degradation (UDVD). The UDVD performs single image super-resolution (SISR) operations for a wide range of variational degradation. Furthermore, the UDVD can also restore image quality from blurring and noise degradation. The variational degradation can occur inter-image and/or intra-image. Inter-image variational degradation is also known as cross-image variational degradation. For example, a first image may be low resolution and blurred, and a second image may be noisy. Intra-image variational degradation is degradation with spatial variations in an image. For example, one region in an image may be blurred and another region in the same image may be noisy. The UDVD can be trained to enhance the quality of images that suffer from inter-image and/or intra-image variational degradation. The UDVD incorporates dynamic convolution, which provides more flexibility in handling different degradation variations than standard convolution. In SISR with a non-blind setting, the UDVD has demonstrated the effectiveness on both synthetic and real images.

Dynamic convolutions have been an active area in neural network research. Brabandere et al. “Dynamic filter networks,” in Proc. Conf. Neural Information Processing Systems (NIPS) 2016, describes a dynamic filter network that dynamically generates filters conditioned on an input. Dynamic filter networks are adaptive to input content and therefore offers increased flexibility.

The UDVD generates dynamic kernels based on the concept of dynamic filter networks with modifications. The dynamic kernels disclosed herein adapt to not only image contents but also diverse variations of degrading effects. The dynamic kernels are effective in handling inter-image and intra-image variational degradation.

The standard convolution uses kernels that are learned from training. Each kernel is applied to all pixel locations. In contrast, the dynamic convolution disclosed herein uses per-grid kernels that are generated by a parameter-generating network. Moreover, the kernels of standard convolution are content-agnostic which are fixed after training is completed. In contrast, the dynamic convolution kernels are content-adaptive and can adapt to different inputs during inference. Due to these properties, the dynamic convolution is a better alternative to the standard convolution in handling variational degradation.

In the following description, two types of dynamic convolutions are disclosed. Moreover, multistage losses are integrated to gradually refine images throughout consecutive dynamic convolutions. Extensive experiments show that the UDVD achieves favorable or comparable performance on both synthetic and real images.

In a practical use case, degrading effects such as blurring, noise, and down-sampling can simultaneously occur. The degradation process is formulated as:

where IHRand ILRrepresent high resolution (HR) and low resolution (LR) images, respectively, k represents a blur kernel, n represents additive noise. Equation (1) indicates that the LR image is equal to the HR image convolved with a blur kernel, downsampled with a scale factor s, and plus noise. An example of the blur kernel is the Isotropic Gaussian blur kernel. An example of additive noise is the additive white Gaussian noise (AWGN) with covariance (noise level). An example of downsampling is the bicubic downsampler. Other degradation operators may also be used to synthesize realistic degradations for SISR training. For real images, a search on degradation parameters is performed area by area to obtain visually satisfying results. In this disclosure, a non-blind setting is adopted. Any degradation estimation methods can be prepended to extend the disclosed method to a blind setting.

FIG.1is a diagram illustrating a UDVD framework100according to one embodiment. The framework100includes a feature extraction network110and a refinement network120. The feature extraction network110operates to extract high-level features of a low-resolution input image (also referred to as a degraded image). The degraded image may contain variational degradation. The refinement network120learns to enhance and up-sample the degraded image based on the extracted high-level features. The output of the refinement network120is a high-resolution image.

The degraded image (denoted as I0) is concatenated with a degradation map (D). The degradation map D, also referred to as a degradation estimation, may be generated based on known degradation parameters in the degraded image; e.g., a known blur kernel and a known noise level σ. For example, the blur kernel may be projected to a t-dimensional vector by using the principal component analysis (PCA) technique. An extra dimension of noise level σ is concatenated to the t-dimensional vector to obtain a (1+t) vector. The (1+t) vector is then stretched to get a degradation map D of size (1+t)×H×w.

The feature extraction network110includes an input convolution111and N residual blocks112. The input convolution111is performed on the degraded image (I0) concatenated with the degradation map (D). The convolution result is sent to the N residual blocks112, and is added to the output of the N residual blocks112to generate feature maps (F).

FIG.2illustrates an example of the residual block112according to one embodiment. Each residual block112performs operations of convolutions210, rectified linear units (ReLU)220, and convolutions230. The output of the residual block112is the pixel-wise sum of the input to the residual block112and the output of the convolutions230. As a non-limiting example, the kernel size of each convolution layer may be set to 3×3, and the number of channels may be set to 128.

The refinement network120includes a sequence of M dynamic blocks123to perform feature transformation. Each dynamic block123receives the feature maps (F) as one input. In one embodiment, the dynamic block123is extended to perform upsampling with an upsampling rate r. Each dynamic block123can learn to upsample and reconstruct the variationally degraded image.

FIG.3is a block diagram illustrating the dynamic block123according to one embodiment. It is understood that the dimensions of the kernels and the channels described below are non-limiting. Each dynamic block m receives the feature maps (F) and an image Im-1as input (m=1, . . . , M). For the first dynamic block in the sequence of M dynamic blocks, the image Im-1is the degraded image (I0) at the input of the framework100. For the subsequent dynamic blocks in the sequence of M dynamic blocks, the image Im-1is an intermediate image output from the prior dynamic block in the sequence. In the example of a dynamic block m, the image Im-1is sent to CONV*3320, which includes three 3×3 convolution layers with 16, 16, and 32 channels, respectively. The feature maps (F) from the feature extraction network110may optionally go through the operations of pixel shuffle310. The output of the pixel shuffle310and the CONV*3320are concatenated and then forwarded to two paths.

Each dynamic block123includes a first path and a second path. The first path predicts dynamic kernels350and then performs dynamic convolution by applying the dynamic kernels350to the image Im-1. The dynamic convolution can be regular or upsampling. An example of the different types of dynamic convolutions is provided in connection withFIG.4. Different dynamic blocks123may perform different types of dynamic convolutions. The second path generates a residual image for enhancing high-frequency details by using standard convolutions. The output of the first path and the output of the second path are combined by pixel-wise additions.

InFIG.3, the lower portion indicated by double lines illustrated the first path. The first path includes a 3×3 convolution layer340to predict and generate the dynamic kernels350. The generated dynamic kernels350are then applied to Im-1to perform dynamic convolutions to generate an output Om. In one embodiment, each dynamic kernel350is a per-grid kernel. The per-grid kernels350are to be applied to corresponding grids of Im-1(m=1, . . . , M). Each per-grid kernel m is generated based on Im-1and the feature maps F. Each corresponding grid contains one or more image pixels sharing and using the same per-grid kernel.

The second path contains two 3×3 convolution layers (shown as CONV*2330) with 16 and 3 channels, respectively, to generate a residual image Rmfor enhancing high-frequency details. The residual image Rmis then added to the output of dynamic convolution Omto generate an image Im. A sub-pixel convolution layer may be used to align the resolutions between the two paths.

FIG.4illustrates two types of dynamic convolutions according to some embodiments. The first type is the regular dynamic convolution, which is used when input resolution is the same as output resolution. The second type is the dynamic convolution with upsampling, which integrates upsampling into the dynamic convolution. Referring to the example inFIG.3, the dynamic kernels350may be for regular dynamic convolutions or dynamic convolutions with upsampling. For regular dynamic convolutions, the dynamic kernels350may be stored in a tensor with (k×k) in channel dimension, where (k×k) is the kernel size for the dynamic kernels350. A dynamic kernel350with up-sampling integrated may be stored in a tensor with (k×k×r×r) in channel dimension, where r is upsampling rate. The refinement network120may include one upsampling dynamic block in the sequence of M dynamic blocks123to produce an upsampled image such as upsampled image410inFIG.4. This upsampling dynamic block can be placed at the first, the last, or anywhere in the sequence of M dynamic blocks. In one embodiment, the upsampling dynamic block is placed as the first block in the sequence. The upsampling dynamic block generates an upsampling dynamic kernel with the channel dimension expanded by r×r; equivalently, this dynamic block generates (r×r) dynamic kernels with each kernel size=k×k. Each of the other dynamic blocks in the sequence of M dynamic blocks123may generate a regular dynamic kernel with kernel size=k×k. All of the M dynamic blocks123in combination perform super-resolution operations in addition to other image refinement operations such as de-noising and de-blurring.

In a regular dynamic convolution, convolutions are conducted by using dynamic kernels K of kernel size k×k. Such operation can be expressed as:

where Iinand Ioutrepresent input and output image, respectively, i and j are the coordinates in an image, u and v are the coordinates in each Ki,j. Note that Δ=floor (k/2). Applying these dynamic kernels is equivalent to computing a weighted sum over nearby pixels to enhance the image quality; different kernels are applied to different grids of the image. In a default setting, there are H×W kernels and the corresponding weights are shared across channels. By introducing an additional dimension C with Equation (2), dynamic convolution can be extended for independent weights across channels.

In a dynamic convolution with upsampling, r×r convolutions are performed on the same corresponding patch to create r×r new pixels, where the patch is the area to which the dynamic kernel is applied. The mathematical form of such operation is defined as:

where x and y are in the coordination of each r×r output block (0≤x; y≤r−1). Here, the resolution of Ioutis r times the resolution of lin. A total of r2HW kernels are used to generate rH×rW pixels as Iout. When performing the dynamic convolution with upsampling, the weights may be shared across channels to avoid excessively high dimensionality.

FIG.5is a diagram illustrating multistage loss computations according to one embodiment. A multistage loss is computed at the outputs of dynamic blocks. The losses are calculated as a difference metric between the HR image (IHR) and Imat the output of each dynamic blocks123. When a ground truth image is available, the difference metric measures the difference between the ground truth image and the output of the dynamic block. The loss is computed as:

where M is the number of dynamic blocks123and F is loss function such as L2 loss or perceptual loss. To obtain a high-quality resultant image, the sum of losses from each dynamic block123is minimized. The sum of losses is used to update the convolution weights in each dynamic block123.

FIG.6is a flow diagram illustrating a method600for image refinement according to one embodiment. The method600may be performed by a computer system; e.g., a system700inFIG.7. The method600begins at step610when the system receives an input including a degraded image concatenated with a degradation estimation of the degraded image. At step620, the system performs feature extraction operations to apply pre-trained weights to the input to generate feature maps. At step630, the system performs operations of a refinement network that includes a sequence of dynamic blocks. One or more of the dynamic blocks dynamically generates per-grid kernels to be applied to corresponding grids of an intermediate image output from a prior dynamic block in the sequence. Each per-grid kernel is generated based on the intermediate image and the feature maps.

FIG.7is a block diagram illustrating a system700operative to perform image refinement operations including dynamic convolutions according to one embodiment. The system700includes processing hardware710which further includes one or more processors730such as central processing units (CPUs), graphics processing units (GPUs), digital processing units (DSPs), field-programmable gate arrays (FPGAs), and other general-purpose processors and/or special-purpose processors. In one embodiment, the processing hardware710includes a neural processing unit (NPU)735to perform neural network operations. The processing hardware710such as the NPU735or other dedicated neural network circuits are operative to perform neural network operations including, but not limited to: convolution, deconvolution, ReLU operations, fully-connected operations, normalization, activation, pooling, resizing, upsampling, element-wise arithmetic, concatenation, etc.

The processing hardware710is coupled to a memory720, which may include memory devices such as dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, and other non-transitory machine-readable storage media; e.g., volatile or non-volatile memory devices. To simplify the illustration, the memory720is represented as one block; however, it is understood that the memory720may represent a hierarchy of memory components such as cache memory, system memory, solid-state or magnetic storage devices, etc. The processing hardware710executes instructions stored in the memory720to perform operating system functionalities and run user applications. For example, the memory720may store framework parameters725, which are the trained parameters of the framework100(FIG.1) such as the kernel weights of the CNN layers in the framework100.

In some embodiments, the memory720may store instructions which, when executed by the processing hardware710, cause the processing hardware710to perform image refinement operations according to the method600inFIG.6.

The operations of the flow diagram ofFIG.6have been described with reference to the exemplary embodiment ofFIG.7. However, it should be understood that the operations of the flow diagram ofFIG.6can be performed by embodiments of the invention other than the embodiment ofFIG.7and the embodiment ofFIG.7can perform operations different than those discussed with reference to the flow diagram. While the flow diagram ofFIG.6shows a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).