Deep Learning for Image Super-Resolution
By: Nabil Madali
Tags: Machine Learning, Convolutional Network, Image Processing, Super Resolution

The image acquisition process, due to the limitation of the imaging distance, the resolution of the imaging device, and other factors, it is difficult for the imaging system to obtain the information in the original scene without distortion.

Imaging systems are usually affected by many factors such as distortion, blurring, downsampling, noise, etc., resulting in a decrease in the quality of the acquired image. Therefore, how to improve the spatial resolution of the image and improve the image quality has always been an urgent need to solve in the field of imaging technology.

According to Fourier Optics, the optical imaging system is equivalent to a low-pass filter. Due to the influence of optical diffraction, the system response above a certain cut-off frequency determined by the diffraction limit resolution is zero. The highest resolution that the system can achieve is first determined by the imaging optics itself. Traditional image restoration techniques can alleviate the influence of various interference factors in the imaging process to a certain extent and improve the quality of the image.

However, these methods are difficult to restore the system Information beyond the cut-off frequency . For this reason, the concept of Super resolution restoration (SR) is proposed. The basic idea of ​​super-resolution restoration technology is to use signal processing methods to reconstruct the imaging system beyond the cut-off frequency. information, i.e., the recovery process of the imaging lost high frequency information, obtaining higher resolution images of the imaging system

There are various super-resolution models for deep learning. These models rely on supervised super-resolution, that is, training with LR images and corresponding ground truth (GT) HR images. Although the differences between these models are very large, they are essentially a combination of a set of components, such as model frameworks, upsampling methods, network design, and learning strategies. From this perspective, researchers combine these components to build an integrated SR model for fitting specific tasks.

Since image super-resolution is an ill-conditioned problem, how to perform upsampling is a key issue. Based on the upsampling operation used and its position in the model, the SR model can be attributed to four model frameworks:

Pre-sampling SR:This method first interpolates the low-resolution image to obtain a coarse high-resolution image .

Source[1]

Post-upsampling SR : Low-resolution images are passed to CNNs. Upsampling is performed in the last layer using a learnable layer.

Source[1]

Move the upsampling operation to the end of the network and learn the mapping in a low-dimensional space.

Progressive upsampling SR : In the above group, although the computational complexity is reduced, only one upsampling convolution is used. This makes the learning process of large-scale scaling more difficult . In order to solve this shortcoming, Laplacian Pyramid SR Network and progressive SR adopt a progressive upsampling framework. In this case, the model uses a cascaded neural network to gradually reconstruct a high-resolution image at each step on a smaller scale.

Source[1]

Iterative upsampling SR:Another popular model architecture is the hourglass (or U-Net) structure . Some variants, such as the Stacked Hourglass network, use several consecutive hourglass structures, effectively alternating between upsampling and downsampling processes.

Source[1]

In addition to the upsampling position in the model, how to perform upsampling is also very important . Although there are many traditional upsampling methods, using CNN to learn end-to-end upsampling has gradually become a trend.

The transposed convolutional layer: The deconvolutional layer, predicts possible inputs based on feature maps whose size is similar to the output of the convolutional layer.

Transposed convolution layer. The blue boxes denote the input, and the green boxes indicate the kernel and the convolution output.Source[1]

Specifically, it expands the image by inserting zero values ​​and performing convolution, thereby increasing the image resolution.

Since the transposed convolutional layer can enlarge the image size in an end-to-end manner while maintaining a connection mode compatible with vanilla convolution, it is widely used as an upsampling layer of the SR model.

The sub-pixel layer is also an up-sampling layer for end-to-end learning. Multiple channels are generated through convolution and then reshaped, as shown in the above figure.

Sub-pixel layer. First, convolution produces an output with s2 channels, where s is the upsampling factor (b). Assuming the input size is h×w×c, the output size is h×w×s2c. After that, perform a shuffle operation to produce an output © of size sh×sw×c. The size of the receptive field can reach 3×3.Source[1]

Due to the end-to-end upsampling method, the sub-pixel layer is also widely used by the SR model.

Compared with the transposed convolutional layer, the biggest advantage of the sub-pixel layer is that it has a larger perceptual field, provides more contextual information, and can help generate more accurate details. However, the distribution of the receptive field of the sub-pixel layer is not uniform, and the block area actually shares the same receptive field, which may cause some distortion near the block boundary.

The Meta upscale module:The previous method requires a pre-defined scaling factor, that is, training different up-sampling modules for different factors, which is inefficient and does not meet actual requirements.

Source[1]

The Meta upscale module is based on meta-learning to solve the SR of any scale factor. Specifically, for each target location on the HR image, this module projects it to a small block on the LR feature map (ie k × k × cin), and predicts the convolution based on the projection offset and scaling factor of the dense layer weight (ie, k×k×cin×cout) and perform convolution.

In this way, the Meta upscale module can continuously scale it up by any factor through a single model . And due to a large amount of training data , the module can show equivalent or even better performance on fixed factors.

However, this method predicts a large number of convolution weights for each target pixel based on multiple values ​​that are not related to the image content, so when facing a large magnification, the prediction result may be unstable and low in efficiency.

Global and local network structure design

In addition to the classic 2D convolution, some interesting variants can be used in the network to improve the results. Dilated convolution can provide more effective receptive fields, so long-distance dependent information can be used. Skip connections, Spatial Pyramid Pooling and Dense Blocks promote the combination of low-level features and high-level features to improve performance.

Residual learning : before ResNet was proposed, the idea of ​​learning residuals in the super-resolution field has appeared in many research work. Among them, global residual learning is just learning the residual between the image obtained after interpolation and the high-resolution image, and recovering high-frequency details by learning a residual image; while local residual learning is similar to the short connection in ResNet.

Source[1]

Recursive learning can greatly increase the receptive field of the network without introducing additional parameters. The method is to recursively use the same module multiple times.The problem of the multiplication factor is broken down into multiple sub-problems, and the recursive network sub-structure is used to solve it and so on. However, recursive learning is prone to the problems of gradient disappearance and explosion. It is necessary to incorporate strategies such as residual learning and multi-level supervision to alleviate these problems.

Source[1]

The main idea of ​​Multi-path learning is to design multiple branches for the network to improve the capacity and expressive ability of the model. It is divided into global multi-path learning and local multi-path learning, scale-specific multi-path learning, etc.

Source[1]

Dense connections are closely related to DenseNet. Dense connections are introduced into the image over-division problem, which can not only reduce the problem of gradient disappearance, but also reuse features to improve the effect. In the use of small growth rate , which can well control the amount of parameters, and has attracted more and more attention and use at present.

Source[1]

Channel re-scaling (channel attention) considers the relationship between different channels in the feature expression, and usually introduces some additional small structures to rescale by channel-wise.

Source[1]

Pyramid pooling usually uses multiple different scale parameters to aggregate global and local context information.

Source[1]

Common loss functions

The loss function is used to measure the difference between the generated high-resolution image and the ground truth high-resolution image. Then use this difference (error) to optimize the supervised learning model. There are several types of loss functions, each of which penalizes different aspects of the generated image.

Pixel loss, which mainly compares the pixel-level difference between two images, including L1 loss and L2 loss , but compared with L1 loss, L2 loss can penalize more Large errors, but have a strong tolerance for smaller errors. Recent studies have shown that L1 loss can achieve better performance and convergence speed.

where h, w and c are the height, width and number of channels of the evaluated images, respectively.

Content loss is mainly to evaluate the image quality from the level of image content understanding and perception, usually using pre-trained image recognition network to compare the feature maps of some middle layers .

The above function calculates the content loss between the ground-truth image and the generated image. Given the pre-trained network (Φ), and the output of the first layer, the network calculates the loss of both. This loss encourages the generated image to be similar in perception to ground-truth images.

Texture loss , the idea comes from the style transfer work, the reconstructed image should have the same style (color, texture, contrast, etc.) as the original image. Therefore, texture loss is also called style reconstruction loss, which is generally measured by the correlation of different feature channels. By using texture loss, the SR model can create real textures and produce more satisfying visual effects. Nevertheless, determining the patch size to match the texture is still a rule of thumb. A patch that is too small will cause artifacts in the texture area, while a patch that is too large will cause artifacts in the entire image, because the texture statistics are the average of different texture areas.

, the idea comes from the style transfer work, the reconstructed image should have the same style (color, texture, contrast, etc.) as the original image. Therefore, texture loss is also called style reconstruction loss, which is generally measured by the correlation of different feature channels. By using texture loss, the SR model can create real textures and produce more satisfying visual effects. Nevertheless, determining the patch size to match the texture is still a rule of thumb. A patch that is too small will cause artifacts in the texture area, while a patch that is too large will cause artifacts in the entire image, because the texture statistics are the average of different texture areas. Adversial loss , with the rise of GAN, the idea of ​​generators and discriminators in the adversarial generation network is introduced into the super division problem. The super division network is the generator, and another discriminator is defined to judge the input Whether the image is generated. In this loss function, the idea of ​​content loss can also be used for reference, and the discriminator uses the high-level expression of the image to make judgments.

Conclusion

Image super-resolution in real scenes is often limited by conditions such as unclear image degradation process and lack of LR-HR image pairs, making the existing SR algorithms difficult to practically apply.

Dealing with Various Degradation, to solve a variety of image degradation problems, for LR images obtained in different ways. There have been some work in this area, but there are some inherent shortcomings, such as the model is difficult to train, and the assumptions are too ideal.

Domain-specific Applications, applications in specific fields, SR algorithms do not necessarily have to be used in specific field data or scenes. SR algorithms can also assist in handling other visual problems, such as video surveillance, face recognition, target tracking, medical images, and scenes. Rendering etc. The SR algorithm can be used for pre-processing or post-processing of such visual problems.

References