Functional Neural Networks for Parametric Image Restoration Problems

1 Introduction . Image restoration [ 40 ] is a classical yet still active topic in low-level computer vision , which estimates the original image from a degraded measurement . For example , single image super-resolution [ 16 ] estimates the high-resolution image from a downsampled one , image denoising [ 4 ] estimates the clean image from a noisy one , and JPEG deblocking [ 7 ] estimates the original image from a compressed one . It is a challenging ill-posed inverse problem which aims to recover the information lost to the image degradation process [ 3 ] , and it is also important since it is an essential step in various image processing and computer vision applications [ 71 , 58 , 2 , 33 , 46 , 31 , 20 ] . Almost every single image restoration problem has a closely related parameter , such as the scale factor in super-resolution , the noise level in image denoising , and the quality factor in JPEG deblocking . The parameter in an image restoration problem tends to have a strong connection with the image degradation process . In the super-resolution problem , the blur kernel of the downsampling process is determined by the scale factor [ 62 ] . In the image denoising problem , the standard deviation of the additive white Gaussian noise is determined by the noise level [ 11 ] . In the JPEG deblocking problem , the quantization table for DCT coefficients is determined by the quality factor [ 42 ] . When we try to restore the clean image from a corrupt one , we might know the value of the corresponding parameter for various reasons . In the super-resolution problem , the scale factor is specified by users [ 28 ] . In the image denoising problem , the noise level could be measured by other devices [ 43 , 37 ] . In the ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . JPEG deblocking problem , the quality factor could be derived from the header of the JPEG file [ 10 ] . Therefore , it is very important to use the known parameter well in such a parametric image restoration problem . Recently , deep convolutional neural network based methods are widely used to tackle the image restoration tasks , including super-resolution [ 14 , 28 , 45 , 34 , 47 , 50 , 68 , 67 , 12 , 41 ] , image denoising [ 25 , 5 , 64 , 32 , 48 , 63 , 65 , 60 , 1 ] , and JPEG deblocking [ 36 , 13 , 55 , 18 , 6 , 17 , 35 , 66 , 15 ] . They have achieved significant improvements over conventional image restoration methods due to their powerful learning ability . However , they have not been paying attention to the parameter involved in an image restoration problem , and handled it in an unsophisticated way . Most previous researchers either treat problems with different parameter levels as independent tasks , and train a specific model for each parameter level [ 14 , 45 , 67 , 63 , 66 ] ; or simply ignore the parameter , and train a single model for all parameter levels [ 28 , 60 ] . The two popular approaches have their own shortcomings . The former is inefficient in computing , because they may have to train and store dozens of models for different parameter levels . The latter is ineffective in performance , since they ignore some important information that could have helped the restoration process . To overcome these weaknesses , we propose a novel system called functional neural network ( FuncNet ) to solve a parametric image restoration problem with a single model . The difference between a plain neural network and our FuncNet is shown in Figure 1 . Unlike a plain neural network , the smallest conceptual element of our FuncNet is no longer a floating-point weight , but a function of the parameter of the problem . When we train a FuncNet , we gradually change these functions to reduce the value of the loss function . When we use the trained FuncNet to restore a degraded image with a given parameter , we first evaluate those functions with the parameter , use the evaluation values as weights of the network , and then do inference as normal . This feature makes it both efficient and effective for a parametric problem . By this way , we neatly blend the parameter information into a neural network model , use it to help the restoration process , and only increase a negligible amount of computation as we will demonstrate later . We apply FuncNet to super-resolution , image denoising , and JPEG deblocking . The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts . The remainder of the paper is organized as follows . Section 2 provides a brief survey of related work . Section 3 presents the proposed FuncNet model , discusses the details of implementation , and analyses its storage and computational efficiency . In Section 4 , extensive experiments are conducted to evaluate FuncNets on three parametric image restoration tasks . Section 5 concludes the paper . 2 Related Work . Neural networks for parametric problems . To the best of our knowledge , there are seven ways to solve a parametric problem with neural network based methods . We list them below roughly in the order of popularity , and discuss their advantages and disadvantages . The first method treats problems with different parameter levels as independent tasks , and trains a specific model for each parameter level [ 14 , 45 , 67 , 63 , 66 ] . The overwhelming majority of previous papers use this approach . This method is easy to understand , and generally has good performance since the parameter level is implied in a model . But it is inefficient in computing , because we may have to train and store dozens of models for different parameter levels . The second method simply ignores the parameter , and trains a single model for all parameter levels [ 28 , 60 ] . This method is also easy to understand , and is very efficient since we only need to train and store a single model . But its performance is typically lower than the first method , since we ignore some important information that could have helped the restoration process . The third method trains a model with a shared backbone for all parameter levels and multiple specific blocks for each parameter level [ 34 ] . It is a compromise between the first and the second method , and has acceptable performance and efficiency . But we may still have to store dozens of specific blocks for different parameter levels . And if the capacity of a specific block is not large enough , the block can not take full advantage of the parameter information . The fourth method converts the parameter scalar into a parameter map , and treats the map as an additional channel of the input degraded image [ 65 ] . It is another way to blend the parameter information into a neural network model . However , the performance of this method is only marginally higher than the second method , and it is still not as good as the first method . Due to the huge semantic difference between the corrupt image and the parameter map , it is hard to make much use of the parameter information for the model . The fifth method conditions a network by modulating all its intermediate features by scalar parameters [ 9 , 21 ] or maps [ 52 ] . It is another way to make a network adapt to different situations . But unlike our FuncNet , the method changes only features rather than parameters . The sixth method trains a model with a relatively shallow backbone network , and each filter of the backbone network is generated by a different filter generating network [ 30 , 26 , 27 , 15 ] . The filter generating networks are usually multilayer perceptrons , and they take the parameter as input . Since the total size of a model is limited , assigning each filter a unique complex network severely limits the size of the backbone network . Such a shallow backbone network only leads to a mediocre performance . Considering the universal approximation ability of the multilayer perceptron , this method is not that different from training a unique shallow model for each parameter level . The seventh method searches in the latent space of a generative model , and returns the most probable result which is not contradicting the degradation model with the parameter [ 57 , 38 ] . It is a general image restoration method which can solve various image restoration problems with a single model , as long as the degradation model is continuously differentiable . However , this method is slower than a feedforward neural network based method , since it requires multiple forward and backward passes in a search . And due to the limited representation ability of the generative model , the performance of this method is also worse than a discriminative model based method . Neural network interpolation . In order to attain a continuous transition between different imagery effects , the neural network interpolation method [ 54 , 53 , 51 ] applies linear interpolation in the parameter space of two trained networks . Although at first glance it is similar to our method , there is a big difference between network interpolation and our FuncNet . The former is a simple interpolation technique while the latter is a regression technique . In the network interpolation method , two CNNs are trained separately for two extreme cases , and then blended in an ad hoc way . This may suffice for tasks [ 54 ] and [ 53 ] , because users will accept roughly characterized visual results , such as `` half GAN half MSE '' or `` half photo half painting '' . However , this is not good enough for an image restoration task whose goal is to restore the signal as accurately as possible . FuncNet is optimized for the entire value range of the task parameter ( e.g. , the noise level , SR scale factor ) , so its accuracy stays high over the entire parameter range , rather than just for the two extreme points like in the network interpolation method . Deep CNN for Single Image Super-Resolution . The first convolutional neural network for single image super-resolution is proposed by Dong et al . [ 14 ] called SRCNN , and it achieved superior performance against previous works . Shi et al . [ 45 ] firstly proposed a real-time super-resolution algorithm ESPCN by proposing the sub-pixel convolution layer . Lim et al . [ 34 ] removed batch normalization layers in the residual blocks , and greatly improved the SR effect . Zhang et al . [ 67 ] introduced the residual channel attention to the SR framework . Hu et al . [ 23 ] proposed the MetaUpscale Module to replace the traditional upscale module . Deep CNN for Image Denoising . Zhang et al . [ 63 ] proposed DnCNN , a plain denoising CNN method which achieves state-of-the-art denoising performance . They showed that residual learning and batch normalization are particularly useful for the success of denoising . Tai et al . [ 48 ] proposed MemNet , a very deep persistent memory network by introducing a memory block to mine persistent memory through an adaptive learning process . Zhang et al . [ 65 ] proposed FFDNet , a fast and flexible denoising convolutional neural network , with a tunable noise level map as the input . Deep CNN for JPEG deblocking . Dong et al . [ 13 ] proposed ARCNN , a compact and efficient network for seamless attenuation of different compression artifacts . Guo and Chao [ 18 ] proposed a highly accurate approach to remove artifacts of JPEG-compressed images , which jointly learned a very deep convolutional network in both DCT and pixel domains . Zhang et al . [ 66 ] proposed DMCNN , a Dual-domain Multi-scale CNN to take full advantage of redundancies on both the pixel and DCT domains . Ehrlich et al . [ 15 ] proposed a novel architecture which is parameterized by the JPEG files quantization matrix .

Summary: The authors propose a technique to build image restoration networks that are robust to variations in the problem parameters (e.g. noise level or upscaling factor). The main idea is to derive the weights of the network as a learnable function of the problem parameter.