Functional Neural Networks for Parametric Image Restoration Problems

1 Introduction . Image restoration [ 40 ] is a classical yet still active topic in low-level computer vision , which estimates the original image from a degraded measurement . For example , single image super-resolution [ 16 ] estimates the high-resolution image from a downsampled one , image denoising [ 4 ] estimates the clean image from a noisy one , and JPEG deblocking [ 7 ] estimates the original image from a compressed one . It is a challenging ill-posed inverse problem which aims to recover the information lost to the image degradation process [ 3 ] , and it is also important since it is an essential step in various image processing and computer vision applications [ 71 , 58 , 2 , 33 , 46 , 31 , 20 ] . Almost every single image restoration problem has a closely related parameter , such as the scale factor in super-resolution , the noise level in image denoising , and the quality factor in JPEG deblocking . The parameter in an image restoration problem tends to have a strong connection with the image degradation process . In the super-resolution problem , the blur kernel of the downsampling process is determined by the scale factor [ 62 ] . In the image denoising problem , the standard deviation of the additive white Gaussian noise is determined by the noise level [ 11 ] . In the JPEG deblocking problem , the quantization table for DCT coefficients is determined by the quality factor [ 42 ] . When we try to restore the clean image from a corrupt one , we might know the value of the corresponding parameter for various reasons . In the super-resolution problem , the scale factor is specified by users [ 28 ] . In the image denoising problem , the noise level could be measured by other devices [ 43 , 37 ] . In the ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . JPEG deblocking problem , the quality factor could be derived from the header of the JPEG file [ 10 ] . Therefore , it is very important to use the known parameter well in such a parametric image restoration problem . Recently , deep convolutional neural network based methods are widely used to tackle the image restoration tasks , including super-resolution [ 14 , 28 , 45 , 34 , 47 , 50 , 68 , 67 , 12 , 41 ] , image denoising [ 25 , 5 , 64 , 32 , 48 , 63 , 65 , 60 , 1 ] , and JPEG deblocking [ 36 , 13 , 55 , 18 , 6 , 17 , 35 , 66 , 15 ] . They have achieved significant improvements over conventional image restoration methods due to their powerful learning ability . However , they have not been paying attention to the parameter involved in an image restoration problem , and handled it in an unsophisticated way . Most previous researchers either treat problems with different parameter levels as independent tasks , and train a specific model for each parameter level [ 14 , 45 , 67 , 63 , 66 ] ; or simply ignore the parameter , and train a single model for all parameter levels [ 28 , 60 ] . The two popular approaches have their own shortcomings . The former is inefficient in computing , because they may have to train and store dozens of models for different parameter levels . The latter is ineffective in performance , since they ignore some important information that could have helped the restoration process . To overcome these weaknesses , we propose a novel system called functional neural network ( FuncNet ) to solve a parametric image restoration problem with a single model . The difference between a plain neural network and our FuncNet is shown in Figure 1 . Unlike a plain neural network , the smallest conceptual element of our FuncNet is no longer a floating-point weight , but a function of the parameter of the problem . When we train a FuncNet , we gradually change these functions to reduce the value of the loss function . When we use the trained FuncNet to restore a degraded image with a given parameter , we first evaluate those functions with the parameter , use the evaluation values as weights of the network , and then do inference as normal . This feature makes it both efficient and effective for a parametric problem . By this way , we neatly blend the parameter information into a neural network model , use it to help the restoration process , and only increase a negligible amount of computation as we will demonstrate later . We apply FuncNet to super-resolution , image denoising , and JPEG deblocking . The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts . The remainder of the paper is organized as follows . Section 2 provides a brief survey of related work . Section 3 presents the proposed FuncNet model , discusses the details of implementation , and analyses its storage and computational efficiency . In Section 4 , extensive experiments are conducted to evaluate FuncNets on three parametric image restoration tasks . Section 5 concludes the paper . 2 Related Work . Neural networks for parametric problems . To the best of our knowledge , there are seven ways to solve a parametric problem with neural network based methods . We list them below roughly in the order of popularity , and discuss their advantages and disadvantages . The first method treats problems with different parameter levels as independent tasks , and trains a specific model for each parameter level [ 14 , 45 , 67 , 63 , 66 ] . The overwhelming majority of previous papers use this approach . This method is easy to understand , and generally has good performance since the parameter level is implied in a model . But it is inefficient in computing , because we may have to train and store dozens of models for different parameter levels . The second method simply ignores the parameter , and trains a single model for all parameter levels [ 28 , 60 ] . This method is also easy to understand , and is very efficient since we only need to train and store a single model . But its performance is typically lower than the first method , since we ignore some important information that could have helped the restoration process . The third method trains a model with a shared backbone for all parameter levels and multiple specific blocks for each parameter level [ 34 ] . It is a compromise between the first and the second method , and has acceptable performance and efficiency . But we may still have to store dozens of specific blocks for different parameter levels . And if the capacity of a specific block is not large enough , the block can not take full advantage of the parameter information . The fourth method converts the parameter scalar into a parameter map , and treats the map as an additional channel of the input degraded image [ 65 ] . It is another way to blend the parameter information into a neural network model . However , the performance of this method is only marginally higher than the second method , and it is still not as good as the first method . Due to the huge semantic difference between the corrupt image and the parameter map , it is hard to make much use of the parameter information for the model . The fifth method conditions a network by modulating all its intermediate features by scalar parameters [ 9 , 21 ] or maps [ 52 ] . It is another way to make a network adapt to different situations . But unlike our FuncNet , the method changes only features rather than parameters . The sixth method trains a model with a relatively shallow backbone network , and each filter of the backbone network is generated by a different filter generating network [ 30 , 26 , 27 , 15 ] . The filter generating networks are usually multilayer perceptrons , and they take the parameter as input . Since the total size of a model is limited , assigning each filter a unique complex network severely limits the size of the backbone network . Such a shallow backbone network only leads to a mediocre performance . Considering the universal approximation ability of the multilayer perceptron , this method is not that different from training a unique shallow model for each parameter level . The seventh method searches in the latent space of a generative model , and returns the most probable result which is not contradicting the degradation model with the parameter [ 57 , 38 ] . It is a general image restoration method which can solve various image restoration problems with a single model , as long as the degradation model is continuously differentiable . However , this method is slower than a feedforward neural network based method , since it requires multiple forward and backward passes in a search . And due to the limited representation ability of the generative model , the performance of this method is also worse than a discriminative model based method . Neural network interpolation . In order to attain a continuous transition between different imagery effects , the neural network interpolation method [ 54 , 53 , 51 ] applies linear interpolation in the parameter space of two trained networks . Although at first glance it is similar to our method , there is a big difference between network interpolation and our FuncNet . The former is a simple interpolation technique while the latter is a regression technique . In the network interpolation method , two CNNs are trained separately for two extreme cases , and then blended in an ad hoc way . This may suffice for tasks [ 54 ] and [ 53 ] , because users will accept roughly characterized visual results , such as `` half GAN half MSE '' or `` half photo half painting '' . However , this is not good enough for an image restoration task whose goal is to restore the signal as accurately as possible . FuncNet is optimized for the entire value range of the task parameter ( e.g. , the noise level , SR scale factor ) , so its accuracy stays high over the entire parameter range , rather than just for the two extreme points like in the network interpolation method . Deep CNN for Single Image Super-Resolution . The first convolutional neural network for single image super-resolution is proposed by Dong et al . [ 14 ] called SRCNN , and it achieved superior performance against previous works . Shi et al . [ 45 ] firstly proposed a real-time super-resolution algorithm ESPCN by proposing the sub-pixel convolution layer . Lim et al . [ 34 ] removed batch normalization layers in the residual blocks , and greatly improved the SR effect . Zhang et al . [ 67 ] introduced the residual channel attention to the SR framework . Hu et al . [ 23 ] proposed the MetaUpscale Module to replace the traditional upscale module . Deep CNN for Image Denoising . Zhang et al . [ 63 ] proposed DnCNN , a plain denoising CNN method which achieves state-of-the-art denoising performance . They showed that residual learning and batch normalization are particularly useful for the success of denoising . Tai et al . [ 48 ] proposed MemNet , a very deep persistent memory network by introducing a memory block to mine persistent memory through an adaptive learning process . Zhang et al . [ 65 ] proposed FFDNet , a fast and flexible denoising convolutional neural network , with a tunable noise level map as the input . Deep CNN for JPEG deblocking . Dong et al . [ 13 ] proposed ARCNN , a compact and efficient network for seamless attenuation of different compression artifacts . Guo and Chao [ 18 ] proposed a highly accurate approach to remove artifacts of JPEG-compressed images , which jointly learned a very deep convolutional network in both DCT and pixel domains . Zhang et al . [ 66 ] proposed DMCNN , a Dual-domain Multi-scale CNN to take full advantage of redundancies on both the pixel and DCT domains . Ehrlich et al . [ 15 ] proposed a novel architecture which is parameterized by the JPEG files quantization matrix .

Summary: This paper proposes a network interpolation scheme for non-blind image restoration problems with a single parameter (e.g. noise level for denoising). The parameter is remapped to [0, 1] using an affine mapping.  Effectively, two networks are trained such that for any given problem parameter in [0,1], an interpolated network is obtained by linearly blending the two networks  parameters. The interpolated model is then applied to the input.