Functional Neural Networks for Parametric Image Restoration Problems
Year: 2022

Abstract:
Almost every single image restoration problem has a closely related parameter, such as the scale factor in super-resolution, the noise level in image denoising, and the quality factor in JPEG deblocking. Although recent studies on image restoration problems have achieved great success due to the development of deep neural networks, they handle the parameter involved in an unsophisticated way. Most previous researchers either treat problems with different parameter levels as independent tasks, and train a specific model for each parameter level; or simply ignore the parameter, and train a single model for all parameter levels. The two popular approaches have their own shortcomings. The former is inefficient in computing and the latter is ineffective in performance. In this work, we propose a novel system called functional neural network (FuncNet) to solve a parametric image restoration problem with a single model. Unlike a plain neural network, the smallest conceptual element of our FuncNet is no longer a floating-point variable, but a function of the parameter of the problem. This feature makes it both efficient and effective for a parametric problem. We apply FuncNet to superresolution, image denoising, and JPEG deblocking. The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts.

heading: 1 Introduction
text: Image restoration [40] is a classical yet still active topic in low-level computer vision, which estimates the original image from a degraded measurement. For example, single image super-resolution [16] estimates the high-resolution image from a downsampled one, image denoising [4] estimates the clean image from a noisy one, and JPEG deblocking [7] estimates the original image from a compressed one. It is a challenging ill-posed inverse problem which aims to recover the information lost to the image degradation process [3], and it is also important since it is an essential step in various image processing and computer vision applications [71, 58, 2, 33, 46, 31, 20].
Almost every single image restoration problem has a closely related parameter, such as the scale factor in super-resolution, the noise level in image denoising, and the quality factor in JPEG deblocking. The parameter in an image restoration problem tends to have a strong connection with the image degradation process. In the super-resolution problem, the blur kernel of the downsampling process is determined by the scale factor [62]. In the image denoising problem, the standard deviation of the additive white Gaussian noise is determined by the noise level [11]. In the JPEG deblocking problem, the quantization table for DCT coefficients is determined by the quality factor [42]. When we try to restore the clean image from a corrupt one, we might know the value of the corresponding parameter for various reasons. In the super-resolution problem, the scale factor is specified by users [28]. In the image denoising problem, the noise level could be measured by other devices [43, 37]. In the
∗Corresponding author
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
JPEG deblocking problem, the quality factor could be derived from the header of the JPEG file [10]. Therefore, it is very important to use the known parameter well in such a parametric image restoration problem.
Recently, deep convolutional neural network based methods are widely used to tackle the image restoration tasks, including super-resolution [14, 28, 45, 34, 47, 50, 68, 67, 12, 41], image denoising [25, 5, 64, 32, 48, 63, 65, 60, 1], and JPEG deblocking [36, 13, 55, 18, 6, 17, 35, 66, 15]. They have achieved significant improvements over conventional image restoration methods due to their powerful learning ability. However, they have not been paying attention to the parameter involved in an image restoration problem, and handled it in an unsophisticated way. Most previous researchers either treat problems with different parameter levels as independent tasks, and train a specific model for each parameter level [14, 45, 67, 63, 66]; or simply ignore the parameter, and train a single model for all parameter levels [28, 60]. The two popular approaches have their own shortcomings. The former is inefficient in computing, because they may have to train and store dozens of models for different parameter levels. The latter is ineffective in performance, since they ignore some important information that could have helped the restoration process.
To overcome these weaknesses, we propose a novel system called functional neural network (FuncNet) to solve a parametric image restoration problem with a single model. The difference between a plain neural network and our FuncNet is shown in Figure 1. Unlike a plain neural network, the smallest conceptual element of our FuncNet is no longer a floating-point weight, but a function of the parameter of the problem. When we train a FuncNet, we gradually change these functions to reduce the value of the loss function. When we use the trained FuncNet to restore a degraded image with a given parameter, we first evaluate those functions with the parameter, use the evaluation values as weights of the network, and then do inference as normal. This feature makes it both efficient and effective for a parametric problem. By this way, we neatly blend the parameter information into a neural network model, use it to help the restoration process, and only increase a negligible amount of computation as we will demonstrate later. We apply FuncNet to super-resolution, image denoising, and JPEG deblocking. The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts.
The remainder of the paper is organized as follows. Section 2 provides a brief survey of related work. Section 3 presents the proposed FuncNet model, discusses the details of implementation, and analyses its storage and computational efficiency. In Section 4, extensive experiments are conducted to evaluate FuncNets on three parametric image restoration tasks. Section 5 concludes the paper.

heading: 2 Related Work
text: Neural networks for parametric problems. To the best of our knowledge, there are seven ways to solve a parametric problem with neural network based methods. We list them below roughly in the order of popularity, and discuss their advantages and disadvantages.
The first method treats problems with different parameter levels as independent tasks, and trains a specific model for each parameter level [14, 45, 67, 63, 66]. The overwhelming majority of previous papers use this approach. This method is easy to understand, and generally has good performance since the parameter level is implied in a model. But it is inefficient in computing, because we may have to train and store dozens of models for different parameter levels.
The second method simply ignores the parameter, and trains a single model for all parameter levels [28, 60]. This method is also easy to understand, and is very efficient since we only need to train and store a single model. But its performance is typically lower than the first method, since we ignore some important information that could have helped the restoration process.
The third method trains a model with a shared backbone for all parameter levels and multiple specific blocks for each parameter level [34]. It is a compromise between the first and the second method, and has acceptable performance and efficiency. But we may still have to store dozens of specific blocks for different parameter levels. And if the capacity of a specific block is not large enough, the block cannot take full advantage of the parameter information.
The fourth method converts the parameter scalar into a parameter map, and treats the map as an additional channel of the input degraded image [65]. It is another way to blend the parameter information into a neural network model. However, the performance of this method is only marginally higher than the second method, and it is still not as good as the first method. Due to the huge semantic difference between the corrupt image and the parameter map, it is hard to make much use of the parameter information for the model.
The fifth method conditions a network by modulating all its intermediate features by scalar parameters [9, 21] or maps [52]. It is another way to make a network adapt to different situations. But unlike our FuncNet, the method changes only features rather than parameters.
The sixth method trains a model with a relatively shallow backbone network, and each filter of the backbone network is generated by a different filter generating network [30, 26, 27, 15]. The filter generating networks are usually multilayer perceptrons, and they take the parameter as input. Since the total size of a model is limited, assigning each filter a unique complex network severely limits the size of the backbone network. Such a shallow backbone network only leads to a mediocre performance. Considering the universal approximation ability of the multilayer perceptron, this method is not that different from training a unique shallow model for each parameter level.
The seventh method searches in the latent space of a generative model, and returns the most probable result which is not contradicting the degradation model with the parameter [57, 38]. It is a general image restoration method which can solve various image restoration problems with a single model, as long as the degradation model is continuously differentiable. However, this method is slower than a feedforward neural network based method, since it requires multiple forward and backward passes in a search. And due to the limited representation ability of the generative model, the performance of this method is also worse than a discriminative model based method.
Neural network interpolation. In order to attain a continuous transition between different imagery effects, the neural network interpolation method [54, 53, 51] applies linear interpolation in the parameter space of two trained networks. Although at first glance it is similar to our method, there is a big difference between network interpolation and our FuncNet. The former is a simple interpolation technique while the latter is a regression technique. In the network interpolation method, two CNNs are trained separately for two extreme cases, and then blended in an ad hoc way. This may suffice for tasks [54] and [53], because users will accept roughly characterized visual results, such as "half GAN half MSE" or "half photo half painting". However, this is not good enough for an image restoration task whose goal is to restore the signal as accurately as possible. FuncNet is optimized for the entire value range of the task parameter (e.g., the noise level, SR scale factor), so its accuracy stays high over the entire parameter range, rather than just for the two extreme points like in the network interpolation method.
Deep CNN for Single Image Super-Resolution. The first convolutional neural network for single image super-resolution is proposed by Dong et al. [14] called SRCNN, and it achieved superior performance against previous works. Shi et al. [45] firstly proposed a real-time super-resolution algorithm ESPCN by proposing the sub-pixel convolution layer. Lim et al. [34] removed batch normalization layers in the residual blocks, and greatly improved the SR effect. Zhang et al. [67] introduced the residual channel attention to the SR framework. Hu et al. [23] proposed the MetaUpscale Module to replace the traditional upscale module.
Deep CNN for Image Denoising. Zhang et al. [63] proposed DnCNN, a plain denoising CNN method which achieves state-of-the-art denoising performance. They showed that residual learning and batch normalization are particularly useful for the success of denoising. Tai et al. [48] proposed MemNet, a very deep persistent memory network by introducing a memory block to mine persistent memory through an adaptive learning process. Zhang et al. [65] proposed FFDNet, a fast and flexible denoising convolutional neural network, with a tunable noise level map as the input.
Deep CNN for JPEG deblocking. Dong et al. [13] proposed ARCNN, a compact and efficient network for seamless attenuation of different compression artifacts. Guo and Chao [18] proposed a highly accurate approach to remove artifacts of JPEG-compressed images, which jointly learned a very deep convolutional network in both DCT and pixel domains. Zhang et al. [66] proposed DMCNN, a Dual-domain Multi-scale CNN to take full advantage of redundancies on both the pixel and DCT domains. Ehrlich et al. [15] proposed a novel architecture which is parameterized by the JPEG files quantization matrix.

heading: 3 Functional Neural Network (FuncNet)
text: In this section, we describe the proposed FuncNet model. To transform a plain neural network into a FuncNet, we replace every trainable variable in a plain neural network by a specific function, such as weights and biases in convolution layers or fully connected layers, affine parameters in Batch Normalization layers [24], and slopes in PReLU activation layers [22]; and keep other layers without trainable variables unchanged, such as pooling layers, identity layers, and pixel shuffle layers [45]. We first describe the method to specify the functions in our FuncNet models, then we describe the initialization, training and inference method for these functions, next we describe the network architectures to contain those functions, and finally we analyse the storage and computational efficiency of our FuncNet models.

heading: 3.1 Specification of Functions
text: The functions used in our FuncNet model should be simple enough. Let us consider the following failure case as a negative example. Suppose the number of the parameter levels is large but still finite, and we use polynomial functions in our FuncNet. If the polynomial function is too complex, and its degree is greater than or equal to the number of the parameter levels minus one, then our FuncNet is no different from training a specific model for each parameter level. In this case, the failing FuncNet takes as much or even more storage space than multiple independent models, and its inference speed is also slightly slower than a same size plain neural network. This negative example demonstrates the necessity of choosing simple functions for our FuncNet.
We choose the simplest and the most basic kind of function, the linear function, as the functions used in our FuncNet model. In this case, the FuncNet model takes exactly double storage space than a same size plain neural network, and it is still more efficient than storing dozens of models for different parameter levels. And it only increases a negligible amount of computations than a same size plain neural network, as we will demonstrate later. Choosing such a simple function does not lead to a poor performance of the final FuncNet model. With multiple activation layers, the final FuncNet model retains the power of nonlinear fitting. The linear function used in our FuncNet model can be defined as:
G(x; θa, θb) = x− xa xb − xa (θb − θa) + θa (1)
where x is the parameter of the problem, xa and xb are lower and upper bound of the support of the parameter distribution respectively, θa and θb are trainable variables, and G(x; θa, θb) is the function used in our FuncNet model to generate variables for different parameter levels.
The parameters have different properties for different problems, and we can make the linear function 1 to suit different problems better by replacing x with H(x), where H(x) is a problem-related function. Then the linear function 1 will become:
G(H(x); θa, θb) = H(x)−H(xa) H(xb)−H(xa) (θb − θa) + θa (2)
The chosen H(x) should have a physical interpretation related to the problem, and of course should make the final FuncNet model perform well. In the super-resolution problem, we use H(x) = 1/x because the reciprocal of the scale factor is the rescaled length on a low-resolution image from a unit length on a high-resolution image. In the image denoising problem, we use H(x) = x because the noise level is equal to the standard deviation of the additive white Gaussian noise. In the JPEG deblocking problem, we use H(x) = 5000/x for x ≤ 50 and H(x) = 200− 2x for x > 50. This is the formula used in JPEG standard [42], and it transforms the quality factor into a scale factor of the quantization table for DCT coefficients. The choices of H(x) for different problems are still empirical, just like when people determine the depth, the width or other configuration for a neural network. But we hope that we can determine H(x) automatically in the future, just like what people do in Neural Architecture Search right now [70].

heading: 3.2 Initialization, Training and Inference
text: Proper initialization is crucial for training a neural network. Even with modern structures and normalization layers, a bad initialization can still hamper the learning of the highly nonlinear system. The goal of initialization for a plain neural network is to set the value of every trainable variable in a proper range, and to avoid reducing or magnifying the magnitude of input signals exponentially. To properly initialize a function in FuncNet, we have to guarantee that all possible output values of the function are in a proper range. Suppose H(x) in function 2 is a monotonic function, then what we need to do is to use initialization algorithm for a plain neural network [22] to initialize θa and θb independently. In this way, all possible output values of the function 2 lie somewhere between θa and θb, and must also be in a proper range if θa and θb are well set. If θa and θb are both sampled from a zero-mean distribution whose standard deviation is σ, then for all possible output values of the function 2, their expected values are still zero, and their standard deviations are between σ/ √ 2 and σ. Experiments have shown that such a small deviation is acceptable for training.
Training a FuncNet is not very different from training a plain neural network. In every iteration, we first sample a fixed number of parameter levels uniformly, use them to construct a minibatch, and then perform stochastic gradient based optimization as normal. As suggested in [69], we train our FuncNet using L1 loss. During the training, we gradually change the values of trainable variables θa and θb to reduce the value of the loss function. The problem can be formulated as
min θa,θb
EIO,ID,x‖F (ID;G(H(x); θa, θb))− IO‖1 (3)
where ID is the degraded version of its original counterpart IO, and F is our FuncNet model.
When we use the trained FuncNet to restore a degraded image with a given parameter x, we first evaluate function 2 with x, trained θa and θb, use the evaluation values as variables of the corresponding plain neural network, and then do inference with the generated plain neural network as normal.

heading: 3.3 Network Architectures
text: The requirements of the networks for the three image restoration problems are very different. For the super-resolution problem, the degradation is deterministic and relatively mild, and the network can concentrate on a relatively small area. For the image denoising problem, the degradation is random and relatively severe, and the network needs to pay attention to a larger area. For the JPEG deblocking problem, the degradation occurs in the DCT domain, and the network should have the ability to utilize the information in the DCT domain. So we use individually designed architectures for the three image restoration problems to meet their own requirements.
We directly use architectures of the state-of-the-art plain neural networks for the three tasks as the architectures of our FuncNet models, and we only make essential modifications to them. For the
super-resolution problem, we use the architecture of RCAN [67], and replace the upscale module for integer scale factors [45] with the meta upscale module for non-integer scale factors [23]. For the image denoising problem, we apply a modified U-net [44] structure as the backbone and use RCAB [67] as residual blocks. For the JPEG deblocking problem, we use the architecture of DMCNN [66] for reference. For the DCT domain branch of our JPEG deblocking model, we use frequency component rearrangement to get a more meaningful DCT representation as suggested in [15]; and for the pixel domain branch, we use the same architecture of our image denoising model. More detailed information can be found in the supplementary material.

heading: 3.4 Storage and Computational Efficiency Analysis
text: Our FuncNet models have high storage efficiency. As we described in Section 3.1, we use two-degreeof-freedom functions in FuncNet models. This takes only twice as much space as what a plain neural network with the same architecture will take. So storing a FuncNet model is much cheaper than storing dozens of plain networks for different parameter levels. Take the super-resolution task as an example. Suppose the scale factor varies from 1.1 to 4 with stride 0.1 as suggested in [23], we can save 93.3% on storage space by using FuncNet rather than plain neural networks.
Our FuncNet models have high computational efficiency as well. For the training phase, we only need to train one FuncNet model rather than to train dozens of plain networks individually. The computational efficiency analysis for this phase is similar to the preceding storage efficiency analysis. For the inference phase, as we described in Section 3.2, we first evaluate functions with the problem related parameter, use the evaluation values as variables of the corresponding plain neural network, and then do inference with the generated plain neural network as normal. So compared to a plain neural network with the same architecture, our FuncNet model only needs a little extra effort to evaluate functions. This part of computation is directly proportional to the number of parameters in the corresponding plain network, and it is several orders of magnitude smaller than the number of multi-adds for a plain image restoration network. Still take the super-resolution task as an example. Suppose we need to double the size of a 360p image, our FuncNet model only needs extra 0.0001% computation than a plain neural network with the same architecture.

heading: 4 Experiments

heading: 4.1 Training Settings
text: Training datasets. Following [67, 23, 60], we use the DIV2K dataset [49] for training. There are 1000 high-quality images in the DIV2K dataset, 800 images for training, 100 images for validation and 100 images for testing. All our three FuncNet models for the three parametric image restoration tasks are trained with the DIV2K training images set.
Parametric settings. In all three parametric problems, the problem related parameters are sampled uniformly. In the super-resolution problem, the training scale factors vary from 1.1 to 4 with stride 0.1. In the image denoising problem, the training noise levels are sampled from the uniform distribution on the interval (0, 75]. In the JPEG deblocking problem, the quality factors vary from 10 to 80 with stride 2.
Degradation models. In the super-resolution problem, we use the bicubic interpolation by adopting the Matlab function imresize to simulate the LR images. In the image denoising problem, we generate the additive white Gaussian noise dynamically by using the Numpy function. In the JPEG deblocking problem, we use the Matlab JPEG encoder to generate the JPEG images.
Data augmentations. In all three parametric problems, we use the same data augmentation method. We randomly augment the image patches by flipping horizontally, flipping vertically and rotating 90◦.
Optimization settings. In the super-resolution problem, we randomly extract 32 LR RGB patches with the size of 40× 40 as a batch input. In the image denoising problem, we randomly extract 32 RGB patches with the size of 96×96 as a batch input. In the JPEG deblocking problem, we randomly extract 32 gray patches with the size of 96× 96 as a batch input, and we make sure that the image patches are aligned with boundaries of Minimum Coded Unit blocks. All our three FuncNet models are trained by ADAM optimizor with β1 = 0.9, β2 = 0.999, and = 10−8. The initial learning
rate is set to 10−4 and then decreases by half for every 2× 105 iterations of back-propagation. All experiments run in parallel on 4 GPUs.

heading: 4.2 Evaluation on Standard Benchmark Datasets
text: In the super-resolution problem, we use the B100 dataset for non-integer scale factor testing, and we use five standard benchmark datasets for integer scale factor testing: Set5, Set14, B100, Urban100, and Manga109. The results are evaluated with PSNR and SSIM [56] on Y channel of transformed YCbCr space. In the image denoising problem, we use three standard benchmark datasets: CBSD68, Kodak24, and McMaster. The results are evaluated with PSNR and SSIM [56] on RGB channel as suggested in [63]. In the JPEG deblocking problem, we use two standard benchmark datasets: LIVE1 and BSDS500. The results are evaluated with PSNR, SSIM [56], and PSNR-B [59] on Y channel.
We compare our results with those of state-of-the-art methods for all three parametric problems. Similar to [34], we also apply a self-ensemble strategy to further improve our FuncNet model and denote the self-ensembled one as FuncNet+. The quantitative results are shown in Table 1, 2, 3, and 4. The visual comparisons are shown in Figure 2, 3, and 4. More detailed information can be found in the supplementary material.

heading: 4.3 Kernel Visualization and Interpretation
text: We visualize kernels of our FuncNet model and try to understand and interpret them. The key point of the analysis is to find out how kernels change with the problem related parameter. Here we show samples of kernels from the first and the last layer of the denoising FuncNet, since the denoising problem has the most definite physical meaning among the three image restoration problems. The first and the last layer are also easier to understand. The results are shown in Figure 5. We can find out that the FuncNet uses more radical kernels for features when the noise level is low. By doing so, the FuncNet can get more information. And the FuncNet uses more moderate kernels when the noise level is high, so the FuncNet can get less error.

heading: 4.4 Ablation Study
text: As we discussed earlier, using functional kernels instead of numerical kernels is the key to making networks perform better for parametric image restoration problems. To verify the effectiveness of our FuncNet models, we train plain counterparts of our FuncNet models, and compare their evaluation results with FuncNets. And to measure the impact of choice on the problem-related function H(x), we train another two versions of FuncNet. The first one always uses the simplest non-trivial mapping H(x) = x, and the second one uses a small multilayer perceptron (MLP) with a hidden layer as a universal function approximator for any possible H(x). We then also compare their evaluation results with FuncNet which uses H(x) with a physical interpretation related to the problem. All the networks for ablation study share the same architectures with their corresponding FuncNet models, and all training and evaluation settings remain unchanged. The evaluation results are shown in Table 5.
This ablation study shows that the adaptability of our FuncNet model is important for parametric image restoration problems. Once our FuncNet degenerates into a plain network, its adaptability to different parameter levels disappears, and its performance drops remarkably. The results also prove that both identity function and MLP are acceptable choices for H(x). We can simply use those functions for a problem which is hard to design a H(x) with a physical interpretation.

heading: 5 Conclusions
text: We propose a novel neural network called FuncNet to solve parametric image restoration problems with a single model. To transform a plain neural network into a FuncNet, all trainable variables in the plain network are replaced by functions of the parameter of the problem. Our FuncNet has both high storage efficiency and high computational efficiency, and the experimental results show the superiority of our FuncNet on three common parametric image restoration tasks over the state of the arts.

heading: Acknowledgement
text: This work is supported by the Natural Sciences and Engineering Research Council of Canada.

Review summaries:
The authors proposed a functional neural network for image restoration. They use a linear function with respect to the task parameter to adjust the weights of the network, to achieve an adaptive restoration for different parameter levels using a single model. The idea is similar to meta-learning on restoration, or weight modulation derived from instance normalization. The experiment shows its better quantitative performance compared with SOTA among three tasks, and better visual quality. 

This paper introduces functional neural networks for image restoration that are parametrized by a variable (e.g. super-resolution factor, noise level, JPEG quality level).  A functional network is one where the neural network parameters themselves are a function of the problem parameter known a priori.  This way, the neural network weights can adapt based on the problem, for example, a denoising CNN can apply stronger denoising when the noise level increases.  Different functions are explored including linear functions which require twice the storage but using a single model.  Compelling experimental results are shown across a range of parameters of the problem.

This paper proposes a network interpolation scheme for non-blind image restoration problems with a single parameter (e.g. noise level for denoising). The parameter is remapped to [0, 1] using an affine mapping.  Effectively, two networks are trained such that for any given problem parameter in [0,1], an interpolated network is obtained by linearly blending the two networks  parameters. The interpolated model is then applied to the input. 

The authors propose a technique to build image restoration networks that are robust to variations in the problem parameters (e.g. noise level or upscaling factor). The main idea is to derive the weights of the network as a learnable function of the problem parameter.