Functional Neural Networks for Parametric Image Restoration Problems
Authors: Fangzhou Luo
Xiaolin Wu
Yanhui Guo
Venue: NeurIPS 2021 Poster (2021)
Keywords: conditional neural network
image restoration
super-resolution
image denoising
JPEG deblocking

Abstract: Almost every single image restoration problem has a closely related parameter, such as the scale factor in super-resolution, the noise level in image denoising, and the quality factor in JPEG deblocking. Although recent studies on image restoration problems have achieved great success due to the development of deep neural networks, they handle the parameter involved in an unsophisticated way. Most previous researchers either treat problems with different parameter levels as independent tasks, and train a specific model for each parameter level; or simply ignore the parameter, and train a single model for all parameter levels. The two popular approaches have their own shortcomings. The former is inefficient in computing and the latter is ineffective in performance. In this work, we propose a novel system called functional neural network (FuncNet) to solve a parametric image restoration problem with a single model. Unlike a plain neural network, the smallest conceptual element of our FuncNet is no longer a floating-point variable, but a function of the parameter of the problem. This feature makes it both efficient and effective for a parametric problem. We apply FuncNet to super-resolution, image denoising, and JPEG deblocking. The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts.

Functional Neural Networks for Parametric Image Restoration Problems
Fangzhou Luo McMaster University luofl @eemmmtte.....m
Xiaolin Wu"
Yanhui Guo McMaster University guoy14 Sooyy44@mmmaster.ccm
McMaster University xwußece, www.Bece.nmmstte..c
Abstract
Almost every single image restoration problem has closely related parameter, such as the scale factor in super-resolution, the noise level in image denoising, and the quality factor in JPEG deblocking. Although recent studies image restoration problems have achieved great success due to the development of deep neural networks, they handle the parameter involved in unsophisticated way. Most previous researchers either treat problems with different parameter levels as independent tasks, and train a specific model for each parameter level; or simply ignore the parameter, and train a single model for all parameter levels. The two popular approaches have their own shortcomings. The former is inefficient in computing and the latter is ineffective in performance. In this work, we propose novel system called functional neural network (FuncNet) to solve parametric image restoration problem with single model. Unlike a plain neural network, the smallest conceptual element of our FuncNet is no longer a floating-pont variable, but a function of the parameter of the problem. This feature makes it both efficient and effective for parametric problem. We apply FuncNet to super- resolution, image denoising, and JPEG deblocking. The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts.
Introduction
Image restoration [40] is a classical yet still active topic in low-level computer vision, which estimates the original image from a degraded measurement. For example, single image super-resolution [16] estimates the high-resolution image from downsampled one, image denoising [4] estimates the clean image from a noisy one, and JPEG deblocking (7 estimates the original image from compressed
one. It is challenging ill- ill-posed inverse problem which aims to recover the information lost to the image degradation process 3, and also important since is an essential step in various image
processing and computer vision applications [7] 58
2
33
46
20
Almost every single image restoration problem has closely related parameter, such the scale factor in super-resolution, the noise level in image denoising, and the quality factor in JPEG deblocking. The parameter in image restoration problem tends have strong connection with the image degradation process. In the super-resolution problem, the blur kernel of the downsampling process is determined by the scale factor 622 In the image denoising problem, the standard deviation of the additive white Gaussian noise is determined by the noise level [II. In the JPEG deblocking problem, the quantization table for DCT coefficients is determined by the quality factor 42 When we try to
restore the clean image from corrupt one, we might know the value of the corresponding parameter for various reasons. the susor-resolution problem, the scale factor is specified by users (2 In
the image denoising problem, the noise level could be measured by other devices
43
37 In the
Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Plain neural network
Functional neural network
W11
W12
W13
W11 W12
W13
Kernel weights w
Kernel weights w
Problem related parameter x
Problem related parameter X
The difference between a plain neural network and our functional neural network (FuncNet). The left and right figure visualize 3 convolution kernel in plain neural network and its counterpart in a FuncNet respectively. For the kernel in a plain network, its weights remain unchanged for different problem related parameter levels, the network only has limited adaptability to parametric image restoration problems. Unlike a plain network, the smallest conceptual element of our FuncNet is longer floating-point variable, but function of the problem related parameter. In other words, the kernel weights of our FuncNet can change for different situations and make our FuncNet perform better for parametric image restoration problems.
Figure 1:
JPEG deblocking problem, the quality factor could be derived from the header of the JPEG file [IO] Therefore, it is very important to use the known parameter well in such parametric image restoration problem.
Recently, deep convolutional neural network based methods are widely used to tackle the image
restoration tasks, including super-resolution [14 28 45
34
47
50
68
67
image denois-
ing 25 5 32
48
63
65
60, []] and JPEG deblocking 33
13
5$
18
35
66
15]. They
have achieved significant improvements over conventional image restoration methods due their powerful learning ability. However, they have not been paying attention to the parameter involved in an image restoration problem, and handled either treat problems with different parameter levels as independent tasks, and train specific model for each parameter level 14
an unsophisticated way. Most previous researchers
[6]; or simply ignore the parameter, and train a single model The two popular approaches have their own shortcomings. The former inefficient computing, because they may have to train and store dozens of models for different parameter levels. The latter is ineffective in performance, since they ignore some important information that could have helped the restoration process.
45
67
63
for all parameter levels 28
60
To overcome these weaknesses, we propose a novel system called functional neural network (FuncNet) solve a parametric image restoration problem with a single model. The difference between plain neural network and our FuncNet is shown in Figure Unlike plain neural network, the smallest conceptual element of our FuncNet is no longer floating-poii point weight, but function of the parameter of the problem. When we train a FuncNet, we gradually change these functions to reduce value of the loss function. When we use the trained FuncNet to restore degraded image with a given parameter, we first evaluate those functions with the parameter, use the evaluation values as weights of the network, and then do inference as normal. This feature makes both efficient and effective for a parametric problem. By this way, neatly blend the parameter information into neural network model, use to help the restoration process, and only increase negligible amount of computation as we will demonstrate later. We apply FuncNet to super-resolution, image denoising, and JPEG deblocking. The experimental results show the superiority of our FuncNet on all three parametric image restoration tasks over the state of the arts.
The remainder of the paper is organized as follows. Section provides a brief survey of related work. Section presents the proposed FuncNet model, discusses the details of implementation, and analyses storage and computational efficiency. In Section 4 extensive experiments are conducted to evaluate FuncNets three parametric image restoration tasks. Section concludes the paper.

---

The fourth method converts the parameter scalar into a parameter map, and treats the map as an additional channel of the input degraded image [65] It is another way to blend the parameter information into neural network model. However, the performance of this method is only marginally Related Work
2
Neural networks for parametric problems. To the best of our knowledge, there are seven ways to solve a parametric problem with neural network based methods. We list them below roughly in the order of popularity, and discuss their advantages and disadvantages.
The first method treats problems with different parameter levels as independent tasks, and trains a specific model for each parameter level [14] 67 63 [66]. The overwhelming majority of previous papers use this approach. This method is easy to understand, and generally has good performance since the parameter level is implied in model. But it is inefficient in computing, because we may have to train and store dozens of models for different parameter levels.
The second method simply ignores the parameter, and trains a single model for all parameter levels [28 60). This method is also easy to understand, and is very efficient since we only need to train and store single model. But its performance is typically lower than the first method, since we ignore some important information that could have helped the restoration process.
The third method trains a model with a shared backbone for all parameter levels and multiple specific blocks for each parameter level [4] It is a compromise between the first and the second method, and has acceptable performance and efficiency. But we may still have to store dozens of specific blocks for different parameter levels. And if the capacity of a specific block is not large enough, the block cannot take full advantage of the parameter information.
higher than the second method, and it is still not as good as the first method. Due to the huge semantic difference between the corrupt image and the parameter map, it is hard to make much use of the parameter information for the model.
The fifth method conditions a network by modulating all its intermediate features by scalar parame- ters 99 21 or maps [52]. It is another way to make a network adapt to different situations. But unlike our FuncNet, the method changes only features rather than parameters.
The sixth method trains a model with a relatively shallow backbone network, and each filter of the backbone network is generated by a different filter generating network 33 26 27 15] The filter generating networks are usually multilayer perceptrons, and they take the parameter as input. Since the total size of a model is limited, assigning each filter a unique complex network severely limits the size of the backbone network. Such shallow backbone network only leads to a mediocre performance. Considering the universal approximation ability of the multilayer perceptron, this method is not that different from training a unique shallow model for each parameter level.
The seventh method searches in the latent space of a generative model, and returns the most probable result which is not contradicting the degradation model with the parameter [57] [38] It general image restoration method which can solve various image restoration problems with a single model, as long as the degradation model is continuously differentiable. However, this method is slower than a feedforward neural network based method, since it requires multiple forward and backward passes in search. And due to the limited representation ability of the generative model, the performance of this method is also worse than a discriminative model based method.
Neural network interpolation. In order to attain continuous transition between different imagery effects, the neural network interpolation method [54] 53 51) applies linear interpolation in the parameter space of two trained networks. Although at first glance is similar to our method, there is a big difference between network interpolation and our FuncNet. The former is simple interpolation technique while the latter is regression technique. In the network interpolation method, two CNNs are trained separately for two extreme cases, and then blended in an ad hoc way. This may suffice for tasks [54] and (53], because users will accept roughly characterized visual results, such as "half GAN half MSE" or "half photo half painting". However, this is not good enough for an image restoration task whose goal is to restore the signal as accurately as possible. FuncNet optimized for the entire value range of the task parameter (e.g., the noise level, SR scale factor), so its accuracy stays high over the entire parameter range, rather than just for the two extreme points like in the network interpolation method.
3

---

Deep CNN for Single Image Super-Resolution. The first convolutional neural network for single image super-resolution is proposed by Dong et al. [14] called SRCNN, and it achieved superior performance against previous works. Shi et al. 45 firstly proposed real-time super-resolution algorithm ESPCN by proposing the sub-pixel convolution layer. Lim et al. [34] removed batch normalization layers in the residual blocks, and greatly improved the SR effect. Zhang et al. [67 introduced the residual channel attention to the SR framework. Hu et al. [23] proposed the Meta- Upscale Module to replace the traditional upscale module.
Deep CNN for Image Denoising. Zhang et al. 63 proposed DnCNN, a plain denoising CNN method which achieves state-of-the-an denoising performance. They showed that residual learning and batch normalization are particularly useful for the success of denoising. Tai al. [4]] proposed MemNet, a very deep persistent memory network by introducing a memory block to mine persistent memory through an adaptive learning process. Zhang et al. (65) proposed FFDNet, a fast and flexible denoising convolutional neural network, with tunable noise level map as the input.
Deep CNN for JPEG deblocking. Dong et al. [13] proposed ARCNN, a compact and efficient network for seamless attenuation of different compression artifacts. Guo and Chao [18] proposed highly accurate approach remove artifacts of JPEG-compressed images, which jointly learned a very deep convolutional network in both DCT and pixel domains. Zhang et al. [66] proposed
DMCNN, Dual-domain Multi-scale CNN to take full advantage of redundancies on both the pixel and DCT domains. Ehrlich et al. [15] proposed novel architecture which is parameterized by the JPEG files quantization matrix.
3 Functional Neural Network (FuncNet)
In this section, we describe the proposed FuncNet model. To transform plain neural network into a FuneNet, we replace every trainable variable in plain neural network by a specific function, such as weights and biases in convolution layers or fully connected layers, affine parameters in Batch Normalization layers [24), and slopes in PReLU activation layers [22]; and keep other layers without trainable variables unchanged, such pooling layers, identity layers, and pixel shuffle layers [55) We first describe the method to specify the functions our FuncNet models, then we describe the initialization, training and inference method for these functions, next we describe the network architectures to contain those functions, and finally we analyse the storage and computational efficiency of our FuncNet models.
3.1 Specification of Functions
The functions used in our FuncNet model should be simple enough. Let us consider the following failure case as negative example. Suppose the number of the parameter levels is large but still finite, and we use polynomial functions in our FuncNet. If the polynomial function is too complex, and its FuncNet is
degree is greater than or equal to the number of the parameter levels minus one, then no different from training a specific model for each parameter level. In this case, the failing FuncNet takes as much even more storage space than multiple independent models, and its inference speed is also slightly slower than same size plain neural network. This negative example demonstrates the necessity of choosing simple functions for our FuncNet
We choose the simplest and the most basic kind of function, the linear function, as the functions used in our FuncNet model. In this case, the FuncNet model takes exactly double storage space than a same size plain neural network, and it is still more efficient than storing dozens of models for different parameter levels. And it only increases a negligible amount of computations than a same size plain neural network, as we will demonstrate later. Choosing such a simple function does not lead to a poor performance of the final FuncNet model. With multiple activation layers, the final FuncNet model retains the power of nonlinear fitting. The linear function used in our FuncNet model can be defined as:
G( Oz, 00)
F. Fa (0. 0s) + +
Tb xa
where x is the parameter of the problem, N, and are lower and upper bound of the support of the parameter distribution respectively, 0,, and f, are trainable variables, and G(æ;; 0a: 0b) is the function used in our FuncNet model to generate variables for different parameter levels.

---

The parameters have different properties for different problems, and we can make the linear function suit different problems better by replacing z with H (x)), where H (z) is a problem-related function. Then the linear function will become:
|
H(z) (x,) G(H(z); 0b) H(x)) 11 (1..) (0p 0a) 0n
(2)
The chosen (x) should have a physical interpretation related to the problem, and of course should
make the final FuncNet model perform well. In the super-resolution problem, we use H (x) 1/1x because the reciprocal of the scale factor is the rescaled length on a low-resolution image from a unit length a high-resolution image. In the image denoising problem, use (xx)
because
the noise level is equal to the standard deviation of the additive white Gaussian noise. In the JPEG deblocking problem,
use (x) 5000/z / for is the formula used in JPEG standard 12. and it transforms the quality factor into a scale factor of the quantization table for DCT coefficients. The choices of H (z) for different problems are still empirical, just like when people determine the depth, the width or other configuration for neural network. But we hope that we can determine H (z) automatically in the future, just like what people do in Neural Architecture Search right now 70
50 and H (x) 200 2x for x > 50. This
3.2 Initialization, Training and Inference
Proper initialization is crucial for training a neural network. Even with modern structures and normalization layers, a bad initialization can still hamper the learning of the highly nonlinear system. The goal of initialization for plain neural network is to set the value of every trainable variable in proper range, and to avoid reducing or magnifying the magnitude of input signals exponentially. To properly initialize a function in FuncNet, we have to guarantee that all possible output values of the function are in a proper range. Suppose H in function 2 is a monotonic function, then what we need to do is to use initialization algorithm for a plain neural network 22 to initialize 9₂ and A, independently. In this way, all possible output values of the function 2 lie somewhere between 9₂ and 6g. and must also be in proper range if 9g and fg are well set. If 8, and A are both sampled from zero-mean distribution whose standard deviation is then for all possible output values of the function 2 their expected values are still zero, and their standard deviations are between g/v2 and a. Experiments have shown that such a small deviation is acceptable for training.
Training a FuncNet is not very different from training a plain neural network. In every iteration, we first sample a fixed number of parameter levels uniformly, use them construct minibatch, and then perform stochastic gradient based optimization as normal. As suggested in [69], we train our FuncNet using LI loss. During the training, we gradually change the values of trainable variables O,. and 0g to reduce the value of the loss function. The problem can be formulated as
(x) min E 11 ..... ((HH)) 0g 0,)) 0..0,
11
(3)
where 1 1 is the degraded version of its original counterpart 10 and F' is our FuncNet model.
When the trained FuncNet to restore degraded image with given parameter Jr, first evaluate function 2 with x, trained O₂ and fg. use the evaluation values as variables of the corresponding plain neural network, and then do inference with the generated plain neural network as normal.
3.3 Network Architectures
The requirements of the networks for the three image restoration problems very different. For the super-resolution problem, the degradation is deterministic and relatively mild, and the network can concentrate on relatively small area. For the image denoising problem, the degradation is random and relatively severe, and the network needs to pay attention to a larger area. For the JPEG deblocking problem, the degradation occurs in the DCT domain, and the network should have the ability to utilize the information in the DCT domain. So we use individually designed architectures for the three image restoration problems meet their own requirements.
We directly use architectures of the state-of-the-arr plain neural networks for the three tasks as the architectures of our FuncNet models, and we only make essential modifications to them. For the
5

---

super- resoureeso problem, we use the architecture of RCAN [67], and replace the upscale module for integer scale factors [45] with the meta upscale module for non non-integer scale factors [23]. For the image denoising problem, we apply modified U-net 4 structure as the backbone and use RCAB 167] as residual blocks. For the JPEG deblocking problem, we use the architecture of DMCNN [66] for reference. For the DCT domain branch of our JPEG deblocking model, we use frequency component rearrangement to get a more meaningful DCT representation as suggested in [15: and for the pixel domain branch, we use the same architecture of our image denoising model. More detailed information can be found in the supplementary material.
3.4 Storage and Computational Efficiency Analysis
Our FuncNet models have high storage efficiency. As we described in Section 3.1 we use two-degree- of-freedom functions in FuncNet models. This takes only twice as much space as what a plain neural network with the same architecture will take. So storing a FuncNet model is much cheaper than storing dozens of plain networks for different parameter levels. Take the super-resolutio task as an example. Suppose the scale factor varies from 1.1 to 4 with stride 0.1 as suggested in [23] we can save 93.3% on storage space by using FuncNet rather than plain neural networks.
Our FuncNet models have high computational efficiency as well. For the training phase, we only need to train one FuneNet model rather than to train dozens of plain networks individually. The computational efficiency analysis for this phase is similar to the preceding storage efficiency analysis. For the inference phase, as we described in Section 3.2 we first evaluate functions with the problem related parameter, use the evaluation values as variables of the corresponding plain neural network, and then do inference with the generated plain neural network as normal. So compared a plain neural network with the same architecture, our FuncNet model only needs little extra effort to evaluate functions. This part of computation directly proportional to the number of parameters in the corresponding plain network, and it is several orders of magnitude smaller than the number of multi-adds for plain image restoration network. Still take the super-resolution task as an example. Suppose we need to double the size of 360p image, our FuncNet model only needs extra 0.0001 % computation than a plain neural network with the same architecture.
4 Experiments
4.1 Training Settings
Training datasets. Following [67 23 60), we use the DIV2K dataset [49] for training. There are 1000 high-quality images in the DIV2K dataset, 800 images for training, 100 images for validation and 100 images for testing. All our three FuncNet models for the three parametric image restoration tasks are trained with the DIV2K training images set.
Parametric settings. In all three parametric problems, the problem related parameters are sampled uniformly. In the super-resolution problem, the training scale factors vary from 1.1 to 4 with stride 0.1. In the image denoising problem, the training noise levels are sampled from the uniform distribution on the interval (0, 75]. In the JPEG deblocking problem, the quality factors vary from 10 to 80 with stride 2.
Degradation models. In the super-resolution problem, we use the bicubic interpolation by adopting the Matlab function imresize simulate the LR images. In the image denoising problem, we generate the additive white Gaussian noise dynamically by using the Numpy function. In the JPRG deblocking problem, we use the Matlab JPEG encoder generate the JPEG images.
Data augmentations. In all three parametric problems, we use the same data augmentation method. We randomly augment the image patches by flipping horizontally, flipping vertically and rotating 90%.
Optimization settings. In the super-resolution problem, we randomly extract 32 LR RGB patches with the size of 40 x 40 as batch input. In the image denoising problem, we randomly extract 32 RGB patches with the size of 96 x 96 as a batch input. In the JPEG deblocking problem, we randomly extract 32 gray patches with the size of 96 x 96 as a batch input, and we make sure that the image patches are aligned with boundaries of Minimum Coded Unit blocks. All our three FuncNet models trained by ADAM optimizor with B1 0.9, B2 0.999, and 10-8, The initial learning

---

HR
Bicubic
EDSR 34
RCAN 67
SAN 12
Urban100 (4x): img046
CSNLN 30
MIRNet 61
Mcta-SR 22
LIIFF
FuneNet
HR
Bicubic
EDSR 34
RCAN 67
SAN 12
Urban 100 (1x): img062
CSNLN 39
MIRNet [61]
Meta-SR (33
LIIF g
FuneNet
Figure 2: Visual comparison between different super-resolution methods
Ground-truth
Noisy
IRCNN 64
DnCNN 63
CBDNet 19
Kodak24 (o = 35): kodim08
RIDNet II
MIRNet 61
FFDNet 65
CResMD 21
FuncNet
Figure 3: Visual comparison between different image denoising methods
Ground-truth
JPEG
ARCNN 1B
DMCNN 66
LIVE1 (g = 20): church
AGARNet 29
CResMD 2I
QGAC 15
FuncNet
Figure 4: Visual comparison between different JPEG deblocking methods

---

Table 1: Results of decimal upscale SR on B100. underlined
Best and second best results highlighted and
Scale
x1.1
x1.2
x1.3
x1.4
x1.5
x1.6
x1.7
x
x x
Method
Meta-SR [23]
42.82
40.04
38.28
36.95
35.86
34.90
34.13
33,45
32.86
FuncNet
43.43
40.41
38.55
37.16
36.02
35.08
34.26
33.60
32.98
FuncNet+
43.36
40.46
38.59
37.21
36.06
35.12
34.30
33.64
33.02
Scale
x2.1
x2.2
x2.3
X2.4
x2.5
x2.6
X2.7
X2.8
x2.9
Method
Meta-SR 23
31.82
31.41
31.06
30.62
30.45
30.13
29.82
29.67
29.40
FuncNet
31.99
31.59
31.23
30.87
30.58
30.30
30.05
29.77
29.59
FuncNett
32.02
31.62
31.26
30.90
30.62
30.34
30.09
29.81
29.63
Scale
x3.1
x3.2
x3.3
x3.4
x3.5
x3.6
x3.7
x3.8
x3.9
Method
Meta-SR 23
28.87
28.79
28.68
28.54
28.32
28.27
28.04
27.92
27.82
FuncNet 29.17
29.02
28.81
28.62
28.46
28.34
28.21
28.06
27.93
FuncNett
29.20
29.06
28.85
28.67
28.51
28.38
28.26
28.11
27.97
Table 2: Results of integer upscale SR. Best and second best results highlighted and underlined. B100, Urban and Manga represent datasets B B100, Urban100, and Manga 199 respectively.
Scale
Scale
Scale Urban
Method
B100
Urban
Manga
B100
Urban
Manga
B100
Manga
EDSR
34
32.32
32.93
39.10
29.25
28.80
34.17
27.71
26.64
31.02
RCAN 67
32.41
33.34
39.44
29.32
29.09
34.44
27.77
26.82
31.22
SAN 12
32.42
33.10
39.32
29.33
28.93
34.30
27.78
26.79
31.18
CSNLN 39
32,40
33.25
39.37
29.33
29.13
34.45
27.80
27.22
31,43
MIRNet
27.04
24.53
26.99
25.96
23.24
25.50
Meta-SR
23
32.35
39.18
29.30
34.14
27.75
31.03
LIIF 8
32.32
32,87
29.26
28.82
27.74
26.68
FuncNet
32.48
33.61
39.73
29.39
29.42
34.90
27.87
27.15
31.71
FuncNet+ 32.51
33.78
39.87
29.43
29.57
35.10
27.90
27.29
31.97
Table 3: Results of image denoising. Best and second best results highlighted and underlined. CBSD, Kodak and Mac represent datasets CBSD68, Kodak24 and McMaster respectively.
15
0 35
75
Method
CBSD
Kodak
Mac
CBSD
Kodak
Mac
CBSD
Kodak
Mac
DrCNN 663
33.89
34.48
33.44
29.58
30.46
30.14
24.47
25.04
25.10
CBDNet 19
32.67
33.32
32.87
28.11
28.87
28.77
24.05
24.64
24.38
MIRNet
61
27.44
28.30
27.92
22.39
23.19
22.47
18.77
18.88
18.76
FFDNet
65
33.87
34.63
34.66
29.58
30.57
30.81
26.24
27.27
27.33
CResMD
21
33.97
34.80
34.80
29.70
30.75
31.00
26.26
27.36
27.39
FuncNet
34.26
35.21
35.39
30.02
31.24
31.61
26.72
27.98
28.18
FuncNet+ 34.28
35.25
35.44
30.05
31.29
31.67
26.76
28.05
28.26
Table 4: Results of JPEG deblocking, Best and second best results are highlighted and underlined. LIVE and BSDS represent datasets LIVEI and BSDS500 respectively.
Quality
10
Quality 20
Quality
30
Quality 40
Method
LIVE
BSDS
LIVE
BSDS
LIVE
BSDS
LIVE
BSDS
ARCNN
13
29.13
29.10
31.40
31.28
32.69
32.64
33.63
33.55
DMCNN 66
29.73
29.67
32.09
31.98
CResMD
21
27.89
27.92
30.58
30.55
32.46
32.37
33.87
33.73
QGAC 15
29.53
29.54
31.86
31.79
33.23
33.12
FuneNet
29.77
29.68
32.20
32.05
33.63
33.44
34.63
34.41
FuncNet+
29.81
29.71
32.23
32.07
33.66
33.47
34.66
34.44
8

---

a= 15
0-35
G=
kernels of the first layer
kernels of the last layer
Figure 5: Kernel visualization of the denoising FuncNet. The left part is sampled from the first layer and the right part is sampled from the last layer. We can find out that the FuncNet uses more radical kernels when the noise level is low, and uses more moderate kernels when the noise level is high.
rate is set to 10- and then decreases by half for every 2 10 iterations of back- nack-propagation. All experiments run in parallel on 4 GPUs.
4.2 Evaluation on Standard Benchmark Datasets
In the super-resolution problem, we use the B100 dataset for non-integer scale factor testing, and we use five standard benchmark datasets for integer scale factor testing: Sct5, Set14, B100, Urban 100, and Mangal09 009. The results are evaluated with PSNR and SSIM [56] on Y channel of transformed YCbCr space. the image denoising problem, we use three standard benchmark datasets: CBSD68. Kodak24, and McMaster. The results are evaluated with PSNR and SSIM [56] on RGB channel as suggested in 663] In the JPEG deblocking problem, we use two standard benchmark datasets: LIVE1 and BSDS500. The results are evaluated with PSNR, SSIM [56), and PSNR-B [59] on Y channel.
We compare our results with those of state-of-the-art methods for all three parametric problems. Similar to (34], we also apply self-ensemble strategy to further improve our FuncNet model and denote the self-ensembled one FuncNett. The quantitative results are shown in Table [ and
The visual comparisons are shown in Figure and More detailed information can be found the supplementary material.
4.3 Kernel Visualization and Interpretation
We visualize kernels of FuncNet model and try to understand and interpret them. The key point of the analysis is to find out how kernels change with the problem related parameter. Here we show samples of kernels from the first and the last layer of the denoising FuncNet, since the denoising problem has the most definite physical meaning among the three image restoration problems. The first and the last layer are also easier understand. The results are shown in Figure We can find out that the FuncNet uses more radical kernels for features when the noise level is low, By doing so, the FuneNet can get more information. And the FuncNet uses more moderate kernels when the noise level is high, so the FuncNet can get less error.
4.4 Ablation Study
As we discussed earlier, using functional kernels instead of numerical kernels is the key to making networks perform better for parametric image restoration problems. verify the effectiveness of our FuncNet models, we train plain counterparts of our FuncNet models, and compare their evaluation results with FuncNets. And to measure the impact choice on the problem-relate function H (x). we train another two versions of FuncNet. The first one always uses the simplest non-trivial mapping (x) x, and the second one uses a small multilayer perceptron (MLP) with hidden layer as a universal function approximator for any possible (x). We then also compare their evaluation results with FuncNet which uses (x) with physical interpretation related to the problem. All networks for ablation study share the same architectures with their corresponding FuncNet models,
and training and evaluation settings remain unchanged. The evaluation results are shown in Table
5

---

Table 5: Results of the ablation study. Super-resolution, denoising and deblocking are tested on Urban100, Kodak24 and LIVE1 respectively.
Super-resolution
Denoising
Deblocking
Method
= 2
4
15
35
q 10
20
FuncNet
33.61
29.42
27.15
35.21
31.24
29.77
32.20
Plain net
33.07
28.93
26.70
34.83
30.89
29.58
31.94
FuncNet (H (x) 2))
33,48
29.33
27.05
35.21
31.24
29.64
32.16
FuncNet (H is MLP)
33.60
29.38
27.02
35.19
31.20
29.69
32.17
This ablation study shows that the adaptability of our FuncNet model is important for parametric image restoration problems. Once our FuncNet degenerates into plain network, its adaptability to different parameter levels disappears, and its performance drops remarkably. The results also prove that both identity function and MLP acceptable choices for (x). We can simply use those functions for problem which is hard to design a H (z) with a physical interpretation.
5 Conclusions
We propose novel neural network called FuneNet to solve parametric image restoration problems with single model. To transform plain neural network into FuneNet, trainable variables in the plain network are replaced by functions of the parameter of the problem. Our FuncNet has both high storage efficiency and high computational efficiency, and the experimental results show the superiority of our FuneNet on three common parametric image restoration tasks over the state of the arts.
Acknowledgement
This work is supported by the Natural Sciences and Engineering Research Council of Canada.
References
[1] Saced Anwar and Nick Barnes. Real image denoising with feature attention. In Proceedings of the IEEE International Conference on Computer Vision, pages 3155-3164, 2019.
Kanwal K Bhatia, Anthony N Price, Wenzhe Shi, Jo V Hajnal, and Danicl Rueckert. Super-resolution re- construction of cardiac mri using coupled dictionary learning. In 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), pages 947-950. IEEE, 2014.
[3] José M Bioucas-Dias and Mário AT Figueiredo. A new twist: Twa-step iterative shrinnaage/hrreesooldngg algorithms for image restoration. IEEE Transactions on Image processing. 16(12):2992-3004, 2007.
[4] Antoni Buades, Bartomeu Coll, and J-M Morel, A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'05), volume 2, pages 60 65. JEEE, 2005.
[5] Harold C Burger, Christian J Schuler, and Stefan Harmeling, Image denoising: Can plain neural networks compete with bm3d? In 2012 IEEE conference computer vision and pattern recognition, pages 2392-2399. IEEE, 2012.
[6] Lukas Cavigelli, Pascal Hager, and Luca Benini. Cas-cun: A deep convolutional neural network for image compression artifact suppression. In 2017 International Joint Conference on Neural Networks (IJCNN). pages 752-759, JEEE, 2017.
[7] Huibin Chang. Michael K Ng, and Tieyong Zeng. Reducing artifacts in jpeg decompression via learned dictionary. IEEE transactions signal processing, 62(3):718 728, 2013.
[8] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8628 8638, 2021.
10

---

[9] Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Variable rate deep image compression with conditional autoencoder. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3146-3154, 2019.
[10] Rémi Cogranne. Determining jpeg image standard quality factor from the quantization tables. arXiv preprint XXix:1802.00992 2018.
[11] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d connfform-mmaail collaborative filtering. IEEE Transactions on image processing. 16(8):2080 2095, 2007.
[12] Tao Dai, Jianrui Cai, Yongbing Zhang, Tau Xia, and Lei Zhang. Sccond-order attention network for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11065-11774 2019.
[13] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by deep convolutional network. In Proceedings of the IEFF International Conference on Computer Vision, pages 576-584, 2015.
[14] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In European conference on computer vision, pages 184-199. Springer, 2014.
[15] Max Ehrlich, Ser-Nam Lim, Larry Davis, and Abhinav Shrivastava. Quantization guided jpeg artifact correction. arXiv, pages arXiv-2004, 2020.
[16] William T Freeman, Egon C Pasztor, and Owen T Carmichael. Learning low-level vision. International journal of computer vision, 40(1):25 47, 2000.
[17] Leonardo Galteri, Lorenzo Seidenari, Marco Bertini, and Alberto Del Bimbo. Deep generative adversarial compression artifact removal. In Proceedings of the IEEE International Conference on Computer Vision, pages 4826 4835, 2017.
[18] Jun Guo and Hongyang Chao, Building dual-domain representations for compression artifacts reduction. In European Conference Computer Vision, pages 628-644. Springer, 2016.
[19] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1712-1722, 2019.
[20] Yanhui Guo, Xi Zhang, and Xiaolin Wu. Deep multi-modality soft-dccoding of very low bit-rate face videos. In Proceedings of 28th ACM International Conference on Multimedia, pages 1947-3955, 2020.
[21] Jingwen Hc, Chao Dong, and Yu Qiao. Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XX 16, pages 53-68. Springer, 2020.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pages 1026-1033, 2015.
[23] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: magnification- arbitrary network for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1575-1584, 2019.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
[25] Viren Jain and Sebastian Seung. Natural image denoising with convolutional networks. Advances in neural information processing systems, 21:769-776, 2008
[26] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In Advances in neural information processing systems, pages 667-675, 675, 2016.
[27] Di Kang, Debarun Dhar, and Antoni Chan. Crowd counting by adapting convolutional neural networks with side information. arXiv preprint 1rXiv:1611.06745 2016.
[28] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages (646-165.. 2016.
11

---

[29] Yoonsik Kim, Jae Woong Soh, and Nam Ik Cho. Agarnet: adaptively gated jpeg compression artifacts removal network for wide range quality factor. IEEE Access, 8:20160-20170, 2020.
[30] Benjamin Klein, Lior Wolf, and Yehuda Afek. A dynamic convolutional layer for short range weather prediction, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4840-4848, 2015.
[31] Younghee Kwon, Kwang In Kim, James Tompkin, Jin Hyung Kim, and Christian Theobalt. Efficient learning of image super-resolution and compression artifact removal with semi-local gaussian processes. IEEE transactions on pattern analysis and machine intelligence, 37(9): 1792-1805 2015.
[32] Stamatios Lefkimmiatis. Non-local color image denoising with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3587-3596,
2017.
[33] Shutao Li, Haitao Yin, and Leyuan Fang. Group-spare representation with dictionary learning for medical image denoising and fusion. IEEE Trunsactions on biomedical engineering. 59(12):3450 -3599, 2012.
[34] Bee Lim, Sanghyun Son, Heewon Kim. Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution In Proceedings the and pattern recognition workshops, pages 136-144, 2017.
the IEEE conference on computer vision
[35] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet-cnn for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 773-782, 2018.
[36] Xiamming Liu, Xiaolin Wu, Jiantao Zhou, and Debin Zhao. Data-driven sparsity- based restoration of ipeg-compressed images in dual transform-pixel domain. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5171-5178, 2015.
[37] Xinhao Liu, Masayuki Tanaka, and Masatoshi Okutomi. Single-image noise level estimation for blind denoising. IEEE transactions image proeesing, 22(12):5226-5237, 2013.
[38] Fangzhou Luo and Xiaolin Wu. Maximum a posteriori on a submanifold: a general image restoration method with gan. In 2020 Internationall Joint Conference on Neural Networks (IJNNN), pages 1-7. IEBE, 2020.
[39] Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S Huang, and Honghui Shi. Image super- resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 5690-5699, 2020.
[40] Peyman Milanfar. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE signal processing magazine, 30(1):106-128. 2012.
[41] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. Single image super-resolution via holistic attention network. In European Conference on Computer Vision, pages 191-207. Springer, 2020.
[42] William B Pennebaker and Joan L Mitchell. JPEG: Still image data compression standard. Springer Science & Business Media, 1992.
[43] Stanislav Pyatykh, Jürgen Hesser, and Lei Zheng. Image noise level estimation by principal component analysis. IEEÉ transactions on image processing, 22(2):687-699 2012.
[44] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical
image segmentation. In International Conference on Medical image computing and computer assiisee intervention, pages 234 241. Springer, 2015.
[45] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and vidco super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pages 1874-1883, 2016.
[46] Vivek Soni, Ashish Kumar Bhandari, Anil Kumar, and Girish Kumar Singh. Improved sub-band adaptive thresholding function for denoising of satellite image based on evolutionary algorithms. IET Signal Processing, 788)720-730, 2013.
[47] Ying Tai, Jian Yang. and Xiaoming Liu. Image resortoesolution via deep recursive residual network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3147-3155, 2017.
12

---

[48] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE international conference on computer vision, pages 4533945577, 2017.
[49] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang. and Lei Zhang. Ntire 2017 challenge on single image apper-resouttion, Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 114-125, 2017.
[50] Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connections. In Proceedings of the IEEE International Conference on Computer Vision, pages 4799-4807, 2017.
[51] Wei Wang, Ruiming Guo, Yapeng Tian, and Wenming Yang. Cisnet: Toward controllable feature space for image restoration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4140 2019.
[52] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super- resolution by deep spatial feature transform. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 606-615, 2018.
[53] Xintao Wang. Ke Yu, Chao Dong, Xiaoou Tang. and Chen Change Loy. Deep network interpolation for continuous imagery effect transition. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages (692-1701, 2019.
[54] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.
[55] Zhangyang Wang, Ding Liu, Shiyu Chang. Qing Ling, Yingzhen Yang. and Thomas S Huang. D3: Deep dual-domain based fast restoration of jpeg-compressed images. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 2764-2772.201
[56] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
[57] Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 5485-5493, 2017.
[58] Deniz Yildırum and Oguz Güngör. A novel image fusion method using ikonos satellite images. Journal of Geodesy and Geoinformation, 1(1):75-83 -7.55-332222 2012.
[59] Changhoon Yim and Alan Conrad Bovik. Quality assessment of deblocked images. JEEE Transactions on Image Processing, 20(1):88-98, 2010.
[60] Songhyun Yu, Bumjun Park, and Jechang Jeong. Deep iterative down-up enn for image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[61] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhancement. In Computer Vision-FCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pages 492-511. 492-551,.prrigeer, 2020.
[62] Kai Zhang, Lue Van Gool, and Radu Timofte. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3217-3226, 2020.
[63] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions Image Processing, 26(7):3142- 3155, 2017.
[64] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang Learning deep enn denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3929-3938, 2017.
[65] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Frdnet: Toward a fast and flexible solution for enn- ba-edaee image denoising. IEEE Transactions Image Processing, 2769)-4008 4622, 2018
13

---

[66] Xiaoshuai Zhang, Wenhan Yang, Yueyu Hu. and Jiaying Liu. Dmenn: Dual-domain multi-scale convolu- tional neural network for compression artifacts removal. In 2018 25th IEEE International Conference Image Prucessing (ICIP), pages 390-394. IEEE, 2018.
[67] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang. Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of European Conference on Computer Vision (ECCV), pages 286-301, 2018.
[68] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network image ruper-nnso.utio. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472 2481, 2018.
[69] Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for neural networks for image processing. arXiv preprint arXiv:15) 1.0886,, 2015.
[70] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint Mxxiv:6611.0177. 2016.
[71] Wilman WW Zou and Pong Yuen. Very low resolution face recognition problem. IEEE Transactions on image processing, 21(1):327-344 -40, 2011.
14


Reviews:
[{"review_id": "vYjNQrBOVQ", "reviewer": "Reviewer_hUCx", "summary": "This paper proposes a network interpolation scheme for non-blind image restoration\nproblems with a single parameter (e.g. noise level for denoising).\nThe parameter is remapped to [0, 1] using an affine mapping. \nEffectively, two networks are trained such that\nfor any given problem parameter in [0,1],\nan interpolated network is obtained by linearly blending the two networks \nparameters. The interpolated model is then applied to the input.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The technical contribution of this paper is rather meager, as network interpolation\nhas been discussed multiple times in the past, including in relation to image\nrestoration. Still, despite some minor  concerns about the evaluation, the proposed\nFuncNet model outperforms reasonable recent baselines, while being trained on\na *range* of parameter values.\n\n\n\n### Strengths\n\n- Improving neural network performance on parameterized problem such that\nthe network adapts to the input parameter is a timely problem with practical\nimportance.\n- Incorporating the parameter as a linear blending weight between two\nsets of network parameters (weights and biases) is interesting.\n- The proposed method quantitatively outperforms recent previous work \non denoising, super-resolution and JPEG deblocking.\n\n\n### Weaknesses\n\n- The proposed technical solution is essentially network interpolation, which\nis not novel, and should therefore be discussed. For\ninstance \"ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks\"\n[Wang 2018] linearly blend between the parameters of a model trained with a GAN loss, and one without.\nEven closer to the domain discussed here, \"Deep Network Interpolation for Continuous Imagery Effect Transition\" [Wang 2019]\ninterpolate between different filter effects.\n- How would the method if the restoration task had more than one parameter (e.g. 2 noise parameters for \na Heteroskedastic Gaussian noise model)?\n- The discussion of related work is insufficient.\n1. For instance, CBDNet \"Toward Convolutional Blind Denoising of Real Photographs\" [Guo 2019]\nuses yet another approach to incorporate the model parameter: an additional prediction branch\nsupervised to regress the ground-truth parameter.\n2. Another largely used approach to condition a network to a scalar parameter,\nis to modulate all its intermediate features based on the condition, using a mechanism like\nFiLM \"FiLM: Visual Reasoning with a General Conditioning Layer\" [Perez 2017]. \n\"Flexible Image Denoising with Multi-layer Conditional Feature Modulation\" [Du 2020] apply this to denoising.\n\"Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration\" [He 2020]\nuse a similar conditioning mechanism for image restoration, and deal with multiple parameters simultaneously.\n- I have some reserves about the evaluation of the proposed model. Why is RIDNet omitted from\nthe quantitative tables? Was the model retrained with the same noise model? \nFFDNet is a bit outdated for a baseline, consider re-training a more recent\nmore with the same procedure as the proposed model (i.e. same noise model, etc).\n- Thank you for the supplemental material. Seeing more images is useful, but\nI would have liked to see more diversity in the input images, rather than\nselected crops. It is difficult to judge the robustness of the method otherwise.\n- I do not think the self-ensembling (FuncNet+) adds much to the paper.\n- It is not clear from the paper whether the baselines were used as-is, retrained\non *all* parameter values (for their respective application), or trained specifically\non each parameter value.\n- In general, it would be useful to understand how FuncNet compares to\nthe \"plain\" version, specialized to each given parameter. This would give \nan upper-bound on the quality the model can reach, and therefore give\na sense of how much quality is lost by using the interpolation scheme instead.\n\n\n\n### Misc\n\n- 224: 0.0001% inaccurate, do you rather mean the linear interpolation of parameters\ntakes this fraction of the total compute?\n- The denoising result in Fig3 is nice. I am surprised RIDNet performs so poorly, since it is more\nrecent, and generally believed to outperform FFDNet.\n\n\n### Update\n\nI bumped by recommendation by +1. My criticism about the method being \"just network interpolation\" was not completely fair since the interpolated models are also used and supervised during training. The contribution therefore has some novelty that I did not fully grasp originally.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not discussed in the paper.", "review_text": "", "extra_scores": {"ethical_concerns": "None of note.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "kDyi1KpATE", "reviewer": "Reviewer_rd8B", "summary": "The authors propose a technique to build image restoration networks that are robust to variations in the problem parameters (e.g. noise level or upscaling factor). The main idea is to derive the weights of the network as a learnable function of the problem parameter.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and the idea clearly explained. The issue of making models that gracefully handle multiple values of a problem parameter is important and I agree with the authors that has been mostly overlooking or treated with unsophisticated techniques. Some works in real image denoising include specialized techniques to estimate the noise level and therefore be robust to variations but the proposed technique is much more general and applicable to different problems. The idea is original and, while simple, elegantly addresses the problem.\n\nThe experimental setting is not very clear. What is the setting used for the baselines? Is a baseline model trained for each parameter value or is a single model trained blindly for all values? Ideally, I think both scenarios should be shown because models typically perform better in the former, although, as the authors say, the storage requirements multiply times the number of parameter configurations.  In the blind case, instead, there would be a problem of fairness of the results because the proposed technique effectively uses twice as many trainable parameters with respect to the baseline and it is possible that the improved performance is due to that rather than the technique. The authors should clarify these points.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "W0cKHTcVdaH", "reviewer": "Reviewer_74Ec", "summary": "The authors proposed a functional neural network for image restoration. They use a linear function with respect to the task parameter to adjust the weights of the network, to achieve an adaptive restoration for different parameter levels using a single model. The idea is similar to meta-learning on restoration, or weight modulation derived from instance normalization. The experiment shows its better quantitative performance compared with SOTA among three tasks, and better visual quality. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros,\n\n- Elegant design for the proposed functional neural network and the writing of the paper is good and very easy to follow.\n- Extensive experiments on SOTA and benchmark dataset revealing the better performance.\n\nCons,\n\n- The idea is not novel enough, and the function design is not that interesting as in [1]. Besides, the task level is given, so the model is not blind for an adaptive inference with any arbitrary inputs, making the practical values very limited. \n- It's better to differentiate task parameter (noise level, scale factor etc.) with network parameter (model weights). Some terms in section 3.1 are somehow confusing, especially 'x' mostly represents input images. When the authors refer x as the parameter, I original though it was the model weights. \n- While comparing with other baselines (like denoising or JPEG deblocking), have the author used one single model trained with all the parameter levels, or each model per level separately? Besides, the parameter size should be controlled for a fair comparison. It seems the double parameter size improves the restoration performance. I understand that one set of parameter can represent multiple models with any given x, but it will be more fair to also double the weights of the single model of the same architecture. \n- In section 4.3, it's not clear what is the 'input' and 'output' layer the authors refer to. The visualization is a little bit confusing to interpret the results. Since the function is linear, it should not be surprising to see the linear effects of kernels.\n- Missing some SR reference like [2]\n\nMy main concern of this paper is the novelty and technical levels for NeurIPS, and its practical values in real image restoration. The better performance may be from some unfair comparison, and there is no explicit design for a single model with adjusted weights to be better than others. \n\n[1] Learning Continuous Image Representation with Local Implicit Image Function, CVPR'21\n[2] Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining, CVPR'20\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the above comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5"}, "extra_text": {}}, {"review_id": "IdFIBp9oTDs", "reviewer": "Reviewer_rgzb", "summary": "This paper introduces functional neural networks for image restoration that are parametrized by a variable (e.g. super-resolution factor, noise level, JPEG quality level).  A functional network is one where the neural network parameters themselves are a function of the problem parameter known a priori.  This way, the neural network weights can adapt based on the problem, for example, a denoising CNN can apply stronger denoising when the noise level increases.  Different functions are explored including linear functions which require twice the storage but using a single model.  Compelling experimental results are shown across a range of parameters of the problem.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important and common problem in image restoration – how to develop a single neural network that can work effectively across a range of settings (e.g. a denoiser that can denoise images with varying levels of noise).  The paper effectively argues that making neural networks for each setting produces good results, but also many networks; whereas a single network that works across a large range of settings may be less effective.  Indeed this a well-known issue.\n\nThe paper doesn’t use this terminology, but essentially it is proposing a solution for a conditional neural network.  That is, the network itself is conditioned based on the parameter (e.g. noise level).  The paper provides a nice taxonomy of different approaches to designing conditional neural networks, categorized into six methods in Section 2.  However, this taxonomy is incomplete; there are other methods proposed for conditional networks that appear to be outside the six methods described, for example:\n\n•\tWang et al., “Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform,” CVPR 2018 introduces Spatial Feature Transform modules which modulate the features, conditioned in their case on a segmentation map but this could instead be a map representing the a priori parameter of the problem. \n\n•\tChoi et al., “Variable Rate Deep Image Compression With a Conditional Autoencoder,” ICCV 2019 introduce a conditioning mechanism using a Lagrange multiplier to produce a conditional autoencoder to avoid training multiple networks in a similar way to the proposed paper. \n\n•\tHe et al., \"Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration,\" ECCV 2020 uses a conditional network to control residual connections.\n\nTherefore, the paper may benefit from recasting the problem as one of learning a conditional neural network for image restoration.  The approach of modifying the network parameters rather than the inputs or features appears to be novel.\n\nThe approach to adjusting the neural network weights based on the parameter is simple and allows the network parameters to change.  Figure 1 gives a nice conceptual visualisation of the concept, and Figure 5 shows the kernel weights being adjusted in practice based on the noise level parameter known a priori.  While this formulation is simple, some details remain unclear to this reviewer.  For example, in practice, how are the lower and upper bounds (x_a, x_b) described below Eq 1 known before the network is trained?  Are there any controls on \\theta_a and \\theta_b in Equation 1 to ensure that \\theta_a < \\theta_b?  If this condition wasn’t satisfied it would seem the network would become unstable.  The paper doesn’t comment on this so perhaps it doesn’t happen?  However, these points are unclear to the reader.\n\nThe visual results are very impressive, particularly those shown in Figures 2 - 4 which greatly outperform the methods that are compared as there is considerably more detail.  Numerically the results are maybe less pronounced in Tables 2 – 4 however do represent an improvement.  Something worrisome though is the methods used in the comparison.  For denoising, the most recent algorithm is from 2018; for super-resolution 2019.  However, the field is moving fast so it is not convincing that the methods in comparison are truly state-of-the-art; recent papers are including methods like attention for example unlike DnCNN or FFDNet.  One might expect papers from 2020 compared to; or referencing recent challenges at the NTIRE 2020 workshops involving denoising or super-resolution.  For example, \n\n•\tZamir et al., “Learning Enriched Features for Real Image Restoration and Enhancement,” ECCV 2020 has the highest score in 2020 on the paperswithcode leaderboard on SIDD and is used for different image restoration tasks.\n\nThe paper has a number of typographical and grammatical issues; please proofread carefully.  There are too many issues to list here.  A common issue is a missing indefinite article, for example, on Line 251, “In super-resolution problem” should be “In the super-resolution problem”.  Note this a common issue in the paper and not limited to Line 251.\n\nOverall, the challenge of conditional image restoration is an important one, and the paper is appreciated by this reviewer.  However, the paper lacks some awareness of the literature and comparisons to other methods are not fully convincing.  If the paper could more thoroughly characterize conditional neural networks and provide more comprehensive comparisons to more recent methods it could be strengthened.  \n\n# Update\nThank you for the additional details and clarifications in the author response.  Having read the other reviews and author responses as well, I've raised my rating by one point.  I think this paper is addressing an important problem in an interesting way that has novelty.  I still have concerns about comparisons to more recent work and issues with presentation and grammar which are holding the paper back a bit. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper appears to be missing a societal impact statement.  The paper also doesn’t really characterise its limitations.\n\nSuggestions would to be prepare a societal impact statement.  For limitations, it would be helpful to understand if the network can become unstable as mentioned above when training.  It would be helpful to see some examples where the method fails to produce an excellent result to fully characterise the method.", "review_text": "", "extra_scores": {"ethical_concerns": "None are identified", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a functional neural network for image restoration/reconstruction tasks. The authors develop an adaptive restoration for different parameter levels using a single model essentially by linearly mixing the two networks. This is somewhat akin to doing meta-learning for restoration. The authors provide various experiments for a variety of restorations tasks showing improvements compared to existing approaches. The reviewers mostly thought the paper's approach was interesting and commended the clarity of the paper. However, they also raised concerns about novelty, differentiation of task and network parameters, baselines. There was a lively discussion in the rebuttal period and both the authors and reviewers engaged in the discussion. As a result most reviewers while somewhat lukewarm lean towards acceptance. I share this opinion. The paper has some interesting ideas but can improve in other aspects. I recommend acceptance but ask the authors to follow the recommendations of the reviewers to improve the final peresentation.
Decision: Accept (Poster)