Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a network interpolation scheme for non-blind image restoration
problems with a single parameter (e.g. noise level for denoising).
The parameter is remapped to [0, 1] using an affine mapping. 
Effectively, two networks are trained such that
for any given problem parameter in [0,1],
an interpolated network is obtained by linearly blending the two networks 
parameters. The interpolated model is then applied to the input.

main_review: The technical contribution of this paper is rather meager, as network interpolation
has been discussed multiple times in the past, including in relation to image
restoration. Still, despite some minor  concerns about the evaluation, the proposed
FuncNet model outperforms reasonable recent baselines, while being trained on
a *range* of parameter values.



### Strengths

- Improving neural network performance on parameterized problem such that
the network adapts to the input parameter is a timely problem with practical
importance.
- Incorporating the parameter as a linear blending weight between two
sets of network parameters (weights and biases) is interesting.
- The proposed method quantitatively outperforms recent previous work 
on denoising, super-resolution and JPEG deblocking.


### Weaknesses

- The proposed technical solution is essentially network interpolation, which
is not novel, and should therefore be discussed. For
instance "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks"
[Wang 2018] linearly blend between the parameters of a model trained with a GAN loss, and one without.
Even closer to the domain discussed here, "Deep Network Interpolation for Continuous Imagery Effect Transition" [Wang 2019]
interpolate between different filter effects.
- How would the method if the restoration task had more than one parameter (e.g. 2 noise parameters for 
a Heteroskedastic Gaussian noise model)?
- The discussion of related work is insufficient.
1. For instance, CBDNet "Toward Convolutional Blind Denoising of Real Photographs" [Guo 2019]
uses yet another approach to incorporate the model parameter: an additional prediction branch
supervised to regress the ground-truth parameter.
2. Another largely used approach to condition a network to a scalar parameter,
is to modulate all its intermediate features based on the condition, using a mechanism like
FiLM "FiLM: Visual Reasoning with a General Conditioning Layer" [Perez 2017]. 
"Flexible Image Denoising with Multi-layer Conditional Feature Modulation" [Du 2020] apply this to denoising.
"Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration" [He 2020]
use a similar conditioning mechanism for image restoration, and deal with multiple parameters simultaneously.
- I have some reserves about the evaluation of the proposed model. Why is RIDNet omitted from
the quantitative tables? Was the model retrained with the same noise model? 
FFDNet is a bit outdated for a baseline, consider re-training a more recent
more with the same procedure as the proposed model (i.e. same noise model, etc).
- Thank you for the supplemental material. Seeing more images is useful, but
I would have liked to see more diversity in the input images, rather than
selected crops. It is difficult to judge the robustness of the method otherwise.
- I do not think the self-ensembling (FuncNet+) adds much to the paper.
- It is not clear from the paper whether the baselines were used as-is, retrained
on *all* parameter values (for their respective application), or trained specifically
on each parameter value.
- In general, it would be useful to understand how FuncNet compares to
the "plain" version, specialized to each given parameter. This would give 
an upper-bound on the quality the model can reach, and therefore give
a sense of how much quality is lost by using the interpolation scheme instead.



### Misc

- 224: 0.0001% inaccurate, do you rather mean the linear interpolation of parameters
takes this fraction of the total compute?
- The denoising result in Fig3 is nice. I am surprised RIDNet performs so poorly, since it is more
recent, and generally believed to outperform FFDNet.


### Update

I bumped by recommendation by +1. My criticism about the method being "just network interpolation" was not completely fair since the interpolated models are also used and supervised during training. The contribution therefore has some novelty that I did not fully grasp originally.
limitations_and_societal_impact: Limitations are not discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a functional neural network for image restoration. They use a linear function with respect to the task parameter to adjust the weights of the network, to achieve an adaptive restoration for different parameter levels using a single model. The idea is similar to meta-learning on restoration, or weight modulation derived from instance normalization. The experiment shows its better quantitative performance compared with SOTA among three tasks, and better visual quality. 
main_review: Pros,

- Elegant design for the proposed functional neural network and the writing of the paper is good and very easy to follow.
- Extensive experiments on SOTA and benchmark dataset revealing the better performance.

Cons,

- The idea is not novel enough, and the function design is not that interesting as in [1]. Besides, the task level is given, so the model is not blind for an adaptive inference with any arbitrary inputs, making the practical values very limited. 
- It's better to differentiate task parameter (noise level, scale factor etc.) with network parameter (model weights). Some terms in section 3.1 are somehow confusing, especially 'x' mostly represents input images. When the authors refer x as the parameter, I original though it was the model weights. 
- While comparing with other baselines (like denoising or JPEG deblocking), have the author used one single model trained with all the parameter levels, or each model per level separately? Besides, the parameter size should be controlled for a fair comparison. It seems the double parameter size improves the restoration performance. I understand that one set of parameter can represent multiple models with any given x, but it will be more fair to also double the weights of the single model of the same architecture. 
- In section 4.3, it's not clear what is the 'input' and 'output' layer the authors refer to. The visualization is a little bit confusing to interpret the results. Since the function is linear, it should not be surprising to see the linear effects of kernels.
- Missing some SR reference like [2]

My main concern of this paper is the novelty and technical levels for NeurIPS, and its practical values in real image restoration. The better performance may be from some unfair comparison, and there is no explicit design for a single model with adjusted weights to be better than others. 

[1] Learning Continuous Image Representation with Local Implicit Image Function, CVPR'21
[2] Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining, CVPR'20


limitations_and_societal_impact: See the above comments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces functional neural networks for image restoration that are parametrized by a variable (e.g. super-resolution factor, noise level, JPEG quality level).  A functional network is one where the neural network parameters themselves are a function of the problem parameter known a priori.  This way, the neural network weights can adapt based on the problem, for example, a denoising CNN can apply stronger denoising when the noise level increases.  Different functions are explored including linear functions which require twice the storage but using a single model.  Compelling experimental results are shown across a range of parameters of the problem.
main_review: The paper addresses an important and common problem in image restoration – how to develop a single neural network that can work effectively across a range of settings (e.g. a denoiser that can denoise images with varying levels of noise).  The paper effectively argues that making neural networks for each setting produces good results, but also many networks; whereas a single network that works across a large range of settings may be less effective.  Indeed this a well-known issue.

The paper doesn’t use this terminology, but essentially it is proposing a solution for a conditional neural network.  That is, the network itself is conditioned based on the parameter (e.g. noise level).  The paper provides a nice taxonomy of different approaches to designing conditional neural networks, categorized into six methods in Section 2.  However, this taxonomy is incomplete; there are other methods proposed for conditional networks that appear to be outside the six methods described, for example:

•	Wang et al., “Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform,” CVPR 2018 introduces Spatial Feature Transform modules which modulate the features, conditioned in their case on a segmentation map but this could instead be a map representing the a priori parameter of the problem. 

•	Choi et al., “Variable Rate Deep Image Compression With a Conditional Autoencoder,” ICCV 2019 introduce a conditioning mechanism using a Lagrange multiplier to produce a conditional autoencoder to avoid training multiple networks in a similar way to the proposed paper. 

•	He et al., "Interactive Multi-Dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration," ECCV 2020 uses a conditional network to control residual connections.

Therefore, the paper may benefit from recasting the problem as one of learning a conditional neural network for image restoration.  The approach of modifying the network parameters rather than the inputs or features appears to be novel.

The approach to adjusting the neural network weights based on the parameter is simple and allows the network parameters to change.  Figure 1 gives a nice conceptual visualisation of the concept, and Figure 5 shows the kernel weights being adjusted in practice based on the noise level parameter known a priori.  While this formulation is simple, some details remain unclear to this reviewer.  For example, in practice, how are the lower and upper bounds (x_a, x_b) described below Eq 1 known before the network is trained?  Are there any controls on \theta_a and \theta_b in Equation 1 to ensure that \theta_a < \theta_b?  If this condition wasn’t satisfied it would seem the network would become unstable.  The paper doesn’t comment on this so perhaps it doesn’t happen?  However, these points are unclear to the reader.

The visual results are very impressive, particularly those shown in Figures 2 - 4 which greatly outperform the methods that are compared as there is considerably more detail.  Numerically the results are maybe less pronounced in Tables 2 – 4 however do represent an improvement.  Something worrisome though is the methods used in the comparison.  For denoising, the most recent algorithm is from 2018; for super-resolution 2019.  However, the field is moving fast so it is not convincing that the methods in comparison are truly state-of-the-art; recent papers are including methods like attention for example unlike DnCNN or FFDNet.  One might expect papers from 2020 compared to; or referencing recent challenges at the NTIRE 2020 workshops involving denoising or super-resolution.  For example, 

•	Zamir et al., “Learning Enriched Features for Real Image Restoration and Enhancement,” ECCV 2020 has the highest score in 2020 on the paperswithcode leaderboard on SIDD and is used for different image restoration tasks.

The paper has a number of typographical and grammatical issues; please proofread carefully.  There are too many issues to list here.  A common issue is a missing indefinite article, for example, on Line 251, “In super-resolution problem” should be “In the super-resolution problem”.  Note this a common issue in the paper and not limited to Line 251.

Overall, the challenge of conditional image restoration is an important one, and the paper is appreciated by this reviewer.  However, the paper lacks some awareness of the literature and comparisons to other methods are not fully convincing.  If the paper could more thoroughly characterize conditional neural networks and provide more comprehensive comparisons to more recent methods it could be strengthened.  

# Update
Thank you for the additional details and clarifications in the author response.  Having read the other reviews and author responses as well, I've raised my rating by one point.  I think this paper is addressing an important problem in an interesting way that has novelty.  I still have concerns about comparisons to more recent work and issues with presentation and grammar which are holding the paper back a bit. 
limitations_and_societal_impact: The paper appears to be missing a societal impact statement.  The paper also doesn’t really characterise its limitations.

Suggestions would to be prepare a societal impact statement.  For limitations, it would be helpful to understand if the network can become unstable as mentioned above when training.  It would be helpful to see some examples where the method fails to produce an excellent result to fully characterise the method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a technique to build image restoration networks that are robust to variations in the problem parameters (e.g. noise level or upscaling factor). The main idea is to derive the weights of the network as a learnable function of the problem parameter.
main_review: Overall, the paper is well written and the idea clearly explained. The issue of making models that gracefully handle multiple values of a problem parameter is important and I agree with the authors that has been mostly overlooking or treated with unsophisticated techniques. Some works in real image denoising include specialized techniques to estimate the noise level and therefore be robust to variations but the proposed technique is much more general and applicable to different problems. The idea is original and, while simple, elegantly addresses the problem.

The experimental setting is not very clear. What is the setting used for the baselines? Is a baseline model trained for each parameter value or is a single model trained blindly for all values? Ideally, I think both scenarios should be shown because models typically perform better in the former, although, as the authors say, the storage requirements multiply times the number of parameter configurations.  In the blind case, instead, there would be a problem of fairness of the results because the proposed technique effectively uses twice as many trainable parameters with respect to the baseline and it is possible that the improved performance is due to that rather than the technique. The authors should clarify these points.
limitations_and_societal_impact: Societal impact is not discussed.
review_title: null

Meta-review: This paper develops a functional neural network for image restoration/reconstruction tasks. The authors develop an adaptive restoration for different parameter levels using a single model essentially by linearly mixing the two networks. This is somewhat akin to doing meta-learning for restoration. The authors provide various experiments for a variety of restorations tasks showing improvements compared to existing approaches. The reviewers mostly thought the paper's approach was interesting and commended the clarity of the paper. However, they also raised concerns about novelty, differentiation of task and network parameters, baselines. There was a lively discussion in the rebuttal period and both the authors and reviewers engaged in the discussion. As a result most reviewers while somewhat lukewarm lean towards acceptance. I share this opinion. The paper has some interesting ideas but can improve in other aspects. I recommend acceptance but ask the authors to follow the recommendations of the reviewers to improve the final peresentation.
Decision: Accept (Poster)