Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper introduces a new perspective for designing image restoration framework by unfolding the Taylor’s Formula. The proposed framework consists of two steps, and can be integrated to existing deep networks.
main_review: Strengths
1. The authors analyze image restoration tasks in a framework of Taylor's Formula. The idea is interesting.
2. The proposed method can be integrated into exisiting methods and boost performance.

Weakness.
1. My main concern is the technical correctness part of the method.
    First, most image restoration tasks are ill-posed problems, thus a analytical and accurate inverse process is impossible by principle, as in Eq (4).
    Second, Taylor's formula is only suitble for functions that are high-order smooth. As for CNN, operators such as dropout or max-pooling are not smooth. Then what kind of CNNs are suible for the proposed method

2. How to supervised G, so its result can approximate k order output ?
    If we cannot ensure the meaning of G's output, is it just some kind of residual learning?
    I would like to see more comparisons between residual learning method and this method.

3. G is a network with additional parameters. When it is integreted to other methods, the total parameters increase.
    Is this the reason for performance gain ?
limitations_and_societal_impact: Mathematically, using Taylor's formula requires several assumpations about the function, which may not be suitable for all restoration tasks.
I think it would be better to be more strict and analyze what kind of restoration tasks are suitble for this method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed to use additional network structures to approximate the derivative parts of Taylor expansion for image restoration. The idea is similar to residual learning by recursively adding details to the images for a better visual quality. 
main_review: Pros,

- The novel idea of combining Taylor formula with image restoration tasks, and good explanation of the motivations.
- Extensive experiments on different image restoration tasks including deraining and deblurring. 

Cons,

- The proposed methods may not be as effective as expected, since the implementation of the derivative parts are the simple additional RNN structures with feature aggregation from different levels. The derivative functions are also approximated by simple convolutional layers. 
- The quantitative comparisons with other baselines after integrating the new frameworks may not be fair enough, since the new model contains additional parameters and it will bring performance improvement.
- The authors claim a better detail restoration, but it cannot be reflected using PSNR or SSIM metrics. PSNR can be still higher even though the results are blur. 
- The visual results are not clearly revealing the better quality, and the 3-order network is not clearly better than 0-order in visual.
- The authors empirically chose 3-order as the best-performance baseline, but what should be the theory support that 3-order network works the best? Table 3 and 4 also show some randomness of different orders, without showing clear evidence of increasing performance of using higher orders. Intuitively, it should be the case that using higher-order will bring back more details. But the results make sense given the current implementation.
- Some typos in line 250-251.

I like the idea of approximate a Taylor formula for image restoration, but the current implementation is not the best way to achieve this idea. The experiment results lack fairness, and more visual comparisons should be provided. Currently, I sit on borderline. 
limitations_and_societal_impact: See the above comments.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The author proposes a novel deep neural network, which leverages the intrinsic knowledge of low-level vision and Taylor’s Approximations, for the image restoration tasks. The author found that the main part and derivative part of Taylor’s Approximations takes the internal similarity as the high-level structure and spatial details of image restoration, respectively. From this, the author breaks down the restoration process into two manageable steps, corresponding to the mapping function and derivative function, respectively. The proposed framework is orthogonal and can be easily integrated into existing deep learning-based methods. The idea of using Taylor’s Approximations to guide deep network design for image restoration is quiteThe limitation and potential negative societal impact of their work have been discussed in their paper. interesting. 
main_review: Strengths: 

1)	The author proposes a novel image restoration framework by leveraging the intrinsic knowledge of image restoration task and Taylor’s Approximations. This idea is interesting and clearly differs from previous related methods. 

2)	The framework is designed by unfolding Taylor’s formula, which is sufficiently supported by theoretical analysis. This design is more reasonable than current roughly layer-stacking deep networks for image restoration.

3)	The proposed framework can be easily integrated with existing methods, which improves their performance by higher PSNR values and sharper details. Moreover, the submission is clearly written, well organized and easy-to-follow.

4)	Extensive experiments over the representative image de-blurring and image de-raining tasks are reported. Quantitative and qualitative results show its effectiveness and scalability. Besides, ablation studies about Taylor’s order are well designed and reported to analyze the order selection.

Weaknesses: 

1)	The author employs Taylor’s Approximation to unfold the universal image restoration model in Line (130-132). However, it is inadequate that only image de-raining and image de-blurring tasks are reported to demonstrate the effect. More comparisons and results of other image restoration tasks, e.g., super-resolution, should be presented.

2)	The author presents the ablation experiments about Taylor’s orders. The results show that that of Order = 3 performs the best. However, the authors have not clearly illustrated the underlying reasons. For example, feature visualization or other technologies should be adopted to do so. 

3)	If the loss is supervised at each layer of the derivative function part, will the effect be better?

4)	How is the generalization ability of the model in real scenes?

limitations_and_societal_impact: The limitation and potential negative societal impact of their work have been discussed in their paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to develop an image restoration framework through a combination of Taylor’s Approximation and deep learning to break the entire recovery process into two manageable steps. Specifically, the former learns high-level contextualized information, and the latter combines it with the degraded input to progressively recover local high-order spatial details. Several promising methods are integrated with the proposed framework and achieve some improvement. Ablation studies about Taylor’s order are also conducted to provide a comprehensive analysis. 
main_review: Pros:
+ The authors innovatively propose a new image restoration framework, in which others researchers or users are likely to utilize it for further restoration performance improvement. The exploration of combining classic methods with deep learning for image restoration is worth trying and encouraging.

+ The claim of this submission is well supported by theoretical analysis. The authors revisit the connection between image restoration process and Taylor’s Formula, and break the restoration process into two manageable steps for the deep network design.

+ Extensive analysis on two image restoration tasks (image de-blurring and de-raining) clearly verify the effectiveness and scalability of the proposed method. 

Cons: 
-Limited reported image restoration tasks. I am curious that whether the proposed method can deal with other ill-posed image restoration tasks, like the classical super-resolution and image de-noising. It would be better if the authors could provide more discussion about it. 

-Some detailed comparison between the original baselines and that integrating with Taylor’s framework is missing, e.g., the computational time and Flops.

-The premise of Taylor’s Formula is that \epsilon in Equation (6) should be small enough. Are there any other image restoration tasks that do not meet this premise?

-What is the difference between the proposed Taylor’s approximation framework and other deep unfolding frameworks, such as HQS [*1, *2] or proximal gradient operator [*3]?

[*1] Learning Deep Priors for Image Dehazing, ICCV, 2019.

[*2] Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels, CVPR, 2019.

[*3] Deep Gradient Projection Networks for Pan-sharpening, CVPR, 2021.

limitations_and_societal_impact: The author has clearly presented and solved the limitation and potential negative societal impact.
review_title: null

Meta-review: This paper proposes a new method for image restoration using deep neural networks based on decomposing the problem using a Taylor approximation. 3 out of 4 reviewers appreciate the contribution as useful and novel. The most negative reviewer (79vC) has the reasonable concern that the proposed method seems mostly empirical and is not well supported by theory. Unfortunately this reviewer only provided a short review and did not engage in committee discussions or acknowledge the author response: for this reason we cannot give it too much weight in making a decision. The other reviewers acknowledged the author response and are satisfied by it. No serious concerns were identified that would invalidate the contribution made by this paper. The author response looks convincing to me as well: Authors, please integrate this discussion and all promised new results into the camera ready version of the paper.
Decision: Accept (Poster)