Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Multi-Scale Adaptive Network (MSANet) for single image denoising with three blocks: Adaptive Feature Block (AFeB), Adaptive Multi-scale Block (AMB) and Adaptive Fusion Block (AFuB). The network architecture design explores the within-scale and cross-scale characteristics of multi-scale networks. Extensive experiments demonstrate that these presented modules gain better performance than the baseline network and some other denoising methods on several real or synthetic datasets.

strengths_and_weaknesses: Strengths:

+This paper is well-written, and the proposed architecture is straightforward to follow.

+The summary of the difference between low-resolution and high-resolution scales in Fig.1 is valuable, and the motivation to explore the within-scale characteristics in architecture design is reasonable.

Weaknesses:

-The relationship between the motivation of exploring the within-scale characteristics and the design of the three proposed modules is farfetched. There is still a lack of related experiments or visualization results that prove that the presented modules can utilize the within-scale characteristics well.

-Comparison with the latest SOTAs such as MPRNet [1], Uformer [2], and DGUNet [3] is missing. The performances reported in this paper seem to be much worse than these previous denoising methods.

-The ablation study in Section 4.4 is limited to demonstrating the different characteristics of each module. Adding the proposed modules seems only to gain some improvements over the baseline network, but the in-depth analysis is missing. I can not get any intuitive conclusions from this part.

[1] Zamir, Syed Waqas, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

[2] Wang, Zhendong, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

[3] Mou, Chong, Qian Wang, and Jian Zhang. Deep Generalized Unfolding Networks for Image Restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

questions: 1) In Fig.1, the authors claim that the robustness of high-level and low-level scales is different. What is the definition of robustness there? Is there any experiment or reference to support this statement?

2) How does the motivation of exploring within-scale characteristics cause the design of the network architecture? The proposed three modules seem to be only simple combinations of residual blocks and deformable/dilated convolution. The performance improvements may come from the additional parameters rather than the designed architecture.

3) How do we combine the features weighted by channel attention and the features weighted by spatial attention in AMB？ The detailed process seems to be omitted.

limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates existing multi-scale methods, and discovers the within-scale characteristics of multi-scale features are ignored. Therefore, the paper reveals this missing piece for multi-scale architecture design, and accordingly proposes a novel Multi-Scale Adaptive Network (MSANet) by simultaneously exploiting the within-scale characteristics and the cross-scale complementarity. MSANet uses AFeB, AMB to build different subnetworks corresponding to diﬀerent scales for exploiting the within-scale characteristics, and AFuB to fuse the multi-scale features with varying characteristics from coarse to ﬁne for exploiting the cross-scale complementarity. Extensive experiments on both three real and six synthetic noisy image datasets show the effectiveness of the proposed designs and the advantages over the previous methods. Moreover, supplementary material shows more results together with the main paper.
strengths_and_weaknesses: Strengths

(1) Simultaneously taking the within-scale characteristics and the cross-scale complementarity into multi-scale architecture design increases the performance of single image denoising task. The reviewer agrees that the within-scale characteristics is significative for multi-scale architecture and single image denoising.

(2) The motivations of the proposed designs, i.e., MSANet and the three neural blocks, are clear and convincing. The experimental results show the effectiveness and robustness of the proposed designs.

(3) The authors conduct adequate experiments on both synthetic and real noise image datasets, and investigate the effectiveness of the proposed three blocks as well as the model complexity. Overall, the experiments are complete.

Weaknesses

(1) The proposed MSANet is not competitive in parameter numbers as there are different subnetworks corresponding to different scale features. This limits the use of the model on memory-constrained devices. In contrast, the running time and FLOPs are competitive due to multi-resolution features.

(2) Using AFeB and AMS together could significantly improve the performance, while using either AFeB or AMB alone slightly improve the performance, i.e., they are bound together for using. This limits the separate use of the two blocks.
questions: (1) Both AFeB and AFuB are based on deformable convolution to adaptively sample and weight the features, but one is used to adaptively select details and ﬁlter noises, the other is used to fuse the multi-scale features with varying characteristics. What caused this crucial difference? Why adaptively sampling and weighting the feature is important for exploiting the within-scale characteristics and the cross-scale complementarity?

(2) Why using AFeB and AMS together could significantly improve the performance while only using one of them slightly improve the performance?

(3) In Table 7, only three baselines are investigated w.r.t. the model complexity, which is inadequate. More baselines should be investigated and compared.
limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Overall, this paper aims to tackle the task of image denoising based on the drawback (not exploring the internal information of each scale in the network) of multi-scale networks. Specifically, the authors propose three modules to extract feature maps, multi-scale construction and feature aggregation to run a noisy image. Extensive experimental results are shown in the paper.
strengths_and_weaknesses: Overall, the architecture of the paper is clear and the writing is good, but the theoretical and methodological parts are not enough to meet me.

Strengths

[+]  The authors seem to propose an effective image denoising solution, described in the motivation, related work, methodology and experiments.

Weaknesses

[-] First of all, the title does not show the author's intention, which is a bad case (No expressed motivation).

[-] The abstract description starts by depicting a big problem, namely the general weakness of multi-scale networks, but focuses on a simple application.

[-] The introduction and related works ignore Transformer-based models.

[-] Figure 1 does not convey the author's intention, which is confusing. Why not use the feature map presentation to illustrate that there is a problem with information extraction within the scale?

[-] Do not insert ‘so on’ to follow ‘such as’.

[-] The biggest problem is that the motivation is not valid and I don't see cases to illustrate the problems with multi-scale networks nowadays.

[-] In addition, the solution is essentially adding convolutional layers (adaptive) between scales, does this solve the problem?

[-] What worries me is that the methods of CVPR'22 have been released, so why are there no relevant experiments to compare these methods?

[-] relu---> ReLU, pytorch--->PyTorch.
questions: [-] My problem is mainly how can the authors prove that the existing approach does not make use of the information within the scale?

[-] The authors' proposed approach does not seem to be innovative.
limitations: Limitations have been illustrated.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: To exploit the potential of the within-scale characteristics and cross-scale complementarity of multi-scale features, three elaborate neural blocks and a novel Multi-Scale Adaptive Network (MSANet) are proposed for single image denoising. Specifically, AFeB and AMB are designed by taking the within-scale characteristics of multi-scale features into consideration, AFuB is designed to exploit the cross-scale complementarity of multi-scale features. Three neural blocks are combined to be the scale-wise subnetwork in MSANet by adapting to the feature characteristics of the corresponding scale. Ablation studies demonstrate the effectiveness of the proposed neural blocks and network. Sufficient comparison experiments on three real and six synthetic noisy image datasets compared with 12 baselines show the advantages of the proposed method.
strengths_and_weaknesses: A. Strengths
1. The idea of simultaneously exploiting the within-scale characteristics and cross-scale complementarity of multi-scale features is interesting and reasonable. The accordingly proposed MSANet incorporates them into multi-scale architecture design, and shows impressive performance on single image denoising task.
2. The design of AFuB is novel and ingenious. Taking the disordered ﬁne-grained image details in high-resolution features and the coarse-grained image context in low-resolution features as input to adaptively sample, weight and transfer the ﬁne-grained image details into the coarse-grained image context.
3. Sufficient experiments are conducted to demonstrate the effectiveness of the proposed designs and the advantages over the previous works. Moreover, the supplementary material also shows more details and results together with the main paper.

B. Weaknesses
1. Since the cross-scale complementarity and the within-scale characteristics are not only restricted to single image denoising task. Therefore, the designs will be more persuasive if they could also achieve improvements in other image restoration tasks.
2. Compared with the three real and the three synthetic color noisy image datasets, the performance improvements over the baselines on the three synthetic grayscale noisy image datasets are less significant.

questions: 1. Although different-scale features are processed by different subnetworks, and the subnetworks in the finest and coarsest scale are clearly scale-specific. However, it’s difficult to understand the scale-specific designs in the two middle subnetworks. More clear interpretations are needed for a better understanding.
2. As shown in Table 6, using AFeB and AMS together could signiﬁcantly improve the performance. However, using either AFeB or AMB alone slightly gains the performance over ResB. Why is that? Some clear explanations are needed for a better understanding.
3. As shown in Table 7, although MSANet contains more parameters, the running time and the FLOPs are not much. Why does this happen? Please give some explanations.
4. Some typos, e.g., “disorderly” -> “disordered” in the paragraph of Adaptive Fusion Block (AFuB).

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Different from existing works treat different scale features equally, in this paper, the authors reveal the different scale features show varying characteristics and should be processed by scale-speciﬁc rather than homologous structures. To simultaneously embrace the within-scale characteristics and cross-scale complementarity of multi-scale features, the authors propose a novel Multi-Scale Adaptive Network (MSANet) for single image denoising. Speciﬁcally, three neural blocks are designed, i.e., adaptive feature block (AFeB) for adaptively sampling and ﬁltering features; adaptive multi-scale block (AMB) for expanding the receptive ﬁeld and adaptively aggregate multi-scale features; adaptive fusion block (AFuB) for adaptively fusing the multi-scale features with varying characteristics. Ablation studies indicate the blocks are eﬀective, and comparison experiments show MSANet achieves better performance than alternatives.

strengths_and_weaknesses: Strengths:
1. The authors reveal the different scale features show varying characteristics and should be processed by scale-speciﬁc structures for this first time. Based on the observation, MSANet is proposed to utilize the within-scale characteristics and cross-scale complementarity of multi-scale features. Extensive experiments are conducted on both three real and six synthetic noisy image datasets to show its superiority.
2. Three neural blocks are designed by considering the within-scale characteristics and cross-scale complementarity of multi-scale features. Ablation studies indicate these blocks are eﬀective.
3. In general, the paper is well written. The related works are clearly illustrated, the shortcomings of existing multi-scale networks are analyzed, and accordingly MSANet is proposed by solving the shortcomings. The framework is clear and easy to understand, and the experimental results are explained in detail.

Weaknesses:
1. The idea of taking advantage of within-scale characteristics and cross-scale complementarity is not limited to denoising task, but is a general idea of multi-scale architecture design. Therefore, exploring and verifying this idea in more tasks and areas would make this work be more significant.
2. MSANet contains more parameters than most of the baselines due to multiple subnetworks for multi-scale features.


questions: 1. In the paper, “adaptive” is frequently used to describe the proposed network as well as the three neural blocks, but itself is not explicitly discussed. Some discussions are recommended for a better understanding to this work.
2. AFeB is designed to adaptively sampling and weighting the input features, which is highly expected for ﬁne-grained features. However, why adaptively sampling and weighting the ﬁne-grained features is good for the denoising task to preserve the image details and filter unpleasant noise?
3. As Table 6 suggests, the performance gains of using either AFeB or AMS alone are slight. However, why using them together could bring significant performance improvements? Please give some interpretations.

limitations: Yes, the authors have discussed the related problems.
review_title: null

Meta-review: metareview: All reviewers are positive about this paper. Although this paper does not achieve the best performance,  it reveals some  insights about some insights about scale characteristics of features, which is  model-agnostic and  potential to design more powerful networks. Also, the proposed method can reduce FlOPs obviously. 
confidence: Certain
Decision: Accept