Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the author proposes a Selective Frequency Network (SFNet) for image restoration. In SFNet, the author designs a multi-branch and content-aware module to decompose the feature into separate frequency sub-bands, and then uses the channel-wise attention mechanism to emphasize the useful information. In addition, to cope with the large-scale degradation kernel, the author proposes an decoupling and modulation module to enlarge the receptive field based on global and window-based average pooling.
strength_and_weaknesses: The paper is well presented and claims better performance on different tasks, while there are some problems in theoretical proof and experimental part.
clarity,_quality,_novelty_and_reproducibility: 1) There are some problems in the proof of this paper. In Eq. 8 and Eq. 9, the author describe that the first row(diag(1,0,0,…,0)) in the Fourier domain is low frequency and the others are high frequency. According to this description, for a feature of size CxHxW, low frequency is the mean value in the spatial dimension and the size of low frequency is Cx1x1, which can be seen in Section 3.2 and Appendix C. This description is contrary to the commonly used low frequency and high frequency. In practice, the size of low frequency and high frequency are CxHxW and low frequency occupies a large part of information and high frequency is only in the edge area. In the two-dimensional Fourier domain (the data has been processed by fftshift), we generally believe that the low frequency is the region within the radius R, and other regions are the high frequency [1-3]. Therefore, the Eq. 8 and Eq. 9 may not be consistent with the actual situation. In other words, the author only provided theoretical proof of MDSF for extreme low frequency, but did not provide theoretical proof of MDSF for other low frequency. More importantly, the author did not demonstrate that the low frequency in the MDSF meets Eq. 8. The low frequency in the MDSF is obtained through learning, not the mean value of the feature. In addition, the size of low frequency is CxHxW, which is completely inconsistent with the low frequency scale defined by the author. Therefore, this proof is problematic and the author confused the definition of low frequency.
2) The author did not write clearly the experimental settings, such as the number of branch in MDSF, the number of group in MDSF.
3) The experiment is not rigorous. In the ablation experiment, the number of epoch is inconsistent in the ablation experiments of different components. Authors should unify the number of epochs.
4) There are some typos in Section 4.8 and Eq. 15.

[1] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal Frequency Loss for Image Reconstruction and Synthesis. In ICCV, 2021.
[2] Mu Cai, Hong Zhang, Huijuan Huang, Qichuan Geng, Yixuan Li, and Gao Huang. Frequency Domain Image Translation: More Photo-realistic, Better Identity-preserving. In ICCV, 2021.
[3] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A Fourier-based Framework for Domain Generalization. In CVPR, 2021.  

summary_of_the_review: The paper is well presented and claims better performance on different tasks, while there are some problems in theoretical proof and experimental part.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works on image restoration and introduces a multi-branch dynamic selective frequency (MDSF) module with learned kernels to generate high and low-frequency features and a multi-branch compact selective frequency (MCSF) module to enhance the features with an attention mechanism. The proposed method achieves several state-of-the-art performances for image restoration tasks.
strength_and_weaknesses: Strength
-The paper works on frequency decomposition and manipulation design, which is important for image restoration.
-The proposed method outperforms many state-of-the-art methods on different image restoration tasks.

Weakness
-Comparison with other attention mechanisms. The proposed MDSF and MCSF are similar to the multi-brach pathways with attention mechanisms. Since the kernel weights are learned, it is hard to tell whether the learned dynamic frequency decomposition benefits more than attention. The authors can replace the modules in Fig 2. (b) with other attention forms to verify the argument.
-Analysis of the high- and low-frequency features. The authors should provide more analysis of the obtained features of MSDF to verify the high- and low-frequency properties and frequency diversity.
-Ablation studies. The authors conduct ablation studies with different settings. It is hard to compare across different components. The authors should give a more unified ablation.
-Comparison to state-of-the-art. The authors should include more recent works, such as NAFNet for image debarring, in the comparison. It is also not obvious whether the proposed modules or training strategies contribute more to the final results.  Liangyu Chen, et al. "Simple baselines for image restoration." arXiv preprint arXiv:2204.04676 (2022).
-The authors can include other related works, e.g., Jaeyoung Yoo, Sang-ho Lee, and Nojun Kwak. "Image restoration by estimating frequency distribution of local patches." CVPR 2018
-The paper quality needs to be improved.
clarity,_quality,_novelty_and_reproducibility: -The paper quality needs to be improved.
-The authors should further refine the manuscript organization and writing, e.g., Section 3.2.1.
-The authors can refine and reorganize the figures, such as Fig. 2 and Fig.3, to make them clearer and more good-looking. Some other analysis figures are also blurry.
-Some citations are missing in the manuscript, e.g., Mao et al. in Frequency based image restoration, Section 2.
summary_of_the_review: The paper achieves good results for different image restoration tasks. However, the paper lacks some validation of the proposed method and comparison to more recent works. Also, the paper quality needs to be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on image restoration tasks in the frequency domain. Specially, a multi-branch dynamic selective frequency module is employed to select the most informative frequency to recover. Besides, the authors propose a multi-branch compact selective frequency module to enlarge the receptive field. Experimental results demonstrate that the network utilizing these two modules performs better than state-of-the-art methods on different image restoration tasks.
strength_and_weaknesses: Strengths:
1. The authors design a useful module to decouple the feature map into different parts and select the most important ones for restoration.
2. The network architecture is well-introduced and it is easy to reproduce the main results.
3. The experiments show that the proposed network achieves state-of-the-art results on five image restoration tasks. The supplementary material shows more results to demonstrate the effectiveness of the proposed method. 

Weaknesses:
1. It will be better if the authors can integrate and simplify Figures 1-3 properly. 
2. How to switch the high-pass and low-pass filters in the Filter Switcher automatically?
3. More visual results or the corresponding PSNRs can be added in Figure 5 to show the superior of the proposed network. Besides, the typesetting of Figure 5 should be modified. If the space is limited, try to delete some compared results in Table1-7.
4. Are the ablation studies all about the deblurring task? If so, add this explanation. 
5. The FLOPs should be added in Table 10 to support that more MCDF modules bring few parameters and complexities.

clarity,_quality,_novelty_and_reproducibility: The manuscript is basically clear. The technical content of the paper is correct. The present paper is reproducible. Incorporating the proposed MDSF and MCSF into the selective frequency network is able to address most image restoration tasks.
summary_of_the_review: I tend to accept this paper based on its solid contributions and novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new deep model framework named Selective Frequency Network (SFN) for multiple image restoration tasks, including single image defocus deblurring, image dehazing, image motion deblurring, image desnowing and image deraining. SFN builds on two types of basic blocks, i.e., Multi-branch Dynamic Selective Frequency module (MDSF) and Multi-branch Compact Selective Frequency module (MCSF). MDSF is proposed to decompose the feature into frequency subbands dynamically and locally for accentuating the effective ones further, while MCSF is designed by using global and window-based average pooling to deal with the large-scale degradation kernel. The proposed SFN exploits a U-type structure to assemble the modules of MDSF and MCSF. Extensive experiments are carried out to validate the superiority of SFN and the effectiveness of block components MDSF and MCSF. 
strength_and_weaknesses: The paper pays attention to the large discrepancy of sharp and degraded image pairs in the frequency domain, and proposes the modules MDSF and MCSF which are used to build SFN net. It is novel in terms of method. The structure of the proposed SFN is clearly presented in the text, and more details are further discussed in Appendex A-H. The content of the paper is very sufficient. The proposed method is validated by extensive experiments to be effective on many image restoration tasks, and perform favorably compared with lots of SOTS methods on various datasets. In addition, ablation studies are made to explain the design of the modules MDSF and MCSF.

1.What are the advantages of the “Shallow Layer and output layer” over existing multi-scale methods? 
2.Why MDSF and MCSF are concatenated in Figure 1? These two modules are used independently, right?
3.How to control the kernel size of different decouplers in MDSF? From figure 2(b), it’s seems that all the kernel size are k^2?
4. There are still some typos in the paper:
(1) In the last paragraph of Section 2, both “Xu et al.” and “Mao et al.” lack reference numbers.
(2) In Figure 2(b), the output of MDSF is the concatenation of the low-frequency feature and the high-frequency feature, while in Figure 2(c), MCSF receives two inputs, i.e., the low-frequency feature and the high-frequency feature from MDSF. I wonder if the concatenation operation in Figure 2(c) is necessary.
(3) The parameter c in equations (2) and (4) takes a real value as in c \in R^{C_i} and c\in R^{2C}. However, c is the index of a channel according to the context of the paper, and could take an integer value.
(4) Section 3.2.1 could be modified to Section 3.3.
clarity,_quality,_novelty_and_reproducibility: The paper is novel enough for its proposed modules MDSF and MCSF as well as SFN established with them. The content is presented clearly and organized reasonably, and the technique described could be reproduced easily. Lastly, the writing is good and fluent.
summary_of_the_review: The paper is novel enough for its proposed modules MDSF and MCSF as well as SFN established with them. The content is presented clearly and organized reasonably, and the technique described could be reproduced easily. Lastly, the writing is good and fluent.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes two new modules, i.e., the multi-branch dynamic selective frequency module (MDSF) and the multi-branch compact selective frequency module (MCSF), and incorporates them into a U-shaped backbone for image restoration. Experiments are conducted to validate the effectiveness of the framework. The proposed method is novel and obtains state-of-the-art results on different image restoration tasks. After the first round of review, Reviewer BBSS pointed out that there were some issues in the proof. Reviewer 4vXg mentioned that the paper lacked some validation of the proposed method and comparison with more recent works.  Reviewer 2iAU provided some suggestions about how to improve the current presentation and experiments. Reviewer Qxbc asked some technical questions about the proposed method. During the rebuttal, the authors provide detailed replies and extra experiment results to address the above issues. The revision has been made in the paper by taking into account all valuable suggestions provided by the reviewers. The paper eventually receives 2 accepts, 1 marginal accept, and 1 marginal reject.  After reading the rebuttal and having an internal discussion, the AC thinks that all major concerns have been addressed by in the rebuttal and recommends accepting the paper due to its novelty and solid experiments.   

justification_for_why_not_higher_score: This paper is a novel and solid application work for image restoration tasks. Considering that it doesn't provide a fundamental contribution or make a breakthrough in the field, the AC recommends "accept with poster".  
justification_for_why_not_lower_score: The proposed method is novel and obtains state-of-the-art results on different image restoration tasks. The AC recommends accepting it.
Decision: Accept: poster