Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a Fourier-transform-composed building block for neural networks. It follows the popular transformer-like backbone but replaces the non-local attention module with Fourier-transform-based operations. The core idea is to leverage the global property of the Fourier space, i.e., manipulating local areas within Fourier space would affect the global spatial data, as a drop-in replacement for the global feature extraction mechanism typically empowered by computation-intensive attention operations. The experiments on multiple image restoration tasks (enhancement, draining, dehazing, pan-sharpening) collectively demonstrate the superiority of the proposed method against the selected methods in terms of restoration quality and computation efficiency. 

strength_and_weaknesses: [Pros]
- The paper is clearly written and easy to follow
- Extensive experiments on multiple image restoration tasks looks sound

[Cons]
- The idea of utilizing Fourier transform as a global modeling tool is not new. There are many works that incorporated Fourier transform in the network design but the authors didn't mention (and compare). A closely related competitor is the Fast Fourier Convolution operation proposed in [A], which  explores an efficient ensemble of local and non-local receptive fields in a single unit. This operation is further utilized in image inpainting [B], demonstrating excellent capability to fill large missing areas via global reasoning. Thus, the novelty of the proposed method is quite limited. 
- One of the key insight driving this paper is erroneous. The authors claimed "Fourier transform is capable of disentangling image degrada-
tion and content component", this is unfortunately wrong. The examples shown in Figure.1 is misleading. In fact, the Fourier phase component represents the major "appearance" of a image, so swaping the phase component of two images swaps the contents of these two images (You could simply verify this through swaping the phases of two irrelevant images). Consequently, the author's argument "Fourier transform is capable of disentangling image degradation and content component and the degradation mainly lies in the amplitude component" is suspicious and not justified. 
- The experimental comparisons against SoTA are not well executed. The authors didn't make "apple to apple" comparisons between the proposed Fourier-based operation and other global modeling operations (e.g., various attention modules). SoTA image restoration methods [C,D] that heavily explore global reasoning are also not compared.

[A] Fast Fourier Convolution, NeurIPS 2020  
[B] Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022  
[C] Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022  
[D] MAXIM: Multi-Axis MLP for Image Processing, CVPR 2022  
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: Though this paper shows some interesting aspects, the technical and experimental flaws abovementioned prevent it from a positive recommendation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to design a simple but effective and efficient global modeling paradigm for image restoration, which achieves the competitive performance on several mainstream image restoration tasks with fewer computational resources.
The main contributions of this paper are list as follows:
1. The authors contribute the first global modelling paradigm for image restoration in a simple but effective and efficient manner; 
2. Following the underlying global modeling rules, this work implicitly embeds the Fourier-based image prior into the core designs: Fourier spatial modeling and Fourier channel evolution. 
3. The proposed paradigm achieves the competitive performance on several mainstream image restoration tasks with fewer computational resources.

strength_and_weaknesses: Strength:
1. The authors introduce a newly-designed loss functions to enable the network for better optimization, thus reconstructing the more pleasing results in both spatial and frequency domains.
2. The authors implicitly embed the Fourier-based image prior into the core designs: Fourier spatial modeling and Fourier channel evolution, which achieves the competitive performance on several mainstream image restoration tasks.
Weaknesses:
1. In section Abstract, “the success may be at the cost of model parameters and FLOPs while the intrinsic characteristics of specific task are ignored”, however, compared to previous studies, it is not clear that how to use the intrinsic characteristics of specific task in this work.
2. As for the core designs: Fourier spatial modeling and Fourier channel evolution, but in section 4.3, there is no ablation experiment to verify the specific functions of these two parts, which is critical to this work.
3. Given that the posed paradigm can achieve the competitive performance on several mainstream image restoration tasks with fewer computational resources, but the necessary theoretical support for the good performance is still lacking.

clarity,_quality,_novelty_and_reproducibility: As for the work, the motivation is clear and the authors first propose a theoretically feasible, simple but effective global modeling paradigm
for image restoration. However, the main claims of the paper not at all supported by theory or empirical results.

summary_of_the_review: An ablation study investigates the performance of an AI system by removing certain components to understand the contribution of the component to the overall system. Thus, it is indispensable to do ablation study on the core designs: Fourier spatial modeling and Fourier channel evolution, which is critical to verify the specific functions of these two parts.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper contributes the first global modelling paradigm for image restoration in a simple but effective and efficient manner. The proposed paradigm achieves the competitive performance on several mainstream image restoration tasks with fewer computational resources. It sparks the realms of the global modelling-based customized image restoration framework with promising performance.
strength_and_weaknesses: Strengths:

+ Novel idea: This paper focuses on providing an alternative global modelling-based customized image restoration framework. It contributes the first global modelling paradigm for image restoration in a simple but effective and efficient manner. This work will spark the realms of global modelling-based customized image restoration framework.
+ Interesting analysis and definition: The authors have summarized the de facto global modeling rule of previous studies as the “spatial interaction + channel evolution”. The summary and definition are interesting and impressive. Following the modeling rule and interesting analysis, this work demonstrates that introducing the image prior into global modeling will promote the development of customized global modeling image restoration paradigm. 
+ Sufficient experiments and good performance: This paper conducts extensive experiments on many mainstream low-level tasks. The extensive experiments suggest that the proposed paradigm achieves the competitive performance with fewer computational resources. 
+ The paper writing is clear and easy to follow.


Weaknesses:

+ The implementation details are confused. For image tasks of image de-hazing/de-raining/enhancement, the proposed paradigm can be directly applied to them as shown in Figure 3. However, in terms of guided image super-resolution with two modalities, the proposed paradigm seems to be modified but the details are not clearly described. 
+ Although the proposed paradigm is verified on several image restoration tasks and achieves good quantitative scores under many IQA metrics, it would be better that the visual results are presented.  The reviewer understands the limited space in the main paper. The visual results could be presented in the supplementary material. 

clarity,_quality,_novelty_and_reproducibility: The contributions of this work are significant; the idea is novel and has not seen in previous works; the simple implementation is easy for reproducibility.
summary_of_the_review: This paper proposes to design the global modeling customized image restoration paradigm for the first time and give an alternative solution in a simple but effective and efficient manner. The idea is novel and interesting. I believe that this work will spark the realms of image restoration.  Therefore, I recommend to accept this work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Different from the global modelling-based image restoration frameworks which provided the remarkable advancement, but at the cost of model parameters and FLOPs while the intrinsic characteristics of specific task are ignored, this paper proposed a simple yet effective global modelling paradigm for image restoration. THe most interesting point of this paper is the proposal of general image restoration prior: Fourier transform is capable of disentangling image degradation and content component, acting as the image degradation prior embedded into image restoration framework. The fruitful experimental results prove the advantage of the proposed paradigm.
strength_and_weaknesses: This paper proposed a new perspective to deal with different kinds of image degradation problem: that is to say, Fourier transform is capable of disentangling image degradation and content component, acting as the image degradation prior embedded into image restoration framework. Furthermore, the super advantage of the experiments also illustrate the efficiency of the proposed method.
clarity,_quality,_novelty_and_reproducibility: 1. The core idea of this paper is that the Fourier transform is capable of disentangling image degradation and content component. However, the deep analysis of this point is inefficient. This is important to demonstrate the advantage of the proposed method.
2. This paper claimed to propose a simple yet effective global modelling paradigm for image restoration, specifically, with lower model parameters and FLOPs. As we know, transformer based networks can always occupy small model parameters. The analysis why the propsoed CNN model can achieve smaller model parameters is unclear.
summary_of_the_review: In summary, the core idea and the experiments are the advantage. However, the deep analysis of the proposed module can be further enhanced.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a model which makes use of Fourier transform modules in image restoration.  The potential advantage of this choice is that a model with relatively few parameters (say 40K) can demonstrate high quality results, beating some previous models which are considerably larger.  This is a potentially valuable contribution, but as I argue now, the paper needs a much more thoughtful and thorough re-presentation in order to be published at the level of this conference.

As pointed out by reviewer vjqt, the results are surpassed by other existing methods, and the authors make clear in the rebuttal and revised paper that (a) these systems perform better - "our method is inferior to such methods", and (b) they do not intend to compare to those methods in the final paper, nor even make the clear statement that they perform better.  The rebuttal suggests that this is because those models have 20M+ parameters, but the paper already includes models up to 30M parameters, so that selection criterion is not consistent with what has already been written.

Several times in the rebuttal or in the updated paper the authors make claims such as " our proposed framework is the first to explore the customized image restoration global modeling paradigm." or  "the objective of our work is orthogonal to previous studies and we thus tailor a simple yet effective global modelling paradigm for image restoration." or  "our main focus is not to beat previous systems".  The authors appear to be arguing that their intentions are more important than the artefacts they have created or are describing.  This is simply not the case.  The proposed system has potential utility: it may be faster or more efficient than other models, at a cost of reduced accuracy.  In order for readers to evaluate the utility of the work, it needs to be made clear what the increase in efficiency is, *and* the reduction in accuracy.  Similarly, with a claim such as  "all above architectures are not fully to explore priors that are specific for image restoration tasks, which is important to lift performance." This may be true, but it needs to be *shown* that is is important to "lift performance" by running the appropriate experiments, and by comparing against the other systems.  The new ablations go some way toward answering this question, showing perhaps 0.5-1.5 PSNR improvements, significant, but only on "middle-performing" models.   To be clear: I am not suggesting that to be accepted this paper needs to beat the SOTA systems for accuracy.  It does not.  However it must *show* the reduction in order to show what is lost by the gain in efficiency.  

In conclusion, the review and rebuttal process has clearly led to improvements in the paper, however the revision as is does not adequately take into account the reviewers' criticisms.  I would strongly encourage the authors to recast the paper to show clearly how their proposal can lead to an efficient system which is not state of the art in terms of accuracy.  A pareto front on a (speed vs accuracy) graph might serve that purpose.

The authors are concerned that reviewer vjqt has misunderstood their figure 1b.  This does not reduce the accuracy of the reviewer's discussions on experimental comparisons. 
justification_for_why_not_higher_score: The rebuttal makes it clear the authors do not intend to compare to SOTA works, despite other 30M parameter models being in the current comparison.
justification_for_why_not_lower_score: n/a
Decision: Reject