Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel up-scaling strategy to upscale the spatial features in an encoder-decoder-based spatial up-down sampling network. Upscaling is performed on the frequency domain to explore the global dependency. The upscaling module has been experimented with several computer vision tasks and experimental results show significant performance improvement in those tasks.
strengths_and_weaknesses: This paper mainly proposes two different types of Fourier upscaling, namely periodic padding and area-interpolation. The main strengths are as follows.
1. This paper explored the relationship between spatial and frequency domains. The theoretical evidence in support of upscaling module design is explained properly.
2. The experiments in comparison with existing upscaling and different variants of Fourier upscaling are performed. The experimental outcome shows promising performance.
3. The paper writing and explanations are good. I actually had this kind of idea but never tried this. It is refreshing to see that this paper has done this. This work will drive the community in a certain direction. 

There is no such major weakness in this paper. I would like to know an explanation behind those results as we can see from Figure 1 that reconstructions in the spatial domain are really bad after periodic padding or area-interpolation. Why does it still work?

questions: Why did this paper not explore the frequency down-sampling strategies and experiments with different variants? Did frequency down-sampling not give satisfactory results? If so, why does only frequency upscaling work?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a theoretically feasible Deep Fourier Up-Sampling (FourierUp) to address up-sampling operators in Fourier domain. Extensive experiments across multiple computer vision tasks are conducted.
strengths_and_weaknesses: Strengths:
- Idea is interesting
- Easy to follow
- Clear writing

Weaknesses:
1. Sec. 3 introduces the details and theoretical evidences of the two kinds of FourierUp Modules, including Periodic Padding interpolation and Area Interpolation. However, it's not clear to me why the proposed FourierUp method is better. Even with the illustration in Fig. 1, the results of the two variants are not that good.  Thus, deeper analysis on why and how FourierUp helps improve the performance should be discussed and illustrated.

2. In experiments, more recent SOTA competitors are required to be compared. For example, PReNet is from CVPR'19, Aod-Net and PanNet are from ICCV'17. I think it's ok to adopt some classical methods as the baselines when proposing a novel framework for the existing task. It would be better if the proposed method is compared with the SOTA cutting-edge methods to provide some deeper understanding on the tasks, which may make the proposed method more insightful and convincing.

3. I am curious about what kind of failure cases the proposed method may have. Sec. 5 actually avoids to show such cases.

4. Complexity comparisons against the competitors are required.
questions: Please see [Weaknesses].
limitations: Yes, the authors discussed the limitations in Sec. 5. However, I am also curious about the failure cases that the proposed method cannot handle currently.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tries to solve a challenging issue in frequency related neural network designs, i.e., how to model the multi-scale frequency patterns. Unlike the simplex up-sampling operations in the spatial domain, up-sampling frequency information in a neural network is non-trivial because the local property of spatial up-sampling cannot be directly used in the frequency domain. According to the spectral convolution theorem, this work proposes an interesting and theoretically sound solution, called Deep Fourier Up-Sampling (DFU). DFU is the first work to consider the modeling of multi-scale frequency patterns and provides key insights for the frequency related network designs. Besides, the DFU can be implemented by different variants with different theoretical support. The DFU is also flexible in its compatibility with existing neural networks. Extensive experiments on multiple vision tasks demonstrate the effectiveness of the proposed DFU.
strengths_and_weaknesses: Strengths:
1. Interesting motivation: This work takes an interesting and challenging issue, multi-scale information modeling in the frequency domain, into account. Although there are some frequency-related neural networks, they cannot effectively and accurately model the relationship of multi-scale frequency information due to the Fourier transformation characteristics. However, the multi-scale frequency information is important for vision tasks. This key issue is the focus of this paper.
2. Theoretical proof: For the proposed Deep Fourier Up-Sampling, this work provides detailed and rigorous proof, making a theoretical sound solution. These proofs provide a new view to understand deep Fourier up-sampling and convince the readers. 
3. Flexibility: The proposed Deep Fourier Up-sampling can be a generic operator, which can be compatibility with existing neural networks and boost their performance. 
4. This paper conducts sufficient experiments to demonstrate the effectiveness of the proposed Deep Fourier Up-sampling. To show the versatility of the proposed method, the experimental comparisons are carried out on diverse vision tasks, including object detection, image segmentation, image de-raining, image de-hazing, and guided image super-resolution. The experiments on multiple vision tasks provide strong supports for the advantages of the proposed multi-scale frequency information modeling.


Weaknesses:
1. Although this paper provides sufficient experimental results, it would be better if the parameter comparisons and more visual results could be provided. The reviewer understands that this paper has conducted extensive experiments, covering low-level and high-level vision takes in the limited space. More results could be provided in the supplementary material. 
2. In Section “Comparison and Analysis”, the underlying reasons why the FourierUp works better are insufficient. More explanations are expected. Furthermore, it is a little confusing that whether each variant of the FourierUp prefers a specified vision task? How to choose the variant when one uses the FourierUp for different vision tasks? Some insightful analysis or suggestion should be provided, which would guide the followers to make full use the capability of the proposed FourierUP. 
3. It would be better if more visual results could be provided in the main paper rather than the supplementary material. It is fine to shrink the sizes of figures for showing more cases. Current format is inconvenient to switch between the main paper and the supplementary material for checking the visual results. 
4. The flowchart of implementation details in supplementary materials are suggested to transfer to main manuscript.   


questions: See the above weaknesses. The authors are suggested to give the detailed responses. 
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose an upsampling layer based on the Fourier transform and properties of the frequency domain. The motivation behind the work is quite vague, only resorting to the fact that the Fourier transform is a global operator rather than local.
strengths_and_weaknesses: The paper is generally poorly written, lacks a clear motivation, and the novelty is unclear. 

Strengths:
- Frequency-based techniques for layer operations are an area of research that deserves more interest

Weaknesses:
- Poor presentation. Readability is very low, especially due to long derivations and proofs in the text rather than in the appendix. It is hard to grasp what the method does at a glance
- Novelty unclear. It seems that the work just exploits well known properties of the Fourier transform of upsampled signals. It is also unclear why the scheme should outperform any other technique
- Experiments show very marginal improvements and lack two very important problems: image classification and denoising
questions: I would like the authors to clarify where the novelty lies in the proposed method and a more precise explanation of why we should expect gains with respect to existing techniques.
limitations: The authors addressed the limitation that two very important benchmarks (classification and denoising) are missing. Still, when proposing a very general technique, the body of proof required to demonstrate it is advantageous needs to be substantially larger than what is presented.
review_title: null

Meta-review: metareview: This paper proposes using Fourier up-sampling for multi-scale modeling. The paper received initial scores of 8 8 5 3. After the rebuttal and in-depth discussions, most reviewers are satisfied with the authors' replies. Reviewer-tPpX who gives negative scores still has concerns about the novelty and presentation of this paper. As I discussed with the reviewer, we finally did not find similar works that share the similar idea as this paper, so I recommend acceptance for this paper. However, I agree with the reviewer-tPpX that the presentation of this paper should be improved. There are actually too many equations and derivations in the manuscript, and this is not friendly for a NeurIPS paper. Please remember to move them into the appendix as much as possible if preparing for the camera-ready version.
confidence: Certain
Decision: Accept