Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors consider the problem of learning in a frequency domain. They note that multiple Fourier domain layers that perform forward and inverse FFT operations can lead to a high computational cost. They also argue that complex valued layers are inefficient, and propose to use a single forward and inverse DCT throughout the architecture. They also discuss how to initialize such a system.
strengths_and_weaknesses: Fourier domain learning is a relatively young but important subfield for machine learning. Therefore the subject is of critical interest. The writing is also clear for most of the manuscript.

However, even though the subject is significant, the contribution to the state of the art is modest. Using a DCT, or any other real-valued harmonic transform instead of the FT is fine, but I'm not really convinced of the argument against using complex valued transforms. If one knows that the signal is real-valued, one can make use of symmetry properties of the FT to avoid any redundant operations. Resorting to the Parseval relation to observe that the error is the same in the transform domain is also perfectly fine, but is a straightforward observation. This leaves a theorem about initialization, as the main contribution. This contribution is welcome, but I think that alone is not sufficient for a Neurips paper. 
questions: - line 94 : Should it be $\mathbb{R}^{m\times n}$ instead of $\mathbb{R}^{n\times m}$?
- in eqn 3 : Is $m$ an optimization parameter -- that is, is $m \in \theta$?
- the eqn between lines 186, 187 : if $m \notin \theta$, minimizers of $J_{\theta}$ and $L_{\theta}$ are the same. $L_{\theta}$ is useful only if you want to compare losses across different values. Is that what you're doing? Are you optimizing over $m$ as well?
- Thm 3.2, line 216, please replace "Then" with "If", and "$\Rightarrow$" with "then". Same comment applies to Corollary 3.1. The statement looks a bit confusing otherwise.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an efficient operator learning scheme by transforming the input data only once to some frequency domain. Numerical results on turbulence simulation indicate its significant speedup and accuracy improvement in several benchmark problems. 
strengths_and_weaknesses: The idea of computing convolution in the Fourier domain was proposed to speed-up the computational time with respect to the Mathieu et al. 2013, this paper shows that there is some overhead by doing so for each convolution operation: it is more direct to learn some equivalent function in the DCT Frequency domain (section 3.1), without using the common FDM layer structure (eq. 1). The strength seems to be its speedup. A reduced order T1 model is then proposed in 3.2 to further restrict the frequency domain. This seems to be a weakness as one may want to model some energy cascade phenomena in turbulence which would require the whole frequency information in the input signal. 
questions: How do you define the architecture of the proposed model in (3)? Also it is not very clear how to define such an architecture which respects symmetry properties of the data, e.g. translation invariance in Navier-Stokes equations. 

What are the operations f_1,…f_d in T1(x)? When you compute the speedup, are you computing the same function defined by the FNO? It seems to me there is some issue regarding the speedup results in Fig 3.1, if they were convolutional (e.g. in Unets) as it depends on the filter size. 

What is the definition of N-MSE in Table 4.1?

When you talk about the sparsity based on Fourier transforms in the introduction, are you considering turbulence as natural signals? Are they really sparse under DCT transforms? 

limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submission develops a new model architecture under the umbrella of _frequency-domain models_. These models are used to learn reduced order models of dynamical systems. The main development is to apply an integral transform (such as an FFT or DCT) once to the signal and then apply a chain of non-linear operations solely in the spectral domain before transforming back. This saves a lot of computation moving back and forth to the frequency domain at every layer.

The authors test their model on a number of dynamical systems modelling problems, showing superior performance in terms of normalized mean squared error and wall clock time. The paper is well written and I support acceptance.
strengths_and_weaknesses: + 153: I like the insight that $\mathcal{T}$ can be amortized across training epochs.

+ Section 3.3: The weight initialization scheme for the reduce-order FDMs looks sound. It is nice that this is shown in the main body of the submission.

+ The experimental results look good. The experiments themselves appear to have been conducted properly (+1 for using multiple training runs).

+ I like the idea of transforming once to the spectral domain and then performing all computations there before transforming back to the spatial domain. It saves computation and thus allows one to deploy much more expressive models in the spectral domain, which is exactly what this submission does (they use a U-Net).

+ The submission is well written and nicely typeset.

- The authors have not included limitations of their work.

- Some of the claims about baselines (see questions) are a bit exaggerated---for instance, FNO does not use hundreds of millions of parameters (see Table 4.1) and these shown be toned down.
questions: 20-21: The original quote is from Feynman (1965)

39: What is meant be a "non-trivial number of channels"? Also, please provide an actual example of how you arrive at hundreds of millions of parameters? If you bandlimit the signals in the frequency domain, as performed in the original Fourier Neural Operator paper, I think the number of learnable parameters is significantly lower than this.

Sec 2.1: Regarding learning in the frequency domain, consider also the line of works in the equivariance literature, where the irreducible representations are used for filters.

89: I would be careful not to use the notation $\mathcal{D}_n$ and $\mathcal{D}_k$ for signal space and frequency space. It is very easy to assume that these are the same kind of space, just with a different dimensionality, when, for example, one may be a real space and the other a complex one.

100: For the Fourier neural operator, is $g$ not purely local; that is, it is a 1x1 filter?

110: How is the complex-valuedness of the layers related to parameter--inefficiency? Since signals are real, there is conjugate symmetry in the frequency domain, so while needing two real parameters to represent each complex parameter, we only need half the filter taps anyway.

- Please use equation numbers for all display-mode equations

186: What is the necessity for measuring the high frequency loss if your task is not reconstruction? $R_o$ will be non-zero, but I do not see why reconstruction error is a good measure of performance unless a downstream task is specified.

- One of the considerations for using FNO is that convolution with large kernels in the spatial domain becomes more efficient in the spectral domain, since the FFT diagonalizes convolution. With bandlimiting of the spectral representation of the kernel, this becomes very efficient. In this work, $f_\theta$ is left free-form and thus while adding capacity, does not lead to a computational complexity reduction necessarily. I'm guessing the main complexity reduction is in the fact that you do not need to transform back and forth from the spectral domain with every layer?

- Is the use of N-MSE the best performance metric for rollout quality? In Fig 4.3 it appears the T1+ loses a lot of fidelity compared to the FNO model baseline. This would not be captured by N-MSE, but by a spectral metric.

- Do this model exhibit generalization over resolution, like the original FNO models? It does not appear to me that this would happen.
limitations: The authors do not address limitations of their work. This should be fixed if the draft is to published. Conversations about resolution independence, choice of integral transform, and restrictions to regular grids would be welcomed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a frequency domain approach for deep learning that is very intuitively summarized in the title: the goal is to have a single transformation into the frequency domain, perform all necessary operations there, and then transform back. While not an overly expensive operation, it is an interesting goal to reduce the number of frequency domain operations. The paper arrives at this goal via truncation and a DCT.

The paper is very nicely written, and motivates the approach well. Frequency domain models (FDMs) are an active topic of research, and the paper thoroughly compares to popular variants, the "fourier neural operators" and a factorized version. It evaluates the proposed approach with a set of flow-centric set examples. This limits the scope somewhat, but I personally found the examples very interesting, and I believe they cover a nice variety of interesting structures for learning problems.

The paper also comes with some theoretical results. They look convincing, but I have to admit I didnt check the proofs in detail.

strengths_and_weaknesses: In terms of main strengths, I think the paper proposes a very attractive idea. The presentation, derivation and discussion are great, I believe. The paper is clear, and very nice to read. The results and presented in a compelling way, and provide a detailed picture of the proposed approach. I think this is a clear strength of the paper.

The advantages of the resulting algorithm are probably a bit less than what one would hope for given the initial discussion of positive properties of the single transformation, but I think they're nonetheless substantial. I definitely found the results in terms of MSE errors and resource usage to be attractive steps forward.

On the side of weaknesses, the results can nonetheless be seen as somewhat underwhelming at times. I think especially the strong low-pass filtering of the frequency truncation is a central limitation of the approach. This is, e.g., visible in the turbulent smoke outputs (fig. 4.3). For this setup, more examples should be provided in the appendix, and I think the implications of inherent low-pass filtering (and the corresponding smoothing effects) should be discussed in the main paper. I hope the authors can also comment on this in the rebuttal, but for now I'm sure the authors could address this in a finalized version of their submission.

Overall, I would argue for accepting this paper to NeurIPS. It proposes an interesting variant of FDMs, thoroughly evaluates and discusses it. Even if some of the results "only" result in smoother outputs with similar MSE and faster runtime, I think the approach is suitable for presentation at NeurIPS as a variant that is highly attractive for some learning scenarios.

questions: One open point I noticed was the introduction and later on omission of the T1+vp scheme. Section 4.1.1 first discussed its clear advantages, but the more complex examples in 4.2 and 4.3 seem to only use T1+. Can the authors explain this?

limitations: As mentioned above, I think the low-pass filtering due to the frequency truncation should be discussed more explicitly.
review_title: null

Meta-review: metareview: The paper received three positive reviews (including a well motivated strong accept) and a negative ones. Overall, the feedback provided by the reviewer seem to be useful. The area chair does agree that the concerns of the negative reviewer are a good reason for a reject, even though their review may trigger interesting discussions. The majority of the reviewers believe that the paper makes an interesting contribution and the area chair is happy to recommend an accept.
confidence: Certain
Decision: Accept