Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: It seems that the authors do not post the rebuttals.
review_title: No rebuttals?

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, the authors proposed a novel neural operator architecture that factorizes the convolution on Fourier space into separate dimensions. Consequentially, the F-FNO model can scale up to a higher number of layers and achieve smaller errors. The paper has a comprehensive numerical study on multiple types of partial differential equations, considering chaotic systems and complex geometries. It also has a careful comparison with numerical solvers on the trade off between speed and accuracy.
strength_and_weaknesses: Strength:
- The work shows significant improvement on previous methods.
- The paper has a comprehensive study on many pde problems.
- It has a careful cost-accuracy study with the numerical solver. I especially love figure 4.

weak:
- The author improve FNO++ with many tricks. I assume F-FNO is also equipped with these tricks. It could be better to clearly list what these tricks are, and to what aspect they contribute to the overall improvement.
- It would be better to provide some intuition why the factorized structure help F-FNO scale with more layers.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. It has good quality. The technical novelty is not very strong.
summary_of_the_review: Overall I find this paper interesting. It has a concrete contribution to the community. I recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper uses the Fourier transform to learn neural operators that can handle long-range spatial dependencies. By factorizing the transform, using better residual connections, and improving the training setup, the proposed F-FNO outperforms the state of the art on PDEs on a variety of geometries and domains. 
strength_and_weaknesses: Pros:  The proposed method outperforms the state of art on PDEs with geometies and domains. 
Cons: The literature reviews on the classical and recent PDE solvers are not very complete. I would like to see some comparison with the classical numerical method, such as the vortex model in approximating 2D Navier-Stokes/Euler with FFO. 


clarity,_quality,_novelty_and_reproducibility: It is a well-written article with high quality. It seems a continuation of the authors' previous work but it is still very important. 
I do not check the reproducibility given the constraint of time. 
summary_of_the_review: I would suggest to accept the current paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces Factorized-FNO, where they consider separable Fourier representation (by taking the fourier transform of each dimension separately and independently) the authors achieve more “stable” models, whose performance increases as the networks are made deeper (something that does not happen with baseline FNOs), for 2D navier stokes equations.

The authors also improve the performance of original FNO baseline by introducing techniques like teacher forcing, a markov property. 

They also test F-FNO on irregular geometries.
strength_and_weaknesses: [Strenghts]

- The main idea introduced by the authors, of factorizing the input dimension and treating the fourier transform of each dimension separately seems to work well for the various 2D Navier Stokes equations considered.
- The method seems to get better with size of depth, and requires less parameters to reach similar performance as normal FNO.

[Weakness]

- The main idea (of treating each dimension independently) is only testing for ONE family of PDEs and that too in just two dimensions. It is hard to know if factorizing will *always* perform better. For example, one scenario where it might not is for PDEs that have cross terms involving derivatives w.r.t different dimensions.
- Most of the contribution made by the authors seem to be utilizing training techniques (like teacher forcing) that have are shown to give better performance in other domains and test them for FNOs. This does not seem to be a very novel.
- The authors test F-FNO on irregular geometries, is it established that normal FNOs definitely do not do well of these irregular geometries considered by the authors. That comparison seems to be lacking in the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow and everything is well explained. 
summary_of_the_review: The authors have tested F-FNO on a single family of PDEs (and only in 2D) and it is hard to judge if it will always improve performance. The paper also lacks comparison with baselines (like FNO) for their abilities to hand irregular geometries.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the author proposee Factorized Fourier Neural Operator (F-FNO), a learning-based approach for simulating partial differential equations (PDEs). This work is seen as an improvement of the work by [Li2021] in which the Fourier Neural Operator was introduced. In this work, the author aims at improving the stability of the algorithm under complex geometries and noisy data by introducing more regularizations in the neural operator. The proposed F-FNO method is able to reduce the size of parameter by an order of magnitude while improving the performance over FNO significantly.


[Li2021] Zongyi Li, Nikola B. Kovachki, K. Azizzadenesheli, Burigede Liu, K. Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021a.

strength_and_weaknesses: This paper has several aspects of strength:

1. Compared to [Li2021], the F-FNO framework has several changes: 1) a residual connection after the feedforward connection; 2) a spatial factorization; and 3) during the training, instead of unrolling the model, the F-FNO use teacher forcing and make online update based on Markov assumption;

2. The performance improvement of F-FNO over FNO is significant. Under several synthesis dataset  and using deep architectures, the reduction on MSE and time until correlation are both significant. 

3. The introduction of forcing function in the F-FNO allows the model to take additional contextual information that helps to solve the equation. 

This paper has several aspects for improvement:

1. While claiming to improve over the state-of-the-art algorithm, the experiments in the main paper only compare with FNO while the comparison with other methods are left in appendix. This is a missed opportunities since it is important to provide a comprehensive review on the related methods in both the parameter size, the MSE and time until correlation, as in [Li2021].  It is suggested to put some of the comprehensive study in the early main page.

2. The spatial factorization is a significant simplification for the model while it may over introduce additional bias since it implicitly assumes that the solution also factorizes spatially. Does it make sense physically in the scenario when the Navier-Stokes equations are applied ? What is the tradeoff for this factorization ? 

3. As a PDE solver in the spectral domain, it is expected to discuss some tradeoff on the choice of spatial and spectral resolution (i.e. the design of the wavelength number vs. the dimension of input). It is known from Sampling Theorem that this choice of sampling rate would affect the performance of the system when the complex geometries require more detailed information to be kept after the discretization. This practical issue seems not discussed but it is interesting to have some thoughts on that.   


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the code provided has been structured well with good comment and documentation. 

summary_of_the_review: This paper has proposed an important update over existing Fourier Neural Operator and has brought it closer to practical use. With additional factorization, the F-FNO framework reduces the parameter size while attaining a great improvement in stability and performance when the layer of the neural network goes deep. This justifies my decision to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents the factorized Fourier neural operator (F-FNO), which brings a set of techniques on FNO. The first technique is the separable Fourier representation to improve model stability and performance. Second, the improved residual connections. Third, the training strategies are carefully designed. Impressive results are found on a variety of benchmarks.
strength_and_weaknesses: # Strengths
1. The results seem to be impressive. 
2. This paper clearly presented their method.
3. Some of the visualizations might inspire other researchers in this domain.

# Weaknesses

## 1. The proposed contributions are incremental and are not even well-defined.
Supporting details:

(1) The residual connections are not novel, and a naive application is not enough for ICLR.

(2) In the abstract, the authors say, "carefully designed training strategies." This claim is too vague.

(3) In the introduction, the authors claim, "In our exploration of various regularization techniques, we find that it is possible to reduce the parameter count by up to an order of magnitude while still outperforming state of the art by a significant margin." This is too incremental. Applying a kind of regularization cannot be a major contribution. Also, how and why the regularization work remains unclear.

(4) The core of the introduction and the method are too few. The major body of the methods is about the formulation of the problem and the baseline methods.

(5) In the introduction, the authors claim three key contributions. The first is about the factorized Fourier neural operator. However, why is it called factorized? I think the only change is the residual connection. This should not be called the factorized Fourier neural operator, and this contribution is still weak.

(6) In the introduction, the authors list two of the experimental findings as their contributions. However, the experimental findings are not that important because the technical contributions are weak, and the experiments are not convincing.

## 2. The experimental results are not convincing. In general, I do not believe the baselines and the improved models are really working well.

(1)  All the error numbers are too large. F-FNO achieves 3.16 errors in elasticity. However, the original FNO paper reports numbers at ~0.01. I understand the settings might be different. If this is the case, why do not the authors put all the methods under the same setting?

(2) Can the authors provide a visualization comparing the GT with the prediction? Why does the modification work under this setting?

(3) Is the baseline strong? Why don't the authors conduct experiments on the official settings of FNO?

(4) In figure captions, the briefing settings should be presented instead of saying, "our models work well, ... ". It is always better for the readers to discover the improvements from the figures rather than reading general over-stated claims.

(5) In Figure 2, how to understand the figure? What are the inputs to each model? What are the outputs? Why are there big holes in the figure? What does the color mean? How to understand the differences between different modalities? Can the baselines achieve this? If the baselines cannot support varying geometries, why? What are the corresponding gt? What do geometries mean here? Are the geometries important in this domain?

(6) The error bars should be shown in each figure.

(7) It seems that the changes are not significant. The improvements should not be so significant.


clarity,_quality,_novelty_and_reproducibility: The clarity is OK but the writing is poor. This means although the authors presented what they have done, they did not bridge the findings to their central claims well. Moreover, the contributions are not well presented.

The quality is incremental and I think the most valuable part of this work has not been presented well through the current draft.

Not too much originality can be found in this work.
summary_of_the_review: Incremental contributions; Clearly below the acceptance bar; results are not convincing.

——————————————————————————————————————————————————————
After discussing with other reviewers via the Zoom meeting, some of my questions are addressed, and it would be better if the authors clarify the mentioned points in the next revision. Also, some senior reviewers with great expertise in this domain insist that this paper has a clear empirical advantage over the baselines. Therefore, I agree to accept this paper right now. I don't think a simple method should not be accepted, whereas the AC also thinks so. I think the authors should clarify their central contribution (eq.8) in this case and not hide their contribution behind a set of tricks. I think a revision would be beneficial to the community and also to the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a relatively simple change to the recently proposed FNO architecture for PDE solvers. The main idea is to, rather than performing a FFT in each FNO layer, to perform an FFT on each of the coordinates separately, then apply the FNO-style trained kernel. While this change is relatively minor, it helps substantially with scaling the depth of the trained architectures, which in terms improves the trained model performance on various families of PDEs in consideration in the paper. The reviewers noted (both in Openreview, and during a Zoom meeting to discuss the paper), that the authors in the camera-ready version need to be forthright that the main original improvement is the factorization (eq. 8), whereas the current writing makes it seem like various other tricks known in the literature before (e.g. enforcing Markovianity, teacher forcing) are also part of the contributions. There were also several suggestions to clarify some plots in the figures (see the individual reviewer edits.)
justification_for_why_not_higher_score: The factorized architecture is a very simple change which yields non-trivial improvements. It deserves to be accepted at the conference, as FNOs are one of the dominant architectures for PDE solvers. On the other hand, the evaluation is fairly small-scale, so it's hard to say that the observations are very robust across PDE families. 
justification_for_why_not_lower_score: The main points of contention with the reviewers revolved around writing and clarity, which I believe can be easily addressed by the authors for the camera-ready version. The reviewers also added specific points to address in their edited reviews.
Decision: Accept: poster