Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work attacks the important problem of learning PDEs with neural networks. To this extent, it proposes a neural network architecture where (in essence), the linearity is not applied in the Fourier space but in the state space. This allows the network to implicitly learn the partial differentials of the equation, and results in a parametrization which is invariant to spatial discretization. This approach yields good results when regressing to data derived from numerical solutions of complex differential equations. 


Strengths:

Overall, the paper is well written. The theoretical and experimental sections are, for the most part, clear and concise, although some important details remain unclear/lacking. 

The method seems to be quite generic and can be applied to a large range of PDEs. 

The inverse problem experiment is interesting and highlights a potentially useful application of the proposed method.


Weaknesses:

Perhaps most importantly, the data generation is not always clear to me. The distinction between training data and test data is not clearly specified. Generalization to different useful initial conditions is obviously of utmost importance in order to properly evaluate the quality of data-driven methods for dynamical systems. 

The loss function and training method is not clearly specified: are observations acquired every t=k \in N steps for NS? Do you use teacher forcing or compute an error on the full sequence? What about the other experiments?


Comments and questions:

It seems that in the proposed model, the Fourier coefficients R do  not vary with the layers. This seems to be quite constraining, could you comment on this?

It would have been interesting to analyze the expressiveness of the architecture. 

You mention that you are not restricted to periodic boundary conditions due to your final decoder (which is never clearly defined), however, does the finite support of the data not lead to artifacts in the Fourier transform, causing you to have to learn a high number of Fourier modes? Could you comment on this? 

In the NS experiment, why have you used N=10 000 when viscosity is set to 1e-4, and N=1000 for the others? It seems as if you have selected the number of samples where your method outperforms the baselines.

Moreover, for the NS experiments, test data seems to be future data (T>10), however this seems to not be the case for the Burgers’ equation, even though it is also time dependent.

As one of your motivations behind this work is to learn the operator, it could have been interesting to test your approach using different sample points as in the training data. 

review_title: Review of Fourier Neural Operator for Parametric Partial Differential Equations 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper Summary: The authors proposed a novel neural Fourier operator that generalizes between different function discretization schemes, and achieves superior performance in terms of speed and accuracy compared to learned baselines.

I have to admit that my understanding of this paper is rather limited and I have lots of conceptual questions about the implementation.
- The authors claimed to novel contributions: one is that the method is fast, one is that the method is discretization invariant. These two seem to be in conflict. The authors leveraged FFT for performing fast convolutions differentiably. However FFT assumes the data to be present on a uniform grid (of 2^n grid cells in each dimension). I understand that since the learned content is a frequency "filter" that can be applied to any discretization, but this can be said for just about any CNN (in that case a learned 3x3 filter in physical space can also just be projected to the spectral space and work with "any discretization"). Furthermore (correct me if I'm wrong), it doesn't seem that the authors experimentally illustrated a case of learning on non-discretized data, and performing inference on discretized versions of it, or vice versa.
- From Fig. 1, it seems that the actual parameters that are being learned is the convolutional kernel R (represented as spectral coefficients), and W. That basically amounts to a single convolution plus bias operation. The expressivity of a single such operation should be pretty limited, since this operation itself is a linear operator, while the underlying equations that the authors demonstrated on are mostly nonlinear. 

More insights into the two questions above are certainly appreciated.

Adding a reference that seems related to this work. The paper below uses a spectral solver step as a differentiable layer within the neural network for enforcing hard linear constraints in CNNs, also taking the FFT -> spectral operator -> IFFT route.

References
---------------
Kashinath, Karthik, and Philip Marcus. "Enforcing Physical Constraints in CNNs through Differentiable PDE Layer." ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations. 2020.

review_title: Several conceptual questions

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Paper summary:

Building on previous work on neural operators, the paper introduces the Fourier neural operator, which uses a convolution operator defined in Fourier space in place of the usual kernel integral operator. Each step of the neural operator then amounts to applying a Fourier transform to a vector (or rather, a set of vectors on a mesh), performing a linear transform (learnt parameters in this model) on the transformed vector, before performing an inverse Fourier transform on the result, recombining it with a linear map of the original vector, and passing the total result through a non-linearity. The Fourier neural operator is by construction (like all neural operators) a map between function spaces, and invariance to discretization follows immediately from the nature of a Fourier transform (just project onto the usual basis). If the underlying domain has a uniform discretization, the fast Fourier transformation (FFT) can be used, allowing for an O(nlogn) evaluation of the aforementioned convolution operator, where n is the number of points in the discretization. Experiments demonstrate that the Fourier neural operator significantly outperforms other neural operators and other deep learning methods on Burgers’ equation, Darcy Flow, and Navier Stokes, and that that it is also significantly faster than traditional PDE solvers.

------------------------------------------
Strengths and weaknesses:

Much of the theoretical legwork for this paper, namely, neural operators, was already carried out in previous papers (Li et al.). The remaining theoretical work, namely writing down the Fourier integral operator and analysing the discrete case, was succinctly explained. The subsequent experimentation was extremely thorough (e.g. demonstrating that activation functions help in recovering high frequency modes) and, of course, the results were very impressive. I liked the paper a lot, and it’s definitely a big step-forward in neural operators. I’m assigning a score of 8 (a very good conference paper), and I think that the paper is more or less ready for publication as is. I’ve included a few questions below (to help my own understanding), as well as some typos I spotted whilst reading the paper.

------------------------------------------
Questions and clarification requests:

1)	Section 4, The Discrete Case and the FFT – could you explain the definition of bounds in the definition of Z_{k_{max}}?
2)	Section 4, Parametrizations of R, sentence 2 – could you explain the definition R_{\phi}? At present I can’t see how the function signature of R matches the definition given.
------------------------------------------
Typos and minor edits:
- Page 3, bullet point 3 – “solving Bayesian inference problem” -> “solving Bayesian inference problems”
- Section 1, final paragraph, sentence 2 - “approximate function with any boundary conditions” -> “approximate functions with any boundary conditions”
- Section 4, The discrete case and the FFT, final paragraph, last sentence - “all the task that we consider” -> “all the tasks that we consider”
- Section 4, Parametrizations of R, last sentence - “while neural networks have the worse performance” -> “while neural networks have the worst performance”
- Section 4, final sentence – “Generally, we have found using FFTs to be very efficient, however a uniform discretization if required.” -> “Generally, we have found using FFTs to be very efficient. However, a uniform discretization is required.”
- Section 5, final paragraph, sentence 2 – “FNO takes 0.005s to evaluate a single instances while the traditional solver” -> “FNO takes 0.005s to evaluate a single instance while the traditional solver”
- Section 6, final sentence – “Traditional Fourier methods work only with periodic boundary conditions, however, our Fourier neural operator does not have this limitation.” -> “Traditional Fourier methods work only with periodic boundary conditions. However, our Fourier neural operator does not have this limitation.”
review_title: Review of "Fourier Neural Operator for Parametric Partial Differential Equations"

final_score: rating: 5: Marginally below acceptance threshold
review_content: The manuscript proposes a neural network for solving partial differential equations. This is performed by a supervised learning of a mapping between the PDE coefficients (a) and the PDE solution (u), where a is drawn from a certain distribution. This work follows  a work by Li et al. 2020b.  Inspired by the properties of Green’s function, in Li et al. the authors suggest to learn a kernel function in an iterative manner.  In this work, the general kernel is replaced by a convolution kernel which is represented in the Fourier space. 




Pros. 

1.	The authors address an important and practical problem 
2.	I like the idea of learning a mapping for a class of PDEs, rather than optimizing per instance
3.	The numerical results are impressive

Cons. 
1.	Novelty. I feel that the novelty is incremental with respect to Li et al. The motivation for replacing the kernel function (3) by a convolution operator is not entirely clear. I guess that efficiency is one of the reasons. If so, I would suggest to discuss the complexity of the proposed scheme vs. Li et al. complexity.  Moreover, how does it affect the expressivity of the network? 
2.	Clarity of the paper.  I didn’t like the idea that on the one hand entire sentences are taken form Li et al. but on the other hand, the paper itself is not self-contained. In practice, the reader should follow the intuition and the motivation given in Li et al in order to grasp the idea of neural operator.  In addition, a concise detailed description of the network architecture is missing. 
3.	Missing references and comparison.  There are two works, aiming at solving PDEs, which are very relevant in the context of the proposed approach. The first copes with the structured case, Greenfeld at al. “Learning to optimize multigrid PDE solvers” and the second handles the unstructured case, Luz et al. “Learning algebraic multigrid using graph neural networks”. It resembles the proposed approach in the sense that the training is performed for a class of problems, yielding a mapping from the coefficients to the solution. In these works, the learning is unsupervised, while generalizing over boundary conditions and domains.
To summarize, I believe that this work should be published. I hope that the authors are able to address my concerns. 

review_title: I vote for reject. However,  I think that the work could be published after addressing some concerns.

Meta-review: Pros:
- Provides a practical technique which can dramatically speed up PDE solving -- this is an important and widely applicable contribution.
- Paper is simultaneously clearly written and mathematically sophisticated.
- The experimental results as impressive.

Cons:
- There were concerns that the paper lacks novelty compared to Li et al 2020b, where the underlying theoretical framework was developed. The primary novelty would seem to be:
- - using Fourier transforms as the specific neural operator
- - the strength of the experimental results

Overall, I recommend acceptance. I believe the techniques in this paper will be practically useful for future research.
Decision: Accept (Poster)