Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper combines the FNO architecture with hypernetworks to learn the solution operator (flow map)
of Markovian PDE systems. A hypernetwork whose input is the time domain (\R_+) is trained to output 
the weights of a FNO which then acts on an initial condition to produce the PDE solution at the time 
input to the hypernetwork. Training is done by uniformly sampling random times on a bounded input domain
and exploiting the Markov property of the flow map, allowing to use as data only the initial condition 
and the solution at a final time without a need for the entire time series. Numerical experiments showing 
a modest improvement over the standard FNO are performed on the 1-d Burgers' equation, generalized Burgers'
equation, and the Chafee–Infante equation as well as on the 2-d Burgers' equation, and the Navier-Stokes
equation.
main_review: Overall, the paper is well-written and easy to follow. I quite like the idea of using a hyper-network
to make the time domain continuous instead of having to add a new dimension to the FNO (the paper does lack
a short review of the FNO which I think is needed as it is not a standard architecture quite yet). The 
idea of learning Markovian systems by exploiting the compositional structure of the flow map
(again using FNO) was earlier proposed in (https://arxiv.org/abs/2106.06898) where the approach is 
to learn a FNO which emulates \Phi_{h} for some fixed time step h > 0 and predict forward by simply
having the FNO act on its own output repeatedly. While this paper considers ergodic systems and tries
to capture the statistics of the invariant measure by emulating very long-time dynamics, I suspect a 
similar approach can be used in the present setting by using a small time-step h along with the loss
function proposed by the authors. A comparison of the two methods would be quite helpful. 

I am slightly worried about the quoted results for the 1-d Burgers' equation where the authors' method
outperforms the standard FNO by almost an order of magnitude. The paper (https://arxiv.org/abs/2108.08481 - Table 2)
shows standard FNO results on 1-d Burgers' that are better than those of the presented method and can be 
improved even more by using a GeLU activation. This casts doubt as to whether the author's hypernetwork
approach is giving the improved results or simply not tuning the FNO. The numerical section is, in my
opinion, the weakest part of the paper as it is not convincing that this method of training which is certainly
more computationally expensive offers a large advantage over standard training. A clear advantage is the 
ability to extrapolate the solution outside the training time window and the authors should make this more 
of a focus, perhaps on problems where shocks don't develop, and show for how many Lyapunov times the 
extrapolated solution can keep up with the true trajectory. Furthermore considering much longer time domains 
would be of great interest as simply taking the solution to t=1 is not enough. Even the original FNO
paper considers NS up time time 50.  Performing such experiments is crucial if the work is to be published.
summary_of_the_review: Nice way of dealing with the time domain by using a hypernetwork, but experiments are not so convincing that
the approach is better than the standard.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a model and a loss function for approximation of the solution map of certain partial differential equations. 
The idea is to use HyperNetworks: the mapping of the initial condition to time t done by model f($\theta$, x), and the parameters are predicted by another neural network. The models utilize the recently proposed Fourier Neural Operator (FNO) for the mapping model, and a fully-connected network for the hypernetwork. Several losses are proposed to ensure consistency of the proposed models, including the composition and reconstruction. The tests are done on a battery of examples, and compared to other neural network baselines the error is smaller. A theorem is proved, but is rather straightforward.
main_review: The paper is in a popular direction, and has all the pros and cons of such kind of papers. The idea of using hypernetworks to model the transition operator is reasonable and of practical interest.  The drawback is that the computed solutions are approximate and do not show (at least, this not considered) systematic convergence to the true solution of the PDEs. Let me explain in more details. A classical FEM/FDM solver typically converges as the number of parameters (mesh points) increase. For neural network based method, a similar phenomenon should be observed when the number of parameters in a neural network representation increases. There is a bunch of work (C. Schwab, G. Kutyonok, D. Yarotsky) that show theoretically for some classes of problems that it is true. But in practice, such results are never reported. Moreover, this paper uses only one fixed architecture and tests it for different grid sizes. So, what is the actual error of the computed solution?

A second question is the methodology of learning the map. As initial condition, a sequence of Gaussian function is used. However, difficult examples for the Burgers equation are "kinks", and there is rich literature on constructing efficient numerical schemes for such kind of tasks. Can you provide any guarantee that the model learned for a small set of function will be able to generalize to other functions? 

Comments&Questions:

- The Theorem one seems more like a Lemma or Statement, since it is rather straightforward.
- What norms is used to measure the reconstruction error? In PDEs, the errors are measure in the relative norms, which are suitable for the corresponding PDE. 
- Computational speed vs accuracy: the error does not depend on the mesh size (Figure 1) . For finite-element, finite volume method there is a systematic convergence of the error with respect to the number of parameters. What can be said about the convergence of the solution with increasing number of parameters? What is the complexity of solving a PDE compared to the classical solver of the same accuracy? 
- Probably, the most interesting part of the model is the potential ability to predict the solution for long time in "one shot".  Can you clarify, who long you can do it? What is the largest time the model can predict?  If smaller steps are needed, then it is more close to the classical time-stepping (also interesting, but not so, since again, a classical solver, maybe with higher-order elements, will outperform the Neural Solver quite significantly).


summary_of_the_review: The paper is in a popular direction, proposes a reasonable improvement, but the presentation and methodology of the study could be significantly improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose to use hyper-network to learn a continuous time-dependent evolution operator based on Fourier neural operator.
main_review: The paper is quite interesting to me. It provides a continuous time representation of FNO using hyper-networks.

Comments:
1. In the original FNO paper, there are two formulations of time: end-to-end spatial mapping G:v(0) -> v(t) and space-time mapping G: v(0) -> (v(0), v(t1), v(t2), ...., v(t)). It seems this paper mainly addresses the end-to-end mapping, but it's indeed more similar to the space-time FNO.

2. The model learns the map G:(v(0), t) -> v(t). In some sense, if once input a full time squeeze (v(0), [0,t1,t2,t3,...,1]), the model will output the full time trajectory (v(0), v(t1), ...., v(1)). In this sense, both this hyper-network model and the original space-time FNO learns one set of parameters for the full trajectory. The major contribution of this work, I think, is to continuously evaluate at any time t, which is quite significant.

3. Same as the space-time FNO, when working on a chaotic system such as NS, the problem will be very challenging if the time interval t is too large. It will be interesting to study the limit of t for the hyper-networks.

4. If remove the batchnorm in the original FNO, its error on 1d Burgers will drop from ~0.01 to ~0.0025, which matches the performance of this hyper-networks model, and then the performance becomes consistent among all other 2d 3d PDEs in the paper (check https://github.com/zongyi-li/fourier_neural_operator/issues/30). In my opinion, it is totally expected that the proposed FNO model does NOT outperform the original FNO at discrete time steps. In general, discrete models get better errors at discrete time steps compared to continuous models (well, in this case, the continuous model is trained using extra information between steps, so it may not be fair). The main advantage of a continuous model is to do continuous evaluations. For a fair and consistent comparison, I suggest the author use the updated FNO code on 1d Burgers equation.

5. the model enforces the composition law (Equation 11) by adding a loss, which seems sub-optimal to me. Ideally one should develop a model that satisfies the composition law intrinsically (like in https://arxiv.org/abs/2106.06898), which is of course a non-trivial problem.

6. some personal tastes:
(a) why call it "semi-supervised"?
(b) It will be great to include some virtualizations of the 2d burgers and 2d/3d NS equation, at least in the appendix.
(c) It is better to call this theorem a remark.
summary_of_the_review: I think the problem is significant and the model is novel. While the paper needs some revision and polish, I would recommend borderline acceptance.
review_title: null

Meta-review: This paper presents an approach to learn the solution operator of Markovian partial differential equations (PDEs) by combining the Fourier Neural Operator (FNO) with a hyper-network.  In short, the hyper-network g_\theta(t) is trained to output the weights of a FNO f_w(x), which (given an initial condition) outputs the PDE solution at the time given to the hyper-network. The main claimed contributions of the proposed approach (as compared to, e.g., the original FNO architecture) are that the obtained solutions improve the learning accuracy at the supervision time points and that the solutions are able to interpolate and extrapolate to arbitrary times.  

The reviewers seemed to like the idea of using hyper-networks for modelling continuous-time FNOs. Several issues were raised by the reviewers, e.g., with respect to related work by Li et al. (2021, https://arxiv.org/abs/2106.06898), which I believe were addressed by the authors satisfactorily. However, it is still concerning that the reported performance for FNO in, e.g., 1d-Burgers does not quite match those recently published  (Kovachki et al, 2021, https://arxiv.org/abs/2108.08481, Table 2).  Although the authors did report additional results using GeLU, these results are still very different to those in  Kovachki et al ( 2021) and it is unclear whether the improved performance is due to a lack of tuning the baseline FNO. I commend the authors for, as suggested by the reviewers, running more extrapolation experiments. However, I believe the reviewers also made a point about considering (training) times much longer than 1, as even the original FNO paper did this for Navier Stokes (NS) with T=50. 

Overall, the paper provides modest improvement wrt FNOs, although it does extend its capabilities to interpolation and extrapolation. The paper will also benefit from providing a brief overview of FNOs.
Decision: Reject