Semi-supervised learning of partial differential operators and dynamical flows

Abstract: The evolution of dynamical systems is generically governed by nonlinear partial differential equations (PDEs), whose solution, in a simulation framework, requires vast amounts of computational resources. For a growing number of specific cases, neural network-based solvers have been shown to provide comparable results to other numerical methods while utilizing fewer resources.
In this work, we present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. Our method treats time and space separately. As a result, it successfully propagates initial conditions in discrete time steps by employing the general composition properties of the partial differential operators. Following previous work, supervision is provided at a specific time point. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is also able to interpolate and extrapolate the solutions to arbitrary times.

Reviews and author responses:

In this paper, the authors propose to use hyper-network to learn a continuous time-dependent evolution operator based on Fourier neural operator. The paper is quite interesting to me. It provides a continuous time representation of FNO using hyper-networks.

Comments:
1. In the original FNO paper, there are two formulations of time: end-to-end spatial mapping G:v(0) -> v(t) and space-time mapping G: v(0) -> (v(0), v(t1), v(t2), ...., v(t)). It seems this paper mainly addresses the end-to-end mapping, but it's indeed more similar to the space-time FNO.

2. The model learns the map G:(v(0), t) -> v(t). In some sense, if once input a full time squeeze (v(0), [0,t1,t2,t3,...,1]), the model will output the full time trajectory (v(0), v(t1), ...., v(1)). In this sense, both this hyper-network model and the original space-time FNO learns one set of parameters for the full trajectory. The major contribution of this work, I think, is to continuously evaluate at any time t, which is quite significant.

3. Same as the space-time FNO, when working on a chaotic system such as NS, the problem will be very challenging if the time interval t is too large. It will be interesting to study the limit of t for the hyper-networks.

4. If remove the batchnorm in the original FNO, its error on 1d Burgers will drop from ~0.01 to ~0.0025, which matches the performance of this hyper-networks model, and then the performance becomes consistent among all other 2d 3d PDEs in the paper (check https://github.com/zongyi-li/fourier_neural_operator/issues/30). In my opinion, it is totally expected that the proposed FNO model does NOT outperform the original FNO at discrete time steps. In general, discrete models get better errors at discrete time steps compared to continuous models (well, in this case, the continuous model is trained using extra information between steps, so it may not be fair). The main advantage of a continuous model is to do continuous evaluations. For a fair and consistent comparison, I suggest the author use the updated FNO code on 1d Burgers equation.

5. the model enforces the composition law (Equation 11) by adding a loss, which seems sub-optimal to me. Ideally one should develop a model that satisfies the composition law intrinsically (like in https://arxiv.org/abs/2106.06898), which is of course a non-trivial problem.

6. some personal tastes:
(a) why call it "semi-supervised"?
(b) It will be great to include some virtualizations of the 2d burgers and 2d/3d NS equation, at least in the appendix.
(c) It is better to call this theorem a remark. I think the problem is significant and the model is novel. While the paper needs some revision and polish, I would recommend borderline acceptance.

 The authors addressed my concerns, and I am willing to raise the score. 

My remaining concern is that the accuracy seems not very impressive, given the extra amount of training time and the additional number of parameters. There exist other neural operators that achieve better accuracy such as the multi-wavelet operator (https://arxiv.org/abs/2109.13459) and the Galerkin transformer (https://arxiv.org/pdf/2105.14995.pdf). Besides the benchmark, I appreciate the contribution brought by this paper. Most of the published works seem somewhat incremental from hindsight. I personally think we should not hold the standard too high.

 Thank you for finding interest in our work. 

As can be seen in the results we posted, FNO with GeLU indeed obtains similar results to the one you’ve obtained. Note, however, that our method benefits from the GeLU activation as well, and still shows superior performance in comparison to FNO.

Our method does not resemble RNN. In an RNN, one has to assume a constant time interval between steps, together with some supervision in intermediate times. Our method, on the other hand, chooses a number of intermediate times at random (our experiments show that 1 is sufficient) and has no supervision over the fields at these times.

We agree that our method has to perform more forward steps than FNO, and therefore is slower during training (we estimated the difference in the reply to referee j3Ma), however, it is as fast as FNO for inference.


 
Q: the model enforces the composition law (Equation 11) by adding a loss, which seems sub-optimal to me. Ideally one should develop a model that satisfies the composition law intrinsically (like in https://arxiv.org/abs/2106.06898), which is of course a non-trivial problem.


A: The issue with compositions such as presented in the contemporaneous work by Li et al. (June 2021; https://arxiv.org/abs/2106.06898) is that in order for the approach to obtain small-time resolution scales, it is required to execute the method with an increasing number of sequential FNO compositions. For instance, to obtain a $\frac{1}{10}$ ratio, one would need to run the FNO network $10$ times, thus the time resolution scales with the network’s depth, and each time scale requires a different architecture. 

Q: some personal tastes: (a) why call it "semi-supervised"? (b) It will be great to include some virtualizations of the 2d burgers and 2d/3d NS equation, at least in the appendix. (c) It is better to call this theorem a remark.

A: 
(a) We chose to call it semi-supervised as there is a supervision at time t=0 and t=1, while we learn for any time $t \in [0, 1]$ with no direct supervision.
(b) We added plots for the difficult solutions of the 2D Burgers equation in section B of the appendix.
(c)  We will change the theorem title to a lemma.


 Thank you for your review.

Q: In the original FNO paper, there are two formulations of time: end-to-end spatial mapping G:v(0) -> v(t) and space-time mapping G: v(0) -> (v(0), v(t1), v(t2), ...., v(t)). It seems this paper mainly addresses the end-to-end mapping, but it's indeed more similar to the space-time FNO.

A: Indeed we address the end-to-end mapping. The second approach in the FNO paper maps several time steps, G: (v(t=0), v(t=1), v(t=2),...,v(t=10)) -> v(t=11). As pointed out in the review, our method is different from the latter approach since we do not require intermediate time steps. Furthermore, FNO uses only discrete time steps whereas our method has the capability of solving for continuous time steps.
Q: Same as the space-time FNO, when working on a chaotic system such as NS, the problem will be very challenging if the time interval t is too large. It will be interesting to study the limit of t for the hyper-networks.
A: Following the referee’s suggestion, we added an appendix section, where we assessed the ability of the hyper-network scheme to extrapolate solutions beyond $t=1$ by simulating additional $20$ solutions ranging from $t=0$ to $t=20$, in increments of $t=0.01$ for the 2D Navier-Stokes equation. Figure 4 in the appendix shows that our method is able to extrapolate solutions in one-shot up to $t\sim 2$ while doubling the error. This linear structure of error accumulation is unlikely to hold at long times due to the general chaotic structure of the solution. On general ground, one expects that for chaotic systems nearby trajectories would diverge exponentially: $\Delta v(t) = \Delta v(0) e^{\lambda t}$, When $\lambda t \ll 1$ we have $\frac{ \Delta v(t) -\Delta v(0)}{\Delta v(0)} \sim \lambda t$ and we can estimate from Figure 3(b) in the time window $[0,1]$ that $\lambda \sim 5\times10^{-2}$.  With such a value we can estimate, $t \sim 20 \ln \left(1+\frac{\Delta v(t) -\Delta v(0)}{\Delta v(0)}\right)$,. Thus, at $t=10$ we expect an error of about $65$ percent.


Q: If remove the batchnorm in the original FNO, its error on 1d Burgers will drop from ~0.01 to ~0.0025, which matches the performance of this hyper-networks model, and then the performance becomes consistent among all other 2d 3d PDEs in the paper (check https://github.com/zongyi-li/fourier_neural_operator/issues/30). In my opinion, it is totally expected that the proposed FNO model does NOT outperform the original FNO at discrete time steps. In general, discrete models get better errors at discrete time steps compared to continuous models (well, in this case, the continuous model is trained using extra information between steps, so it may not be fair). The main advantage of a continuous model is to do continuous evaluations. For a fair and consistent comparison, I suggest the author use the updated FNO code on 1d Burgers equation.
A: Indeed, we’ve used a different version of the repository than the current one. These implementations differ by the GeLU and ReLU activation functions. Specifically, both for 1D and 2D, both implementations do not contain a BatchNorm layer. Following the Reviewer’s comments, we’ve trained and tested our method, both with the GeLU and ReLU activation functions. The results are:

1D results:
 
| Method          | burgers         | chafee          | genburgers2     |
 |:----------------|:----------------|:----------------|:----------------|
| FNO             | 0.0096          | 0.0028          | 0.0090          |
| FNO GeLU        | 0.0022          | 0.0010          | 0.0032          |
| Ours $n=1$      | 0.0022          | **0.0008** | 0.0031          |
| Ours $n=1$ GeLU | **0.0015** | 0.0011          | **0.0024** |
 

2D results:
 

| Method    | 2D Burgers      | 2D Navier-Stokes   |
|:-----------------------------------------------|:----------------|:-------------------|
| FNO                                            | 0.0336          | 0.0467             |
| FNO GeLU                                       | 0.0248          | 0.0458             |
| Ours $P=1$                                     | 0.0335          | 0.0442             |
| Ours $P=1$ GeLU                                | **0.0214**          | **0.0429**            |



This shows that not only that our method is still superior in comparison to FNO, but also that it is robust with respect to the chosen activation function.
We would like to emphasize that our model does not use extra information between timesteps, but only consistency conditions. The only supervised information is given at times t=0 and t=1.


 
Q: “Probably, the most interesting part of the model is the potential ability to predict the solution for a long time in "one shot". Can you clarify, how long you can do it? What is the largest time the model can predict? If smaller steps are needed, then it is closer to the classical time-stepping (also interesting, but not so, since again, a classical solver, maybe with higher-order elements, will outperform the Neural Solver quite significantly).”

A:  Following the referee’s suggestion, we added an appendix section, where we assessed the ability of the hyper-network scheme to extrapolate solutions beyond $t=1$ by simulating additional $20$ solutions ranging from $t=0$ to $t=20$, in increments of $t=0.01$ for the 2D Navier-Stokes equation. Figure 4 in the appendix shows that our method is able to extrapolate solutions in one-shot up to $t\sim 2$ while doubling the error. This linear structure of error accumulation is unlikely to hold at long times due to the general chaotic structure of the solution. On general ground, one expects that for chaotic systems nearby trajectories would diverge exponentially: $\Delta v(t) = \Delta v(0) e^{\lambda t}$, When $\lambda t \ll 1$ we have $\frac{ \Delta v(t) -\Delta v(0)}{\Delta v(0)} \sim \lambda t$ and we can estimate from Figure 3(b) in the time window $[0,1]$ that $\lambda \sim 5\times10^{-2}$.  With such a value we can estimate, $t \sim 20 \ln \left(1+\frac{\Delta v(t) -\Delta v(0)}{\Delta v(0)}\right)$,. Thus, at $t=10$ we expect an error of about $65$ percent and at $t=2$ we expect the error to increase by a factor of $2$ compared to the supervised error at $t=1$.


 Thank you for your review.

Q: “A classical FEM/FDM solver typically converges as the number of parameters (mesh points) increase. For neural network based methods, a similar phenomenon should be observed when the number of parameters in a neural network representation increases. There is a bunch of work (C. Schwab, G. Kutyonok, D. Yarotsky) that show theoretically for some classes of problems that it is true. But in practice, such results are never reported. Moreover, this paper uses only one fixed architecture and tests it for different grid sizes. So, what is the actual error of the computed solution?”


A: We have shown that changing the grid size in the case of the Burgers equation does not influence the accuracy of our network learning for regular solutions (Figure 1). This means that with the number of parameters that we use we have already reached convergence. 
The same holds for solutions that are close to shocks. Note that the FNO paper has shown convergence for regular solutions of the Navier-Stokes equations, by considering training on low-resolution grids and inferring on high-resolution ones. Our method is based on the same FNO architecture.


Q: “A second question is the methodology of learning the map. As initial condition, a sequence of Gaussian function is used. However, difficult examples for the Burgers equation are "kinks", and there is rich literature on constructing efficient numerical schemes for such kind of tasks. Can you provide any guarantee that the model learned for a small set of functions will be able to generalize to other functions?”

A: Since the dynamical system is nonlinear one cannot guarantee a generalization to an arbitrary set of functions. For the 2D-Burgers equation shocks begin to appear around t=1.15 as discussed in section 6.2. Although our model learns from data that does not include shocks at $t=1$, we are able to extrapolate solutions close to shocks as can be seen in Figure 3 (a). Following the reviewer’s concern, visualizations of these shocks and the ability of our method to handle them have been added to the appendix (section B).
The generalization question for Navier-Stokes equations, in particular for solutions with large velocity gradients, is very important and its systematic study will require a comprehensive set of accurate simulations.

Q: “Theorem one seems more like a Lemma or Statement, since it is rather straightforward.”

A: We changed the title Theorem to a Lemma. Note that the lemma has an important value since the different ways of violating it point to different possible error loss terms that we studied in detail in this work.

Q: “What norms are used to measure the reconstruction error? In PDEs, the errors are measured in the relative norms, which are suitable for the corresponding PDE.”

A: Indeed we use a relative norm error, see equation (13) in the manuscript.

Q: “Computational speed vs accuracy: the error does not depend on the mesh size (Figure 1) . For finite-element, finite volume method there is a systematic convergence of the error with respect to the number of parameters. What can be said about the convergence of the solution with increasing number of parameters? What is the complexity of solving a PDE compared to the classical solver of the same accuracy?”

A: Note that since the underlying method we use is FNO, where for each time we simply replace the network’s weights, the complexity of our method during inference is similar to the original FNO. The complexity of FNO is quasilinear which is at least an order of magnitude faster than traditional solvers as explained in the original FNO paper (see, https://openreview.net/forum?id=c8P9NQVtmnO).

 Q: “The numerical section is, in my opinion, the weakest part of the paper as it is not convincing that this method of training, which is certainly more computationally expensive, offers a large advantage over standard training. A clear advantage is the ability to extrapolate the solution outside the training time window and the authors should make this more of a focus, perhaps on problems where shocks don't develop, and show for how many Lyapunov times the extrapolated solution can keep up with the true trajectory. Furthermore considering much longer time domains would be of great interest as simply taking the solution to t=1 is not enough. Even the original FNO paper considers NS up time time 50. Performing such experiments is crucial if the work is to be published.”

A: As can be seen in Figures 2 and 3, effectively only one composition is needed to surpass FNO performance. In terms of training time, both the intermediate loss term and the composition loss term require additional 4 forward FNO executions, therefore the running time is x5 of the original FNO (neglecting the hyper network’s small MLP computation). For inference, the runtimes of both methods are approximately the same.

On the issue of late time behavior: Indeed, on general grounds, the chaotic nature of the dynamical system one expects that after a certain number of e-folds any network will not be able to learn the exponentially diverging trajectory.
Following the referee’s suggestion, we added an appendix section, where we assessed the ability of the hyper-network scheme to extrapolate solutions beyond $t=1$ by simulating additional $20$ solutions ranging from $t=0$ to $t=20$, in increments of $t=0.01$ for the 2D Navier-Stokes equation. Figure 4 in the appendix shows that our method is able to extrapolate solutions in one-shot up to $t\sim 2$ while doubling the error. This linear structure of error accumulation is unlikely to hold at long times due to the general chaotic structure of the solution. On general ground, one expects that for chaotic systems nearby trajectories would diverge exponentially: $\Delta v(t) = \Delta v(0) e^{\lambda t}$, When $\lambda t \ll 1$ we have $\frac{ \Delta v(t) -\Delta v(0)}{\Delta v(0)} \sim \lambda t$ and we can estimate from Figure 3(b) in the time window $[0,1]$ that $\lambda \sim 5\times10^{-2}$.  With such a value we can estimate, $t \sim 20 \ln \left(1+\frac{\Delta v(t) -\Delta v(0)}{\Delta v(0)}\right)$,. Thus, at $t=10$ we expect an error of about $65$ percent and at $t=2$ we expect the error to increase by a factor of $2$ compared to the supervised error at $t=1$ (as is validated in Figure 4).

Furthermore, FNO does not extrapolate to time scales of $T=50$, but rather learns them as they appear in its training set. This is in contrast to the extrapolation and interpolation capabilities of our method, which does not encounter physical states at times different than $T=0$ and $T=1$.


 Thank you for your review.

Q: “The idea of learning Markovian systems by exploiting the compositional structure of the flow map (again using FNO) was earlier proposed in (https://arxiv.org/abs/2106.06898) where the approach is to learn a FNO which emulates \Phi_{h} for some fixed time step h > 0 and predict forward by simply having the FNO act on its own output repeatedly. While this paper considers ergodic systems and tries to capture the statistics of the invariant measure by emulating very long-time dynamics, I suspect a similar approach can be used in the present setting by using a small time-step h along with the loss function proposed by the authors. “

A: In contrast to Li et al. (June, 2021 https://arxiv.org/abs/2106.06898), which propagates in time using a fixed time step, our approach is able to propagate at an arbitrary continuous-time step. Furthermore, in order for the approach described by Li et al. to obtain small-time resolution scales, it is required to execute the method with an increasing number of sequential FNO compositions. For instance, to obtain a $\frac{1}{10}$ ratio, one would need to run the FNO network $10$ times, thus the time resolution scales with the network’s depth, and each time scale requires a different architecture. 

Q: “I am slightly worried about the quoted results for the 1-d Burgers' equation where the authors' method outperforms the standard FNO by almost an order of magnitude. The paper (https://arxiv.org/abs/2108.08481 - Table 2) shows standard FNO results on 1-d Burgers' that are better than those of the presented method and can be improved even more by using a GeLU activation. This casts doubt as to whether the author's hyper network approach is giving the improved results or simply not tuning the FNO. “

A: Thank you for mentioning this contemporaneous work. Following the reviewer’s remark, we’ve tested our method both with the GeLU activation (Kovachki et al., Aug 2021 https://arxiv.org/abs/2108.08481) and without it. The addition of GeLU activation slightly improved the results of our method. In all our experiments, our results with GeLU activations are better than FNO with GeLU activation. Thus, not only that our method outperforms FNO, it is also robust with respect to the chosen activation function. In detail: 

1D results:
 
| Method          | burgers         | chafee          | genburgers2     |
 |:----------------|:----------------|:----------------|:----------------|
| FNO             | 0.0096          | 0.0028          | 0.0090          | 
| FNO GeLU        | 0.0022          | 0.0010          | 0.0032          | 
| Ours $n=1$      | 0.0022          | **0.0008** | 0.0031          | 
| Ours $n=1$ GeLU | **0.0015** | 0.0011          | **0.0024** |
 

2D results:
 

| Method    | 2D Burgers      | 2D Navier-Stokes   |
|:-----------------------------------------------|:----------------|:-------------------|
| FNO                                            | 0.0336          | 0.0467             |
| FNO GeLU                                       | 0.0248          | 0.0458             |
| Ours $P=1$                                     | 0.0335          | 0.0442             |
| Ours $P=1$ GeLU                                | **0.0214**          | **0.0429**            |




 I can confirm that using the re-implementation here [https://github.com/scaomath/galerkin-transformer](https://github.com/scaomath/galerkin-transformer/blob/a9c2dc4427bfaba051d7e0154f110e460050c1df/libs/layers.py#L1040),
- change `nn.Conv1d` to a faster `nn.Linear`;
- remove batch norm, optional layer norm in the wrapper;
- change ReLU to SiLU (similar to GeLU);
(see also the FNO official repo's issue in Review 5dc6's comments)

If we use FNO original repo's dataset. The 1d Burgers benchmark for FNO1d with 500 epochs of `MultiStepLR` with starting LR `1e-3`, weight decay `1e-4`, for 10 seeds the lowest evaluation error is `2.174e-3` with an average of `2.290e-3`.

Plus RNN-like temporal composition will be slow (equation (16) in the paper). No method can dethrone FNO in terms of speed (not surprising since it is FFT + dense matrix multiplication along the temporal direction).

The paper combines the FNO architecture with hypernetworks to learn the solution operator (flow map)
of Markovian PDE systems. A hypernetwork whose input is the time domain (\R_+) is trained to output 
the weights of a FNO which then acts on an initial condition to produce the PDE solution at the time 
input to the hypernetwork. Training is done by uniformly sampling random times on a bounded input domain
and exploiting the Markov property of the flow map, allowing to use as data only the initial condition 
and the solution at a final time without a need for the entire time series. Numerical experiments showing 
a modest improvement over the standard FNO are performed on the 1-d Burgers' equation, generalized Burgers'
equation, and the Chafee–Infante equation as well as on the 2-d Burgers' equation, and the Navier-Stokes
equation. Overall, the paper is well-written and easy to follow. I quite like the idea of using a hyper-network
to make the time domain continuous instead of having to add a new dimension to the FNO (the paper does lack
a short review of the FNO which I think is needed as it is not a standard architecture quite yet). The 
idea of learning Markovian systems by exploiting the compositional structure of the flow map
(again using FNO) was earlier proposed in (https://arxiv.org/abs/2106.06898) where the approach is 
to learn a FNO which emulates \Phi_{h} for some fixed time step h > 0 and predict forward by simply
having the FNO act on its own output repeatedly. While this paper considers ergodic systems and tries
to capture the statistics of the invariant measure by emulating very long-time dynamics, I suspect a 
similar approach can be used in the present setting by using a small time-step h along with the loss
function proposed by the authors. A comparison of the two methods would be quite helpful. 

I am slightly worried about the quoted results for the 1-d Burgers' equation where the authors' method
outperforms the standard FNO by almost an order of magnitude. The paper (https://arxiv.org/abs/2108.08481 - Table 2)
shows standard FNO results on 1-d Burgers' that are better than those of the presented method and can be 
improved even more by using a GeLU activation. This casts doubt as to whether the author's hypernetwork
approach is giving the improved results or simply not tuning the FNO. The numerical section is, in my
opinion, the weakest part of the paper as it is not convincing that this method of training which is certainly
more computationally expensive offers a large advantage over standard training. A clear advantage is the 
ability to extrapolate the solution outside the training time window and the authors should make this more 
of a focus, perhaps on problems where shocks don't develop, and show for how many Lyapunov times the 
extrapolated solution can keep up with the true trajectory. Furthermore considering much longer time domains 
would be of great interest as simply taking the solution to t=1 is not enough. Even the original FNO
paper considers NS up time time 50.  Performing such experiments is crucial if the work is to be published. Nice way of dealing with the time domain by using a hypernetwork, but experiments are not so convincing that
the approach is better than the standard.  

The paper presents a model and a loss function for approximation of the solution map of certain partial differential equations. 
The idea is to use HyperNetworks: the mapping of the initial condition to time t done by model f($\theta$, x), and the parameters are predicted by another neural network. The models utilize the recently proposed Fourier Neural Operator (FNO) for the mapping model, and a fully-connected network for the hypernetwork. Several losses are proposed to ensure consistency of the proposed models, including the composition and reconstruction. The tests are done on a battery of examples, and compared to other neural network baselines the error is smaller. A theorem is proved, but is rather straightforward. The paper is in a popular direction, and has all the pros and cons of such kind of papers. The idea of using hypernetworks to model the transition operator is reasonable and of practical interest.  The drawback is that the computed solutions are approximate and do not show (at least, this not considered) systematic convergence to the true solution of the PDEs. Let me explain in more details. A classical FEM/FDM solver typically converges as the number of parameters (mesh points) increase. For neural network based method, a similar phenomenon should be observed when the number of parameters in a neural network representation increases. There is a bunch of work (C. Schwab, G. Kutyonok, D. Yarotsky) that show theoretically for some classes of problems that it is true. But in practice, such results are never reported. Moreover, this paper uses only one fixed architecture and tests it for different grid sizes. So, what is the actual error of the computed solution?

A second question is the methodology of learning the map. As initial condition, a sequence of Gaussian function is used. However, difficult examples for the Burgers equation are "kinks", and there is rich literature on constructing efficient numerical schemes for such kind of tasks. Can you provide any guarantee that the model learned for a small set of function will be able to generalize to other functions? 

Comments&Questions:

- The Theorem one seems more like a Lemma or Statement, since it is rather straightforward.
- What norms is used to measure the reconstruction error? In PDEs, the errors are measure in the relative norms, which are suitable for the corresponding PDE. 
- Computational speed vs accuracy: the error does not depend on the mesh size (Figure 1) . For finite-element, finite volume method there is a systematic convergence of the error with respect to the number of parameters. What can be said about the convergence of the solution with increasing number of parameters? What is the complexity of solving a PDE compared to the classical solver of the same accuracy? 
- Probably, the most interesting part of the model is the potential ability to predict the solution for long time in "one shot".  Can you clarify, who long you can do it? What is the largest time the model can predict?  If smaller steps are needed, then it is more close to the classical time-stepping (also interesting, but not so, since again, a classical solver, maybe with higher-order elements, will outperform the Neural Solver quite significantly).

 The paper is in a popular direction, proposes a reasonable improvement, but the presentation and methodology of the study could be significantly improved.

Meta-review: This paper presents an approach to learn the solution operator of Markovian partial differential equations (PDEs) by combining the Fourier Neural Operator (FNO) with a hyper-network.  In short, the hyper-network g_\theta(t) is trained to output the weights of a FNO f_w(x), which (given an initial condition) outputs the PDE solution at the time given to the hyper-network. The main claimed contributions of the proposed approach (as compared to, e.g., the original FNO architecture) are that the obtained solutions improve the learning accuracy at the supervision time points and that the solutions are able to interpolate and extrapolate to arbitrary times.  

The reviewers seemed to like the idea of using hyper-networks for modelling continuous-time FNOs. Several issues were raised by the reviewers, e.g., with respect to related work by Li et al. (2021, https://arxiv.org/abs/2106.06898), which I believe were addressed by the authors satisfactorily. However, it is still concerning that the reported performance for FNO in, e.g., 1d-Burgers does not quite match those recently published  (Kovachki et al, 2021, https://arxiv.org/abs/2108.08481, Table 2).  Although the authors did report additional results using GeLU, these results are still very different to those in  Kovachki et al ( 2021) and it is unclear whether the improved performance is due to a lack of tuning the baseline FNO. I commend the authors for, as suggested by the reviewers, running more extrapolation experiments. However, I believe the reviewers also made a point about considering (training) times much longer than 1, as even the original FNO paper did this for Navier Stokes (NS) with T=50. 

Overall, the paper provides modest improvement wrt FNOs, although it does extend its capabilities to interpolation and extrapolation. The paper will also benefit from providing a brief overview of FNOs.

Decision: Reject