Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an approach to modeling PDE solutions using a trainable hyper-network that generates the parameters of an FNO network. The hyper-network gets as an input only the desired continuous time t (>0) and the FNO transforms an initial condition under the dynamics specified by the output of the hyper-network. While the performance of this method at the discrete training time steps already improves on FNO, the main innovation is allowing for interpolated solutions at any continuous time. 
strength_and_weaknesses: The main strength of the paper is the pursuit to learn the underlying dynamics over the continuous time domain. As the authors mention in the discussion section (Sec. 6), this might indeed be very useful as a reduced-order model e.g. of the NSE. 

Comments:
- Given that the main comparison baseline is FNO and FNO is also a major building block in this architecture, I would have expected a brief introduction to the used FNO formalism. 
- Where are CNNs and GNNs? Given the recent popularity of GNNs for the solution of PDEs (e.g. https://arxiv.org/abs/2202.03376, https://arxiv.org/abs/2002.09405) as well as CNNs (e.g. https://arxiv.org/abs/1810.08217), one should at least mention these lines of research in the "Related Work" section and justify why they are not (or are) relevant here. 
- I see the intention to learn meaningful interpolated solutions of PDEs via the hyper-network, but given the lack of results on extrapolation over longer horizons, e.g. t=10...100, I would speculate that the results are not promising. Any comment on that?  Given the practical relevance of extrapolating over longer times, this aspect should be addressed somewhere. And this regime is something that has already been studied by the so-called direct-time methods, e.g. in this workshop paper by Meyer et al., 2021 using GNNs: https://arxiv.org/abs/2112.10296. I understand that all the PDEs you consider are dissipative, thus too long trajectories wouldn't make sense, but why not look at some inviscid equations (e.g. Burgers, Euler) and discuss e.g. the survival time as done in https://arxiv.org/abs/2202.03376.
- I like the loss terms defined in the paper quite a lot: they incorporate all possible types of interpolation. However, I know other papers that do similar things (ignoring L_final and L_initial for a second): The construction of L_comp^(P) is very closely related to the "temporal bundling" presented here https://arxiv.org/abs/2202.03376, and L_inter reminds me of the "pushforward training" from the same paper. I'm sure that there are more analogies from other papers.

Two minor problems I detected are:
1. some formatting issues with the equations leading to "Equation equation [...]"
2. "Miltiple Graph Neural Operator" -> Multipole Graph Neural Operator"
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to digest. As mentioned in the main review, some more information on FNO could be useful.

I haven't had the chance to work with FNO-based approaches yet, thus I cannot evaluate the figures and tables more than what I see in the paper. The presented results look ok to me and the improvement over vanilla FNO in the interpolation regime seems quite clear.

Regarding novelty, the paper seems to introduce the combination of hyper-networks and FNO for the first time. And although the hyper-network and FNO networks seem to be "off the shelf", given the rather specialized loss makes for a novel approach.

The supplementary material seems to do the job in terms of reproducibility. The 1D data is downloadable, and the 2D and 3D data can be generated using PhiFlow. I successfully started a training run on the 1D data.
summary_of_the_review: The idea of the paper is very interesting, but out of the 21 mentioned references, I didn't find a single one from 2022. This suggests that the authors lack knowledge of the recent literature, which includes e.g. GNNs. This issue and the weak justification of the chosen methods and experiments (see above) are the reason why I believe that this paper is not yet ready for publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a hyper-network to solve partial differential equations. 
strength_and_weaknesses: The strengths of this paper are:

$\mathbf{1}.$ The novelty of separating time and space by the hyper-network architecture. 

$\mathbf{2}.$ This paper is clearly written and well organized, with detailed explanation of theories and experiments. 

The weakness of this paper are:

$\mathbf{1}.$ The proposed approach is a combination of known methods, although it might be novel to use hyper-network.

$\mathbf{2}.$ The baseline used in the experiments might not be strong enough. For example, why using FNO and/or MGNO in the experiments? Why other network-based methods were not used? Also, is it possible to use some numerical methods as a baseline? The main concern is whether or not the chosen baseline is good and comprehensive to use as references. 

Other concerns (minor if compared with the above weakness):

$\mathbf{1}.$ The literature review claims the model base method, such as (Raissi 2019), does not require training dataset. Physics informed network does require training the network with data, as most of the network-based method do. 

$\mathbf{2}.$  The literature review also claims that the methods referred do not predict intermediate time, which could be false. Even if it is right, there exists methods which can predict solutions at intermediate time. 

$\mathbf{3}.$ The loss function presented in Eq (17) is a simple addition, which could be improved by tuning hyper-parameters as a convex or other types of combinations. 
clarity,_quality,_novelty_and_reproducibility: $\cdot$ This paper is clearly written and organized. 

$\cdot$ The novelty of this paper is not obvious as it merely combines known methods. It might be ok to qualify a good novelty as long as there are significant contribution from other aspects, such as empirical results, etc. However, $\mathbf{1}.$ It is not clear how innovative improvement on the architecture of hyper-network; $\mathbf{2}.$ it is not convincing enough the proposed method outperforms other methods as the chose of baseline is limited; $\mathbf{3}.$ predicting intermediate time point is not a novel contribution. Based on the above three points, along with the combo of known methods, the novelty is not obvious. 

$\cdot$ The reproducibility is feasible. 
summary_of_the_review: As stated in the above comments, this paper might not deliver a solid contribution, and the novelty could be limited.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work considers neural operators in the semisupervised setting. It extends Fourier neural operator with a hypernetwork structure, which allows composition in time. The paper provides numerical experiments on 1d Burger equation and 2d and 3d Navier-Stokes equation. The proposed method has a slightly improved error rate but better interpolation performance in time.
strength_and_weaknesses: Strength:
- extend operator learning to semisupervised learning
- new hyper-network architecture
- continuous interpolation in time

Weakness:
- learning accuracy at the supervision time-points is not very impressive (vs FNO+)
- need more justification for a semi-supervised learning setting.

Comment:
- Is this model data efficient? Maybe the proposed method will show an advantage at a smaller amount of data sets.
- How does the continuous-time learning compare with the standard autoregressive model (t ->t+1, e.g. Markov neural operator)? The autoregressive automatically enforces composition in time. Since the Navier-Stokes is markovian, one probably does not need history. 
clarity,_quality,_novelty_and_reproducibility: The paper is overall understandable, but the clarity can be improved. The novelty is not very impressive.
summary_of_the_review: Overall, I think the paper provides a concrete study, but its contribution and novelty may not meet the threshold. Therefore, I think this work is marginally below the bar for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Fourier Neural Operator (FNO) is a data-based approach for learning an infinite-dimensional mapping, which is helpful for numerically solving PDEs. The paper extends FNO to an evolutional setup. FNO can be used in an evolutional setup too, for which one can learn a mapping that maps the solution at time t to that at time t+dt, and then iterate this mapping. This paper however aims at directly learning the evolutional semigroup $\Phi(t): \mathcal{H} \to \mathcal{H}$, as a function of both the solution/initial condition and $t$. For this, properties of a semigroup are encouraged by the construction of loss and the time-dependence was accounted for by a specific hyper-network architecture. Interesting results are empirically shown.
strength_and_weaknesses: Strength: 
Overall, this is an interesting paper. I especially enjoy the construction of the loss, which prompts what is learned to be like a semigroup.

Weakness: 
I wish more empirical results are provided for a more comprehensive evaluation of the performance. I hope I didn’t read incorrectly, but at this moment I have several questions: 

(1) Has the method been compared to iterations of vanilla FNO? That is, FNO for characterizing the evolution map over time-step dt, and then iterated T/dt times for reaching time T? I hope the new results still outperform FNO used in this way, for both small dt and large dt. 

(2) What kind of Navier-Stokes problem is being considered? What is the initial condition, what should the exact solution look like, and what does the learned solution look like?

(3) In existing methods like NODE, time dependence was often just used as a part of the input of a neural network. Is it possible to justify, experimentally, why the new hierarchical hyper-network structure is better? Is there any performance/computational-cost trade-off?

clarity,_quality,_novelty_and_reproducibility: The loss seems innovative. The architecture is not new but its application to evolutional PDE solve seems new.

Regarding quality, perhaps it is better to weaken some of the claims. For instance:

* Abstract writes “The evolution of dynamical systems is generically governed by nonlinear partial differential equations”. But it is not “generically”. An example is https://en.wikipedia.org/wiki/Tent_map .
Another example is non-local “PDE”.

* Introduction paragraph 2: “all analytical techniques fail”. This is false and disrespecting an entire field (e.g., [Foias, Manley, Rosa, Temam. Naiver-Stokes equations and turbulence. 2001]).

These are easy to revise though.

summary_of_the_review: Overall I feel this is an interesting paper. Simple but good idea, which can be a solid step toward an important problem (ML for PDE solve). I still have some reservation at this moment due to comments made above about the results, but I can be convinced.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a hyper-network for FNO’s that allow it to predict intermediate future states. These are used to augment the loss for a better trained model. 

strength_and_weaknesses: 
s: the result analysis is interesting and quite insightsful.

s: the method and loss seem to be well derived, and are sensible.

s/w: The results are not strong enough. The method does roughly hover around the baseline, getting either marginal improvements or similar results.

w: the writing is ok but could be improved. The introduction could do a better job at describing the problem and the issues in earlier works. This paper is quite incremental by focusing largely on FNO, and making it more powerful. The paper could try to position its contributions on the wider context better.
clarity,_quality,_novelty_and_reproducibility: 
Clarity adequate, but could be improved. Quality adequate. Novelty adequate. Reproducibility high. 

summary_of_the_review: This is an ok paper with a sensible idea, but ultimately I feel this is not enough and nothing really stands out. The method is simple and incremental, and the results are not impressive enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces an extension of Fourier Neural Operator (FNO) for solving PDEs and forecasting dynamical flows. The novelty consists in using a hypernetwork for modeling the weight evolution of FNO inside the forecasting horizon. Compared to the baseline FNO, supervision is added at intermediate time steps in the forecasting interval for training and the loss function is enriched with additional terms. Evaluation is performed on a series of simulated PDEs in 1 and 2 dimensions.

The main innovation w.r.t. FNO is the possibility to predict in continuous time via the hypernetwork. All the reviewers consider that the novelty is limited and that the evaluation should be enriched with additional baselines. The paper is interesting but too preliminary for acceptance.

justification_for_why_not_higher_score: Marginal innovation and experimental validation insufficient
Decision: Reject