Semi-supervised learning of partial differential operators and dynamical flows
Authors: Michael Rotman
Amit Dekel
Ran Ilan Ber
Lior Wolf
Yaron Oz
Venue: Submitted to ICLR 2023 (2023)

Abstract: The evolution of dynamical systems is generically governed by nonlinear partial differential equations (PDEs), whose solution, in a simulation framework, requires vast amounts of computational resources. In this work, we present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. Our method treats time and space separately and as a result, it successfully propagates initial conditions in continuous time steps by employing the general composition properties of the partial differential operators. Following previous works, supervision is provided at a specific time point. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, or three spatial dimensions. The results show that the new method improves the learning accuracy at the time of the supervision point, and can interpolate the solutions to any intermediate time.

Under review as a conference paper at ICLR 2023
SEMI-SUPERVISED LEARNING OF PARTIAL DIFFEREN- TIAL OPERATORS AND DYNAMICAL FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
The evolution of many dynamical systems is generically governed by nonlinear partial differential equations (PDEs), whose solution, in a simulation framework, requires vast amounts of computational resources. In this work, we present novel method that combines hyper-netword, solver with Fourier Neural Operator architecture. Our method treats time and space separately and as result, it successfully propagates initial conditions in continuous time steps by employing the general composition properties of the partial differential operators. Following previous works, supervision is provided at specific time point. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, or three spatial dimensions. The results show that the new method improves the learning accuracy at the time of the supervision point, and can interpolate the solutions to any intermediate time.
INTRODUCTION
The evolution of classical and quantum physical dynamical systems in space and time is generically modeled by non-linear partial differential equations. Such are, for instance, Einstein equations of General Relativity, Maxwell equations of Electromagnetism, Schrödinger equation of Quantum Mechanics and Navier-Stoker (NS) equations of fluid flows. These equations, together with appro- priate initial and boundary conditions, provide a complete quantitative description of the physical world within their regime of validity. Since these dynamic evolution settings are governed by partial differential operators that are often highly non-linear, it is rare to have analytical solutions for dynamic systems. This especially true when the system contains a large number of interacting degrees of freedom in the non- non-linear regime.
Consider, as an example, the NS equations, which describe the motion of viscous fluids. In the regime of high Reynolds numbers of the order of one thousand, one observes turbulences, in which all symmetries are broken. Despite much effort, two basic questions have remained unanswered, that is the existence and uniqueness of the solutions to the 3D NS equations and the anomalous scaling of the fluid observables in statistical turbulence. The solution to the (deterministic) NS equations seems almost random and is very sensitive the initial conditions. Many numerical techniques have been developed for constructing and analysing the solutions to fluid dynamics systems. However, the complexity of these solvers grows quickly as the spacing the grid that is used for approximating the solution is reduced and the degrees of freedom of the interacting fluid increases.
Given the theoretical and practical importance of constructing solutions to these equations, it natural to ask whether neural networks can learn such evolution equations and construct new solutions. The two fundamental questions are: (i) The ability to generalize to initial conditions that are different from those presented in the training set, and (ii) The ability to generalize to unseen time points, not provided during training.The The reason to hope that such tasks can be performed by machine learning that despite the scemingly random behaviour of, e.g. fluid flows in the turbulent regime, there is an underlying low-entropy structure that can be learnt. Indeed, in diverse cases, neural network- solvers have been shown provide comparable results to other numerical methods, while utilizing fewer resources.
Our Contributions
We present neter-n based solver combined with Fourier Neural Operator architecture which is able to learn non- lonerr partial differential operators that govern the dynamics of chaotic and out of the equilibrium flows.

---

Under review as a conference paper at ICLR 2023
1. Our hyper-network. architecture treats time and space separately. Utilizing data set of initial conditions and the corresponding solutions at a labeled fixed time, the network learns large class of time evolution PDEs.
2. Our approach enables interpolation to arbitrary (unlabelled) continuous times without additional data points.
3. Our solutions improve the learning accuracy at the supervision time-points.
4. We thoroughly test our method on various time evolution PDEs, including non-linear fluid flows in one, two and three spatial dimensions.
2 RELATED WORK
Hyper-networks While conventional networks employ a fixed set of pre-determined parameters, which is independent of the input, the hyper-network scheme, invented multiple times, and coined by Ha et al. (2017), allows the parameters of a neural network to explicitly rely on the input by combining two neural networks. The first neural network, called the hyper-network, processes the input or part of it and outputs the weights of second neural network. The second network, called the primary network. has fixed architecture and weights that vary based on the input. It returns, given its input, the final output.
This framework was used successfully in a variety of tasks, ranging from computer vision (Litwin & Wolf 2019), continual learning (von Oswald et al. (020), and language modeling (Suarez) (017). While is natural to learn functions with hyper-networks since the primary network can be seen as a dynamic, input-dependent function, we are not aware of any previous work that applies this scheme for recovering physical operators.
Neural network-based PDE solvers Due to the well-known limitations of traditional PDE solvers on one hand, and in light of new advances made in the field of neural networks on the other, lately we have witnessed very significant progress in the field of neural network- based PDE solvers (Karniadakis et al. 2021). These solvers can be roughly divided into two groups according to the resource they utilize for learning: data-driven and model-based.
Model-based solvers, known as Physics Informed Neural Networks (PINNs) (Raissi et al., 2019). harness the differential operator itself for supervision. This is done by defining loss, the residual of the PDE. These solvers also require a training dataset and can provide solutions for arbitrary times.
Data-driven solvers trained over large dataset containing initial conditions and observed final states after the same time interval, T. Among these solvers there are encoder-decode based ones, either fully convolutional (Zhu & Zabaras 2018) one, U-Net shaped one (Thuerey et al.) 2020 the PCANN (Bhattacharya et al. (021) 1) which applies Principal Component Analysis (PCA) as an auto-encoder and interpolates between the latent spaces using neural network. The Multipole Graph Neural Operator (MGNO), the DeepONet which encodes input function and locations separately using two neural networks (Lu et al. 2019b), Graph Neural Simulators Sanchez-Gonzal (2220) which utilize graph neural networks, and the MP-PDE (Brandstetter et al. 2021], which learns neural operator on the graph These solvers learn solutions on a specific discretized grid, which poses a limitation for any practical applications.
al.
Recently, a mesh-invariant data-driven direction has been proposed (Lu et al., 2019a Nelsen Stuart 2021 Anandkumar et al.) 2020; Patel et al. (202)). The mesh invariance is obtained by learning operators rather than mappings between initial and final states. This is achieved using architectures that enforce mesh invariant network parameters. Such networks have the ability train and infer on different meshes. As a result, these networks can be trained on small grids and run, during inference, on very large grids. Li et al. (2020b) have advanced the Fourier Neural Operators (FNO). FNOs utilize both a convolution layer in real space and a Fourier layer in the Fourier domain. Each Fourier layer first transforms its input, Rd to Fourier space, point-wise multiples the transformed input by a set of learned coefficients, w E TR (note that since is a real vector, only coefficients are required), and applies an inverse Fourier transform to obtain the output, y F-1 {w c F {x)) E Rdd. Since the learned parameters reside in Fourier space, it allows training on a specific scale, and inferring on other scales. has been shown that the FNO solver outperforms previous solvers in number of important PDEs. The current data-driven methods
line of works by introducing
2

---

Under review as a conference paper at ICLR 2023
successfully evolve solution to the supervised time step. By successive applications of these methods, solutions may be evolved increments of T. Though the accuracy of these solutions may degrade, some statistical properties
still be obtained (Li et al. 202T). However, these methods are not designed for evolving solutions to intermediate times. Knowing the solutions along the complete time evolution trajectory is highly valuable for theoretical and practical reasons and provides means for learning new dynamical principles.
3 PARTIAL DIFFERENTIAL EQUATIONS
Consider a d-dimensional vector field ū( t) T x R + R., where I = (II,..., are periodic spatial coordinates 2; = 2, + 2x on the d-dimensional torus. TV. and f is the time coordinate. The vector field evolves dynamically from the initial condition i( t = 0) T - R according to non-linear PDE of the form,
0,ū(Z,t) = Lo)
(1)
where L is a differential operator that does not depend explicitly on time, and has expansion in and its spatial derivatives. We assume that given regular bounded initial vector field there is unique regular solution equation
[
Since L does not depend explicitly time, can formally
write the solution of such equations as
ग(ū. () = e
(2)
Because of dissipation terms, such as the viscosity term in the NS equations, solutions to equation] generically break time reversal invariance (1 -1, + -ū). Furthermore, the total energy of the system is non-increasing as a function of time since we are not injecting energy at t * 0, 0, f dd S 0, and serves as consistency on the network solutions.
In this work, we consider the following non-linear PDEs:
Generalized Burgers equation. Describes dieddimeniioall compressible fluid flows with scalar velocity field v(r, !). We use the generalization parameter with q 1, 3, (g case corresponds to the regular Burgers equation),
0,0 + V v/v,
where v is the kinematic viscosity, V the gradient and A is the Laplacian
Chafee-Infante equation. One-dimensional equation with constant parameter that models reaction-diffusion dynamics,
Oe + A(v v) = Av.
(4)
Two-dimensional Burgers. Describes compressible fluid flows in two space dimensions for the vector field, ū(ū 1) (vt,²²),
0,0 (U V)ū= vAE.
(5)
Two and three-dimensiooall NS. Two and three-dimensional incompressible NS equations,
0, (ū. V)ū Vp vAØ V v 0, where p is the fluid's pressure, ((Ztt) (u',..., vd) and y is the kinematic viscosity.
66)
4 METHOD
Typically, data-driven PDE solver, such neural network, evolves unseen velocity fields, र(ह =0 0) to a fixed time T, PTū 0) = (()
by learning from set of i 1...N initial conditions sampled at t 0, T. (H, 0), and their corresponding time-evolved solutions of ūn (#, = T) of equation |
We generalize 4y, to propagate solutions at intermediate times, 0 < t T, by elevating a standard neural architecture to a hyper-netword one. A hyper-netwon architecture a composition two neural networks, a primary network (xr and hypernetwork go (t) with learned parameters such
3

---

Under review as a conference paper at ICLR 2023
that the parameters of f are given the output of g. Unlike common neural architectures, where single input is mapped to a single output, in this construction, the imput is mapped to function, (o)))) which maps its output. Applying this to case have,
2, 0) (o 0)) ((.,
This architecture may be used to learn not only the time-evolved solutions at t T, but also intermediate solutions, at 0 < < T, without explicitly providing the network with any intermediate time solution. This task may be accomplished by utilizing general consistency conditions, which apply to any equation of the form of equation
21, D22 = Lr Dt, = 11,
Notice that t La follows from equation 9 Consider the differential equation Suppose we construct map d such that T (I, t 0) ह (æ, t T), where ū7 ( 0) are all initial conditions (or form an appropriate complete set that spans all possible initial conditions) for equation | and oe (xz, whether 2,0% (z, 0) = U. (ã, t) for all times 2 0, i.c. that the map & evolves the initial condition conrectly as the solutions of equation for any time.
7) are the corresponding solutions of the equation at time T. We would like to know
Lemma 4.1. A map 4 that propagates complete set of initial conditions at l. = O to the corre- sponding solutions of equation F at fixed time propagates the initial conditions to the corresponding solutions for any time t 0.
and satisfies the composition law in equation
Proof. Denote by the map that propagates correctly at any time t the initial conditions to solutions
of equation 1 The space of initial conditions ह( 1 0) T - TRd is complete set of
functions and the same holds for ग(ū, T). Since yor agrees with Φr on this set of initial condition follows that 4P
TT. Consider next a partition of the interval [0, T] to p identical parts,
[0,
T T 27
1)2 T). We have the composition of maps:
p
4 $
& ΦT p PT
PrPT
PI
(10)
from which we conclude that Pr = Φ (there can be an overall sign in the relation between them when p taken to odd which is fixed to one by continuity). We can perform this division for any p and conclude using the composition rule in equation that 4; = frr for any t.
4.1 LOSS FUNCTIONS
In this section we introduce the loss functions used to train our models. In order to simplify our notation, we will suppress the explicit space dependence our equations, so 7 t 0) will be denoted by T()). We denote by N the size of the training set of solutions. Our loss function consists of several terms, supervised and unsupervised.
The information of the specific PDE is given to our model via supervised term,
N
Línal Σ
Err (2rū, (0), v (T)),
(1I)
which is responsible for propagating the initial conditions to their final state at / = T. The recon- struction error function Err is defined as,
Σa
C 5.n
Er (2.b)) (2.b)=
(12)
Eg D3
where the ax 3 indices run over the grid coordinates.
To impose the consistency condition + = Ia, we use the loss function term,
Linitial =
Σ
Err (Bo7, (0), (0)
(13)
4

---

Under review as a conference paper at ICLR 2023
The composition law in equation 9 is implemented by the loss function term,
Cinter Σ Err +13 Uy (0), Peg 24 o (0))
(14)
where 7= 14, + 12 is drawn uniformly from 0|0, 7] and ( is drawn uniformly from U[0, 77, so the sum of tj + 1/ does not exceed T.
The last term concerns the composition of the maps, 2, to generate Φr by p-sub-intervals of the interval [0, T].
p T
II
11
Σ =T=
t²
(15)
j=1
and it reads,
N P
P II I
C ((P)
ErT
(0) 0 11
(16)
where P is the maximal number of intervals used. The time intervals are sampled using ty ~ U j j- (T) and the last interval is dictated by the constraint in equation 15 The P = term is
omitted, since it corresponds Linal- The total loss function reads,
c(Pe Limal Linitial + Linter <
(17)
EXPERIMENTS
We present a battery of experiments ID, 2D and 3D. The main baseline we use for comparison is FNO (Anandkumar et al., (020). which is the current state of the art and shares the same architecture as our primary network For the 1D Burgers equations, we also compare with the results of Multipole Graph Neural Operator (MGNO) (Li et al. 2020a). We do not have the results of this method for other datasets, due its high runtime and memory complexity.
5.1 EXPERIMENTAL SETUP
Our neural operator is represented by a hyper-network composition of two networks. For the primary network f, we use the original FNO architecture. This architecture consists of four Fourier integral operators followed by ReLU activation function. The width of the Fourier integral operators is 64, 32 and 20, and the operators integrate up to cutoff of 16. 12, 4 modes in Fourier space, for one, two, and three dimensions, respectively. Recently, Kovachki et al. (202T) demonstrated that the FNO architecture may benefit from replacing the ReLUU activation with GeLU activation. Therefore, our results are reported using both variants, denoted by + suffix for the GeLUU variant.
For the hypernetwork we use fully-connected network with three layers, each with a width of 32 units, and ReLU activation function. 9 generates the weights of network. for given time point I, thus mapping the scalar / to the dimensionality of the parameter space of f. A tanh activation function is applied to the time dimension, f, before is fed to g. Together, f and g constitute the operator 4, appears in equation
All experiments are trained for 500 epochs, with an ADAM Kingma Ba 2015)) optimizer and learning rate of 0.001, weight decay of 0.0001, and batch size of 20. For MGNO, used a learning rate of 0.00001, weight decay of 0.0005 and a batch size of 1, as in its official github repository. For our method, at each iteration and for each sample, sample different time points for obtaining the time intervals of C. and Linter as detailed in Section 4.1
For the one-dimensional PDEs, we randomly shift both the input and the output to leverage their periodic structure. For one- and two-dimensional problems, the four loss terms are equally weighted. For the three dimenssini NS equation, the Linter term was multiplied by a factor of 0.1, the reason being that as we increase the number of space dimensions we encounter a complexity due to the growing number of degrees of freedom (i.e. the minimum number of grid points per integral scale)
5

---

Under review as a conference paper at ICLR 2023
required to accurately describe fluid flow. Standard Kolmogorov's type scaling (Frisch 1995 implies that the number of degrees of freedom needed scales as Na R. where R. is the Reynolds number. This complexity is seen in numerical simulations and we observe it in our framework of learning, as well. All experiments were executed using an NVIDIA RTX 2080 TI with 24GB.
5.2 DATA PREPARATION
Our experiments require a set of initial conditions at t = 0 and time-evolved solutions at time t = T for our PDEs. For the simplicity of notations we will set T = 1 in the following description. In well as the time-evolved solutions are periodic in all spatial
all cases, the boundary conditions dimensions. The creation of these datasets consists of two steps. The first step is to sample the initial conditions from some distribution. For all of our one-dimensional experiments we use the same initial conditions from the dataset et al. (2020b). which was originally used for Burgers equations and sampled using Gaussian random fields. For Burgers equation, we used solutions sampled at 512, 1024, 2048, 4096, 8192, while for the other PDEs we used a grid size of 1024. All one-dimensional PDEs assume a viscosity, V 0.1.
For the two-dimensional Burgers equation our data is sampled using a Gaussian process with a periodic kernel,
k(x, y; = oze 2 (1-x111
Pl-- 12)
(18) with σ = 1 and = 0.6, and with a grid size 64 x 64 and a viscosity of v = 0.001. For the two and dinensimmeiional NS equation
the droww package (Holl et al.) 2020 to generate samples 64 x 64 and 128 x 128x x 128 grids, later resampled to 64 x4 x 64. For the two and three-dimensional NS, we use viscosity of 0.001. In all cases the training and test sets consist of 1000 and 100 samples, respectively.
The second step is compute the final state the two-dimensional Burgers equations we use the py pde python package (Zwicker (MIT) License) to evolve the initial state for the final solution. To evaluate the time evolution capabilities of our method, we compute all intermediate solutions with At 0.01 time-step increments for the one-dimensional Burgers equation. For the two-dimensional Burgers equations we do this up to f = 1.15, since at this regime the solutions develop shock waves due to low viscosity and we learn regular solutions. For the two-dimensional and three-dimensional NS equations we use O' package (Holl et al. (MIT License) to increment the solutions with At = 0.1.
1. For the equations in one-dimension and
5.3 RESULTS
We compare the performance of our method on the one dimensional PDEs with FNO Li et al. 2020b) and MGNO (Li et al. 2020a). In this scenario, we use a loss term with two intervals, P 2. Since MGNO may only use batch size of and requires a long time to execute, we present it only for the mimedimesioonal Burgers equation. As seen in Table original FNO by at least factor of 3. The addition of a GeLU activation function, suggested by Kovachki et al. (202)) is beneficial to FNO, but offers only a slight improvement to our method, and only some of the benchmarks. We further compare the -ie-dimeniional Burgers equation for various grid resolutions at t = 1 in Figure T As previously noted by Li et al. (2020b), the grid resolution does not influence the reconstruction error for any of the methods.
all PDEs, our method outperforms the
To assess the contribution of the Limer term and to account for the influence of the number of intervals induced by Clipp we evaluate the reconstruction error for the task of interpolating in t e 1) in comparison to the ground truth solutions in Figure 2 As a baseline for the interpolation capabilities of our network. we further apply linear interpolation between the initial and final solution, a(t) = (1 - t))(O) + te(1) (even though one uses the t = solution explicitly for that which gives clear and unfair advantage at t = 1). The horizontal dashed cyan line marks the reconstruction error of the the FNO method at = 1, thus the interpolation our approach provides exceeds the prediction ability of FNO at discrete time steps. For extrapolation, & > 1, we divide the time into unit intervals plus a residual interval (e.g. for t = 2.3 we have the three intervals 1, 1, 0.3), and we let the network evolve the initial condition to the solution at t 1, and then repeatedly use the solution as initial condition to estimate the solution at later times. As can be seen, the addition of the Linter term dramatically improves the reconstruction error. Furthermore, increasing the number

---

Under review as a conference paper at ICLR 2023
Table
The reconstruction eπrоr on the one-dimensional PDEs 1, grid size 1024. B refers Burgers equations, GB to generalized Burgers equations, and CI to Chafee- Infante equation.
Method
CI
GB q=2
GB q=3
GB q=4
MGNO
0,0552
FNO
0,0096
0.0028
0.0090
0.0095
0.0098
FNO+
0.0022
0.0010
0.0032
0.0025
0.0055
Ours
0.0022
0.0008
0.0031
0.0030
0.0030
Ours+
0.0015
0.0011
0.0024
0.0026
0.0030
Roconstruction Error
END
SGNNO
10-3
512
1,024
2,048
1000
8,192
Grid Size
Figure 1: The reconstruction error the one-dimensional Burgers equation for different grid resolutions.
of intervals P induced by (P)) reduces the reconstruction error, but with smaller effect. The reconstruction error in the extrapolation region is low mainly because the high energy dissipation. Note that for all of the presented variations, the reconstruction error of our method / is lower than FNO.
Table contains comparison of our method to FNO the two dimensional PDEs, Burgers and NS equations. In two-dimensions the advantage of our method over the baseline at f = is relatively modest. However, a much larger advantage revealed on intermediate time, can be seen in Figures 3a and in 33 In both PDEs it is apparent that Limer is an essential ingredient for providing a satisfactory interpolation, and that the required number of intervals, P. not large. This is because Liner enforces time-translation invariance, and therefore smooths the solution.
[ooleB]smmmarize the reconstruction error on the three-dimensional NS equation
1. this
case, we applied our method with P = 1 and ReLU activations for computational reasons, as larger number of intervals required significantly more memory (during training only; during inference requires memory size similar to that of FNO) and GeLU activation cannot be applied "in-place". As can be seen, our method outperforms FNO and performs the best with the Cinter term in this benchmark as well.
DISCUSSION AND OPEN QUESTIONS
Lemma 1 points to practical sources of possible errors in learning the map T.. The first source of emror concerned with the training data set. Quantifying the set of initial conditions and solutions at time T' from which we can learn the map ΦT with required accuracy, depends the complexity of the PDE. Burgers equation is integrable (for regular solutions) and can be analytically solved. In contrast. NS equations not integrable dynamical flows and cannot be solved analytically. Thus, for a given learning accuracy, we expect the data set needed for Burgers equation to be much smaller than the needed for NS equations. Indeed, we numerically the difference in the learning accuracy
7

---

Under review as a conference paper at ICLR 2023
0.12
Bazeline
0.1
p=
Reconstruction Error
P
P
8 9-10-
P 2 w/o Conter
2
Donecpolaciid
Eatrap-latiin
Region
4-10-2
2-10-2
0.5
1.5
2 2.5 3
Figure 2: The reconstruction error on the one-onnenntt one-dimensional Burgers equation for intermediate time predictions and the py-pde solver solutions for different numbers of intervals P the composition loss term, Ceomp- We show the median as well as the 10th, 25th, 75th and 90th percentiles. The statisties is based on 100 samples from the test set. The dashed horizontal cyan line marks the reconstruction error of FNO at t 1 for comparison.
Table The reconstruction emor on the two-dimensional PDEs 1, Burgers and NS equations.
Method
2D Burgers
2D NS
FNO
0.0336
0.4340
FNO+
0.0248
0.4511
Ours
0.0335
0.4302
Ours+
0.0214
0.4163
of these PDEs. Second, there is eиror in the learning of the consistency conditions in equation 9 Third, we impose only limited set partitions equation [3] which leads to an error that scales as I where P is the partition number of the time interval. We decreased this error by applying various such partitions. Note, that the non-convexity of the loss minimization problem implies, as we observe in the network performance, that the second source of error tends to increase when decreasing the third one and vice versa. The difference in the complexity of the NS equations compared to Burgers equation is reflected also in the interpolation performance, which significantly better for the Burgers flows in Figure Ba compared to the NS flows in Figure 3b
Our work opens up several directions for further interesting studies. First, it is straightforward to generalize our framework to other PDEs, such as having higher-order time derivatives, as in relativistic systems where the time derivative is of order two.
Second, we only considered regular solutions in our work. It is known that singular solutions such as shock waves in compressible fluids play an important role in the system's dynamics. The continuation of the solutions passed the singularity is known to be unique in one-dimensional Burgers system. This is not the case general as, instance, for inviscid fluid flows modelled by the incompressible Euler equations. It would be of interest to study how well our network learns such solutions and extends them beyond the singularity.
Third, we learnt the dynamical equations at a fixed grid size of the spatial coordinates. It would be highly valuable if the network could learn to interpolate to smaller grid sizes. FNO allows a super-resolution and hence interpolation to smaller grid size. It does by employing the same physics of the larger seale. Learning the renormalization group itself can open up window to learning physics that appears different scales.
8

---

Under review as a conference paper at ICLR 2023
0.25
p= w/o Cinter Cinter p=1
p- w/o Lister
p= Liwte
Recomsrrction Eroor
Recoratraation Eirorr
p p-2
Began
0.15
Kermpilaties
0.6
0.4
10
0.2
0.5
1.5
0.2
0.4
0.6
0.8
(b)
(a)
Figure 3: The reconstruction error for intermediate time predictions and ground truth solver solutions for different number intervals in the composition loss term, Lcomp. We show the median as well as the 10th, 25th, 75th and 90th percentiles. The statistics is based on 100 samples the test set. The dashed horizontal cyan line marks the reconstruction error of FNO at = 1 for comparison. (a) wwo-dimensional Burgers equation. (b) two-dimensional NS equation.
Table The reconstruction enror on the three-dimensional NS equation 1.
Method
3D NS
FNO
0.2778
FNO+
0.2644
Ours w/o Cinter
0.2675
Ours+ w/o Cinter
0.2672
Ours
0.2504
Fourth, studying the space of solutions and its statistics (in contrast to individual solutions) for PDEs such as NS equations is expected to reveal insights about the universal structure of the physical system. For example, it would be valuable to automatically gain insights into the major unsolved problem of anomalous scaling turbulence.
In our work we considered a decaying turbulence. In order to reach steady-state turbulence and study the statistics the fluid velocity distribution one needs to introduce the network random force pumping energy to the system the rate that of dissipation. On general ground, one expects that for chaotic systems nearby trajectories would diverge exponentially in time, Av(t) = (O))e> This imposes, in general, limit the ability learn single chaotic solution at long times due the emror accumulation. We note, however, that this a limitation of any numerical solver.
CONCLUSIONS
Learning the dynamical evolution of non-linear physical systems governed by PDEs is of much importance, both theoretically and practically. In this work presented network scheme that learns the map from initial conditions to solutions of PDEs. The scheme bridges an important gap between data- and model-driven approaches, by allowing data-driven models provide results for arbitrary times.
Our network based solver, combined with Fourier Neural Operator architecture, propagates the initial conditions, while ensuring the general composition properties the partial differential operators. It improves the learning accuracy at the supervision time-points and interpolates and

---

Under review as a conference paper at ICLR 2023
extrapolates the solutions arbitrary (unlabelled) times. We tested our scheme successfully on non-linear PDEs in one, two and three dimensions.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anima Anandkumar, Kamyar Arizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020. URL https::/ http://ppenneeieeeeeeetrrrmmmmme f2g22mmE-
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021. doi: 10.5802/smaijjem..
URL https: / ssaa jem centre - mersenne org/ http//atiilleeeeeeeeee /10. 10. 58C2/ smal- jem. 74/
Johannes Brandstetter, Daniel E Worrall, and Max Welling. Message passing neural pde solvers. In International Conference Learning Representations, 2021.
Uriel Frisch. Turbulence: the legacy of AN Kolmogorov. Cambridge university press, 1995.
David Ha, Andrew M. Dai, and Quoc V. Le. Hyperettorks. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. httph:tts///oo htttps://openreviert net. /eee/fforumt forum?id=
IkpACel1x
Philipp Holl, Vladlen Koltun, Kiwon Um, Telecom Paris LTCI, IP Paris, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In NeurIPS Workshop, 2020.
George Em Karniadakis, loannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physies-informe machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. CoRR, abs/2108... 084811, 2021. URL htttt::/tt//iiii// org/ abs/ / 21 08481
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances Neural Information Processing Systems, 33:6755 6766, 2020a.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, et al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2020b.
Zongyi Li, Nikola B. Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Markov neural operatons for learning chaotic systems. CoRR, abs/2106.06m 06898, 2021. URL htt.t:tss:///ee htttps::/aax....gg///.oaao
06898
Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1824-1833, 2019.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. CoRR, abs/1910.03193, 2019a.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. 2019b.
Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between banach spaces. SIAM Journal Scientific Computing, (3(5)-33212-A3244 2021.
Ravi G Patel, Nathaniel A Trask, Mitchell A Wood, and Eric C Cyr. A physics- operator regression framework for extracting data-driven continuum models. Computer Methods in Applied Mechanics and Engineering, 373:1 13500, 2021.
11

---

Under review as a conference paper at ICLR 2023
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics- insfrcs- neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 378:686-707, 2019.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning. pp. 1459-8468. PMLR, 2020.
Joseph Suarez. Language modeling with recurrent highway hypernetwrkks. In Advances in neural information processing systems, pp. 3267-3276, 2017.
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for -veeragd navier-stokes simulations airfoil flows. AIAA Journal, 58(1):55-36, 2020.
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual
learning with hypernetworks. In International Conference on Learning Representations, 2020.
URL https: /tpp:s///peeeewiwww net Eorum id SJgwNerKvB
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for
surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018.
David Zwicker. py-pde: A python package for solving partial differential equations. Journal of Open Source Software, 5148) 2020. doi: 10.21 105/joss.02/ 02158. URL wtt.:hhtt////t :/do:s://ddii /ttp:::://oooo///11//0/ 21105/joss 02158
A VISUALIZATION OF SHOCKS
Figure 4 shows the time evolution of the states of two-dimensional Burgers equations. Even though only regular solution appeared in the training set, our approach is able generalize the vicinity of non-regular solutions as can be seen at 1.15, where shocks start to develop.
B RUNTIME ANALYSIS
Let Teno be the time required for applying FNO once. Both Linitall and Limaa require single FNO application, and therefore each is executed in time TeNo- The intermediate term, Liners requires three FNO calls, and therefore requires 3T and Lcomp requires Teno for each of the partitions, totaling in 5TENO Ep pTPNo
4TENo 1 P(P 1)T calls.
C ADDITIONAL ABLATIONS
To assess the contribution of each term the loss function in equation 17 different weight coefficients are added,
CIP Ainitial Cinitial AinterCinter + Asppervision Csupervision Suuperision.Cuuprrriiinnn
(19)
where Lxupervision Lrial Ceomp (Note that Lúnal is the zeroth order of C6P"))) The reconstruction
(PP)
error function is computed on the ID Burgers equation using our method with one partition (P 1). For Asupervision two values
and Asupervision = 1 For the other weight coefficients we use the values 0.1, 1. All the combinations are summarized in Table 4 for As can be seen, while the precise value of the error
used, Asupervision
Aupervision 1 and 5 for Asupervvsion varies, the method is robust with respect to these coefficients, except for the case where X is much smaller than both Asupervision and Ainerr In this case, is unable converge since the initial conditions at = 0 are not learned properly, influencing the hyper-network predictions at = 0 and the whole time evolution.
12

---

Under review as a conference paper at ICLR 2023
E-O
1.15
Grand
Proliction
Grunn tratt
Predlcto
Geoan traths
Vecticity
Protinioo
0-4
RA
Figure 4: The velocity vector fields and the corresponding vorticity of the two-dimensional Burgers equation. Shocks are being formed in the final state.
Table 4: The reconstruction enror the one-dimensional Burgers equation PDEs at using grid size 1024 and Asueervision
Ainitial
Xinter
Err
0.1
0.1
0.0012
0.1
0.5
0.0014
0.1
1.0
1.0000
0.5
0.1
0.0019
0.5
0.5
0.0021
0.5
1.0
0.0014
1.0
0.1
0.0016
1.0
0.5
0.0022
1.0
1.0
0.0015
13

---

Under review as a conference paper at ICLR 2023
Table 5: The reconstruction error on the one-dimensional Burgers equation PDEs at using grid size = 1024 and Asupervision
Annitial
Amner
Err
0.1
0.1
0.0023
0.1
0.5
0.0019
0.1
1.0
0.0019
0.5
0.1
0.0024
0.5
0.5
0.0014
0.5
1.0
0,0013
1.0
0.1
0.0021
1.0
0.5
0.0020
1.0
1.0
0.0018
14


Reviews:
[{"review_id": "aHsqC2BJit", "reviewer": "Reviewer_2qHC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers neural operators in the semisupervised setting. It extends Fourier neural operator with a hypernetwork structure, which allows composition in time. The paper provides numerical experiments on 1d Burger equation and 2d and 3d Navier-Stokes equation. The proposed method has a slightly improved error rate but better interpolation performance in time.", "summary_of_the_review": "Overall, I think the paper provides a concrete study, but its contribution and novelty may not meet the threshold. Therefore, I think this work is marginally below the bar for acceptance.", "strength_and_weaknesses": "Strength:\n- extend operator learning to semisupervised learning\n- new hyper-network architecture\n- continuous interpolation in time\n\nWeakness:\n- learning accuracy at the supervision time-points is not very impressive (vs FNO+)\n- need more justification for a semi-supervised learning setting.\n\nComment:\n- Is this model data efficient? Maybe the proposed method will show an advantage at a smaller amount of data sets.\n- How does the continuous-time learning compare with the standard autoregressive model (t ->t+1, e.g. Markov neural operator)? The autoregressive automatically enforces composition in time. Since the Navier-Stokes is markovian, one probably does not need history. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PRHiIYGF8M", "reviewer": "Reviewer_cCio", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hyper-network to solve partial differential equations. ", "summary_of_the_review": "As stated in the above comments, this paper might not deliver a solid contribution, and the novelty could be limited. ", "strength_and_weaknesses": "The strengths of this paper are:\n\n$\\mathbf{1}.$ The novelty of separating time and space by the hyper-network architecture. \n\n$\\mathbf{2}.$ This paper is clearly written and well organized, with detailed explanation of theories and experiments. \n\nThe weakness of this paper are:\n\n$\\mathbf{1}.$ The proposed approach is a combination of known methods, although it might be novel to use hyper-network.\n\n$\\mathbf{2}.$ The baseline used in the experiments might not be strong enough. For example, why using FNO and/or MGNO in the experiments? Why other network-based methods were not used? Also, is it possible to use some numerical methods as a baseline? The main concern is whether or not the chosen baseline is good and comprehensive to use as references. \n\nOther concerns (minor if compared with the above weakness):\n\n$\\mathbf{1}.$ The literature review claims the model base method, such as (Raissi 2019), does not require training dataset. Physics informed network does require training the network with data, as most of the network-based method do. \n\n$\\mathbf{2}.$  The literature review also claims that the methods referred do not predict intermediate time, which could be false. Even if it is right, there exists methods which can predict solutions at intermediate time. \n\n$\\mathbf{3}.$ The loss function presented in Eq (17) is a simple addition, which could be improved by tuning hyper-parameters as a convex or other types of combinations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NzAxzhaF93", "reviewer": "Reviewer_NVyD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach to modeling PDE solutions using a trainable hyper-network that generates the parameters of an FNO network. The hyper-network gets as an input only the desired continuous time t (>0) and the FNO transforms an initial condition under the dynamics specified by the output of the hyper-network. While the performance of this method at the discrete training time steps already improves on FNO, the main innovation is allowing for interpolated solutions at any continuous time. ", "summary_of_the_review": "The idea of the paper is very interesting, but out of the 21 mentioned references, I didn't find a single one from 2022. This suggests that the authors lack knowledge of the recent literature, which includes e.g. GNNs. This issue and the weak justification of the chosen methods and experiments (see above) are the reason why I believe that this paper is not yet ready for publication.", "strength_and_weaknesses": "The main strength of the paper is the pursuit to learn the underlying dynamics over the continuous time domain. As the authors mention in the discussion section (Sec. 6), this might indeed be very useful as a reduced-order model e.g. of the NSE. \n\nComments:\n- Given that the main comparison baseline is FNO and FNO is also a major building block in this architecture, I would have expected a brief introduction to the used FNO formalism. \n- Where are CNNs and GNNs? Given the recent popularity of GNNs for the solution of PDEs (e.g. https://arxiv.org/abs/2202.03376, https://arxiv.org/abs/2002.09405) as well as CNNs (e.g. https://arxiv.org/abs/1810.08217), one should at least mention these lines of research in the \"Related Work\" section and justify why they are not (or are) relevant here. \n- I see the intention to learn meaningful interpolated solutions of PDEs via the hyper-network, but given the lack of results on extrapolation over longer horizons, e.g. t=10...100, I would speculate that the results are not promising. Any comment on that?  Given the practical relevance of extrapolating over longer times, this aspect should be addressed somewhere. And this regime is something that has already been studied by the so-called direct-time methods, e.g. in this workshop paper by Meyer et al., 2021 using GNNs: https://arxiv.org/abs/2112.10296. I understand that all the PDEs you consider are dissipative, thus too long trajectories wouldn't make sense, but why not look at some inviscid equations (e.g. Burgers, Euler) and discuss e.g. the survival time as done in https://arxiv.org/abs/2202.03376.\n- I like the loss terms defined in the paper quite a lot: they incorporate all possible types of interpolation. However, I know other papers that do similar things (ignoring L_final and L_initial for a second): The construction of L_comp^(P) is very closely related to the \"temporal bundling\" presented here https://arxiv.org/abs/2202.03376, and L_inter reminds me of the \"pushforward training\" from the same paper. I'm sure that there are more analogies from other papers.\n\nTwo minor problems I detected are:\n1. some formatting issues with the equations leading to \"Equation equation [...]\"\n2. \"Miltiple Graph Neural Operator\" -> Multipole Graph Neural Operator\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Md9_sTYpf2s", "reviewer": "Reviewer_rv2Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Fourier Neural Operator (FNO) is a data-based approach for learning an infinite-dimensional mapping, which is helpful for numerically solving PDEs. The paper extends FNO to an evolutional setup. FNO can be used in an evolutional setup too, for which one can learn a mapping that maps the solution at time t to that at time t+dt, and then iterate this mapping. This paper however aims at directly learning the evolutional semigroup $\\Phi(t): \\mathcal{H} \\to \\mathcal{H}$, as a function of both the solution/initial condition and $t$. For this, properties of a semigroup are encouraged by the construction of loss and the time-dependence was accounted for by a specific hyper-network architecture. Interesting results are empirically shown.", "summary_of_the_review": "Overall I feel this is an interesting paper. Simple but good idea, which can be a solid step toward an important problem (ML for PDE solve). I still have some reservation at this moment due to comments made above about the results, but I can be convinced.", "strength_and_weaknesses": "Strength: \nOverall, this is an interesting paper. I especially enjoy the construction of the loss, which prompts what is learned to be like a semigroup.\n\nWeakness: \nI wish more empirical results are provided for a more comprehensive evaluation of the performance. I hope I didn’t read incorrectly, but at this moment I have several questions: \n\n(1) Has the method been compared to iterations of vanilla FNO? That is, FNO for characterizing the evolution map over time-step dt, and then iterated T/dt times for reaching time T? I hope the new results still outperform FNO used in this way, for both small dt and large dt. \n\n(2) What kind of Navier-Stokes problem is being considered? What is the initial condition, what should the exact solution look like, and what does the learned solution look like?\n\n(3) In existing methods like NODE, time dependence was often just used as a part of the input of a neural network. Is it possible to justify, experimentally, why the new hierarchical hyper-network structure is better? Is there any performance/computational-cost trade-off?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2TtIfXF_nR", "reviewer": "Reviewer_ykCW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a hyper-network for FNO’s that allow it to predict intermediate future states. These are used to augment the loss for a better trained model. \n", "summary_of_the_review": "This is an ok paper with a sensible idea, but ultimately I feel this is not enough and nothing really stands out. The method is simple and incremental, and the results are not impressive enough.\n", "strength_and_weaknesses": "\ns: the result analysis is interesting and quite insightsful.\n\ns: the method and loss seem to be well derived, and are sensible.\n\ns/w: The results are not strong enough. The method does roughly hover around the baseline, getting either marginal improvements or similar results.\n\nw: the writing is ok but could be improved. The introduction could do a better job at describing the problem and the issues in earlier works. This paper is quite incremental by focusing largely on FNO, and making it more powerful. The paper could try to position its contributions on the wider context better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject