SineNet: Learning Temporal Dynamics in Time-Dependent Partial Differential Equations
Authors: Xuan Zhang
Jacob Helwig
Yuchao Lin
Yaochen Xie
Cong Fu
Stephan Wojtowytsch
Shuiwang Ji
Venue: ICLR 2024 poster (2024)
Keywords: Partial differential equations
Physics simulation
Dynamics learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Multi-stage U-Net for time-evolving PDEs

Abstract: We consider using deep neural networks to solve time-dependent partial differential equations (PDEs), where multi-scale processing is crucial for modeling complex, time-evolving dynamics. While the U-Net architecture with skip connections is commonly used by prior studies to enable multi-scale processing, our analysis shows that the need for features to evolve across layers results in temporally misaligned features in skip connections, which limits the model’s performance. To address this limitation, we propose SineNet, consisting of multiple sequentially connected U-shaped network blocks, referred to as waves. In SineNet, high-resolution features are evolved progressively through multiple stages, thereby reducing the amount of misalignment within each stage. We furthermore analyze the role of skip connections in enabling both parallel and sequential processing of multi-scale information. Our method is rigorously tested on multiple PDE datasets, including the Navier-Stokes equations and shallow water equations, showcasing the advantages of our proposed approach over conventional U-Nets with a comparable parameter budget. We further demonstrate that increasing the number of waves in SineNet while maintaining the same number of parameters leads to a monotonically improved performance. The results highlight the effectiveness of SineNet and the potential of our approach in advancing the state-of-the-art in neural PDE solver design. Our code is available as part of AIRS (https://github.com/divelab/AIRS).

Published as a conference paper at ICLR 2024
SINENET: LEARNING TEMPORAL DYNAMICS IN TIME- DEPENDENT PARTIAL DIFFERENTIAL EQUATIONS
Xuan Zhang': Helwig'; Yuchao Lin', Yaochen Xie', Cong Fu', Stephan Wojtowytsch² & Shuiwang Ji'
'Department of Computer Science & Engineering, Texas A&M University DDeparmeent of Mathematics, University of Pittsburgh
ABSTRACT
We consider using deep neural networks to solve dime-dependent partial differential equations (PDEs), where multi-scale processing is crucial for modeling complex, time-evolving dynamics. While the U-Net architecture with skip connections is commonly used by prior studies to enable multi-scale processing, our analysis shows that the need for features to evolve across layers results in temporally misaligned features in skip connections, which limits the model's performance. To address this limitation, we propose SineNet, consisting of multiple sequentially connected U-shaped network blocks, referred to as waves. In SineNet, high- resolution features are evolved progressively through multiple stages, thereby reducing the amount of misalignment within each stage. We furthermore analyze the role of skip connections in enabling both parallel and sequential processing of multi-scale information. Our method is rigorously tested on multiple PDE datasets, including the Navier-Stokes equations and shallow water equations, showcasing the advantages of our proposed approach over conventional U-Nets with comparable parameter budget. We further demonstrate that increasing the number of waves SineNet while maintaining the same number of parameters leads to monotonically improved performance. The results highlight the effectiveness of SineNet and the potential of our approach in advancing the state-of-the-art in neural PDE solver design. Our code is available as part of AIRS (htpss//gitub..ommddilla//aaili
INTRODUCTION
Partial differential equations (PDEs) describe physics from the quantum length scale (Schrödinger equation) to the scale of space-time (Einstein field equations) and everywhere in between. Formally, a PDE is an expression in which the rates of change of one or multiple quantities with respect to spatial or temporal variation are balanced. PDEs are popular modeling tools across many scientific and engineering fields to encode force balances, including stresses that are combinations of spatial derivatives, velocity and acceleration that time derivatives, and external forces acting fluid solid. Due to their widespread use in many fields, solving PDEs using numerical methods has been studied extensively. With advances deep learning methods, there has been a recent surge of interest
in using deep learning methods for solving PDEs (Raissi et al. 2019 Kochkov et al. 2021 Lu et al.
2021 Stachenfeld et al., 2022 Li al. 2021a Pfaff et al.] 2021 Takamoto et al.
2022 Gupta &
Brandstetter 2023). ), which is the subject of this work.
Our main focus here on fluid dynamics, where we typically encounter two phenomena: advection
and diffusion. Diffusion in isotropic media is modeled using the heat equation (a, A ))u= 0, which equates the rate of change time 0, u to the spatial Laplacian of the function u. an expression of sccond-order derivatives. The fact that first-order derivative in time is compared to second derivative in space leads to the parabolic scaling in which / and x behave comparably. In many numerical methods such as finite element discretizations, this behavior requires discretizations for which the length-scale in time òt is much smaller than the spatial length-scale da: ôt & (dzr)², To avoid excessive computation time when computing in fine resolution, multi-scale models are attractive
* Equul contribution. Correspondence to: Xuan Zhang <xuan.zhang @ tamu. tamm.aduu. Jacob Helwig <jacoba..hlligge edm and Shuiwang Ji <sji @tamu.edu.

---

Published as a conference paper at ICLR 2024
which accurately resolve both local interfaces and long-range interactions over larger time interval per computation step. Methods which are not localized in space, e.g., heat kernel convolutions or spectral methods, are able to overcome the parabolic scaling.
Unsurprisingly, the situation becomes much more challenging when advection is considered. While diffusion models how heavily localized quantities spread out to larger regions over time, advection describes the transport of quantities throughout space without spreading out. In the equations of fluid dynamics, the transport term (u V)u non-linear in u and famously challenging both analytically and numerically. In particular, any numerical approximation to the solution operator which propagates forward in time must be non-linear to comply with the non- non-linearity in the equation.
For its strength in multi-scale processing. the U-Net architecture has recently become a popular choice as an inexpensive surrogate model for classical numerical methods mapping dynamics forward in time (Gupta & Brandstetter 2023 Takamoto et al. 2022 Wang et al. 2020). However, because the input to the network is earlier time than the prediction target, latent evolution occurs in the
U-Net feature maps. This implies that skip connections between the down and upsampling paths of the U-Net, which are crucial for faithfully restoring high-resolution details to the output, will contain out-of-date information, and poses challenges in settings where advection plays significant role.
In this work, we propose SineNet, an architecture designed to handle the challenges arising from dynamics wherein both diffusion and advection interact. SineNet features multi-stage architecture in which each stage is a U-Net block, referred to as a wave. By distributing the latent evolution across multiple stages, the degree of spatial misalignment between the input and target encountered by each wave is reduced, thereby alleviating the challenges associated with modeling advection. At the same time, the multi-scale processing strategy employed by each wave effectively models multi-scale phenomena arising from diffusion. Based on the proposed SineNet, we further analyze the role of skip connections in enabling multi-scale processing in both parallel and sequential manners. Finally, we demonstrate the importance of selecting an appropriate padding strategy to encode boundary conditions in convolution layers, a consideration we find particularly relevant for periodic boundaries. We conduct empirical evaluations of SineNet across multiple challenging PDE datasets, demonstrating consistent performance improvements over existing baseline methods. We perform an ablation study in which we demonstrate performance monotonically improving with number of waves for a fixed parameter budget. The results highlight the potential of our approach in advancing the state-of-tte oftte-of-the-ar
the-art art in neural PDE solver design and open new avenues for future research in this field.
NEURAL APPROXIMATIONS TO PDE SOLUTION OPERATORS
Time-evolving partial differential equations describe the behavior of physical systems using partial derivatives of an unknown multivariate function of space-time u Ω x R + RMM where Q is the spatial domain and It accounts for the temporal dimension. The A -dimennsional codomain of u consists of scalar and/or vector fields such as density, pressure, or velocity describing the state of the system. Given the solution ul.,tu at time fg. our task is to learn the forward operator mapping u(- to) to the solution at later time f given by u(-, t). In practice, u is numerical solution to the PDE, that is, discretized in space-time onto a finite grid. As opposed to neural operators (Kovachki et al. (021), which aim to learn this operator independent of the resolution of the discretization, we instead focus on learning the operator for a fixed, uniform discretization. Concretely, for d-dimensional spatial domain Ω, the discretized solution at time step t is given by u E TR. M X X for t t= 1, 1,..., T, where T is the temporal resolution and Ni,..., Na are the spatial resolutions along each dimension of Ω (d = 2 in our experiments). Moreover, is typical in numerical solvers, we condition our models on h historical time steps. Formally, we seek to learn the mapping M M N. M XXXxX.xXX X with M(U:-h+15 Uz) = Uc+1. for L = h....,--
3 LEARNING TEMPORAL DYNAMICS IN CONTINUUM FIELDS
3.1 MULTI-SCALE LEARNING AND U-NETS
In Fourier space, low frequency modes correspond to global information, and high frequency modes
to local. Convolutions can be performed in the frequency domain via pointwise multiplication of
2

---

Published as a conference paper at ICLR 2024
Cal
t-0
Upsample
Figure I: Illustration of the misalignment issue in U-Nets. The left side shows the one-step prediction of trained U-Net, with each intermediate layer's feature maps averaged over the channel dimension displayed in the top row. On the right side, the misalignment between é = 0 and = 3 is demonstrated. is upsampled and then concatenated with é = 0. The time-
Specifically, the feature map of evolving effect of the preceding U-Net layers results in misalignment of the corresponding physical features, as indicated by the bounding boxes (a) and (b). This misalignment is particularly problematic for convolutions. Since the kernel is localized, information from misaligned high frequency features, such as those visualized in (a) and (b), cannot be optimally integrated in updating the feature map. Mitigating this misalignment key for improving the performance of U-Nets.
frequency modes, resulting in parallel processing of multi-scale information (Gupaa & Brandstetter
2023). This approach is taken by Fourier Neural Operators (FNOs) (Li al. [2021a] Tran al.
2023 Li et al. 2022 Rahman et al., [2022] Helwig et al. 2023). which directly parameterize their
convolution kernel the frequency domain.
In the spatial domain, U-Nets (Ronneberger et al. 2015) are commonly used to process multi-scale information in sequential manner. U-Nets extract information on various scales hierarchically by performing convolutions on increasingly downsampled feature maps before invoking a symmetric upsampling path. Skip connections between corresponding downsampling and upsampling layers are established to restore high-resolution information. To map fug-h+1,-.. uz) to u441, the U-Net encodes inputs Bg and then processes Zo sequence of I. feature maps along a downsampling path 11, ................................
....... 1L
To = P((u- 4));; æt Se (d (xx--1) (= I,....,,
where P is an encoder, d is a downsampling function, and fe composes convolutions and non- linearities. Next, the U-Net upsamples to a sequence feature maps 1r.41..... 225. along the upsampling path and decodes the final feature map Ø21, to u++1
De = (w (Z -2) e = I. +
21; 4 = Q (#22),
where Q is the decoder, v is an upsampling function, and ga concatenates the upsampled feature map with the skip connected downsampled feature map before applying convolutions and non-linearities.
3.2 MISALIGNMENTS IN LEARNING TEMPORAL DYNAMICS
U-Nets capture global context by aggregating localized features with downsampling layers. Using this global context, the output is constructed with upsampling layers, wherein skip connections to the downsampled feature maps play a key role in restoring high-resolution details. While we consider their application neural solvers, U-Nets were originally developed for image segmentation (Ronneberger ct al., 2015). segmentation tasks, features in the input image and target segmentation mask are spatially aligned. By contrast, in the context of learning temporal dynamics, the spatial localization property no longer holds, as temporal evolution results in misalignment between the input and prediction target. Here, misalignment refers to the displacement of local patterns between two feature maps, which occurs between the PDE solutions at two consecutive time steps due to advection. To resolve this misalignment, feature maps undergo latent evolution. However, due the local
3

---

Published as a conference paper at ICLR 2024
L+1
P
Skip Connection
2/
V
Figure 2: Illustration of the proposed SineNet for learning temporal dynamics in PDEs. Multiple U-Net waves are composed to perform one-step prediction, with the output of each wave averaged over the channel dimension displayed in the top row, demonstrating the time-evolving process from to tot+1, 1. The orange sinusoidal line illustrates propagation between resolutions and is not part of the model architecture. Feature maps in this figure are from SINENET without wave residuals for clarity and with transposed convolutions for upsampling. In Appendix A.2 we visualize the feature maps from the SINENET-8 & presented in Section 5.3
processing of convolution layers, feature maps closer to the input (or output) will be more aligned with the input (or output), respectively. This leads to misalignment between the feature maps propagated via skip connections and those obtained from the upsampling path.
To empirically demonstrate this inconsistency, we visualize the feature maps averaged over the channel dimension at each block within well-trained U-Net in Figure I Specifically, we examine the feature map from the skip connection at the highest resolution (E 0) and compare it with the corresponding one from the upsampling path (6 3). We observe misaligned features between e and the upsampled & = 3, as highlighted by the fcatures contained in the bounding boxes (a) and (b). As kernels are most often local, e.g., x 3, following the concatenation, convolution operations will be either partially or fully unable to incorporate the evolved features with the corresponding features in skip connections as a result of the misalignment.
3.3 MULTI-STAGE MODELING OF TEMPORAL DYNAMICS WITH SINENET
Despite the inconsistency, skip connections in U-Nets remain crucial to restoring high-frequency details and thereby enable multi-scale computation via downsampling and upsampling. To mitigate the misalignment issue while maintaining the benefits of the U-Net architecture, we construct the SINENET- K architecture by partitioning the latent evolution across K lightweight U-Nets Vs, referred to as waves. This approach reduces the latent evolution between consecutive downsampling and upsampling paths, which in turn reduces the degree of misalignment in skip connections. Each wave V& responsible for advancing the system's latent state by an interval of & E [0. 1), with EisEek dR = 1. Specifically, V&3 'ss objective is evolve the latent solution from time t + AR-1, achieved through the temporal evolution by the preceding h 1 waves, to time t + A,, where A represents the cumulative sum all 8, up to the k-th interval. The overall mapping is then given by
Φt P (f (-lu-b+1..... u.))) U.+1 = Q (x++1);
T++d. Vi(z++Ar-),, k= 1,...,,,
(4)
where Fz+AA, denotes the latent map of U++d at time f + As. and P and Q are 3x3 convolution layers to linearly encode and decode the solutions into or from latent maps, respectively. During training, the intermediate sub-steps as determined by 6% are implicit and optimized end-to-end with the K waves using pairs of data ((1u) ((1u--t1--.. ut) ut 1., to minimize the objective
Eie [<(Wxe(trt) (WWx((u/-A , u ) Wx = Q Vo.o....oioo
(5)
4

---

Published as a conference paper at ICLR 2024
where L is suitably chosen loss and j indexes the solutions in the training set. is worth noting that the intervals 65 are not necessarily evenly divided. They can be input-dependent and optimized for superior performance during training to enable adaptable temporal resolution in the latent evolution. For example, when the temporal evolution involves acceleration, it is generally more effective to use smaller time steps for temporal windows with larger velocities. We demonstrate connection between this formulation and Neural ODE (Chen et al. 2018) in Appendix F.2
The proposed approach effectively reduces the time interval each wave is tasked with managing, thereby simplifying the learning task and mitigating the extent of misalignment in skip connections. We empirically demonstrate such effect in Appendix A..I] This strategy not only fosters a more efficient learning process but also significantly enhances the model's ' ability to handle complex, time-evolving PDEs. Through this approach, we present robust way to reconcile the essential role of skip connections with the challenges they pose in the context of PDEs.
3.4 WAVE ARCHITECTURE
Here we discuss the construction of each of the K waves V2 in SineNet mapping the latent solution Tt+A,-1 to T in Equation 4. V& is implemented as conventional U-Net Uk. as described in Equations 1 andd with a wave residual as
Vi (#I+A...) 1t+ + Ux(
(6)
Each wave is constructed with a downsampling and upsampling path, both of length L = 4, i.e. 22 = fe (d (x L 2,3, 4} for downsampling and æy = ge (u (xx-1) xz-()) E E (5, 7.8} for upsampling. To maintain light-weight architecture, the downsampling and upsampling func- tions d and e are chosen as average pooling and bicubic interpolation. f is constructed with a convolution block ch, that is, two 3 convolutions with layer norm (Ba et al. 2016) and GeLU activation (Hendrycks & Gimpell 2016). with the first convolution increasing the number of channels in the feature map by a multiplier mx. which is chosen dependent on the number of waves K the SineNet such that the number of parameters is roughly constant in K. of is constructed similarly, except the first convolution decreases the number of channels by factor of mg. Both f and ge additionally include a block residual, whose role we discuss further in Section 3.5 as
17 (xx) = w (d (a:) + et (d (x:)
gt (v y)) =w (v (x:) &e (cat (u (x),
(8)
where wp is point-wise linear transformation acting to increase or decrease the number of channels in the residual connection.
3.5 DUAL MULTI-SCALE PROCESSING MECHANISMS
While in Equation 6 wave residuals serve to improve the optimization of SineNet (He et al. 2016), the block residuals utilized in Equation 7 additionally allow each wave V/2 to process scale information following both the sequential and parallel paradigms analyzed by Gupta & Brandstetter (2023). Parallel processing mechanisms, of which the Fourier convolutions utilized by FNOs (Li et al.) (021a) are an example, process features directly from the input various spatial scales independently of one another. In contrast, under the sequential paradigm inherent to U-Nets, features at each scale are extracted from the features extracted at the previous scale. Because this approach does not include a direct path to the input features, processing capacity at a given scale may be partially devoted toward maintaining information from the input for processing at later scales rather than extracting or evolving features.
To improve the efficiency of the sequential paradigm, SineNet utilizes block residuals as described in Equations 7 and 8 which we now show allows both parallel and sequential branches of processing. Consider the input Zo. a linear downsampling function d such as average pooling, and convolution blocks C12 C22 C3. all of which are composed as the downsampling path of a U-Net, that is, downsam- pling followed by convolution blocks. Assume for simplicity that the input and output channels for each of the convolution blocks equal and w is omitted. Then, by including block residuals, the feature map following the first downsampling and convolution block is given by
. d (2o) + ca (d (ma))
5

---

Published as a conference paper at ICLR 2024
and thus, the imput to og is given by
d (m)) = de (2r) + d (ci (d (2x))),
(10)
where de (zn) == d (d (2r)) represents the direct flow of information from the input in the parallel branch, enabling 02 process information following both the parallel and sequential paradigms. From the input to given by
d (x2) = di (xx) + d dd (c1 (d + 02 (x1))),
(11)
can be seen that this dual-branch framework is maintained later layers. Specifically, the input to the k-th layer will be comprised of zo downsampled & times summed with a feature map processed sequentially. However, since addition will entangle the parallel and sequential branch, we propose to disentangle the parallel branch by concatenating it to the input of each convolution block in the downsampling path. Therefore, the downsampling path in SineNet replaces Equation with
f (x)) = w (x))) + e (cat (d (x), de (zo))),
(12)
where d (ara) is the result of the projection layer P downsampled & times. We note that although latent evolution in the sequential branch will result in misalignment with the parallel branch, we empirically observe a performance gain by including the parallel branch, particularly once it has been disentangled. Additionally, this inconsistency between branches is mitigated by the multi-stage processing strategy adopted by SineNet.
3.6 ENCODING BOUNDARY CONDITIONS
The boundary conditions of a PDE determine the behavior of the field along the boundary. Our experiments with the incompressible Navier-Stokes equations use a Dirichlet boundary condition on the velocity field such that the velocity on the boundary zero, and Neumann boundary condition on the scalar particle concentration such that the spatial derivative along the boundary is zero. These conditions are encoded in feature maps using standard zero padding.
The remaining PDEs we consider have periodic boundary conditions, wherein points on opposite boundaries are identified with one another such that the field "wraps around". For example, for function u on the unit torus, that is, periodic function with domain [0, 1]2, we have that for u11, For such boundary condition, we found
E (0, 1). u( 0) uur, 1) and (0. circular padding to be a simple yet crucial component for achieving optimal performance (Dresdner ct al. 2023), as otherwise, a great deal of model capacity will be spent towards sharing information between two boundary points that appear spatially distant, but are actually immediately adjacent due to periodicity. As Fourier convolutions implicitly assume periodicity, FNO-type architectures are are ideally suited for such PDEs.
RELATED WORK
Neural PDE solvers. Many recent studies explore solving PDEs using neural networks and are often
applied to solve time-dependent PDEs (Poli et al.) 2022 Lienn & Günnemann, (022). Physics-
informed neural networks (PINNs) (Raissi al. 2019 Wang et al. 2021] Li et al. (02 lb) and
hybrid solvers (Um et al. 2020 Kochkov et al. 2021 Holl et al. 2020b) share similar philosophies with classical solvers, where the former directly minimizes PDE objectives and the latter learns to improve the accuracy of classical solvers. On the other hand, many works focus on purely data- driven learning of mappings between PDE solutions, which the approach we take here. |Wang al.) (2020) and (Stachenfeld et al. (2022) apply their convolutional architectures to modeling turbulent flows in spatial dimensions, while Lienen et al.) (2023) use diffusion models to simulate 3-dimensional turbulence. Li ct al. (2021a) developed the FNO architecture which has been the subject of several follow-up works (Poli et al. 2022 Tran et al. 2023 Helwig et al. 2023}. Similar to SineNet, PDE-Refiner (Lippe et al. (023) solves PDES via iterative application of a U-Net, however, instead of advancing time with each forward pass, U-Net applications after the first serve to refine the prediction.
Stacked U-Nets. There are other studies exploring stacked U-Nets, however, most focus computer vision tasks such as image segmentation (Xia & Kulis 2017 Shah et al., 2018 Zhuang 20188) Fu et al. 2019) and human pose estimation (Newell et al. 2016). There are also researchers exploring

---

Published as a conference paper at ICLR 2024
U-Nets and variants therein to model PDE solutions. Chen al. (2019) apply variations of U-Nets, including stacked U-Nets, to predict the steady state of a fluid flow, although they do not study temporal evolution and furthermore do not consider stack greater than 2. Raonić et al. (2023) propose to use CNNs to solve PDEs, but they do not have the repeated downsampling and upsampling wave structure. We differ from these works by making the key observation of feature evolution and feature alignment.
EXPERIMENTS
As fluid dynamics are descrihed by time-evolving PDEs where diffusion and advection play major roles, perform experiments on multiple fluid dynamics datasets derived from various forms of the Navier-Stokes equation. We begin by describing our experimental setup, datasets and models in Sections 5.1 5.2 before presenting our primary results in Section [5.3 We close with an abalation study in Section S.4 Furthermore, in Appendix A..]] we conduct an experiment to validate our claim of reduced latent evolution per wave, and demonstrate that this results in reduced misalignment in skip connections in Appendix A.2
5.1 SETUP AND DATASETS
where the inputs are the fields at current and historical steps and the target is the field at the next time step. During validation and testing, models perform an autoregressive rollout for several time steps into the future. Scaled L2 loss (uupta & Brandstetter, 2023 Tran et al. 2023 Li et al. 202 is used as training loss and evaluation metric, which we describe in Appendix D.2 rollout test errors.
In all experiments, models are trained with data pairs ((u.-++1.....
u(+1) .. ut),
We report both the one-step and
We consider three datasets in our experiments. Each dataset consists of numerical solutions for given time-evolving PDE in two spatial dimensions with randomly sampled initial conditions. Detailed dataset descriptions can be found in Appendix B.2
Incompressible Navier-Stokes (INS). The incompressible Navier-Stokes equations model the flow of a fluid wherein the density is assumed to independent of the pressure, but may not be constant due to properties of the fluid such salinity temperature (Vreugdenhil [994). The equations given by
dv 0t
=
Vo 2V Vp V ==
(13)
where v is velocity, is internal pressure, and f is an external force. We use the dataset from Gupta Brandstetter (2023), simulated with numerical solver from the & package (Holl et al. 2020a).
In Appendix F.1 we present results on the conditional version of this dataset considered by Gupta & Brandstetter 2023), where the task to generalize over different time step sizes and forcing terms.
Compressible Navier-Stokes (CNS). Compressibility is generally consideration most relevant for fast-moving fluids (Anderson, 2017). We generate our CNS dataset using the numerical solver from Takamoto et al. (022). The dynamics in this data are more turbulent than those in the INS data, with the viscosity as 10 and the initial Mach number, which quantifies the ratio of the flow velocity to the speed of sound in the fluid AAdersson] 2017). as 0.1.
Shallow Water Equations (SWE). The shallow water equations are derived by depth-integrating the incompressible favier-Stokes equations and find applications in modeling atmospheric flows (Vreug denhil 1994). We use the dataset from Gupta & Brandstetter (2023) for modeling the velocity and pressure fields for global atmospheric winds with periodic boundary condition.
5.2 MODELS
Here, we overview the models used in our experiments, and provide further details in Appendix D
SINENET-8/16. SineNet with 8 16 waves and 64 channels at the highest resolution. Multiplier m, = 1.425/1.2435 is used, resulting in channels along the downsampling path of each wave being arranged as (64, 91, 129, 185, 263) (64, 79, 98, 123, 153).

---

Published as a conference paper at ICLR 2024
F-FNO. Fourier Neural Operators (Li et al. convolutions in the frequency domain (Gupta & Brandstetter 2023), and were originally developed for PDE modeling. As one of the primary components of SineNet depth, compare state- of-the-art FNO variant, the Factorized FNO (F-FNO) (Tran et al. 2023), optimized specifically for depth. In each of the 24 layers, the number of Fourier modes used is 32 and the number of channels is 96.
a) process multi-scale features by performing
DIL-RESNET. As opposed downsampling and upsampling, the dilated ResNet (DIL-RESNET) proposed by Stachenfeld et al. (2022) is neural PDE solver that processes multi-scale features by composing blocks of convolution layers with sequentially increasing dilation rates followed by sequentially decreasing dilation rates (Zhang et al. (202)).
U-NET-128 and U-NET-MOD. U-NET-128 is equivalent SINENET-1 with the multiplier mk = 2 and 128 channels at the highest resolution, with channels along the downsampling path arranged (128, 256, 512, 1024, 2048). U-NET-MOD modern version of the U-Net architecture which parameterizes downsampling using strided convolutions and upsampling using transposed convolutions. Additionally, doubles number of convolutions and skip connections per resolution.
For convolution layers in SINENET, DIS.-RENNET, U-NET-128 and U-NET-MOD, zero padding is used on the INS dataset and circular padding is used on CNS and SWE.
5.3 RESULTS
We present results for INS, CNS, and SWE in Table 1 On all datasets, SINENET-8 has the lowest 1-step and rollout errors. Out of the remaining baselines, F-FNO has the strongest performance in terms of rollout error on CNS and SWE, while DIL-RESNE has the best baseline rollout error INS. visualize SINENET-8 predictions on each dataset Appendix G
Table 1:
Summary of rollout test error and one-step test error. The best performance is shown in bold and the second best underlined.
INS
CNS
SWE
METHOD
PAR. (M)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
I-STEP (%)
ROLLOUT (%)
SINENET-8
35.5
1.66
19.25
11.3
2.10
1.02
1.78
SINENET-16
35.4
1.46
17.63
0.87
2.04
0.92
1.64
F-FNO
30.1
2.41
22.58
1.46
2.91
1.22
2.46
DIL-RESNET
16.7
1.72
19.29
1.17
3.76
2.23
4.12
U-NET-128
135.1
2.69
24.94
1.62
3.05
1.63
3.38
U-NET-MOD
144.3
2.43
23.65
1.32
3.34
1.60
3.02
5.4
ABLATION STUDY
Table 2: Ablation study results. The best performance is shown in bold.
INS
CNS
SWE
METHOD
# PAR. (M)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
SINENETS. ENTANGLED
32.8
1.69
19.46
1.01
2.39
1.14
1.97
U-NET-8
We construct several models to ablate various components of SineNet and present results for these models in 2
DEEPER U-NET-8, Conventional U-Net, but increases the number of convolutions per resolution along the down and upsampling paths by factor of relative SINENET-8, resulting in an architec- ture with similar number of layers to SINENET-8, as visualized in Figure in Appendix [ This serves to ablate SineNet's multi-stage processing strategy as means of resolving the misalignment issue, and furthermore validate that observed improvements of SineNet are not mere result of increased depth. Table 2 The largest improvements
we see that on all datasets, SINENET-& outperforms DEEPER U-NET-8.
the SWE data, where the time interval between consecutive solutions 48 hours. While the large timestep results substantial misalignment in the DEEPER U-NET skip connections, the multi-stage strategy of SINENET handles it effectively.
8

---

Published as a conference paper at ICLR 2024
INS
CNS
SWE
33.0
Rolout 1-Step
Rollou
2:50
Rollout
Step
2.25
S-Step
21.5
1.00
21.0
2.00
20.0
3.25
1.75
Brror
18.5
2.50
1.50
17.5
1.25
2.5
1.00
1.00
14
10
Figure 3: Number of waves K versus test error. Solid line is rollout error, dashed line is 1-step error. Numerical results are presented in Appendix F.8
SINENET-8-EE --ENT---ETTA SINENET-8, but the parallel and sequential processing branches entangled, i.., the downsampling path is constructed as in Equation as opposed to the disentangled downsampling in Equation [T] In Table we observe that disentangling the parallel branch results in consistent performance gains.
SINENET-K. SINENET with waves, and the channel multiplier mı chosen such that the number of parameters is roughly constant across all models, which we discuss further in Appendix [D.3] In Figure 3 we present results for K = 2, 4, 6, 8, 10, 12, 14, 16 and find that on 3 datasets, errors monotonically improve with K, although improvements appear to plateau around K 16. This result is consistent with our assumption of the latent evolution of features, as a greater number of waves leads to smaller amount of evolution managed by each wave, thereby leading to improved modeling accuracy through reduced misalignment. In Appendix C we present the inference time and space requirements for SINENET-K.
Effect of circular padding, In Table 3 we replace circular padding with zero padding in SINENET-8 on SWE. The periodic boundaries result in boundary points opposite each other being spatially close, how- ever, without appropriate padding to encode this into feature maps, the ability model advection across boundaries is severely limited.
Table 3: Comparison between zero and circu- lar padding on SWE with SINENET-8.
PADDING
PADDING
1-STRP (%6
ROLLOUT (%)
1-STEP (%)
ROLLOUUT (%)
1.50
4.19
1.02
1,78
DISCUSSION
We discuss potential limitations in Appendix E including out-of-distribution (OOD) generalization, applicability to PDEs with less temporal evolution, computational cost, and handling irregular spatiotemporal grids. Additionally, while SineNet is CNN and therefore cannot directly generalize to discretizations differing from that in the training set, inference at increased spatial resolution can be achieved through dilated network operations or interpolations without any re-training, which we discuss and experiment with further in Appendix F3
CONCLUSION
In this work, we have presented SineNet. a neural PDE solver designed to evolve temporal dynamics arising in time-dependent PDEs. We have identified and addressed a key challenge in temporal dynamics modeling, the misalignment issue, which results a performance drop for conventional U- Nets. By reframing the U-Net architecture into multiple waves, SineNet mitigates this misalignment, leading to consistent improvements in solution quality over a range challenging PDEs. Further, we analyze the role of skip connections in enabling both parallel and sequential processing of multi-scale information. Additionally, we demonstrate that increasing the number waves, while keeping the number of parameters constant, consistently improves the performance of SineNet. Our empirical evaluation across multiple challenging PDE datasets highlights the effectiveness SineNet and its superiority over existing baselines.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by National Science Foundation grants IIS-2243850 and IIS IIS-200686.
REFERENCES
Dallas Albritton, Elia Brué, and Maria Colombo. Non-uniqueness of leray solutions of the forced navier-stokes equations. Annals of Mathematics, 196(1):415 455, 2022.
John Anderson. Fundamentals of Aerodynamics. McGraw- Hilla, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16X1v:1677006400 2016.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations, 2022. URL https///peemeeiiw..e///oorrn
id=wSix3PYKKSU
Junfeng Chen, Jonathan Viquerat, and Elie Hachem. U-net architectures for fast prediction in fluid mechanics. 2019.
Ricky T. Q. Chen. torchdiffeq, 2018 URL http:://ggthhb..comm/nccc//////t
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Peter Constantin and Ciprian Foias. Navier-stokes equations. University of Chicago Press, 2020.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision,
pp. 764-773, 2017.
Gideon Dresdner, Dmitrii Kochkov, Peter Christian Norgaard, Leonardo Zepeda-Nunez, Jamie Smith, Michael Brenner, and Stephan Hoyer. Learning to correct spectral methods for simulating turbulent flows. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL
https://ppnnevie..n///oouuu hiw.:/oouummiddddeBRB..........
Jun Fu, Jing Liu, Yuhang Wang, Jin Zhou, Changyong Wang, and Hanqing Lu. Stacked deconvolu- tional network for semantic segmentation. IEEE Transactions on Image Processing, 2019.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning, pp. 2083 2092. PMLR, 2019.
Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporllscccl generalized PDE modeling. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: //ppameiieonee/ommmntEESSSSSTS
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Jacob Helwig, Xuan Zhang, Cong Fu, Jerry Kurtin, Stephan Wojtowytsch, and Shuiwang Ji. Group equivariant Fourier neural operators for partial differential equations. Proceedings of the 40th International Conference Machine Learning, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 106.11606.0.5 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 53:6840-6851. 2020.
Philipp Holl, Vladlen Koltun, Kiwon Um, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In NeurIPS workshop, volume 2, 2020a.
10

---

Published as a conference paper at ICLR 2024
Philipp Holl, Nils Thuerey, and Vladlen Koltun. Learning to control pdes with differentiable physies. In International Conference on Learning Representations, 2020b. URL hutpss//openreview.neel Corummidd-HeSinnPP.e
Masanobu Horie and Naoto Mitsume. Physics-embedded neural networks: Graph neural pde solvers with mixed boundary conditions. Advances in Neural Information Processing Systems, 35:2321 88- 23229, 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. 2015.
Matthieu Kirchmeyer; Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In International Conference on Machine Learning, pp. 11283-11301. PMLR, 2022.
Milan Klöwer, Tom Kimpson, Alistair White, and Mosè Giordano. milank//speedyweather jl: vO. 2.1. Version vO, 2, 2022.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine learning-accelerannn accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21)c2101184111 2021.
Nikola Kovachki, Zongyi Li. Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2 108.0881,, 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 (11111))2278222222 2324,, 1998.
Zongyi Li, Nikola Kovachki, Kamyar Arizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755-6766, 2020.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Arizzadenesheli, Burigede Liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021a. URL
https://openeviie.e//////um//ii/ http://rrummiddddoooooo//VVoom
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-infe informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03777 2021b.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.0522 0000299 2022.
Marten Lienen and Stephan Günnemann. Learning the dynamics of physical systems from sparse ob- servations with finite element networks. International Conference on Learning Representations, 2022. URL https://openneevee.e///ffommmmm wat.://rmmmmmd---HmAaAAuu-u-
Marten Lienen, Jan Mansen-Palmus David Lüdke, and Stephan Günnemann. Generative diffusion for 3d turbulent flows. arXiv preprint 2rx6v:2306.01776, 2023.
Phillip Lippe, Bastiaan S Veeling, Paris Perdikaris, Richard E Turner, and Johannes Brandstetter. Pde- refiner: Achieving accurate long rollouts with neural pde solvers. arXiv preprint (3XY::308..0532. 2023.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DecpONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-229, 2021.
S Chandra Mouli, Muhammad Alam, and Bruno Ribeiro. Metaphysica: Ood robustness in physics- informed machine learning. In ICLR 2023 Workshop Physics for Machine Learning, 2023.

---

Published as a conference paper at ICLR 2024
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Computer Vision-ECCO 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIIII 14, pp. 483-499 499. Springer, 2016.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8177 8171. PMLR, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzule and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021. XP
URL http:://ppnneeeevv..e///o/uuultd
Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Ré, and Stefano Ermon. Transform once: Efficient operator learning in frequency domain. Advances in Neural Information Processing Systems, 35:7947-7959 2022.
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli. U-no: U-shaped neural operators. arXiv preprint arXiv:2204. ///2,, 2022.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-7017 707, 2019.
Bogdan Raonić, Roberto Molinaro, Tobias Rohner, Siddhartha Mishra, and Emmanuel de Bezenac. Convolutional neural operators. arXiv preprint arXiv:2302.0/178 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015.
Alvaro Sanchez-
Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning simulate complex physics with graph networks. In International conference on machine learning, pp. 8459-8468 PMLR, 2020.
Sohil Atul Shah, Pallabi Ghosh, Larry S. Davis, and Tom Goldstein. Stacked u-nets:a no-frills approach to natural image segmentation. arXiv: 10343, 2018.
Kim Stachenfeld, Drummond Buschman Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Conference Learning coarse models for efficient turbulence simulatation. In International Representations, 2022. URL https::/openneviieweefffoff http://www....eeeoooooocccconoo. Bod=mssBooTZ-Nhh
Learned
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1611, 2022.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference Learning Representations, 2023. URL
Kiwon Um, Robert Brand, Yun Raymond Fei, Philipp Holl, and Nils Thuerey. Solver-in-ther loop: Learning from differentiable physics to interact with iterative poe-solvers. Advances in Neural Information Processing Systems, 33:6111 1-6122, 2020.
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Coreelis Boudewijn Vreugdenhil. Numerical methods for shallow-water volume 13. Springer Science & Business Media, 1994.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1457-1466, 2020.
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). ), Advances Neural Information Processing Systems, 2022. URL https://ppanneree.ett/fommmm 7id BxSPPP7P7GOOOOOO
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021.
Tailin Wu, Takashi Maruyama, Qingqing Zhao, Gordon Wetzstein, and Jure Leskovec. Learning controllable adaptive simulation for multi-scale physics. NewrlPS 2022 Al for Science: Progress and Promises, 2022. URL http::///wwwooooooo/////////////
Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprint arXiv: /71 1.1 188506, 2017.
Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et Artificial intelligence for science quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08223, 2023.
Juntang Zhuang. Laddernet: Multi-path networks based on u-net for medical image segmentation. arXiv preprint arXiv: 1810.00781 2018.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A MISALIGNMENT ANALYSIS
A.1 SLOWING DOWN EVOLUTION IN LATENT SPACE
We now conduct an empirical analysis to validate hypothesis concerning the manageable inconsis- tency due to reduced misalignment within each wave. Specifically, we demonstrate the progressive evolu- tion of the system's 's state over the waves by injecting a perturbation into the model input and monitoring the ensuing response in the latent maps at each wave. The response this perturbation serves as an indica- tor of the influence of the perturbation at each time step, thereby enabling us to track the evolution of the system. To quantify this response, a forward pass of the input u, through our trained model and capture the ê-th feature map, denoted as Ae We then introduce a localized perturbation to u, in the form of random noise concentrated in small central region and re-record the l-th feature map, denoted Æz+Ac- The absolute difference ac+A<= = [i+ - x++ae is computed and visual- ized to represent the response. We add such noise to samples from the test set as it allows us to see the feature propagation from an excitation with a small spatial extent. Otherwise, if we only provide the per- turbation as input to the network, it will become an out-of-distribution problem for the network since there are no similar samples in the training set. However, if we inject small amount of noise to sample in the test set, the resulting input remains close to the original test sample, and therefore is still in- dn-distrributonn and can be used to analyze the behavior of the trained model.
U-Net
SINENET
first carry out
Mews
Meea]
Figure 4: Feature map responses to noise in- jection for trained U-NET-122 and SINENET- 8. As opposed to the U-NET, in SINENET, the perturbation influence propagates grad- ually from the first to the last feature map, demonstrating the reduced latent evolution managed by each wave.
Figure 4 showcases the evolution of this response across the latent maps ac+A...... Q++Ax in a well-trained SINENET-8 model. For comparison, also include the response from well-trained U-NET- 128, using the output of the first projection layer and the output of the last upsampling block. In both the U-NET and SINENET models, the perturbation influence expands spatially from the input to the output. However, for SINENET-8, the perturbation influence propagates incrementally from the first to the last feature map, indicating that the spatial evolution within each wave reduced. It's important to note that this progressive propagation is not merely consequence of limited receptive fields, as the receptive field of each wave substantially larger than the area influenced by the perturbation. Furthermore, the receptive field for each wave of SINENET is equal to that of the U-NET.
A.2 FEATURE MAP VISUALIZATION
We visualize feature maps for each wave of SINENET-8 in Figure 5 randomly selected example from the INS data. Specifically, for each wave, we show the highest-resolution skip connection from the downsampling path alongside the result of the upsampling path to which the skip-connected feature map is concatenated, as due to latent evolution, these two feature maps will have the greatest misalignment out of all of the skip connections in a given wave. Visualization is done by averaging over the channel dimension of each feature map. To increase the overall contrast, we clip the 1at and th quantiles. As can be seen relative to the feature maps from the U-Net in Figure the misalignment skip connections is substantially mitigated thanks to the multi-stage feature evolution in Sin
14

---

Published as a conference paper at ICLR 2024
Skip connected (128 x288)
Aller upsampling (128 x
Wave
Wave 2
Wave 3
Wave
Wave 5
Wave
Wave 7
Figure 5: Visualization of SINENET-8 feature maps averaged over the channel dimension for INS data. Skip connections from the highest resolution in the downsampling path of each wave are visualized the left, which concatenated with the upsampled feature maps visualized on the right. Compared to U-Net FFigure[]in main text), the degree of misalignment in skip connections is far less severe, feature maps progressively evolve more gradually across waves.
15

---

Published as a conference paper at ICLR 2024
skip
slop
P
C2
C4
Q
P
C4
d
C1
C3
C1
C1
(a) SINENET
(b) U-NET
(c) DEEPER U-NET
Figure Comparison of SINENET, U-NET and DEEPER U-NET architectures.
B DATASET DETAILS
B.1 THE NAVIER-STOKES EQUATION
The Navier-Stokes equations (Constantin Foias 2020j are the fundamental model which encodes the conservation of mass and momentum in fluid flows in manifold applications ranging from weather research to aerospace engineering and stellar magneto-hydrrodynmmic. system of non-linear partial differential equations for the density p. velocity field and pressure p of fluid (compressible case) or velocity u and pressure p (incompressible case). The Navier- Stokes equations are the dominant model of fluid dynamics from which other equations such as the Euler equation or the shallow water equations are derived in limiting regimes (zero viscosity, shallow water). Due its foundational role, the subject of an unsolved Millennium Problem of the Clay Mathematics Institute and has attracted attention from mathematical analysts as well as applied scientists. Challenges abound, especially for mast-ngving turbulent flows, and despite recent breakthroughs (Albritton et al., 2022). even the existence and uniqueness of solutions is not fully understood.
comprises a
B.2 DETAILED SETUP
Incompressible Navier-Stokes (INS). Each trajectory is 14 time steps spaced 1.5 seconds apart and models the evolution of a vector velocity field with Dirichlet boundary condition and scalar field with a Neumann boundary conditon representing the concentration of particles advected by the velocity field. Trajectories are spatially discretized onto a 128 x 128 grid. The data follow train/valid/test split of 1.200/1,300/1..1
and the length of the time history trajectories are unrolled for 10 steps during evaluation. The viscosity controlling the level of turbulence in the flow is 0.01.
steps such that
Compressible Navier-Stokes (CNS). Each trajectory has 21 time steps and consists of scalar pressure field, a scalar density field, and vector velocity field, each with periodic boundary conditions. Trajectories are generated split as 5,400/1,300// ,300 and, following Takamoto et al. (002)), models use time history h = 10 such that trajectories are unrolled for 11 steps during evaluation.
512 512 spatial grid and downsampled to 128 128. The dataset is
Shallow Water Equations (SWE). The trajectories are generated using the numerical solver 96 192 spatial discretization. Each trajectory consists of 11 time steps, with 48 hours between each step. The data are split as 3,600/1,400/1, ,400, and h = historical time steps are used such that trajectories are unrolled for steps during evaluation.
from Klöwer al. (2022)
C TIME AND SPACE COMPLEXITY
In Table analyze the inference time, training time, and GPU footprint for all models a batch of size 32 randomly selected from the INS dataset. The times presented here are an average over 1,000 batches single 80GB A 100 GPU.
16

---

Published as a conference paper at ICLR 2024
Table 4: Analysis of inference and training time, as well memory required for all models on batch of size 32 randomly selected from the INS dataset. A batch size of 16 is used for SINENET- NEURAL-ODE in the Forward+Backwann complexity test cannot fit into the GPU memory with batch size 32.
FORWARD
FORWARD+BACKWAN
METHOD
# PAR. (M)
TIME (S)
MEMORY (GB)
TIME (S)
MEMORY (GB)
SINENET-2
35.5
0.122
4.19
0.398
10.63
SINENET-4
35.5
0.180
3.86
0,596
15.11
SINENET-6
35.5
0.225
3.74
0.729
19.18
SINENET-8
35.5
0.286
3.54
0,917
23.31
SINENET- 10
35.5
0.328
3.52
1.043
27.21
SINENET- 12
35,5
0,368
3.87
1.157
30,90
SINENET- 14
35,4
0,410
3.70
1,286
34.61
SINENET- 16
35.5
0,447
3.65
1,400
38.21
SINENET-NEURAL-ODE
12.7
1.822
8.60
2.977
49.53*
F-FNO
30.1
0.399
3.91
1.175
41.05
MIL-RESNET-12 128
4.2
0.182
3.83
0.387
28.77
HIL-RESNET-256
16.7
0.469
8.37
1,044
56.93
U-NET-128
135.1
0.169
7.58
0,532
13.24
U-NET-MOD
144.3
0.097
4.71
0,236
14.12
D IMPLEMENTATION DETAILS
D.1 TRAINING
Our code is implemented in PyTorch (Paszke et al. 2019). Models are trained and evaluated 2 NVIDIA A 100 80GB GPUs. All models are optimized for 50 epochs with batch size 32 and the model with the best validation rollout results is used for testing. Following Gupta & Brandstetter (023), each epoch consists of T iterations over the data. For each trajectory, a start time is Performing T cycles per epoch ensures that each possible randomly sampled from h .., }. ux+1) is sampled more than once in expectation for each 1-step input-target pair ((44--+1,
trajectory. We calculate statistics from the training data along the field dimension and normalize model inputs and targets to have 0 mean and unit variance. During rollouts, we apply the inverse normalization.
All models are optimized with the AdamW optimizer (Kngmma & Ba 2015 Loshchilov & Hutter 2019), using an initial learning rate of Mui 2 10 except for the F-FNO on SWE and CNS. where we found larger learning rate to improve performance (see Table 6). The learning rate was warmed up linearly for epochs from V/min = 10-7 to 7/mit before being decayed for remaining 45 epochs using cosine scheduler.
D.2 LOSS
We use the Scaled-L, L2 loss for both training and evaluation. We observe that the magnitude can vary significantly across different fields. Therefore, compute the Scaled-L.p loss separately for different fields to account for differences magnitude and then calculate the average across all fields. Given a prediction ú, and target u, time step f, composed of M fields, the 1-step loss is computed as
1 M
M
u/l12
clsstep (is,,
(ir, u.) =
(14)
u 1|
where & denotes the k-th field of u and the norm is taken over all spatial dimensions. Note that consider each scalar component as a separate field. For example, the velocities along x direction and y direction are considered two different fields and normalized independently. For validation and test, the rollout loss is computed as the average 1-step loss over rollout time steps
rrlloou
Clstte (ū ut),
(15)
T-
t=
17

---

Published as a conference paper at ICLR 2024
where T is the number of total time steps and h is the number of conditioning historical time steps.
D.3 SELECTION MULTIPLIER HERPARAMEEEER
Table 5: Multiplier hyperparameter and number channels in each wave for SINENET-K across various choices of K. The multiplier determines how the number of channels is upscaled along the downsampling path and downscaled along the upsampling path. We adjust this multiplier manage the number of parameters and computational cost of SineNet. We also list the number of channels produced by each of the blocks from the beginning to the end of the downsampling path. The input number of channels to each wave in all models is 64.
METHOD
# PAR. (M)
MULTIPLIER
# OF CHANNELS
SINENET-2
35.5
1.8075
(115, 209, 377, 683)
SINENET-4
35.5
1.6110
(103, 166, 267, 431)
SINENET-6
35.5
1,5000
(96, 144, 216, 324)
SINENET-8
35.5
1.4250
(91, 129, 185, 263)
SINENET-10
35.5
1.3660
(87, 119, 163, 222)
SINENET-IE 12
35.5
1.3190
(84, 111, 146, 193)
SINENET- 14
35.4
1.2790
(81, 104, 133, 171)
SINENET-16 16 35.5
1.2435
(79, 98, 123, 153)
We discuss the choice of the number of channels in the feature maps along the down and upsampling paths. The number channels in the feature map output by the 8-th downsampling and upsampling blocks, respectively, is given by
21
m 20]
24 = [ 1- 20]
(16)
where zo the number of channels following the projection by the encoder P and there are L downsampling and upsampling blocks. While conventional U-Nets use multiplier mı 2, we manage the number parameters and complexity of our SINENET-K architectures by selecting m such that the number of parameters is roughly constant in K. In Table we present the multiplier mK and number of channels along the downsampling path for the SINENER-- architecture with varying number of waves K. All architectures presented here use 64.
D.4 BASELINE TUNING
Here we present hyperparameter tuning results for baseline methods the SWE dataset. As discussed in Appendix D.I we found that for F-FNO, using larger learning rate improved results. Additionally, Tran al. (2023) found that in their setting, sharing weights between Fourier convolution layers improved F F-FNO performance. However, we find not sharing to give better results. We additionally found that increasing the number of channels in DES-RE improved performance. We present validation emrors for these experiments in Table
Table 6: Best validation results for different baseline hyperparameter settings on SWE. We evaluate F-FNO with different learning rates, as well as with and without weight sharing. We additionally evaluate the effect of number of channels
METHOD
# PAR. (M)
1-STEP VALID (%)
ROLLOUT VALID (%)
FFNO, Tinit 10 10
30.1
1.21
2.44
FFNO, Jinit = 2 14
30.1
1.88
3.72
FFNO-SHARED, Thnit = x 10-3
3.0
1.77
3.44
DIL-RESNET-122
4.2
3.40
6.60
16.7
2.21
4.09
18

---

Published as a conference paper at ICLR 2024
E DISCUSSION ON LIMITATIONS
cut-of-distribution (OOD) generalization. Unlike classical solvers which based on intrinsic mathematical model of PDEs and can be applied to almost any initial and boundary conditions, data-driven surrogate models are only guaranteed generalize well to scenarios which are close to the training distribution. For example, performance drops substantially when simulating the same dynamics but on larger domain than observed during training (Stachenfeld et al., 2022). It is possible to improve OOD robustness by combining surrogate models with classical solvers (Kochkov ct al., 2021). However, pure surrogate models remain attractive due to flexibility model design and fast inference. It remains to be seen how training methods can be adapted to improve OOD robustness, with initial works in this direction focusing on meta-learning approaches (Wang et al. 2022, Mouli et al. 2023 Kirchmeyer et al. 2022). In Appendix F.5] we found noise injection during training (Sanchez Gonzalez et al. 2020; Stachenfeld et al. 2022) improved robustness to difficult
intial conditions and generalization rollouts 10x longer than training rollouts.
PDEs with reduced temporal evolution. SineNet was developed specifically for time-evolving PDEs wherein the input fields are spatially misaligned with respect the target fields. Therefore, further experimentation is needed to determine the benefits of SineNet in learning dynamics where time evolution does not play as large of role, such as the steady-state Darey flow equations considered by] Li et al. 202
Computational cost. As shown in Appendix C using more waves increases the inference time and training memory due to the added depth. Nevertheless, as we did in the experiments, channel multipliers can be adjusted to control the computational cost.
Irregular spatial grids and time intervals. For modeling dynamics on irregular discretizations, graph neural networks have been used (Brandstetter al., [022) Pfaff et al., (202T] Wu et al., (2022).
As with all CNNs, SineNet can only handle inputs on rectangular grids with uniformly spaced mesh points. However, is possible to extend SineNet irregular meshes with graph neural networks (Gao) & Ji 2019: Li et al.) (020), where the key idea of combining multi-resolution processing and multi- stage processing remains valid. Furthermore, because SineNet is trained for autoregressive prediction, it only advance time by fixed-size timestep A., and thus, cannot predict the solution time points that are not multiples of A, However, as we show in Appendix F. SineNet can be trained to generalize over Ae.
Mixed and non-null boundaries. As discussed in Section 3.6 the Dirichlet boundary conditions we consider for the particle concentration field can be effectively encoded via zero padding. Although the Neumann boundary conditions on the velocity field would ideally be encoded in feature maps using reflection padding to represent the null spatial derivative, encoding both the Neumann boundary condition and the Dirichlet boundary condition is not straightforward. This is because it is unclear which feature maps correspond the particle concentration, and which correspond to the velocity field. To further complicate matters, feature maps earlier in the architecture may correspond to both. Although here we choose to only use zero-padding for simplicity, future work should explore principled approaches to encoding mixed boundary conditions for convolutional architectures, with Horie & Mitsume (2022) having initiated this line of work for graph neural networks.
Furthermore, while we consider null Dirichlet and null Neumann boundaries here, non-null conditions are also of interest, For non-null Dirichlet conditions, the value the field on the boundary is known and can be encoded simply by padding with the known value. In the case of the non-null Neumann condition, one approach could be to pad with a finite difference-type approximation using the feature map's boundary value and the boundary derivative given by the condition. Further work is needed to validate the effectiveness of both approaches.
F EXTENDED RESULTS
F.1 CONDITIONAL INS
In the conditional task considered by Gupta & Brandstetter (222)) models trained to generalize over variable-sin time steps and variable forcing terms. Specifically. ly, in Equation f is an extermal force acting along the and y axes as f (0, f) IR2. In the conditional task, the y-component
19

---

Published as a conference paper at ICLR 2024
Table 7: Summary of rollout test error and one-step test error for various time step sizes on Conditional INS.
1-STEP (%)
METHOD
PAR. (M)
A, 0.375
A, = 0.75
A,
A, = 3.0
A, 6.0
ROLLOUT (%)
SINENET-8-AN
38.0
2.29
2.82
19.79
5.27
SINENET-8-ADAGN
40.3
2.05
2.59
3.63
6.64
19.21
4.60
U-NET-MOD-ADI 146.6
2.46
3.23
4.94
9.50
24.69
5.85
SNET-MOD-ADARR
2.16
5.00
of the forcing term varies between trajectories as [0.2, Furthermore, the time step size is reduced by a factor of relative to the INS data from 1.5 seconds to 0.375 seconds such that the number of time steps increases from 14 56. The one-step training objective then becomes the prediction of u given Uy where the time step A, takes values in {0.375k k 1,..., .5))
To model the variable forcing term and time step size, network inputs (u, R)). where is the time-step and buoyancy к == (f, Gupta & Brandstetter (2023) condition models on 6 by learning embedding of obtained by applying sinusoidal embeddings (Vaswani et al. 2017) and an MLP to f and A, In each convolution block, is linearly projected and added along the channel dimension of feature maps. Gupta & Brandstetter (2023) additionally consider scale-shift approach, where a linear projection of is multiplied element wise along the channel dimension prior addition along the channel dimension with a second linear projection of (Perez et al. 2018). Cupta & Brandstetter (2023) refer to this approach as adaptive group normalization, since the scale and shift are applied directly following normalization (Nichol Dhariwal, (02T). Since SineNet does not use vreeactivation as in the U-NET-MOD considered by Gupta & Brandstetter (2023), wherein normalization and activation functions are applied prior to convolution layers, the scale and shift in SineNet are instead applied following the activation function. Nonetheless, still refer to the scale-shift approach as adaptive group normalization for consistency with Gupta & Brandstetter (2023).
In 7 we compare SINENET-S 8 to U-NET-MOR using both the additive (ADD) and adaptive group normalization (ADAGN) approaches the conditional task. Models trained split of 2,496/95/608, where all trajectories have 56 time steps and the buoyancies / appearing in each split are distinct from those appearing in the remaining splits. Following Gupta & Brandstetter (2023). we evaluate each of the models on I-step prediction with A, E {00.37,, 1.5, 3.0, 6.0). We additionally report errors on rollouts of length 10 with A, 0.375 beginning from all possible time steps in each trajectory. Consistent with Gupta & Brandstetter (2023), we find the ADAGN versions of SineNet and U-NET-MoN to outperform the ADD versions. SINENET8-ADACEE top performer in all metrics.
the
Table Results on INS for SINENET-8, SINENET- ODE, and SINENET-8 with reduced time history h.
METHOD
# PAR. (M)
1-STEP (%)
ROLLOUT (%)
SINENET-8 SINENET-NEURAI-OOEE
35.5
1.66
19.25
12.7
1.83
19.88
SINENET-8, / =
35.5
1.62
18.86
SINENET-8, h
35.5
1.64
19.07
SINENET-B 1 h=
35.5
1.62
18.90
F.2 CONNECTION TO NEURAL ODE
Neural ODE Chen al. 2018 g is general neural framework for learning the mapping from an initial
latent representation (0) to an output latent representation te(1) by parameterizing the derivative of with a neural network v9 as
dh(r) = (0())T),
dr
20

---

Published as a conference paper at ICLR 2024
To map from (0) to h(1), the derivative then be integrated
h(1) = h(0) + vo(h(F)), T),,,
where the integral approximated using numerical integrator, Chen al. (2018) derive method for backpropagating through the operations of an arbitrary numerical integrator in a stable and memory- efficient manner, enabling training of tg. Their method includes the case of adaptive time-stepping dh(r) integrators, which evaluate the derivative dr an adaptive number of times dependent on h(T) to induce stability in the integration. This allows to be applied variable number of times in mapping h0) to (1), thereby formulating a continuous-depth neural network.
number of common machine learning tasks fit in this framework, including image classification, image generation, and time series modeling. Taking h(T) as the latent solution D and vo as a U-Net frames the mapping from 2, Fz+1 neural ODE formulation of SineNet which refer to as HINENET-NERRALE SEEEEERRRL-DIA -DD.. We train and evaluate EINENET-NEURALE AARRAALODI -DD on INS using the fifth order Dormand- Seandd--riiee---aaaeeeee Shampine numerical integrator, which the default in the Neural-ODE PyTorch library (Chen 2018). Following the MNIST (LeCun et al., 1998) experiment from Chen al. (2018), we reduce the parameters of SENENET-NEURA-OOUT
roughly by factor of relative to SineNet. We incorporate the coordinate via learned embedding vector added along the channel dimension in each convolution block of tg as done the conditional task in Appendix
Although Chen et al. (2088) showed that could be optimized with space complexity independent the number of evaluations, we instead train SINENET-
Number of Function Evaluations by Epoch
35.0
32.5
NEURAL-ODE with direct back propagation to speed up training. As shown in Figure 7 the number of function evaluations for SINENET-NEURAL-ODD increases as training progresses, which is consis-
30.0
27.5
25.0
tent with the findings of Chen et al. (2018). Due to the high number of U-Net calls per forward pass, SINENET-NEURAL-OE
22.5
is expensive to train in terms of space and time despite the reduction in the number of parameters, as we show in Table 4 We present test results for SINENET-NEURAL ADEE in Table where we find its performance to be compet- itive with SINENET-8.
10
20
30
40
50
Epoch
Figure 7: Average number of forward passes per batch by epoch when training SINENET- NEURAL-ODE Consistent with Chen et al. (2018), find that the number of forward passes increases as training progresses.
F.3 SUPER-RESOLUTION ANALYSIS
Table 9: Summary of rollout test error and one-step test eтror for the super-resolution task on 1,000 CNS trajectories downloaded from PDEBench (Takamoto et 2022).
128 x 128
512 x 512
METHOD
PAR. (M)
1-STEP (%) ROLLOUT (%)
SR METHOD
1-STEP (%)
ROLLOUT (%)
DILATION
1.06
2.65
SINENET-8
35.5
1.06
2.64
INTERPOLATION
2.44
3.54
F-FNO
30.1
1.46
2.93
DIRECT
1.44
2.90
As discussed in Section neural operators (Kovachki et al., 202T) aim to learn PDE solution operators independently of the resolution of the training data. This enables generalization beyond the discretization the training data such that a trained neural operator can perform zero-shot super resolution (Li et al. 2021a), wherein the task is to solve the PDE higher resolution than during architectures learn their kernel functions on
training. As U-Nets, SineNets, and other CNN- the same grid as the training data, they cannot perform this task directly, although recent work has adapted CNNs to the neural operator framework (Raonić et al.) 2023).
To perform super-resolution with Nine consider two approaches: interpolation and dilation.
21

---

Published as a conference paper at ICLR 2024
Dilation Abs. Err.
Interp. Abs. Err.
Ground Truth
0.15
0.15
Density
0.10
0.10
0.05
0.05
0.00
0.00
0.4
0.4
Pressure
0.2
0.2
13
0.0
0.0
12
0.03
0.03
V
0.02
0.02
0.01
0.01
0.2
0.00
0.00
00.4
0.03
0.02
0.0
0.01
0.01
-0.2
0.00
0.00
Figure 8: Absolute error for super-resolved one-step prediction on CNS using dilation and interpola- tion.
The input initial 10 time steps are downsampled to the training resolution of 128 x 128. SineNet then solves the PDE at the lower resolution, after which the solution is interpolated the higher resolution.
Interpolation.
Dilation. Alternatively, SineNet can operate directly on the high resolution data using dilation. Convolutions, downsampling, and upsampling operations from the SineNet trained on 128 x 128 are all dilated by factor of 4. Intuitively, the grid can be viewed pattern, where each grid point interacts exclusively with other grid points that share the same type (color) as designated by this checkerboard arrangement. This ensures that even a higher resolution, each feature map grid point interacts only with grid points spaced equidistant to those it would interact with at the training resolution. Although dilation is a standard operation convolution layers, we highlight that it is crucial to also apply dilation in pooling (for downsampling) and in interpolation (for upsampling). For example, in the 2 pooling operation, instead averaging and reducing 2 2 regions of immediate neighboring grid points, average and reduce over the corner grid points x 5 regions.
divided using a checkerboard
We download 1,000 512 512 CNS trajectories from PDEBench (Takamoto al. 2022) and evaluate models trained on the 128 x 128 CNS data presented in the main text. We compare to F-FNO, neural operator which does not require interpolation dilation perform super-resolution. Results are presented in Table As in the main text, SINENET-8 outperforms F-FNO at training resolution. At the higher resolution, interpolation introduces error, while dilation achieves nearly identical error to at training resolution. In Figure 8 we visualize the super-resolved errors using both approaches for one-step prediction on randomly chosen example. can be seen, unlike dilation, interpolation introduces errors in regions of high gradient. However, dilation is only applicable for super-resolving at integer multiples of the training resolution. For non-integer super-resolution multiples, similar techniques in deformable convolutions (Dai et al.) 2017) could be considered to offset the operations to non- non-integer locations during testing. Furthermore, all super-resolution approaches introduce error in settings where the higher resolution solutions contain high- tigh-frequency details not present training resolution, or when numerical simulation at higher resolution alters the behavior of the dynamics relative to training resolution, e.g., by introducing smaller eddies in fluid simulation.
22

---

Published as a conference paper at ICLR 2024
INS
CNS
SWE
SineNet-16
4.0
SineNet-16
3.5
SineNet-16
Nen-
Di-ResNeet
3.5
P-PNO
3.0
F-FNO
Error (%)
20
2.5
2.5
15-
2.0
2.0
1.5
1.5
1.0
12
Figure 9: Evolution of rollout error for SINENET-8, SINENET- 16, and the best baseline each dataset.
F.4 TEMPORAL EVOLUTION OF ROLLOUT ERROR
In Figure visualize the evolution of the rollout error of SineNet against the best baseline each the considered datasets.
F,5 LONG-TIME PREDICTION ON CNS
To evaluate SineNet on longer time horizons, we generated 100 CNS trajectories with 120 time steps. As opposed to the CNS task considered in the main text where solvers unroll trajectories for steps given the initial 10 steps. we increase the time horizon by a factor of 10 in unrolling 110 steps given the initial 10 steps. Although the initially turbulent dynamics stabilize the lengthy trajectory and therefore present more stable prediction target, we evaluated models trained on the original CNS task such that beyond 21, the dynamics were out-of-distribution.
One approach to increase long-term stability is noise injection Sancaez-Gonzalez al. 2020: Stachen feld et al., (022). where O-mean Gaussian noise is added to training inputs simulate noise encountered by the solver during rollouts due to errors in previ- ous predictions. Recent work by Lippe et al. (2023)
CNS Rollout Loss, T= 120
SineNet-8-Noise
SineNet-8
F-FNO
Error (%)
20
extended this approach to consider multiple levels of noise that the model is trained to remove, which
shares connection with denoising diffusion proba-
10-
bilistic models (Ho et al.) 2020). Tran et al. (2023) found noise injection to be important for stable train- ing of F-FNO, which we employed in training F-FNO for tasks. For this longer rollout task, addi- tionally evaluated SINENET-8 8--OSSE trained on the 21-step CNS data with the noise level 0.01, which is the same as F-FNO.
0
20
40
60
80
100
120
t
Figure 10: Rollout error averaged across
89 CNS trajectories of length T' = 120 for SINENET-8-NOISE. SINENET-8, and F-
We present results Table TO We find subset of the generated trajectories led to poor performance FNO.
across all models, even causing several of the baselines diverge entirely and produce solutions with greater than 100% error in 1 1//10 cases. The cause is unclear, although the initial density and/or pressure fields for each of these trajectories have small mean values relative to the remaining trajectories, potentially creating out-of-distribution dynamics. We therefore present results both with and without these trajectories. SINENET-S --eer---Noo S-NOISE has the lowest rollout error in both cases, and more robust to the 11 difficult cases than the remaining models.
In Figure 10
we present the rollout error by time step for SINENET-8 8 trained with and without noise, well F-FNO. As can be seen, noise injection has regularizing effect that causes the predictions of SINENET-8 to have better emrors for short time horizons than models trained with noise injection. However, at longer time horizons, models trained with noise injection surpass SINENET-8 due to
23

---

Published as a conference paper at ICLR 2024
their robustness the accumulation of rollout error. In Figures II 14 selected trajectory predicted by SINENET-8 8-NOISE on this data.
visualize randomly
Table 10: Summary of rollout test errors on CNS with T = 120. ROLLOUT-89 corresponds to the dataset with the 11 trajectories resulting in baseline error greater than 100% removed, while ROLLOUT- 100 is for the full dataset. '-' indicates rollout error greater than 100%.
METHOD
PAR. (M)
ROLLOUT- (%)
ROLLOUT-89 (%)
SINENET-8-NOISE
35.5
15.94
14.62
SINENET-8
35.5
27.97
15.44
F-FNO
30.1
15.15
DIL-RESNET
16.7
23.44
U-NET-122 128
135.1
29.97
15.25
U-NET-MOD
144.3
18.95
F.6 ABLATION ON NUMBER OF CONDITIONING STEPS
In Table 8 we ablate the number of historical conditioning steps used on INS. We find that = 1 gives the lowest test error, followed by h 3. That h performs well or better than h is in line with findings reported by Tran et al. (2023). our experiments, we instead adhere to the number of historical steps used by the respective benchmarks for SWE). However, we believe that principled approaches for choosing the number of conditioning steps, as well as how to incorporate them into model predictions, is an interesting topic for future research.
each dataset (4 for INS, 10 for CNS and
F.7
WAVE BOTTLENECK
Table 11: Results on INS, CNS, and SWE for SINENET-8 and SINENET-8 SINENET---BOTTLENEKC BOTTLENECK.
METHON
PAR. (M)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
SINENET-8
19.25
2.10
1.78
35.5
1.68
19.19
1.20
3.14
1.63
2.51
In addition to the dual processing mechanism which we discuss in Section 3.5 and ablate with in Section 5.4 primary difference between the waves V& comprising SineNet and conventional U-Net is the encoder and decoder present in conventional U-Nets (P and Q in Equations and SineNet instead maintains a nigh-dimensional representation between waves. We ablate this design choice with SINENET-8 MENEEET---BTLLENEN BOTTLENECK, which decodes the latent solution Lt+ output by V& to the lower-dimensional base space before encoding back to the latent space for input to VA++. Aside from the dual processing mechanism, this renders the architecture of each wave V& closer to that of a conventional U-Net, however, it creates a bottleneck between waves.
In Table I1I we present results for SINENET-S 8- BOTTLENECCCCC evaluated on INS, CNS, and SWE. On all datasets, SINENET-8 outperforms SINENET-8 has lower rollout error than substantial margin, SINENET-8 LINENET-8-BOTTLENNER
in terms of 1-step error. While on CNS and SWE by BOTTLENECK tops the rollout error of SINENET-8 8 on INS.
The mixed results on these datasets could be due the difficulty of the initial time steps in the INS rollouts relative to the remaining time steps. Compared to the 1-step error of 1.66 for SINENET-8 averaged over all possible time steps in INS trajectories, Figure shows that the errors starting from the beginning of the trajectory are over twice high. This is Tikely due to the large velocities in earlier timesteps creating dynamics which are faster-evolving and with greater degree of local variability than at later timesteps, which is exemplified in Figures early on in predicted trajectories, as well as the accumulation of error through autoregressive rollout, there is substantially larger gap between 1-step errors and rollout errors INS. We hypothesize that the bottleneck serves as form of regularization which decreases performance in terms I-step prediction, but increases robustness the difficult initial steps in INS. Performing similar analysis
18 20
As result of larger errors
24

---

Published as a conference paper at ICLR 2024
for CNS and SWE, the difficulty of the initial timesteps instead appear similar to the remaining timesteps on both datasets. Thus, the regularization not beneficial as before, and fact leads to performance drop, likely due to the information bottleneck between waves.
We visualize the decoded feature maps between each wave for SINENET-8 8- BOTTLENECK on INS, CNS, and SWE in Figures 15| T6. and 17, respectively. Interestingly, SWE is global weather forecasting task, we can see the outline of the world map in the visualized feature maps (e.g. Feature Map I of Wave in Figure (7). which implies that the primary objective of these feature maps is for modeling the evolution of dynamics about the boundaries of continents, potentially encouraged by the information compression the wave bottleneck.
F.8 SINENET-K KK RESULTS
In Table
12
present numerical results for the SineNets with varying K visualized in Figure 3
Table 12: Summary of rollout test enror and one-step test enror for SineNet with varying K.
INS
CNS
SWE
METHOD
PAR. (M)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
1-STEP (%)
ROLLOUT (%)
SINEÑET-2 35.5
23.71
1.56
4.98
1.38
2.54
SINENET-4
35.5
1.91
21.06
1.26
2.63
1.13
1.97
SINENET-6
35.5
1.73
19.58
1.12
2.40
1.10
1.91
SINENET-8
35.5
19.25
2.10
1.02
1.78
SINENET-10 10
35.5
1.58
18.68
0.94
2.24
0.97
1.73
SINENET-12
35.5
1.53
18.09
0.93
2.12
0.92
1.64
SINENET-14
35.4
1.49
17.66
0.89
1.97
0.94
1,66
SINENET-16
35.5
1.46
17.63
0.87
2.04
0.92
1.64
G DATASET AND PREDICTION VISUALIZATION
Here, we visualize rollout predictions from SINENET-8 on INS (Figures 18 20).
CNS (Figures 21 24
and SWE (Figures 25 ground truth field in the left column, the predicted field in the middle column, and the absolute error the right column.
randomly selected trajectory from each In each, show the
25

---

Published as a conference paper at ICLR 2024
CNS, T= 120: Density
Ground Truth
Predicted
Absolute Error
t- 20
0.2
5.0
5.0
2.5
2.5
8
8
1.0
t- 30
0.5
4
4
7.5
4=
5.0
5.0
2.5
2.5
6
6
t=
4
4
8
t- 70
7.5
7.5
5.0
5.0
2.5
7.5
t= 90
5.0
5.0
2.5
2.5
8
8
100
4
4
B
6
1=110
t= 120
Figure 11: Compressible Navier-Stokes density field for T 120 downsampled every 10 time steps.
26

---

Published as a conference paper at ICLR 2024
CNS, T= 120: Pressure
Ground Truth
Predicted
Absolute Error
0.33
4
4
0.2
== 20
t= 30
0.2
r- 40
0.2
0.4
t=50
0.2
0.50
0.25
0.75
0.50
t= 70
1.0
t= 80
0.25
0.50
t- 100
0.25
1.0
3.5
3.5
t= 110
3.0
3.0
0.75
3.5
3.5
0.50
t- 120
3.0
3.0
0.25
Figure 12: Compressible Navier-Stokes pressure field for T 120 downsampled every 10 time steps.
27

---

Published as a conference paper at ICLR 2024
CNS, T= 120: Velocity x
Ground Truth
Predicted
Absolute Error
0.2
00.2
0.02
c- 20
0.0
0.01
-0.2
-0.2
t-
0,0
0.0
0.02
-0.2
-.2
0.2
0.2
0.050
t- 40
0.0
0.0
0.025
0.2
0.2
0.2
0.2
0.050
C- 50
0.0
-0.0
0.025
0.075
0.050
0.025
0.1
0.1
t- 70
0.0
0.0
0.05
-0.1
0.10
t= 80
0,0
0.0
-0.2
-0.
0.2
0.2
t- 90
0.0
0.0
0.05
0.2
0.2
0.10
t- 100
0.05
0.2
t= 110
0.0
0.0
-0.1
-0.1
1..1
0.1
0..10
0.0
0.0
0.05
-0.1
-0.1
Figure 13: Compressible Navier-Stokes velocity x component for T = 120 downsampled to every 10 time steps.
28

---

Published as a conference paper at ICLR 2024
CNS, T= 120: Velocity y
Ground Truth
Predicted
Absolute Error
0.2
0.2
0.04
t- 20
-0.2
-0.2
0.2
0.2
0.04
t- 30
0.02
-0.2
-0.2
0.2
0.2
0.050
t- 40
0.0
0.0
0.025
0.2
0.10
0.2
0.2
0.10
t- 60
0.0
0.0
0.05
-.2
-0.2
-0.1
0.1
t= 70
-0.1
-0.1
0.100
C- 80
0.0
0.0
0.05
0.10
0.0
0.0
0.05
0.2
0.2
0.10
0.0
0.0
0.05
0.2
-0.2
0.2
0.2
0.100
t= 110
0.0
0.0
0.05
-02
0.10
t- 120
0.0
0.0
Figure 14: Compressible Navier-Stokes velocity y component for T 120 downsampled every 10 time steps.
29

---

Published as a conference paper at ICLR 2024
Feature Map
Feature Map
Feature Map
Wave 1
2
Wave 3
Wave 1
Wave 5
Wave 6
Wave 7
Wave B
Figure 15; Visualization of SINENET-8 BLee----BoTTIE BOTTLENECK decoded feature maps between each wave on INS. The Wave 8 feature maps are the predicted particle concentration and velocity field.
30

---

Published as a conference paper at ICLR 2024
Feature Map
Peature Map 2
Feature Map 3
Feature Map
Wave 1
Wave
Wave 3
Wave 1
Wave 5
Wave 6
Wave 7
Wave 8
Figure 16: Visualization of SINENET-8 SANENETT--BBTTLLEE BOTTLENECK decoded feature maps between each wave on CNS. The Wave 8 feature maps are the predicted density, pressure and velocity fields.
31

---

Published as a conference paper at ICLR 2024
Feature Map
Feature Map
Peature Map
Wave
Wave
Wave
Wave
Wave
Wave
Wave
Figure 17: Visualization of SINENET-8 8- BOTTLENECK decoded feature maps between each wave on SWE. The Wave 8 feature maps are the predicted pressure and velocity field.
32

---

Published as a conference paper at ICLR 2024
INS: Particle Concentration Predicted
Ground Truth
Absolute Error
t=
t-7
t=
t= 11
t= 12
t= 13
t= 14
0.938 1.823
0.053
0.938
1.823
0.001
0.418
0.836
Figure 18: Incompressible Navier-Stokes particle concentration.
33

---

Published as a conference paper at ICLR 2024
INS: Velocity
Ground Truth
Predicted
Absolute Error
t-7
t-
t= 10
t= 11
t- 12
t= 13
t= 14
-1.425 1.092
-1.425 -0.667
1.092
0.001
0.344
0.688
Figure 19: Incompressible Navier- Stokes velocity x component.
34

---

Published as a conference paper at ICLR 2024
INS: Velocity y Predicted
Ground Truth
Absolute Error
t-7
t-
t= 10
t= 11
t- 12
t= 13
t= 14
-2.014-0.199 1.657
-2.044 -0.193 1.657
0.001 0.458
0.915
Figure 20: Incompressible Navier-Stokes velocity y component.
35

---

Published as a conference paper at ICLR 2024
CNS: Density Predicted
Ground Truth
Absolute Error
tm 11
t=
t= 14
t= 15
t=
t=
t- 19
t= 20
1.37
6.97
12.57
1.37
0.000
1.176
Figure 21: Compressible Navier-Stokes density field.
36

---

Published as a conference paper at ICLR 2024
CNS: Pressure Predicted
Ground Truth
Absolute Error
tm 11
t=
t= 14
t= 15
t=
t=
t- 19
t= 20
29.17 40.12
18.21
0.000 1.425
Figure 22: Compressible Navier-Stokes pressure field.
37

---

Published as a conference paper at ICLR 2024
CNS: Velocity x Predicted
Ground Truth
Absolute Error
t=
t= 12
t= 13
t-
t=15
t= 16
t= 17
t=
t0
t=
-0.736 0.067 0.871
-0.736 0.067
0.871
0.000 0.035 0.069
Figure 23: Compressible Navier-Stokes velocity x component.
38

---

Published as a conference paper at ICLR 2024
CNS: Velocity y Predicted
Ground Truth
Absolute Error
t=
t= 12
t= 13
t=15
t= 16
t= 17
t=
t=
t=
0.652
0.016 0.684
0.652 0.016 0.684
0.001 0.041 0.082
Figure 24: Compressible Navier-Stokes velocity y component.
39

---

Published as a conference paper at ICLR 2024
SWE: Pressure
Ground Truth
Predicted
Absolute Error
t= 4
t= 5
t=7
t- 9
t= 10
[=11
-4.125
-0.201
3.723
-4.125
-0.201
3.723
0.001
0.059
0.117
Figure 25: Shallow water equations pressure field.
40

---

Published as a conference paper at ICLR 2024
SWE: Velocity
Ground Truth
Predicted
Absolute Error
t= 3
t= 4
t=
t= 6
t= 7
t=
t= 10
[=11
16.95
0.14
17.22
-16.95
0.14
17.22
0.001
0.299
0.598
Figure 26: Shallow water equations velocity component.
41

---

Published as a conference paper at ICLR 2024
SWE: Velocity y
Ground Truth
Predicted
Absolute Error
t= 3
t= 4
t=
t= 6
t= 7
t=
t= 10
[=11
-17.56
-0.24
17.08
-17.56
-0.24
17.08
0.001
0.351
0.702
Figure 27: Shallow water equations velocity component.
42


Reviews:
[{"review_id": "3oPOhxGzId", "reviewer": "Reviewer_uUSa", "summary": "SineNet, a sequential U-Net with comparably small channel depths, is introduced to model the temporal evolution of 2D PDEs. By design, SineNet mitigates the spatial misalignment between skip connections in the traditional U-Net by subdividing the task of spatial propagation over multiple small U-Nets, referred to as waves. Experiments on multiple benchmarks demonstrate the superiority of SineNet against state-of-the-art methods.", "questions": "1. In Section 3.4, does $L=4$ mean that four downsampling plus convolution combinations followed by four upsampling plus convolution combinations are employed? That is, $x=f_l(d(x))$ (encoding) followed by $x=g_l(v(x))$ (decoding) for $l\\in[0, 1, 2, 3]$? Or do you employ only two downsampling and upsampling operations, as visualized in Figure 1? Maybe you can clarify right after Equation (7). \n2. Have you tried to set $\\delta_k$ as learnable parameter, or is the tuning of the time-subscale subject to the practitioner and model designer?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "_Originality:_ of spatial information misalignment between different hierarchies in the U-Net is an elemental observation and finding. Related work is exhaustive and cited adequately.\n\n_Quality:_ All claims are well supported by appropriate visualizations and experiments.\n\n_Clarity:_ The manuscript is mostly clear, well organized and written in approachable language. One point remained unclear to me and should be clarified upon publication (see question below).\n\n_Significance:_ Solving the spatial misalignment problem with the proposed method has a potentially large impact on time-series forecasting with U-Nets and will advance the field. The ML community that is interested in simulating spatiotemporal dynamics (which also relates to video prediction) will likely benefit from the observations presented in this manuscript.\n\n_Further comments_:\n- The demonstration of spatially misaligned features in Figure 1 is very indicative and comprehensive.\n- I like the neat property to have adaptive $\\delta_k$ per wave to adequately model velocities of different speeds.\n- Great to have runtime analysis and memory footprint of all models provided in Table 4 of Appendix C!", "weaknesses": "1. Unclear what limitations the method bears. Under what circumstances do you expect SineNet to break and fail?\n2. Why does Dil-ResNet which you benchmark against only have 4.2M parameters (compared to 35M in your model)? This is a fairly critical point, since neither the inference time nor the required memory in Table 4 seem to justify such a small parameter count for Dil-ResNet. Indeed, I was curious to see how vanilla ResNet would perform and like that you have included Dil-ResNet. This actually forced me to only rate your submission as \"marginally above threshold\" instead of voting for a clear accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tv6UhQmOxd", "reviewer": "Reviewer_XLvR", "summary": "The paper addresses the misalignment problem caused by the U-Net architecture in the context of PDE simulations and proposes multi-stage modeling of temporal dynamics with SineNet. SineNet consists of multiple sequentially connected U-shaped network blocks, which the authors call waves. High-resolution features evolve progressively through multiple stages, and contains misalignment within each stage. The method is evaluated on typical problems where multi-scalability can be seen, and its effectiveness are shown compared to representative baselines.", "questions": "See the weakness above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper points out an interesting problem that happens when employing the U-Net architecture to simulate spatiotemporal dynamics. The paper is well-organized and is generally easy to follow. The idea of the proposed model is simple and the choice of the architecture is based on reasonable analyzation of existing works. Wide range of ablation study is conducted and the proposed method outperforms strong baselines in rollout experiments.", "weaknesses": "I still do not fully understand what the authors mean by misalignment, since the metric for misalignment does not look properly introduced. Although the loss $a_{t+\\Delta l}$ is introduced in the Appendix A.1, the loss does not seem to serve as the metric because it (and the experiment in Appendix A.1) looks more like for the robustness against noise injected. Thus, it is also still unclear if the improvement in the rollout result reported in Table 1 and Table 2 is achieved by the reduction of misalignment. \n\n\nSome ablation study is missing. For example, what about if we replace each $V_{k}$ with standard U-Net with the equations (2) and (3)? Can the authors also provide visualization on this ablation version of the model in a way similar to Figure 5?\n \n\n \n**Minor comments**\n\nTypo Section 3.3: … benefits of the U-Net architecture, we contruct the ….", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRfcI95UoD", "reviewer": "Reviewer_NgLk", "summary": "When applying U-Net architectures with skip connections to dynamical problems, the features do not evolve across layers, thus limiting the efficiency of these methods for predicting dynamical systems. The method proposed in the article tries to overcome this challenge by having features across layers that evolve with the time. More precisely, the architecture is an encode-process-decode architecture consisting of several U-Net architectures stacked together with convolutions to encode and decode the solutions. Each U-Net, or wave, predicts the latent solution at a small time step instead of at the final time step, thus allowing evolving features. In addition to this global architecture, the downsampling and upsampling operations include residual blocks from the input, thus connecting the evolution of the features directly with the input. The architecture is tested on fluid mechanics problems, namely incompressible Navier-Stokes, compressible Navier-Stokes and Shallow Water equations. The experiments consist in forecasting these dynamical systems.", "questions": "- Have you tried with a high number of rollout steps ? It would be interesting to see how stable the method is when pushed to its limits.\n- In Figure 3, the error seems to keep decreasing with the number of waves. Have you tested with more waves? This would be interesting to see when the error arrives at a plateau and this could help improve the performances of the model if the error keeps decreasing with the number of waves.\n- Have you tried using noise to improve the rollout stability? (as has been done for instance in [4])\n- In the experiments, null Dirichlet and Neumann boundary conditions were chosen. What happens if different values for these conditions are chosen? Will zero padding still be effective? \n\n[4]: Stachenfeld, K., Fielding, D. B., Kochkov, D., Cranmer, M., Pfaff, T., Godwin, J., ..., Sanchez-Gonzalez, A. (2021). Learned coarse models for efficient turbulence simulation. arXiv preprint arXiv:2112.15275.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The method solves a common problems when directly applying U-Net architectures to dynamical systems.\n\n- The paper is well written and the justifications of the architecture are documented.\n \n- The performances of the method seem to be competitive with several baselines.", "weaknesses": "- Stacked U-net have already been developed  for different tasks as detailed in the article (section Stacked U-Nets in the related work). The implementation here differs in some minor points and thus is more effective for PDE prediction but this is a marginal improvement over the previous works.\n- The experiments are not extensive. First, more baselines should be used, with at least DeepONet [1], MP-PDE ([2], cited but not implemented) or Neural ODE [3], which closely resembles the formulation of SineNet. Second, the evolution of the error with the time is not shown, only at a given number of rollout steps for each datasets, thus limiting the impact of the results. Third, no ablation on the influence of the number of historical time steps is performed.\n- As other U-Net architectures, SineNet cannot handle irregular grids or grids of different sizes, which are very common in various dynamics resolution problems. This limits the applicability of this line of methods to more practical problems.\n- This is more minor. As with other auto-regressive architectures, the forecasting process is done at regular steps, so the architecture cannot produce values at any time $t$, but only at multiples of $\\Delta t$.\n\n[1]:Lu, L., Jin, P.,  Karniadakis, G. E. (2019). Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193.\n\n[2]:Brandstetter, J., Worrall, D.,  Welling, M. (2022). Message passing neural PDE solvers. arXiv preprint arXiv:2202.03376.\n\n[3]: Chen, R. T., Rubanova, Y., Bettencourt, J., Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cM5X41OfvB", "reviewer": "Reviewer_BgSq", "summary": "This paper introduces SineNet as sequential / multi-stage UNet which improves temporal modeling for time-dependent partial differential equation (PDE) surrogates. This is motivated by misalignment in UNet architectures between downsampling and upsampling paths. The misalignment is caused by the temporal update of the underlying PDE which is in stark contrast to segmentation tasks for which the U-Net was initially introduced. SineNet architectures consist of multiple waves which are individual small UNet components, and thus, enable a smooth transfer between input timesteps and predicted timestep. SineNet models are testes against Factorized Fourier Neural Operators, Dilated ResNets, and various U-Net variants on 2D benchmarks of compressible / incompressible Navier-Stokes equations and shallow water equations.", "questions": "- The iterative refinement process is conceptually similar to PDE Refiner (Lippe et al.) - with some major differences to my understanding. SineNets iteratively refines the temporal update in an one end-to-end approach whereas PDE Refiner looks at the input several times and refines different frequency components. It might be interesting to contrast those two approaches in the discussion.\n- The paper strongly references the original paper of Gupta & Brandstetter. This paper put some emphasis on conditioning on equation parameters as well as flexible lead time predictions. It would be interesting to have statements / experiments on this as well. Similar FiLM like encodings would work for SineNet I suppose?  \n- The misalignment made me realize why ViTs or Swin Transformers might be better fit for PDE modeling compared to standard U-Nets. Can you comment on this?  \n\nI am happy to raise my score if weaknesses and questions are addressed properly.\n\n\n\n\nLippe, P., Veeling, B. S., Perdikaris, P., Turner, R. E., & Brandstetter, J. (2023). Pde-refiner: Achieving accurate long rollouts with neural pde solvers. arXiv preprint arXiv:2308.05732.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated. It presents an intriguing observation on temporal PDE modeling for U-Nets. Whereas U-Nets are ideally suited for e.g. segmentation tasks (U-Nets extract information on various scales hierarchically before invoking a symmetric upsampling path), for PDE modeling this symmetric upsampling is broken since the time evolution of the system is incorrectly modeled. The visualizations help a lot understanding this problematic.\n- The architecture design is novel and thought-through.\n- Experimental results are strong and ablations are convincing, personally I like Figure 3 a lot!  \n- The code is clean and easy to follow.", "weaknesses": "- As clearly stated in the paper, U-Nets in this setting only work for fixed grid resolutions. There are however already works which design operators for different grids - see e.g.,  Raonic et al. Would it be possible to extend the experiments to some \"super-resolution\" experiments (i.e., evaluating at a different (higher) resolution than the one used during training)? I am pretty convinced SineNets are doing great there as well.\n- One advantage U-Nets have is that they are storage efficient since the highest resolution is only used in the first and last block. I am not sure how SineNets handle this - probably through P and Q networks? It would be great to have some information on this. Toward this end I am wondering if the number of waves is increasing the memory footprint and by how much? \n\n\n\nRaonić, B., Molinaro, R., Rohner, T., Mishra, S., & de Bezenac, E. (2023). Convolutional Neural Operators. arXiv preprint arXiv:2302.01178.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)