Advection Augmented Convolutional Neural Networks
Authors: Niloufar Zakariaei
Siddharth Rout
Eldad Haber
Moshe Eliasof
Venue: NeurIPS 2024 poster (2024)
Keywords: Reaction-Advection-Diffusion System
Partial Differential Equation
Semi-Lagrangian Scheme
Spatio-temporal Prediction
Primary area: machine_learning_for_physical_sciences
TL;DR: We introduce an advection augmented CNN and show its effectiveness on several real-world datasets.

Abstract: Many problems in physical sciences are characterized by the prediction of space-time sequences. Such problems range from weather prediction to the analysis of disease propagation and video prediction. Modern techniques for the solution of these problems typically combine Convolution Neural Networks (CNN) architecture with a time prediction mechanism. However, oftentimes, such approaches underperform in the long-range propagation of information and lack explainability. In this work, we introduce a physically inspired architecture for the solution of such problems. Namely, we propose to augment CNNs with advection by designing a novel semi-Lagrangian push operator. We show that the proposed operator allows for the non-local transformation of information compared with standard convolutional kernels. We then complement it with Reaction and Diffusion neural components to form a network that mimics the Reaction-Advection-Diffusion equation, in high dimensions. We demonstrate the effectiveness of our network on a number of spatio-temporal datasets that show their merit. Our code is available at https://github.com/Siddharth-Rout/deepADRnet.

Advection Augmented Convolutional Neural Networks
Niloufar Zakariaei University British Columbia Vancouver, Canada nilouzk@student. ubc.ca
Siddharth Rout University of British Columbia Vancouver, Canada Sonihharthrooutuube.. ca
Eldad Haber
Moshe Eliasof
University of British Columbia
University of Cambridge Cambridge, United Kingdom me5320cam ac
Vancouver, Canada ehaber@eoas. .ut.eeeosssu....c.ooooo.
Abstract
Many problems physical sciences are characterized by the prediction of space- time sequences. Such problems range from weather prediction to the analysis of disease propagation and video prediction. Modern techniques for the solution these problems typically combine Convolution Neural Networks (CNN) architec- ture with a time prediction mechanism. However, oftentimes, such approaches underperform in the long-range propagation of information and lack explainability. this work, we introduce a physically inspired architecture for the solution such problems. Namely, we propose augment CNNs with advection by design- ing a novel semi-Lagrangian push operator. We show that the proposed operator allows for the non-local transformation of information compared with standard convolutional kernels. We then complement it with Reaction and Diffusion neural components form a network that mimies the equation, in high dimensions. We demonstrate the effectiveness of our network on number of apatio-temport datasets that show their merit. Our code is available attps://gitann com Sidhrtth Sothtoot/// cuut/deeppDRnet.
Introduction and Motivation
Convolution Neural Networks (CNNs) have long been established as one of the most fundamental
and powerful family of algorithms for image and video processing tasks, in applications that range
from image classification [27, 211, denoising [5] and reconstruction [26], generative models [17]. More examples of the impact of CNNs on various fields and applications can be found in [40, 18, 32] and references within.
At the core of CNNs, stands the convolution operation - a simple linear operation that local and spatially rotation and translation equivariant. The locality of the convolution, coupled with nonlinear activation functions and deep architectures have been the force driving CNN architectures to the forefront of machine learning and artificial intelligence research [50, 21J. One way understand the success of CNNs and attempt to generate an explainable framework for them is to view CNNs from a Partial Differential Equation (PDE) point of view [43, 7]. In this framework, the convolution is viewed as a mix of discretized differential operators of varying order. The layers of the network are then associated with time. Hence, the deep network can be thought as a discretization of nonlinear Line-dependen PDE. Such observations have motivated parabolic network design that
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

D711
Look
ADR
Reshet
Epochs
(A) Source image
(B) Target image
(C) Convergence
Figure 1: A simple task of moving information from one side of the image to the other. The source image in A is moved to the target image in B. The convergence of a simple ResNet and an ADRnet proposed in this work is in (C).
smooth and denoise images [44] as well as to networks that are based on hyperbolic equation [28] and semi-implicit architectures [20].
However, it is known from the literature [34], and also demonstrated in our experiments, that CNN architectures tend to under-perform in tasks that require rapid transportation (also known as advection) of information from one side of image the other. particular, in this paper, we focus on the prediction of the spatio-temporal behavior of image features, where significant transportation present in the data. Examples of such data include the prediction of weather, traffic flow, and crowd movement.
Related work: In recent years, significant research was devoted to addressing spatio-temporal problems. Most of the works known to us are built on a combination of CNN to capture spatial dependencies and Recurrent Neural Networks (RNN) to capture temporal dependencies, A sample of papers that address this problem and the related problem of video prediction can be found in [4, 53, 46, 31, 22, 35, 13] and reference within. See also [65] and [39] for recent comparison between different methods. Such methods typically behave as black boxes, in the sense that while they offer strong downstream performance, they often times lack a profound understanding of the learned underlying dynamics from the data. In addition, networks that try to predict the optical flow in videos [9, 64, 48] were proposed to estimate the flow in the original image domain. However, predictions on the image domain may be limited and not capture hidden dynamics. Another type of work that designed for scientific datasets [51], which uses Fourier-based based methods to build the operators. See also [3] for review on the topic.
Notably, while CNN is versatile tool that allows learning spatial dependencies, can have significant challenges in learning simple operations that require transportation. As an example, let us consider the problem of predicting the motion in the simple case that the input data an image, where all pixels take the value of 0 except for a pixel on the bottom left (marked in gray), ), and the output is an image where the value transported a pixel on the top right. This example is illustrated in Figure 1. Clearly, no local operation, for example, a convolution of say, 3 x 3 or even 7 x 7 can be used move the information from the bottom left of the image to the top right. Therefore, the architecture to achieve this task requires either many convolutions layers, downsampling the image via pooling, where the operations are local, performing convolutions on the downsampled image, and then upsampling the image via unpooling, followed by additional convolutions "clean" coarsening and interpolation artifacts, as is typical in UNets [42, 8). To demonstrate, we attempt to fit the data with simple convolution residual network and with residual network that has an advection block, as discussed in this paper. The convergence history for the two methods is plotted in Figure I. We that while residual network is incapable of fitting the data, adding an advection block allows it to fit the data to machine precision.
Motivation:
This set of problems, as well the relatively poor performance it offers on data that contains advection as in simple task in Figure sets the motivation for our work. Our aim to extend the set of tools that is available in CNNs beyond simple and local convolutions. For dependent PDEs, is well known that is possible to model most phenomena by a set of advection-diffusion reaction equations (see, e.g., [12, 11] and references within). Motivated by the connection between the discretization of PDEs and deep network [43, 7]. and our observations on the shorteomings of existing operations in CNNs, we propose reformulating CNNs into three different components.

---

Namely, (i) pointwise term, also known as a reaction term, where channels interact locally. (ii) A diffusion term, where features are exchanged between neighboring pixels smooth manner. And, (iii) an advection term, where features are passed from pixels to other pixels, potentially not only among neighboring pixels, while preserving feature mass or color loss². As we discuss in Section the combination of diffusion and reaction is equivalent to a standard CNN. However, there is no CNN mechanism that is equivalent to the advection term. Introducing this new term equips the network with flexibility in cases where information is carried directly.
Contributions:
The contributions of this paper are three-fold. First, we form the spatio-temporal dnttection-iiffsioon-eeaae
process, which is novel and has not been studied in CNNs prior to our work. Second, we propose the use of the semi-Lagrangian approach for its solution, introducing a new type of learnable linear layer, that is sparse yet non- local. This is in contrast to standard convolutional layers, which act locally. In contrast to advection, other mechanisms for non-local interactions, require dense interactions, which are computationally expensive [57]. Specifically, our use of sami-Lagrangian methods offers a bridge between particle- based methods and convolutions [29]. Thus, we present a new operation the context CNNs, that we call the push operator to implement the advection term. This operator allows us to transport features anywhere on the image in a single step an operation that cannot be modeled with small local convolution kernels. It is thus a simple yet efficient replacement to the standard techniques that are used move information on an image. Third, we propose a methodology to learn these layers based the splitting operator approach, and show that they can successfully model advective processes that appear in different datasets.
dynamics in high dimensions
The advection diffusion reaction model is optimal when applied to the prediction of images where the information for the prediction is somehow present in the given images. Such scenarios are often present in scientific applications. For example, for the prediction of the propagation of fluids or gasses, all we need to know is the state of the fluid now (and in some cases, in a few earlier time frames). A more complex scenario is the prediction of video. In this case, the next frame may have new features that were not present in previous frames. To this end, the prediction of video requires some generative power. While we show that our network can be used for video prediction and even obtain close to the state-of-the results, we observe that it performs best for scientific datasets.
Limitations:
2 Model Formulation
Notations and assumptions. We consider spatio-temporal vector function of the form (t,,x) [q (t, x)...., 4m. (t, x)] E Q. where Q is the space vector function with channels. The function is defined over the domain x E C Rd. and time interval [O, Łj]. Our goal is to predict the function time L for some la > Lz. given the inputs up time j. For the problem we consider here, the time is sampled on a uniform grid with equal spacing. Below, we define the
system that renders the blueprint of the method proposed in this paper achieve our goal.
System. Given the input function q. we first embed it in higher dimensional space. We denote the embedding function by :E I, defined as
I(I,, x) = (L.(( x), 0tn)
where Mn R. + R is multi-layer preceptron (MLP) that embeds the function q from m to > channels with trainable parameters On.
To represent the evolution of q we evolve I in the hidden dimension, c, and then project it back into the space Q. One useful way to represent the evolution of a spatio-temporal process is by combining three different processes, as follows:
Reaction: A pointwise process where channels interact pointwise (sometimes referred to as 1x 1 convolutions)
Diifiusion: A process where features are being communicated and diffused locally.
That is the sum of the features is constant.
3

---

B
C
Output
Input
Advection: tx 1k+1/3
Diffusion: 1k+1/3 1R+2/3
Reaction: 2 - tx+1
Figure 2: An illustration of the advection-diffusion reaction process. In the first step, Column (advection), pixel on the lower left of the image is transported into the middle of the mesh. In the second step, Column B (diffusion), the information is diffused its neighbors, and finally, in the last step, Column (reaction), each pixel interacts locally to change its value.
Advection: A process where information transports along mediums.
These three processes are also illustrated in Figure 2, and their composition defines the advection- diffusion-reaction differential equation on the embedded vector I.
The equation can be written as
it x)
KAI((, x) + V (UU((U + R(I(,,)
(2)
Ø1.
T(t = 0, x) = M(q(t = 0))
(3)
Here A is the Laplacian, and V is the divergence operator, as classically defined in PDEs [12]. The equation is equipped with initial condition and some boundary conditions. Here, for simplicity of implementation, we choose the Neumann boundary conditions, but other boundary conditions can also be chosen. The diffusivity coefficient K, velocity field U, and the parameters that control the reaction term R are trainable and are discussed in Section 3.
The equation is integrated second MLP, that projects the hidden features in I(t = T, x) to the desired output dimension, which in our is the same the input dimension, i.e.,
some interval [0. 7 and finally obtains qT,, x) by applying
(7,x) Mont (I(T, x), Pout),
where Out trainable parameters for the projection MLP.
Remark (Equation 2 Reformulation). The discretization of Equation can be challenging due to conservation properties of the term V (UI(t, x)). An alternative equation, which may be easier to discretize in our context, can be obtained by noting that
OI x
ÖI Øt
(UT)
+U. VI
(5)
It
The operator on the left-hand side in Equation 5 is the continuity equation [12], where the mass of I conserved. The first two terms on the right hand side, namely, I, + U VI are sometimes refer to as the color equation [12] as they conserve the intensity of I. For divergent free velocity fields, that is, when V U O, these are equivalent, however, for non-divergent fields, the term IV U pointwise operator on I, that is, it is reaction term. When training model, one can use either Equation its continuity form or replace the term with Equation and learn the term IV U as part of the reaction term, R. We discuss this in discretization of our model in Section 3.3.

---

3 From a Partial Differential Equation to a Neural Network
To formulate neural network from the differential equation in Equation needs be discretized in time and space. In this work, we assume data that resides on regular, structured mesh grid, such as 2D images, and the spatial operators to discretize Equation are described below. To discretize Equation 2 in time, we turn to Operator Splitting methods [1] that are common for the discretization of equations with similar structures, and were shown to be effective in deep learning frameworks [11]. As we see next, such discretization leads to neural network that has three types of layers that are composed of each other, resulting in an effectively deeper neural network.
3.1 Operator Splitting
The idea behind operator splitting is to split the integration of the ODE into parts [30]. Specifically, consider a linear differential equation of the form
aI x AI(t, + DI(t,x) + RI(,,
(6)
dt
where A, D and R are matrices. The solution to this system at time i is well known [12] and reads
I(t, x) = exp (tA + +D + [R) x),
(7)
where exp denotes the matrix exponentiation operation. It is also possible to approximate the exact solution presented in Equation 7 as follows
exp (tA + tD +R) x) = exp(tA) ((sxp(tD
(8)
The approximation is of order t, and it stems from the fact that the eigenvalues of the matrices A, D and R do not commute (see [1] for a thorough discussion). Equation can also be interpreted in the following way. The solution, for short time integration time t, can be approximated by first a1(t.x) RI x), Io I(0,x) obtaining solution (t x), followed by the
solving the system St (t(x.)) DI (t. x) In- = In obtaining the solution Lnd x) and finally solving the system o4 AIRD (f, x), 1o LRD. The advantage of this approach is that it allows the use of different techniques for the solution of different problems. The derivation of the approach employed in this work is presented in Appendix A.4, which also provides a detailed explanation of the invariance the order of splitting.
solution of the system
at
@I(xx)
Let R be the solution operator that advances I(ty,>) x) to (t(t)+1, x). Similarly, let be the solution operator that advances [n((++, x) to (I x) and lastly, let A be the solution of the advection problem that advances InD (4j+1, to [lf++1, x). Then, a layer in the system can be written as the composite of three-layer
CI(t,, x) = AoDo (t,,
(9)
That is, the resulting discretization in time yields neural network architecture of a layer that is composed of three distinct parts. We now discuss each part separately.
3.2 Advection
The innovative part of our network is advection. The advection approximately solves the equation
ØT = v. (U),,
(10)
Øt
for a general velocity field U. For the solution of this equation, we now introduce a linear operation that we use to enhance the performance of our network. Our goal is to allow for information to pass over large distances. To this end, consider displacement field U (U,, U2) and consider the push operation, A (U)I as the operation that takes every pixel in I and displaces it from point x to X, x + U. Since the point x, does not necessarily reside on a grid point, the information from x spread over four grid points neighbors, weights that are proportional to the distance from these points. A sketch of this process is plotted in Figure 3 (a). The operator discussed above conserves that mass of the features. different implementation, as discussed in Remark 1, is discretize the color equation. This is done by looking backward and using the interpolated value shown in
5

---

Figure 3: Discretization of the push operator. (a) Left: Semi-Lagrangien mass preserving transport,
discretizing the continuity. (b) Right: Semi-I.grangiaa color-preserving transport.
Figure 3(b). It possible to show [14] that these linear operators transposed of each other. Here, for each implementation, we chose to use the color equation. We show in ablation studies that the results when using either formulation are equivalent.
The process allows for a different displacement vector u for every grid point. The displacement field U in has 2c channels and vary in space and time. To model the displacement field, propose to use the data at times,
Qs = [q(1x-j, x). ((lx-(+1,,)) 4(e, x)],
(11)
where is the length of history used to learn the displacements.
Using Qs. the displacement field computed by simple residual convolution network, which we formally write as
U RN (Q 7),
(12)
where RN is the residual network parameterized by r7.
3.3 Reaction
The reaction term is a nonlinear 1 x convolution. This yields a residual network of the form
1 I, +
= R,)
(13)
where M is standard, double-layer ML.P with parameters 0, and h is step size that is hyper- parameter. We may choose to have more than a single reaction step per iteration.
3.4 Diffusion
For the diffusion step, we need to discretize the Laplacian on the image. We use the standard 5-point Laplacian [16] that can also be expressed as 2D group convolution [37]. Let An be the discrete Laplacian. The diffusion equation reads
1 I hKAR
we choose j we obtain an explicit scheme
I,+1 = I, + hEA
(14)
Note that the diffusion layer can be thought of as group convolution where each channel is convolved with the same convolution and then scaled with different The forward Euler method for the diffusion requires lx to be small if we want retain stability. By choosing & j + we obtain the backward Euler method, which unconditionally stable
1 = (I hx I, = D(K)
(15)
To invert the matrix the cosine transform [25] which yields log complexity for this step.

---

Table 1: Datasets statistics. Training and testing splits, image sequences, and resolutions
Dataset
Niain
Nist
(C, H, W)
History
Prediction
PDEBench-SWE
900
100
(1, 128, 128)
10
1
CloudCast
5241
1741
(1, 128, 128)
4
4,
12, 16
Moving MNIST
10000
10000
(1, 64, 64)
10
10
KITTI
2042
1983
(3, 154, 512)
2
1,3
Combining Diffusion and Reaction to Single Layer. handled by an implicit method (that a matrix inversion) and the reaction is handled by an explicit method. For datasets where the diffusion is significant, this may be important; however, in many datasets where diffusion is very small, it is possible use an explicit method for the diffusion. Furthermore, since both the diffusion and reaction are computed by convolutions, it is possible to combine them into structure that is very similar to classical Convolutional Residual Network that replaces the diffusion and reaction steps. For the datasets used in this paper, we noted that this modest architecture was sufficient to obtain results that were close to state-of-the-art.
In the above network the diffusion is
3 convolution (see [43] and [20] for additional discussions). This yields a
3.5 Implementing the ADR Network
Implementing the diffusion and reaction terms, either jointly or combined, we use standard Convolutional Residual Network. The advection term is implemented by using the sampleGrid command in PyTocc [41], which uses an efficient implementation to interpolate the images.
While the network can be used as described above, we found that better results can be obtained by denoising the output of the network. To this end, have used a standard UNet and applied the output. As we show in our numerical experiments, this allows us to further improve downstream performance. The complete network is summarized in Algorithm
Algorithm 1 The ADR network
Set 1o < M (qu 0,), Qa as in equation 11. for j = 0, m. 1 do
Diffusion-Reaacionn IDR t DR, Re I Compute displacement U, = (IDR n7) as in equation 12 Push the image 1 = DR
end for
Set qı+t M (Im. 0r) (Optional) Denoise qu+e = UNet(qu+t)
Experiments
Our goal to develop architectures that perform well for scientific-related datasets that require advection. In our experiments, we use two such datasets, CloudCast [70], and the Shallow Water Equation in PDEbench [51]. However, our ADRNet can also be used for the solution of video prediction. While such problems behave differently than scientific datasets, we show that our ADRNet can perform reasonably well for those applications as well. Below, we elaborate on the utilized datasets. We run our codes using a single NVIDIA RTX-A6000 GPU with 48GB of memory. Besides the experimental results reported in this Section, we provide additional results, from ablations to visualizations and measured runtimes in Appendix A.
4.1 Datasets
We now describe the datasets considered in our experiments, which are categorized below.
Scientific Datasets: We consider the following datasets which arise from scientific problems and communities: (1) SWE, The shallow-water equations are derived from the compressible Navier-
7

---

Stokes equations. The data is comprised of 900 sets of 101 images, each of which is a time step. (2) CloudCast. The CloudCast dataset comprises 70,080 satellite images captured every 15 minutes and has resolution of 3712 x 3712 pixels, covering the entire disk of Earth.
Video Prediction Datasets: These datasets mainly from the Computer Vision community, where the goal is to predict future frames in videos. The datasets are as follows: (1) Moving MNIST. The Moving MNIST dataset is synthetic video dataset designed features 20-frame sequences where two MNIST digits move with random trajectories. (2) KITTI. The KITTI widely recognized dataset extensively used in mobile robotics and autonomous driving, and it also serves as benchmark for computer vision algorithms.
sequence prediction models.
The statisties of the datasets summarized in Table 1, and in Appendix A, provide results additional datasets, namely TaxiBJ [70] and KTH [45].
4.2 Evaluation
Ranking of Methods. Throughout all experiments where other methods are considered, we rank the top 3 methods using the color scheme of First, Second, and Third.
Performance on Scientific Datasets. We start our comparisons with the SWE and CloudCast datasets. These datasets fit the description of our ADRNet as future images depend on the history alone (that is, the history should be sufficient to recover the future). Indeed, Table 2 and Table show that ADRNet performs much better than other networks for these goals. Additional experiments on the Navier-Stokes dataset are provided in Appendix A.3
Table 2: Results PDEBench SWE Dataset.
Table 3: Results on CloudCast dataset.
Method
NRMSE
UNet [52]
8.3e-2
Method
SSIM (t)
PSNR (1)
PINN [52]
1.7e-2
AE-Conv LSTM [70]
0.66
8.06
MPP-AVIT-TI [36]
6.6e-3
MD-GAN [67]
0.60
7.83
ORCA-SWIN-B [47]
6.0e-3
TVL1 [56]
0.58
7.50
FNO [52]
4,4e-3
Persistent [70]
0.55
7.41
MPP-AVIT-B [36]
2.4e-3
MPP-AVIT-L [36]
2.2e-3
ADRNet
0.83
38.17
ADRNet
1.3e-4
3/t
4:00
2001
Ground-Tran
Prediction
Error
Figure 4: Prediction and error for the SWE problem using our ADRNet.
Examples the predictions of the SWE dataset and the CloudCast datasets are plotted in Figure and Figure 6. For the SWE dataset, the errors are very small and close machine precision. For CloudCast, the data is noisy, and it is not clear how well it should fit. Predicting a single-time step, while useful, has limited applicability. Our goal is push the prediction for longer, hence providing an alternative to expensive numerical integration. The results with SWE for long-time prediction (i.e. using the same 10 timesteps from history to predict 10. 20, 50 timesteps the future) are presented in Table 4, together with comparison of the celebrated state of the art FNO method [33] where we see that our model performs well even long-time prediction. As summarized Table ADRNet outperforms other models on long-range predictions for the SWE dataset. This is further illustrated in Figure 5, which shows the prediction accuracy of ADRNet at future time steps of 10, 20, and 50, demonstrating its ability to capture extended dependencies effectively.

---

Figure 5: Predictions on the SWE dataset at future time steps 10,20,50 (left to right). Our results demonstrate the large receptive field learned by ADRNet.
Table 4: Comparison of ADRNet and FNO long range- suneepredictions. We consider the prediction of different numbers of steps given different numbers of input steps. For example, the setting of 10 input steps and prediction steps is denoted by 10 5.
Metric
10 5
10 + 10
10
20
10 -> 50
ADRNee
FNO
ADRNet
FNO
ADRNet
FNO
ADRNet
FNO
MSE
9.2e-08
4.0e-07
1.5e-07
5.8e-07
2.1e-07
6.7e-07
8.5e-07
1.4e-06
aMSE L
8.5e-08
3.7e-07
1.4e-07
5.4e-07
1.9e-07
6.2e-07
7.8e-07
1.3e-06
RMSE
3.0c-04
6.3e-04
3.9c-04
7.6c-04
4,5c-04
8.1c-04
9.2c-04
1.2c-03
HRMSE
2.9c-04
6.1e-04
3.7c-04
7.3c-04
4.4c-04
7.8c-04
8.8c-04
1.1c-03
MAE 4
2.0c-04
2.8c-04
1.7c-04
3.7c-04
1.9c-04
3.6c-04
4.1c-04
5.7c-04
25
Ground Truth
22
Prediction
22
Difference
Figure 6: Example of the forecast by ADRNet compared to the ground truth over four time steps. '(' denotes the forecast time 15-minute intervals. We use four input images to predict subsequent four images. While changes in the CloudCast dataset in the two subsequent frames are slow, ADRNet achieved superior results terms of PSNR and SSIM. A quantitative comparison is shown in Table 3.
To evaluate the generalization capability of ADRNet, we conducted additional experiments using pre-trained ADRNet models on different datasets. These experiments, detailed in Appendix A.6, demonstrate the effectiveness ADRNet in transfer learning tasks, such as adapting from the Navier-Stokes dataset to the SWE dataset.

---

Table 5: Moving MNIST.
Table 6: Results on KITTI.
Method
MSE 4 MAE +
Method
MS- (x10 3)
LPIPS (x10 3)
MSPred [58]
34,4
MAU [6]
27.6
t
t 3
t+1 1
t 3
PhyDNet [19]
24.4
70.3
SADM [2]
83.06
72.44
14.41
24.58
VimVP [53]
23.8
68.9
MCNET [59]
75.35
63.52
24.04
37.71
CrevNet [69]
22.3
Wise [15]
82.00
N/A
17.20
N/A
TAU [54]
19.8
60.3
OPT [66]
82.71
69.50
12.34
20.29
SwinLSTM [55]
17.7
DMVEN (w/o R) [23]
88.06
76.53
10.70
19.28
IAM4VP [46]
15.3
49.2
DMVEN [23]
88.53
78.01
10.74
19.27
ADRNet
16.1
50.3
ADRNet
85.86
83.62
9.26
Video Prediction Performance. We have used number of video datasets test ADRNet. The results of two of them (Moving MNIST and KITTI) are reported in Table 5 and Table We perform additional experiments for the KTH Action and TaxiBJ datasets in the appendix A. The moving MNIST dataset adheres to the assumptions of our ADRNet. Indeed, for this dataset, we obtain results that are very close to state-of-the-art methods.
The KTH Action dataset is more complex as not all frames can be predicted from the previous frames without generation power. Nonetheless, even for this dataset our ADRNet performs close to the state of the art. This limiting aspect video synthesis studied through experiments in appendix A.I.
Conclusion
In this paper, have presented network for tasks that reside regular mesh that be viewed as multi-channel image. The method combines standard convolutions with a linear operator that transports information from one part of the image another. The transportation vector field is learned from previous images (that is, history), allowing for information to pass from different parts of the image others without loss. We combine this information within diffusion-reactio process that can be coded by itself or by using a standard ResNet.
Acknowledgments
ME funded by the Blavatiik-Cambridge fellowship, the Cambridge Accelerate Programme for Scientific Discovery, and the Maths4DL EPSRC Programme.
10

---

References
[1] U.M. Ascher. Numerical methods for Evolutionary Differential Equations. SIAM, Philadelphia, 2010.
[2] Xinzhu Bei, Yanehao Yang, and Stefano Soatto. Learning semantic-aware dynamics for video prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 902-912, 2021.
[3] Jan Blechschmidt and Oliver G Ernst. Three ways to solve partial differential equations with neural networks-a review. GAMM-Mittillnneen. (4(2xc202100006 2021.
[4] Thomas Bohnstingl, Stanislaw Wožniak, Angeliki Pantazi, and Evangelos Eleftheriou. Online spatio-temporal learning in deep neural networks. IEEE Transactions Neural Networks and Learning Systems, 2022.
[5] Harold C Burger, Christian Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with bm3d? 2012 IEEE conference on computer vision and pattern recognition, pages 2392-2399. IEEE, 2012.
[6] Zheng Chang, Xinfeng Zhang, Shanshe Wang. Siwei Ma. Yan Ye, Xiang Xinguang, and Wen Gao. Mau: A motion-aware unit for video prediction and beyond. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 26950-26960 Curran Associates, Inc., 2021.
[7] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6571-6583, 2018.
[8] Özgün Çiçek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. Sebastien Ourselin, Leo Joskowicz, Mert R. Sabuncu, Gozde Unal, and William Wells, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2016, pages 424-432, Cham, 2016. Springer International Publishing.
[9] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[10] Moshe Eliasof, Eldad Haber, and Eran Treister. PDE-GCN: Novel architectures for graph neural networks motivated by partial differential equations. Advances in Neural Information Processing Systems, 34:3836-3849, 2021.
[11] Moshe Eliasof, Eldad Haber, and Eran Treister. Adr-gnn: Advection-diffusionrrraaion graph neural networks. arXiv preprint 3XXi:2307711092 2023.
[12] L. C. Evans. Partial Differential Equations. American Mathematical Society, San Francisco, 1998.
[13] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical in- teraction through video prediction. Advances in neural information processing systems, 29, 2016.
[14] J. Fohring, E. Haber, and L. Ruthutto. Geophysical imaging of fluid flow in porous media. SIAM J. on Sci. Comp, appar, 2014.
[15] Daniel Geng, Max Hamilton, and Andrew Owens. Comparing correspondences: Video pre- diction with correspondence-wiss losses. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3365-3376, 2022.
[16] G.H. Golub and C.F. Van Loan. Matrix Computations. Johns Hopkins University Press, 1983.

---

[17] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Farde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in neural information processing systems, pages 2672-2680, 2014.
[18) Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern recognition, 77:354-377, 2018.
[19] Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11471-1149, 2020.
[20] Eldad Haber, Keegan Lensink, Eran Treister, and Lars Ruthotto. Imexnet forward stable deep neural network. In International Conference on Machine Learning, pages 2525-2534. PMLR, 2019.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[22] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems, 31, 2018.
[23] Xiaotao Hu, Zhewei Huang. Ailin Huang, Jun Xu, and Shuchang Zhou. dynamic multi-scale voxel flow network for video prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6121-6131, 2023.
[24] Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1771-1779. PMLR, 2017.
[25] C.T. Kelley, Iterative Methods for Linear and Nonlinear Equations. SIAM, Philadelphia, 1995.
[26] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[27] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
(28) Keegan Lensink, Bas Peters, and Eldad Haber. Fully hyperbolic convolutional neural networks. Research in the Mathematical Sciences, 9(4):60, 2022.
[29] M. Lentine, T. Gretarsson, and R. Fedkiw. An Unconditionally Stable Fully Conservative Semi-Lagrangian Method. Journal of Computational Physics, 230:2857-2879, 2011.
[30] R.J. Vequee Numerical Methods for Conservation Laws. Birkhauser, 1990.
[31] Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Li. Moganet: Multi-order gated aggregation network. In The Twelfth International Conference on Learning Representations, 2023.
[32] Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. survey of convolutional neural networks: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems, 33(12):6999-7019 2021.
[33] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.08885 2020.
12

---

[34] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Newral Information Processing Systems, 33:6755-6766, 2020.
[35] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, arXiv preprint arXiv: 1.004440, 2015.
[36] Michael McCabe, Bruno Régaldo-Saing Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Fickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, al. Multiple physics pretraining for physical surrogate models. In NeurlPS 2023 Al for Science Workshop, 2023.
[37] J. Nagy and P.C. Hansen. Deblurring Images. SIAM, Philadelphia, 2006.
[38] Mare Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 716-731, 2018.
[39] Sergiu Oprea, Pablo Martinez-Gonzalee Alberto Garcia-Garcia John Alejandro Castro- Vargas, Sergio Orts-Escolano Jose Garcia- Rodriige,, and Antonis Argyros. A review on deep learn- ing techniques for video prediction. IEEE Transactions Pattern Analysis and Machine Intelligence, (4(6)) 2826, 2020.
[40] Keiron O'shea and Ryan Nash. An introduction to convolutional neural networks. arXiv preprint 15 2015.
[41] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitoo,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems, 2017.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505.04555 2015.
[43] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. arXiv preprint arXiv: 1804 04272, 2018
[44] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pages 1-13, 2019.
[45] Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: a local svm approach. In Proceedings of the 17th International Conference Pattern Recognition, 2004. ICPR 2004., volume 3, pages 32-36. IEEE, 2004.
[46] Minseok Seo, Hakjin Lee, Doyi Kim, and Junghoon Seo. Implicit stacked autoregressive model for video prediction. arXiv preprint arXiv:2303.078844 077.99709000..,... 2023.
[47] Junhong Shen, Liam Li, Lucio M. Dery. Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. ICML, 2023.
[48] Xiaoyu Shi, Zhaoyang Huang. Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung. Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12469-12480, 2023.
[49] Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung. Wai-kin Wong, and Wang-chun Woo. Convolutional Istm network: machine learning approach for precipitation nowcasting. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1, NIPS' 15, page 802-810, Cambridge, MA, USA, 2015. MIT Press.
[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint weXXv::409.15550 2014.
13

---

[51] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning, 2023.
[52] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1 196-11611,, 2022.
[53] Cheng Tan, Zhangyang Gao, Siyuan Li, and Stan Z Li. Simvp: Towards simple yet powerful spatiotemporal predictive learning. arXiv preprint arXiv:2211.12500 2022.
[54] Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-18782, 2023.
[55] Song Tang, Chuang Li, Pu Zhang, and RongNian Tang. Swinlstm: Improving spatiotempo- ral prediction accuracy using swin transformer and Istm. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13470-13479, 2023.
[56] Isabel Urbich, Jörg Bendix, and Richard Müller. A novel approach for the short-term forecast of the effective cloud albedo. Remote Sensing, 10(6))955, 2018.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] Angel Villar-Corrales Ani Karapetyan, Andreas Boltres, and Sven Behnke. Mspred: Video prediction at multiple spatio-temporal scales with hierarchical recurrent networks. In British Machine Vision Conference, 2022.
[59] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations, 2017.
[60] Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. PredRNN++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 5123-5132. PMLR, 10-15 Jul 2018.
[61] Yunbo Wang, Lu Jiang. Ming- Hsuan Yang. Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d LSTM: A model for video prediction and beyond. In International Conference on Learning Representations, 2019.
[62] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[63] Yunbo Wang. Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9154-91622 2019.
[64] Henglai Wei, Xiaochuan Yin, and Penghong Lin. Novel video prediction for large-scale scene using optical flow. arXiv preprint 1805. 12243,, 2018
[65] Christopher K Wikle Comparison of deep neural networks and deep hierarchical models for spatio-temporal data. Journal of Agricultural, Biological and Environmental Statisties, 14(2):155-003 2019.
14

---

[66] Yue Wu, Qiang Wen, and Qifeng Chen. Optimizing video prediction via video frame interpola- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 17814-17823, 2022.
[67] Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2364-2373, 2018.
[68] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
[69] Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and information inforenation-prseevving future frame prediction and beyond. In International Conference Learning Representations, 2020.
[70] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
15

---

A Ablation Studies and Additional Experiments
A.1 Limited Generative Synthesis
Two real-life video datasets are taken to predict future time frames. Their statistics can be found in Table The specific challenge posed by these datasets is due to the dissimilarity the train and test sets. This is evident from the notable difference in the training and validation/test losses, which can be seen in Figure The validation loss starts increasing with more epochs. For example, the KTH Action uses the movement behavior of 16 people for training while the models are tested on the movement behavior of 9 other people slightly altered scenario. So, we can say that the problem is to learn the general logic to predict unseen scenarios. Thus generative capability of a model could be crucial for better prediction.
Thamnng wilidriinn
Tarrnng
wiliderion
los (na00
KTH Action
TaxiBJ
Figure 7: Bias in training and testing samples in KTH Action and TaxiBJ datasets
KTH Action The KTH dataset features 25 individuals executing six types of actions: walking, jogging, running, boxing, hand waving, and hand clapping. Following methodologies established in references [59, 61], we utilize individuals 1-16 for training and individuals 17-25 for testing. The models are trained to predict the subsequent frames based on the preceding 10 observations.
TaxiBJ TaxiBJ is collection of real-world GPS spatiotemporal data of taxis recorded as frames of 32x32x2 heat maps every half an hour, quantifying traffic flow in Beijing. We split the whole dataset into a training set and test set as described in [70]. We train the networks to predict 4 future time frames from 4 observations.
Results Our model is easily able to predict and outperform the saate-of-the-an models in real-life video examples as well. The results for KTH Action and TaxiBJ can be seen in and 9.
Table 7: Additional Dataset Statisties: Details Resolutions
Training and Testing, Image Sequences, and
Dataset
Nunin
Nies
(C, H. W)
History Prediction
KTH Action
5200
3167
(1, 128, 128)
10
20
TaxiBJ
19627
1334
(2, 32, 32)
4
A.2 CloudCast
The CloudCast dataset used for multiple long-range predictions like 4, 8, 12, and 16 timesteps. can be noticed that even if the MSE or the quality degrades, the degradation is noticeably minimal. It can be seen in Table 10, the figures for predicting 16 steps in future still better than the state of art for 4 steps in future.
16

---

Table 8: Comparison of Our Method for KTH Action Dataset.
Method
SSIM
PSNR (dB) 1
ConVLSTM [49]
0.712
23.58
PredRNN [62]
0.839
27.55
CausalLSTM [60]
0.865
28.47
MSPred [58]
0.930
28.93
E3D-LSTM [61]
0.879
29.31
SimVP [53]
0.905
33.72
TAU [54]
0.911
34.13
SwinLSTM [55]
0.903
34.34
ADRNet
0.808
31.58
Table 9: Comparison of Our Method for TaxiBJ Dataset.
Method
MSE L
MAE +
SSIM t
ST-ResNet [70]
0.616
VPN [24]
0.585
ConvLSTM[49]
0,485
17.7
0.978
FRNN [38]
0.482
PredRNN [62]
0,464
17.1
0.971
CausalL.TTy LSMM [60]
0.448
16.9
0.977
MIM [63]
0.429
16.6
0.971
E3D-LSTM [61]
0.432
16.9
0.979
PhyDNet [19]
0.419
16.2
0.982
SimVP [53]
0.414
16.2
0.982
SWinLSTM [55]
0.390
0.980
IAM4VP [46]
0.372
16.4
0.983
TAU [54]
0.344
15.6
0.983
ADRNet
0.445
16.6
0.975
Table 10: Results for CloudCast dataset, Comparison of our model (ADRNet) with state of art models
ADRNet Predictive performance
Metric
t+4
t 8
t 12
t 16
MSE (L)
0.015
0.016
0.018
0.019
SSIM (1)
0.83
0.79
0.76
0.74
PSNR (1)
38.17
37.89
37.35
37.23
A.3 Navier-Stokes Dataset
To further demonstrate the effectiveness of our ADRNet scientific data, conduct experiments on an additional dataset from PDEbench, specifically the Navier- -Sokkrr-Stokss equations. This large dataset consists of 21,000 images, each with resolution of 5 112 Our ADRNet ranks second among various methods, positioning it in line with state-of-the- approaches, as shown in Table 11 and Figure 8. Additional metrics this dataset obtained with UNet, FNO, and our ADRNet are reported Table 12.
Table 11: Comparison of methods based normalized Mean Squared Error (nMSH). Our ADRNet achieves competitive performance, ranking second among state-of-the-art methods.
Method
UNet
FNO
MPP-AViT-TI MPP-AViT-S MPP-AVIT-B MPP-AViT-L
ADRNet (Ours)
MSSE
1.67
0.243
0.0312
0.0213
0.0172
0.0142
0.0168
17

---

Va.
Vy Reference
Vy Prediction
Vy Emor
10
12
10
56
-10
40
Figure 8:
Visualization of ADRNet on Slaier-Stokes dataset from PDEBench. Left column: ground- truth velocity maps. Middle column: ADRNet prediction velocity maps prediction. Right Column: Error between ground-truth and prediction.
Table 12: Comparison of different metrics UNet, FNO, and our ADRNet, on the PDEBench Navier-Stokes Inviscid Compressible at M = 0.1 with Turbulent Initial Condition.
Metric
UNet
FNO
ADRNet (Ours)
RMSE
3.3 x 10-1
2.8 x 10
1.2676 x 10-1
mRMSE
1.9 x 10-1
1.6 x 10-1
1.53 x 10-2
Max Error
2.2 100
1.8 10
6.8378 10 10-1
cRMSE
1.5 x 10-2
1,2 x 10
4.0903 x 10 10-3
bRMSE
3.6 x 10-1
2.8 x 10
8.3314 x 10-2
FRMSE (low)
6.5 10-2
5.0 10
1.4105
10
fRMSE (mid)
3.2 10
3.1 X 10
2.6598
10
IRMSE (high)
8.5 X 10-3
6.5 x 10
1.7822 x 10
A.4 Order of Operator Splitting
We now show that numerically, the network computations are agnostic to the order of operator splitting. Following that, we perform an experiment to verify our theoretical derivation.
Numerical Behavior. The following derivation demonstrates the validity of the operator split- ting approach used, as detailed below. In numerical PDEs, for an initial value problem (IVP) dæ dt Ax, the solution is x(t) exp(tA)x(O)). For dr/dt
Ar + Bx, the solution is axt) exp(t(A+++ + (0)) If matrix exponentials were treated as scalars, the solution would be :(t) exp(tA) (0) implying that the order of operations (reaction-diffusion and advection) is invariant.
To analyze this, review matrix exponentials. The matrix exponential exp(A) is defined by:
exp(A)
k!
A==
For the sum of matrices A and B, the expansion is:
exp(t(A + B)) = 1 +(A B) 0.5t2 (A2 B + AB + BA) O(P)
For the product of two matrix exponentials:
exp(tA) exp(tB) (I tA 0.5t 42 O(t2) (1+ +
18

---

Generally, for matrices A and B, AB * BA unless they share eigenvectors. Expanding Equation (2) and collecting terms, we get:
exp(tA) exp(ttB) I 0.51 (A B² 2AB) +0(1)
Comparing this with:
exp(t(A B)) = + 1. 1B 0.5 B AB + BA) + O(t")
find the approximation enror is O
exp(LA) exp(IB) - exp(1 (A + B)) = 0.5l 0.5 - BA) +(())
This error depends on how AB differs from B.A.
Regarding the sequence of operations, changing the order of A and B yields:
exp(t(A + B)) = exp(tA) exp(tB) + 0( = exp(tB) exp(tA) 0(t²)
Thus, the order of operations (advection vs. reaction-diffusion) does not fundamentally affect numerical accuracy.
Drffttion-iiffuioon--- Reaction (ADR) vs. Diffusion-Reactin DrffusionnReaciion-Advvetiiooon (DRA). To verify our understanding of the numerical behavior of the order of splitting, we conduct an experiment that compares two possible orders of operations: advection followed by reaction-diffusion, and vice versa, on the Moving MNIST dataset. Figure 9 shows that the convergence plots and obtained predictions for the two possible orderings are similar. addition, we report the obtained test set performance on the Moving MNIST dataset obtained with the two considered variants (ADRNet and DRANet) in Table 13.
ADR
DRA
10"-
Loss
101-
Epochs
8
8
Figure 9: Convergence plot comparing the order of operations: ADR vs. DRA, along with examples of predictions made by the models (from left to right, respectively). The different order of layers in the ADR model yields similar results, consistent with our theoretical analysis.
19

---

A.5 Advection vs. Dilated Convolutions
Dilated convolutions are known to be mechanism that allows wide field of view [68], which is also obtained by our Advection operator. To demonstrate the benefit of our advection operator, we consider two possible uses of dilated convolutions: (i) Using dilation instead of the diffusion mechanism, combined with our advection operator, and (ii) using dilation instead of advection. We compare their performance Moving MNIST dataset, which requires the ability to transport information across distant pixels. Our results are presented in Table 14, and visualized in Figure 10. As can be seen, the performance obtained when utilizing the advection operator is significantly improved, both in terms of training convergence and the obtained downstream performance, highlighting importance of advection in tasks that require long-range transportation of features.
10a
Loss
10-2
No dilation with Advection
10-3
Dilation with Advection
Dilation without Advection
20
40
60
80
100
Epochs
Figure 10: Comparison of ADRNet vs. using dilated convolutions. Blue: ADRNet. Orange: Using dilation to implement diffusion, coupled with the advection operator. Green: Using dilation place of advection, with standard convolutions for reaction-difflsiinn ADRNet demonstrates superior convergence.
Method
MSE (4)
MAE (4)
ADRNet
16.1
50.3
DRANet
16.2
50.3
Table 13: Moving MNIST Test set performance with ADR vs. DRA operator splitting ordering. Both cases yield similar results.
Method
MSE (1)
MAE (L)
Dilation with Advection
16.6
51.1
Dilation without Advection
25.7
72.8
ADRNet
16.1
50.3
Table 14: A comparison two possible uses of Dilated Convolutions vs. our ADRNet, on the Moving MNIST test set.
20

---

A.6 Generalization and Transferability
Our ADRNet can be classified as tion equipped with implicit bias and mathematically grounded behavior. It is therefore, interesting study whether training ADRNet on one dataset and task can be useful for a different dataset and task. To this end, in Figure 11 illustrates the performance of ADRNet on the SWE dataset, leveraging pre-trained model initially trained on the Navier-Stokes dataset from PDEBench. Our results indicate that using pre-trained ADRNet, even with minimal fine-tuning (a single linear adaptation layer), significantly improves predictive accuracy compared to a randomly initialized model. This outcome highlights the adaptability of ADRNet and generalization capability across scientific datasets, showcasing its potential for tasks requiring domain transfer.
neural network [43, 10]. Thus, it by construc-
sathematicalyy-grondeed
Ground-Trutt Truth
Error Prediction
Randomly
Pre-trained PORNet fine-tuning
Pre-trained ADRNet with fine-tuning
initialized
ADRNet
Figure 11: Results on SWE dataset using ADRNet pre-trained on the Navier-Stokes dataset. Using a pre-trained ADRNet is beneficial both with and without fine-tuning single linear adaptation layer compared with a randomly initialized ADRNet.
A.7 Illustration of the Learned Velocity Fields
In Figure 12, we illustrate the learned advection field and attention maps. The obtained velocity fields and their application confirm the concept described Figure and Figure 2 in the paper. We note that the advection operator works on all channels in the embedded space. For the example at hand (Moving MNIST), we have 64 channels and 5 convolution layers, that blend 10 previous time steps given the input. To generate the figure, we inspect one of the channels across all layers, and plot a quiver plot of advection field. This quiver plot shows the direction which the advection guided to solve the task in the moving MNIST dataset. In addition, advection field. The absolute value of the advection field is equivalent to an attention map. because shows the areas in the original image that move the most to generate the final image. Including this figure allows us to visualize which areas in the input need to be moved to obtain the desired target. As can be seen from Figure 5, the pixels that correspond the digits in the input images are the that obtain larger values displacement in the learned advection field this result is in accordance with the concept of learning advection fields, done in our ADRNet.
plot the absolute value of the
A.8 Runtimes
In Table 15, we provide a runtimes comparison of ResNet and our ADRNer. The runtimes were measured using NVIDIA RTX-A6000 GPU, with ADRNet and ResNet, both with 32 hidden
21

---

18
8
8
8
8
3
8
8
8
Figure 12: Top Row: 10 input time-steps to ADRNet. Middle Row: Quiver maps of the learned advection fields. Bottom Row: Attention maps are defined by the magnitude of the advection field at each pixel. The advection fields and attention maps are in layers O..... 3. from left to right.
dimensions, 10 input features, output feature, batch size of 32, and 2 layers, a varying image size input. The results show that while our ADRNet requires more runtimes than standard CNNs, it maintains reasonable computational cost.
Image Size
32x32
64x64
128x 128
256x256
ResNet
0.6
20.3 10.4
79.2 / 45.8
192.8 / 58.2
ADRNet (Ours)
12.8/3..
53.3 19.1
175.8 / 72.8
484.3 / 179.1
Table 15: Runtimes (training/inference) comparison of ResNet and our ADRNet, in milliseconds.
22

---

B Evaluation Metrics
Moving MNIST, KTH Action, TaxiBJ, CloudCast These specific video prediction datasets have been using MAE (Mean Absolute Error), MSE (Mean Squared Error), SSIM (Structural Similarity) and PSNR (Peak Signal-to-Nor Egnoiee Ratio). The evaluated SSIM and PSNR are averaged over each image. The MSE and MAE have specific way to calculate, where the pixel-wise evaluation values are summed up for all the pixels in the image.
N
H
W
C
MSE
y 0)
(16)
N
W
MAE ΣEΣ
y 01
(17)
PSNR 1 Σ 10 log10
MAX²
(18)
N 1=1
MSE
(2 -Ci)(2aay + C2)
SSIM(x.y)
(19)
(n² + 12 + C1)(o? + o C2)
SSIM 1
Σ SSIM(x,))
(20)
N
where:
is the number of images in the dataset,
H is the height of the images,
W is the width of the images,
C is the number of channels (e.g., for RGB images),
y is the true pixel value at position (i, h, w, c), and
ÿ is the predicted pixel value position (i, h, w. c).
MAX is the maximum possible pixel value of the image (e.g.. 255 for an 8-bit image),
MSE is the Mean Squared Error between the original and compressed image.
Pr is the average x,
My is the average of y,
of is the variance of x;
y is the variance of y,
Dry is the covariance and y,
G (K,L)² and C2 L22 are two variables to stabilize the division with weak denominator,
L is the dynamic range of the pixel values (typically, this is 255 for 8-bit images),
K1 and K2 are small constants (typically, Ki = 0.01 K₂ = 0.03).
SDEE PDEBench uses the concept of pixel-wise mean squared error (MSE) and normalized mean squared eгror (nMSE) validate scaled variables in simulated PDEs. Along with these, we also use root mean squared error (RMSE) and normalized root mean squared error (nRMSE).
W ΣΣΣΣ
N N
1
MSE
xx xx-1²
(21)
N. H W
C
h=
23

---

N H
W
2)2
nMSE
(22)
N. H. W
x²
W ΣΣLΣ
N H
RMSE
8)2
(23)
N
H W. C
m cml
1
ΣΣΣΣ
N
W
$)
nRMSE
(24)
N
H.W.C C KITE
x²
where:
N is the number of images in the dataset,
H is the height of the images,
W is the width the images,
C is the number of channels (e.g., 3 for RGB images),
x is the true pixel value at position (n, h, w, c), and
is the predicted pixel value at position (n, h, c)..
KITTI
MS MS-SSIM(2,) (2, 1o
II
(2, w) [5,(2, v)
(25)
j=1
N
MS-SSIM
MS-SSIM(I,)) v)
(26)
where:
N is the number of images in the dataset,
las (a,, y) the luminance comparison at the coarsest scale M,
Cy (ar, y) is the contrast comparison at scale j,
(a, y) the structure comparison at scale j,
αM, 3j. 7i are the weights applied to the luminance, contrast, and structure terms at each scale respectively,
M is the number of scales used the comparison.
The luminance, contrast, and structure comparisons given by:
24 + C1
(x,,v) v)
1h + 11 + Ci
20,0, + C2
c(x, v)
0 + o C2
s(x, v) = xzy + C33
o20y Css
where:
Pes My are the local means of x and y,
Tx Fy are the local standard deviations of and y.
Day is the local covariance of x and y,
C1, Cz, C33 are constants stabilize the division.
Wi
LPIPS(x; x)
Σ H
ΣΣ
mm c (d(()) (t))
(27)
24

---

LPIPS
LPIPS (2IPS(x,)) a)
(28)
where:
N is the number of images in the dataset,
(x(x) is the activation the /-th layer deep network for the image x, (#) is the activation of the 1-th layer of deep network for the image i;
wi is a learned weight vector for the I-th layer,
Hi and W, are the height and width of the 1-th layer activations,
c denotes element- wise multiplication.
C
Hyperparameter Settings and Computational Resources
C.1 ADRNet Training PDEBench-SWE
Hyperparameter
Symbol
Value
Learning Rate
Batch Size
B
64
Number of Epochs
N
200
Optimizer
Adam
Number of Layers
1
Hidden Channels
128
Activation Function
SiLU
Table 16: Neural Network Hyperparameters
C.2 ADRNet Training on Other Datasets
Hyperparameter
Symbol
Value
Learning Rate
2e 06
Batch Size
16
Number of Epochs Optimizer
N
1000
Adam
Number of Layers
8
Hidden Channels
192
Activation Function
SiLU
Learning Rate Scheduler
ExponentialLR
Table 17: Neural Network Hyperparameters
C.3 Computational Resources
All our experiments are conducted using an NVIDIA RTX-A6000 GPU with 48GB of memory.
25

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The contribution paragraph in the introduction (Section 1) explicitly describes the main contributions of the paper. The claims are supported by theory (Section 3) and experiments (Section 4). The results show our ADRNet exceptionally beating state-of-the-art methods (Section 4.2) spatiotemporal prediction with wide variety of datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The proposed method is very much generalizable for spatiotemporal predic- tions. Yet, the evaluation section (Section 4) and appendix (Appendix A. include separate paragraphs discussing the limiting generative synthesis with experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Section presents our main theoretical motivation, and it contains the full set assumptions in devising our ADRNet. However, no theoretical result, theorem or lemma is proposed.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 3 fully describes our proposed method, and Appendix A contains experimental details to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
27

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We offer comprchensive details regarding the implementation and evalua- tion our ADRNet (Appendix A), and our code is available at https::/g/t //tthubbtttttt com/ wecchatth-Rout/desppllll
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cublic/guides/ hholli//guides//ddettttiiiss ceriiisssiooiiiiiy cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //niss. cctpublic/guieeeeeccccccccocccce /tte//gueeeeeeeeemmmiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 and Appendix C.1 contains all the details, including the hyperparam- eters and other experimental choices.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
We follow the same report practice in preceding papers in the field, which also do not report the standard deviation their results. To ensure faireess, we still ran our model 5 different seeds and reported the average result.
Justification:
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix C.3 describes the type of GPUs used in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttts////orrpps...p/i////tttii
Answer: [Yes]
Justification:
The research conforms with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiciio..
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work fundamental and not tied to any particular application. Hence, there is no associated socictal impact.
29

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have credited the owners of the assets and the licenses in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "ROvE1Z4h1z", "reviewer": "Reviewer_2i2R", "summary": "This study addresses spatio-temporal prediction problems in the physical sciences. Since existing methods for temporal prediction mechanisms based on convolutional neural networks (CNNs) often underperform when propagating long-range information, the authors proposed a physics-inspired architecture that mimics the reaction-advection-diffusion equation in high dimensions.", "questions": "- The order of reaction, diffusion, and advection seems to be crucial for the design of the neural network. In Fig. 2, the process is shown in the order advection-diffusion-reaction, while the proposed operator splitting processes the data in the order reaction-diffusion-advection. Is there a particular reason for adopting this sequence in operator splitting? In addition, it would be necessary to empirically investigate whether this method can be generalized to various scientific datasets that do not follow this sequence of processes.\n- In Table.6, DMVFN [21] is duplicated. Please check it.", "limitations": "See the weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation of the proposed algorithm is straightforward.\n- The idea of operator splits to construct a neural network from the advection-diffusion-reaction differential equation is interesting.", "weaknesses": "- The authors need to include many essential steps so that readers can fully understand the proposed idea. For example, the approximation described in eq. (8) needs to be explained in detail; the authors merely refer the readers to [1]. This approximation is a crucial process that enables operator splitting, and therefore, a detailed analysis should at least be included in the appendix.\n- I wonder why the diffusion and the reaction can be combined in a 3$\\times$3 convolution layer. I agree that the reaction step can be implemented with a nonlinear 1$\\times$1 convolution layer. However, for a diffusion step, as shown in Fig. 2, the dilated convolution is better suited to describe the diffusion step. \n- There is no quantitative comparison between the proposed ADRNet and other algorithms. Even though Fig. 5 illustrates the results of ADRNet, these frames are not suitable for demonstrating spatio-temporal prediction as there are no significant transitions between temporal frames.\n- As can be seen in Table. 8 and Table. 9, ADRNet performs poorly compared to the other algorithms and even shows quite a gap to SOTA performance, which raises doubts that the proposed ADRNet can consistently address the spatio-temporal prediction problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank you for the detailed and constructive feedback. We are glad you found our submission well-motivated and our approach interesting. We have addressed all your comments and suggestions and are open to further discussion. We hope you find our responses satisfactory and consider revising your score accordingly.**\n\n**Regarding W1&W4 (Derivations, and order of reaction-diffusion-advection)**:  \n\nDue to space limitations, we initially omitted the derivation for Equation (8) and referred readers to [1]. However, we have now included the derivation in the revised paper.\n\nOur derivation demonstrates the validity of the operator splitting approach used, as detailed below:\n\nIn numerical PDEs, for an initial value problem (IVP)  $dx/dt = Ax$, the solution is $x(t) = exp(tA)x(0)$. For $dx/dt = Ax + Bx$, the solution is $x(t) = exp(t(A+B))x(0)$. If matrix exponentials were treated as scalars, the solution would be $x(t) = exp(tA)exp(tB)x(0)$, implying that the order of operations (reaction-diffusion and advection) is irrelevant.\n\nUnfortunately this analysis does not work for matrices. To analyze this, we review matrix exponentials. The matrix exponential $exp(A)$ is defined by:\n\n$$exp(A) = \\sum_{k=0}^{\\infty} \\frac{1}{k!} A^k $$\n\nFor the sum of matrices $A$  and $B$, the expansion, using Taylor's theorem, is:\n\n$$exp(t(A+B)) = I + t(A+B) + 0.5t^2(A^2 + B^2 + AB + BA) + O(t^3) \\quad (1)$$ \n\nFor the product of two matrix exponentials:\n\n$$ exp(tA) \\cdot exp(tB) = (I + tA + 0.5t^2A^2 + O(t^3)) \\cdot (I + tB + 0.5t^2B^2 + O(t^3)) \\quad (2)$$ \n\n\n\n**Remark 1**: Generally, for matrices $A$ and $B$, $AB \\neq BA$ unless they share eigenvectors.\n\nExpanding Equation (2) and collecting terms, we get:\n\n$$ exp(tA) \\cdot exp(tB) = I + tA + tB + 0.5t^2(A^2 + B^2 + 2AB) + O(t^3) $$\n\nComparing this with Equation (1), we find the approximation error is \\( O(t^2) \\):\n\n$$ exp(tA) \\cdot exp(tB) - exp(t(A+B)) = 0.5t^2(AB - BA) + O(t^3) $$\n\nThis error depends on how $AB$ differs from $BA$.\n\n**Regarding the order of operations**: Changing the order of $A$ and $B$ yields:\n\n$$exp(t(A+B)) = exp(tA) \\cdot \\exp(tB) + O(t^2) = exp(tB) \\cdot exp(tA) + O(t^2)$$\n\nThus, the order of operations (advection vs. reaction-diffusion) does not fundamentally affect numerical accuracy with order of $O(t^2)$.\n\n\nEncouraged by your question, we added an experiment comparing two orders of operations: advection followed by reaction-diffusion, and vice versa. Training convergence and sample prediction results on the moving MNIST dataset, shown in **Figure 4 of the attached PDF**, are similar. These results support our earlier analysis. We appreciate your insightful comments and have included this analysis, results, and discussion in the revised paper.\n\n**Regarding W2 (reaction-diffusion and dilated convolution)**: \nThank you for the important discussion. We address it in two parts: first, why diffusion and reaction can be combined into a 3x3 convolution; second, using dilated convolution for diffusion as suggested.\n\nCombining diffusion and reaction into a single step stems from solving a PDE with linear reaction. For a 1D diffusion-reaction PDE:\n\n$$I_{xx} + r I = q$$\n\nIn Fourier space:\n\n$$\\omega^2 \\hat{I} + r \\hat{I} = \\hat{q}$$\n\nThus, $I = F^{-1}\\left(\\frac{1}{\\omega^2 + r} \\hat{q}\\right)$. The filter $\\frac{1}{\\omega^2 + r} $ combines diffusion and reaction. Implementing this requires Fourier or Cosine transforms, which can be computationally expensive. A simpler approximation is a 3x1 kernel, like 1/4*[1,2,1] in 1D, used in computer vision and image processing (e.g., Nagy and Hansen, reference [35]).\nWe have added this discussion to our revised paper.\n\n**Regarding the use of dilated convolutions for diffusion**, we agree with the reviewer, that it is another option to implement diffusion and have added relevant experiments. Specifically, we tested: (i) ADRNet with dilated convolutions for diffusion versus standard convolutions, and (ii) dilated convolutions for advection while using standard convolutions for reaction-diffusion, as suggested by Reviewer zzNS.\n\nThe convergence plots in Figure 3 of the attached PDF show that our ADRNet implementation outperforms dilated convolutions on the moving MNIST dataset. We also found dilated convolutions more effective for diffusion than for advection. These results and discussions have been included in the revised paper. Thank you.\n\n**Regarding W3&W5 (comparison with other methods and additional data)**: \n\nWe would like to kindly ask the reviewer to refer to our **global response for detailed information regarding our extensive experiments**. Briefly, *we conducted over 8 quantitative comparisons across 6 diverse datasets (including synthetic, scientific, atmospheric, natural images, real-world videos, and GPS spatio-temporal data), benchmarked against multiple SOTA methods*. Our results, discussed comprehensively in the appendix, highlight both the strengths and limitations of ADRNet. \n\nRegarding Tables 8 and 9, we present these results in Appendix A to highlight the limitations of ADRNet and identify scenarios where it may not achieve state-of-the-art performance. This rigorous evaluation is crucial in scientific studies, including Machine Learning. Our results show that ADRNet performs comparably or better than many recent methods on the main datasets and works well on two additional datasets, despite not being state-of-the-art. We hope you appreciate our thorough analysis of ADRNet's strengths and limitations, which we have further enhanced in the revised paper.\n\nTo further demonstrate ADRNet on scientific data, as suggested by you in W5, we have added results from the Navier-Stokes equations dataset from PDEbench. These results are included in our general response to all reviewers.\n\n**Regarding W6 (Table 6):**\n\nThanks for noting this error. In DMVFN [21], two versions of the method were proposed. It was a typo on our side not to include the specific versions. We have fixed the table accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nVnYnBRjzw", "reviewer": "Reviewer_Lq8m", "summary": "This paper proposes a method to enhance the performance of networks in solving PDE problems using CNNs. To achieve this, the authors present a computational approach for efficiently retrieving information from distant points in spatio-temporal datasets. Their main contributions can be summarized as follows: 1) They propose an interpretation method for information using the advection-diffusion-reaction step. To implement this accurately, 2) they introduce a semi-Lagrangian linear layer. 3) They present a training approach for these components. Despite minor issues such as experimental validation, computational efficiency, or the readability of figures, this paper is well-structured mathematically and demonstrates high performance in the field.", "questions": "Check Weakness", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The interpretative range of information in a CNN, considered a function, is limited to its receptive field. This limitation makes it challenging to interpret information outside of this domain. The paper addresses this issue by proposing the Advection-Diffusion-Reaction approach, significantly improving performance.\n\n2. The equations proposed in the paper are clear regarding variable declarations and usage, and they are written in a way that is easy to follow. While aspects like rigor may require a more detailed review, there appear to be no major issues based on my examination. The paper effectively formalizes the challenges faced by existing networks and provides solutions to address these issues.", "weaknesses": "1. The most notable drawback is the absence of visualizations, such as attention maps, to confirm that the concept shown in Figures 1 and 2 operates as expected at each feature level, even in toy examples. While the network demonstrates significant performance improvements, experiments are needed to verify whether the concept functions correctly compared to the computational complexity involved. In particular, it would be impressive if a pre-trained model could observe the proposed problem of long-range information being advected and diffused.\n\n2. Overall, there is a need to enhance readability, including improving the concept figures. Specifically, Figures 2 and 5 require better clarity. Additionally, the graph in Figure 1-(c) could benefit from readability improvements. The absence of source code makes it difficult to understand how each step was implemented, which adds to the difficulty of reading and evaluating the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the Reviewer for the overall positive assessment of our paper. In particular, we are delighted to read that the Reviewer finds that our paper addresses existing interpretability issues in CNNs,  that the paper is clear and easy to follow, and that our ADRNet demonstrates high performance. We are also very grateful for your constructive feedback, which we made great efforts to address during the rebuttal period. We also found your feedback to be highly insightful and valuable, and it allowed us to improve our paper. Below, we provide our responses to each of your queries and suggestions. We hope that you find them satisfactory and that you will consider revising your score accordingly. We are also happy to discuss any additional comments or questions you may have.**\n\n**Regarding W1 (Visualizations):**\n\nWe thank the reviewer for their suggestions. We have now added figures of the learned advection field and attention maps (please see **Figure 5 in the attached rebuttal PDF**). These figures confirm the concept described in Figures 1 and 2 in the paper. We note that the advection works on all channels in the embedded space. For the example at hand (moving MNIST), we have 64 channels and 5 convolution layers, that blend 10 previous time steps given as the input. In the added Figure (Figure 5 in the attached PDF)  we inspect one of the channels across all layers, and plot a quiver plot of the advection field. This quiver plot shows the direction in which the advection is guided to solve the task in the moving MNIST dataset.\nIn addition, we plot the absolute value of the advection field. The absolute value of the advection field is equivalent to an attention map, because it shows the areas in the original image that move the most to generate the final image. Including this figure allows us to visualize which areas in the input need to be moved to obtain the desired target. As can be seen from Figure 5, the pixels that correspond to the digits in the input images (10 time channels) are the ones that obtain larger values of displacement in the learned advection field – this result is in accordance with the concept of learning advection fields, as done in our ADRNet. \n\nWe would like to thank the reviewer for the important suggestions, and we added the figures suggested by the reviewer to our revised paper. \n\n**Regarding W2 (Pre-trained model):**\n\nWe would like to thank the reviewer for the insightful proposal of using a pre-trained model. We believe that such a success can improve the acceptability of our approach by the ML community. We have therefore added an experiment to test your proposal.\nBefore moving to the description of the added experiment, we would like to kindly note that the computational complexity of our method, despite adding the advection term, is reasonable considering the fact that it effectively adds another layer to the network compared to a standard off-the-shelf Convolutional ResNet. Furthermore, note that some problems (e.g., our synthetic problem of moving a pixel from one corner of the image to another, cannot be solved effectively using a standard Convolutional ResNet, as shown in Figure 1 in our paper).\nIn particular, we provide a table with measured training and inference runtimes for varying image sizes in {32x32,64x64,128x128,256x256}. The runtimes are provided below:  \n\n| Image Size    | 32                             | 64                               | 128                               | 256                               |   |\n|---------------|--------------------------------|----------------------------------|-----------------------------------|-----------------------------------|---|\n| ResNet        | Train:4.2ms / Inference:0.6ms  | Train:20.3ms / Inference: 10.4ms | Train:79.2ms / Inference: 45.8ms  | Train:192.8ms / Inference:58.2ms  |   |\n| ADRNet (Ours) | Train:12.8ms / Inference:3.7ms | Train:53.3ms / Inference:19.1ms  |  Train:175.8ms / Inference:72.8ms | Train:484.3ms / Inference:179.1ms |   |\n\nThe runtimes were measured on an ADRNet and a ResNet,  both with 32 hidden dimensions, 10 input features, 1 output feature, batch size of 32, and 2 layers. We also added them to our revised paper. \n\n\nWith regards to showing **results on a pre-trained model**, we have now experimented with a new, larger dataset from the PDEbench suite of datasets. Namely, we used the Navier Stokes (NS) equations dataset. We report the results of this experiment in our general response to all Reviewers.\n\n**Given this pre-trained ADRNet for the Navier-Stokes dataset**, we now **adopt it to predict the SWE system**, which is also studied (from scratch) in our paper in Table 2. Because the number of inputs and outputs is different between the NS and SWE datasets, we trained a **single linear layer followed by a SiLU activation** and **used the output of the ADRNet trained on NS data to predict the solution for SWE**. As can be observed in **Figure 2 in the attached PDF**, as well as in the results reported in the Table below, the results obtained with this pre-trained model are better than using a random ADR model, and given that only a single layer was trained, are reasonably well compared to the training the complete network from scratch. These results indicate that the network can be utilized to generalize the advection process which is known to appear in the NS equations as well as the SWE problem. We added these results and discussions to our revised paper. Thank you for your suggestion, as it truly helped us to improve the paper.\n\n**Regarding W3 (Readability and Clarity):**\n\nThank you for the concrete suggestions. We have revised the caption text in the figures and visually improved them (for example, we changed the font size in Figure 2 in the paper, which we agree was hard to read).  Regarding source code, as promised in our submission checklist and main paper text, upon acceptance, we will publicly share our code on GitHub. We confirm that this is indeed the case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P8xACfyEPP", "reviewer": "Reviewer_zzNS", "summary": "This work is based on an analogy of CNNs to Advection-Diffusion-Reaction PDEs: Classical ResNets can be understood as pure Diffusion-Reaction PDEs, without an Advection Term. Thus, this work proposes to introduce a new layer resembling the Advection Term, to allow for long-range transportation of information. More specifically, they propose to learn the velocity fields, with which to advect the coordinates with a CNN. Advected quantities can then be computed by simply moving the quantities on the old grid to the new coordinates, and then interpolating back to the old grid (which they propose to do in a mass-conserving way, i.e. resembling conservative semi-lagrangian advection). Leveraging this new advection layer, the authors find performance on par with state-of-the-art approaches on a few benchmark datasets from PDE emulation, weather radar forecasting and video prediction. In addition they find indication, that the added advection layer can improve direct longer horizon forecasts for the Shallow Water Equation.", "questions": "- How about dilated Convolutions? The recent GraphCast paper introduces long-range connections in their multimesh to solve the issue of propagating information over longer distances. The equivalent for CNNs would be dilated convolutions, that is, the filters contain holes/zeros, and thus allow for propagating information from further away without heavy oversmoothing. It would be cool to see an ablation study against this.\n- A common application of models, like the PDE emulators in the study, is to use them in long-term autoregressive rollouts. Standard CNNs are often unstable over such longer rollouts. It would be super interesting to see how the ADRNet does in such a setting.\n- Would have been great if this submission would have had a code repo, i would have liked to try the network myself.", "limitations": "The study of limitation is very short and only existing in the survey. Please discuss in the main text, do you observe any artifacts from introducing the advection layer? (e.g. how about at the image boundaries?) How well does training go, is it easier or harder to train?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Introduces a new neural network layer that mimics semi-lagrangian advection\n2. Shows on multiple common (and not too easy) benchmarks against strong baselines that the derived ADRNet architecture is competitive\n3. Paper is reasonably well written, with some less-important details moved into the appendix.", "weaknesses": "Major points:\n1. If the CFL conditions are fulfilled, then Advection should be local. In other words, if the time step is small enough (compared to velocity), advection only moves quantities to neighboring pixels. Then, a regular CNN with local filters should be able to capture advection. Hence, i'd expect your layer to be most useful for simulations with large time steps compared to velocity. I am unsure if the datasets presented in this paper fall into this category! You already partially discuss this, in Line 204-207. I suggest to extend the discussion and maybe also study how important long-range advection is in the datasets presented here, e.g. by performing image co-registration between the time steps (you could use 2D cross correlation for this), and then plotting the average displacement distance for each dataset. Potentially compare this with the average displacement your trained models learn inside the advection layers. You should also compare this to the receptive field, that the CNNs have without the advection layer. You could also tailor the benchmark datasets to be more suitable for your model by deliberately increasing their time step and thus the need for your advection layer. Of course if you do this, communicate it clearly, and best report scores for multiple step sizes, which should ideally show how the advection layer becomes more and more important, the longer the time step.\n2. For video prediction, optical flow is still often integrated in SOTA approaches. It is similar to advection, and since you propose your model may be useful for video prediction, I suggest you discuss this in the related works.\n3. Your experiment section is very short. As is, it is not clear what exactly you evaluate. Please elaborate in the text. \n4. The description of the Advection layer should be central, as it is the main contribution. I found it a bit short, in particular the Pseudocode Algorithm 1 does not detail how the Push-Operator works. Also section 3.2 could be extended to describe more in-depth how the conservative regridding works in practice. Maybe even add a figure that describes your approach.\n5. Section 2 seems a bit long. I mean it is very nice to draw this analogue of CNNs to the ADR equation to introduce the advection layer, but it is neither essential for understanding the layer itself, nor is it very novel. I suggest to shorten this part. Same holds for Section 3.1\n\nMinor Points:\n1. Please place your floating elements on the top of the page, [t], as is pages 7 & 8 are somewhat visually confusing.\n2. Typos and missing words, e.g. L248 Figure _\n3. Tab4 what do 10->5, 10->10 and so on mean? Describe it in the caption!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the positive, thorough, and detailed feedback, which combines profound knowledge from both ML and numerical PDEs perspectives. We address each of your comments and suggestions below and hope our responses are satisfactory, and that you will consider revising your score. We welcome any further questions or suggestions and are happy to discuss them.**\n\n**Regarding W1 (CFL):**  \n\nWe appreciate your suggestion and feedback. Unlike convolutions, which follow the CFL condition, the explicit semi-Lagrangian method can be thought of as an implicit method (please see reference [28]) and therefore is not limited by the CFL condition, making it suitable for advection in PDEs and inspiring its use in ADRNet.\n\nWe agree that ADRNet is effective for long-range predictions requiring large receptive fields. In Table 4, ADRNet outperforms FNO in predicting 50 time steps ahead on the SWE dataset, and Table 10 shows its effectiveness on the CloudCast dataset. **Figure 6 in the rebuttal PDF illustrates receptive fields from 1 to 14 pixels on a 64x64 image.**\n\nFollowing your and Reviewer 2i2R's suggestions, we compared long-range predictions using dilated convolutions. **Figure 3 in the attached PDF** shows our advection mechanism performs better. Further details are in our response to W7.\n\n**Regarding W2 (Optical Flow):** \n\nThe Reviewer is correct about the relationship between learning advection in ADRNet and optical flow methods. The two key differences are that (i) optical flow finds displacement fields in the image domain, while ADRNet learns fields for each channel in the hidden space and (ii) the goal of ADRnet is to predict the next frame while in optical flow one uses the frame to estimate the velocity. We added a discussion and references like [R1,R2,R3]. Thank you.\n\n[R1]VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation\n\n[R2]Novel Video Prediction for Large-scale Scene using Optical Flow\n\n[R3]FlowNet: Learning Optical Flow with Convolutional Networks\n\n**Regarding W3 (Experiments):** \n\nPlease refer to our global response for details on our extensive experiments. We performed comparisons across 6 diverse datasets (including synthetic, scientific, natural images, real-world videos, and GPS data) and benchmarked against SOTA methods.\n\nOur Appendix covers ADRNet's strengths, limitations, dataset details, evaluation metrics, and hyperparameters. We have improved the connection between the main text and the appendix as suggested.\n\nAdditionally, we added a 7th dataset, the Navier-Stokes dataset from PDEBench, with results detailed in our global.\n\n**Regarding W4 (Description of Advection):** \n\nThank you for the comments and suggestions. We explain the method in the paragraph below Equation (10) and in Figure 3. For the Push-Operator implementation, please refer to Jan Modersitzki's book 'FAIR: Flexible Algorithms for Image Registration,' which dedicates a chapter to this problem and includes code examples.\n\nWe used a Pytorch implementation with the grid_sample function to advect the current state using the learned advection field. We will add a figure to the revised paper.\n\nAlso, as we state in our paper, it is our full intention to publicly publish the code (including all scripts of the experiments and figures in our paper) on GitHub if the paper is accepted.\n\n**Regarding W5 (Sections 2 and 3.1):**\n\nWe thank the referee for this comment. The referee clearly has a background in numerical methods for PDEs. Our aim was to write a paper that is readable to the larger audience in the ML community. In fact, Reviewer 2i2R requested the addition of more background material for Section 2. We are aiming to strike a balance between two different and important communities, and we are using your invaluable guidance to improve our paper. \n\n**Regarding W6 (typos, floating elements, Table 4 caption):** \n\nThank you for thoroughly reading our paper. We have now placed all floating elements at the top of the page and fixed the typos. We also added an explanation to Table 4, ensuring it is self-contained. The term'10->5' means the network was trained on 10 previous time steps to predict the next 5. These configurations show that ADRNet offers strong performance on various prediction horizons. We clarified it in the revised paper.\n\n**Regarding W7 (Dilated Convolutions):** \n\nWe appreciate the suggestion from both you and Reviewer 2i2R to use dilated convolutions. We have added experiments with two architectures: one using dilated convolutions for diffusion and another with dilation without advection.\n\nSpecifically, we tested: (i) ADR with dilated convolutions for diffusion, and (ii) dilated convolution to replace the advection layer (by larger receptive field)  while keeping standard convolution for reaction-diffusion.\n\nConvergence plots are in **Figure 3 of the attached rebuttal PDF**. Our ADRNet shows better performance than dilated convolutions on the moving MNIST dataset. The results and discussions are included in the revised paper. Thank you.\n\n**Regarding W8 (Applications of ADRNet):**\n\n Thank you for your interest in our method. We are eager to publish our code, including all scripts for the visualizations and experiments, on GitHub upon acceptance. We look forward to seeing our concepts applied to additional applications.\n\n\n**Regarding W9 (Limitations):** \n\nThank you for the question. In addition to the discussion of the limitations of the method in Section 1, we discuss the generative limitation of physics-inspired architectures such as our ADRNet, in Appendix A. \n\nOur semi-Lagrangian advection method operates on an infinite domain, which implies that information can move out of the image domain without generating boundary artifacts. \n\nRegarding training, we did not encounter training difficulties, as reflected in our choice of hyperparameter, using the same learning rate for all parameter, with a relatively small grid search . In all cases, the training procedure was stable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bub8170L1e", "reviewer": "Reviewer_wodQ", "summary": "The paper introduces Advection Augmented Convolutional Neural Networks (ADRNet), integrating a semi-Lagrangian push operator into CNNs to improve spatio-temporal sequence prediction. Mimicking the Reaction-Advection-Diffusion equation, ADRNet addresses challenges in long-range information propagation and explainability. Evaluations on scientific datasets (CloudCast, SWE) and video prediction datasets (Moving MNIST, KITTI) show ADRNet's superior performance over baseline models.", "questions": "What criteria or methods did you use to determine hyperparameters for different datasets and models?\n\nHave you tested ADRNet on larger and more diverse datasets? What were the results?\n\nAre there ways to enhance the generative capabilities of ADRNet for more complex video prediction tasks?", "limitations": "**Addressing Limitations**:\n   - The paper acknowledges limitations in generative capabilities for video prediction.\n\n**Suggestions**:\n   - Explore methods to enhance generative aspects.\n   - Test on larger, more complex datasets and discuss results and challenges.\n   - Discuss potential societal impacts, such as ethical use of predictive models in sensitive areas like healthcare.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality**:\n   - Innovative use of a semi-Lagrangian push operator for long-range information in CNNs.\n\n\n**Significance**:\n   - Addresses a key limitation in CNNs for spatio-temporal prediction tasks.\n   - Potential impact in fields like weather prediction and traffic flow analysis.", "weaknesses": "**readability**:\n- The introduction of the method in this paper primarily relies on formulas, lacking the necessary explanations and descriptions, making it less readable. \n- There are writing errors in the article, making it difficult to understand, such as in line 246. \n\n**Scalability**:\n   - Scalability to larger, real-world datasets and more complex models is not thoroughly explored.\n\n**Replication of results**:\n   - Although the experimental parameter descriptions are provided, due to the lack of running code, going to reproduce the results of the experiment, however, requires quite a bit of work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the Reviewer’s positive assessment of our paper. We’re pleased that our innovative use of the semi-Lagrangian push forward operator and its potential impact on spatio-temporal tasks, such as weather prediction and traffic flow, were well-received. We are grateful for the constructive feedback, which has helped us improve the paper. We hope our responses address your concerns satisfactorily and encourage you to consider revising your score. We are happy to discuss any further questions or comments you may have.**\n\n**Regarding W1 (Readability):**\n\n Our work blends concepts from numerical methods for PDEs with Neural Network architectures. We have tried to strike a balance between known material in the PDE literature to ML knowledge. We would like to note that our aim is to cater to a wide audience in the ML community, and we therefore follow your proposal. Specifically, we added a number of explanations (e.g. the Operator Splitting approach and the role of advection in allowing long-range information transfer) to the manuscript. Also, the problem indicated by the reviewer in Line 246 was due to the floating figure not being aligned to the top, and we have now corrected it. We sincerely appreciate your comments and have implemented them into our revised paper for enhanced readability. Thank you.\n\n**Regarding W1 (Scalability):**\n\nWe are thankful for your suggestion. Following your guidance, we have now measured the scalability of the method.\nImportantly, please note that in our ADRNet, there are two main operations: reaction-diffusion, which is implemented similarly to a standard ResNet, and the advection mechanism. In order to provide a comprehensive benchmarking of the scalability of the method, we time each component in ADRNet, that is, the (i) reaction-diffusion and (ii)  advection, on different image sizes. This allows to directly measure the added computational cost by using ADRNet compared with a standard network like ResNet. The results are reported in the Table below, and they were also added to our revised paper.\n\n| Image Size    | 32                             | 64                               | 128                               | 256                               |   |\n|---------------|--------------------------------|----------------------------------|-----------------------------------|-----------------------------------|---|\n| ResNet        | Train:4.2ms / Inference:0.6ms  | Train:20.3ms / Inference: 10.4ms | Train:79.2ms / Inference: 45.8ms  | Train:192.8ms / Inference:58.2ms  |   |\n| ADRNet (Ours) | Train:12.8ms / Inference:3.7ms | Train:53.3ms / Inference:19.1ms  |  Train:175.8ms / Inference:72.8ms | Train:484.3ms / Inference:179.1ms |   |\n\nFurthermore, to accommodate your suggestion of demonstrating our ADRNet on a large dataset, we have added a new result for the Navier-Stokes dataset from the PDEBench suite of datasets. The Navier-Stokes dataset  is large and consists of 21000 images of resolution 512x512. The results are provided in the Table below, and we also include visualizations of the results in **Figure 1 in the attached PDF rebuttal file**. \n\n**Regarding W3 (Code):**\n\nThank you for the query. As we state in our paper, it is our full intention to publicly publish the code on GitHub if the paper is accepted for publication. The code includes all the scripts for the different problems and figures presented in our paper for a full reproduction of our results.\n\n**Regarding W4 (Hyperparameters):**\n\n\n\nWe have used a grid-search approach to determine the hyper-parameters based on the validation set in each respective dataset. The hyperparameter ranges are $lr \\in \\\\{1e-4,1e-5,1e-6,2e-6\\\\}$, batch_size $\\in \\\\{64,128,256\\\\}$,   number_of_layers $\\in \\\\{1,2,4,8\\\\}$, number_of_channels $\\in \\\\{128,196,256\\\\}$. In our Appendix we provided the selected hyperparameters, and we added the details provided here to the revised Appendix. Thank you.\n\n**Regarding W5 (Large dataset and diversity):**\n\nThank you for the question. We used six diverse datasets in our submission: moving MNIST (synthetic), PDEbench (scientific), CloudCast (atmospheric), KITTI (natural images), KTH Action (real-world videos), and Taxi-BJ (spatio-temporal locations). These diverse datasets cover a range of properties and domains, demonstrating the effectiveness of our method across different types of data.\n\nFurthermore, following your request we add a new result for the Navier-Stokes dataset from the PDEBench. The Navier-Stokes dataset is large and contains 21,000 images with a resolution of 512x512.  The **results are provided in our general response** to all Reviewers, showing that our ADRNet achieves the 2nd place among several leading methods. Also, we added a **visualization of the results in the attached rebuttal PDF in Figure 1**. All the results were also added to our revised paper.\n\n**Regarding W6 (Generative capabilities):**\n\nWe thank the referee for this question. We believe that this indeed can be done, by using training similar to recent score matching techniques. This is a large excellent project that we are currently looking at. We added this  discussion to the revised paper.\n\n\n**Regarding Suggestions :** \n\nThank you for your thoughtful suggestions. We address them as follows:\n\n(i) Generative aspects: We are exploring this as a future direction, which could benefit scientific tasks like weather prediction.\n\n(ii) Larger datasets and additional tasks: Please refer to our response and results for W5.\n\n(iii) Societal impact: We agree this is important. ADRNet's ability to provide competitive performance across various datasets highlights its potential applications. We emphasize the need for ethical use of strong predictive models. Our paper includes results on publicly available datasets, which we believe do not involve unethical data. \n\nWe have added these discussions to the revised paper. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response:

We thank the four Reviewers for their detailed and constructive feedback, which overall was positive, stating that our method and paper are (i) **motivated and interesting** (*Reviewer 2i2R*), (ii) **addressing issues in existing CNNs,  clear and easy to follow, and demonstrating high performance** (*Reviewer Lq8m*), (iii)  **introduce a new neural network layer, reporting results on multiple “(not too easy) benchmarks”** showing competitive results, as well as that our paper is reasonably well written (*Reviewer zzNS*), and, finally, (iv) makes an **innovative use of the semi-Lagrangian push forward operator**, addresses a key limitation in CNNs for spatio-temporal prediction tasks, and can have **potential impact in fields like weather prediction and traffic flow analysis** (*Reviewer wodQ*). 

**Your feedback included questions and suggestions that helped us to improve our paper. A non-exhaustive list of your suggestions and our revisions includes:**
1. The addition of derivation of operator splitting approach, and the experimentation with a different order of the terms in ADRNet (*Reviewer 2i2R*). Our experimental results support the analysis of the operator splitting scheme, presented in **Figure 4 in the rebuttal PDF**.

2.  Experimenting with dilated convolutions mechanisms to implement diffusion and to allow long range propagation compared with Advection (*Reviewers 2i2R and zzNS*). Our results show the strength of our ADRNet compared with other existing mechanisms, and are , presented in **Figure 3 in the rebuttal PDF**.

3. Using of pre-trained ADRNet (*Reviewer Lq8m*). We have shown that we can transfer learned ADRNet to different datasets, showing an additional benefit of ADRNet. The results are discussed in our responses and provided in **Figure 2 in the rebuttal PDF**.

4. We added visualization of the learned advection fields and attention maps, to understand the behavior of ADRNet (*Reviewer Lq8m*). The result are provided in **Figure 5 in the rebuttal PDF**. 

5. Showing results on an additional large dataset (*Reviewer wodQ*).  Please see our results below and in **Figure 1 in the rebuttal PDF**.
***

While we provided a dedicated response for each Reviewer, we found a repeated point in some of the reviews, and we would like to address it globally:

**Regarding Evaluation:** Several reviewers commented about the scope of our experiments. We would like to enlist the **set of experiments conducted in our paper, which are extensive and contain diverse datasets from different sources, for different types of problems, which are evaluated and benchmarked against multiple SOTA methods from recent years.** In particular, we have done more than 8 quantitative comparisons on a wide range of datasets and tasks, including standard metric comparison as-is done in other well-known papers in the field, as well as ablation studies that show the ability of ADRNet to perform long-range predictions). The experimental results in the paper include the following:

* Table 2 compares our method to 7 other methods on the SWE dataset from PDEbench, a scientific dataset. 

* Table 3 compares our ADRNet to 4 other methods of the CloudCast data set, which is a real-world dataset of spatio-temporal climate prediction.

* Table 4 compares our method to FNOs on long-range predictions for the data from PDEbench. In all these comparisons, our method performs better and in some cases, much better **, 30% improvement, than SOTA on this data set**.

* Table 5 compares our ADRNet to 8 different methods on the moving MNIST dataset. As discussed in Section 4.1, this dataset contains movements of digits and is widely used to show performance on spatio-temporal capabilities on video prediction.
Our results in Table 5 show that ADRNet is the second-best among leading methods. 

* Table 6 compares our method to 6 other methods on KITTI, a real-world natural images dataset. From Table 6, we see that our ADRNet is the leading method on 3 out of 4 of the considered cases and is 3rd best on the remaining metric.
* We provide an ablation study on the impact of learning to predict various future frames ahead in Tables 4 and Table 10, which show the effectiveness of ADRNet  for problems that require a large receptive field.
* We provide results on two additional datasets (KTH Action and TaxiBJ) , as well as providing training and validation curves, all reported in Appendix A.


Overall, we have experimented with **6 different datasets with different prediction tasks**. Note that these datasets are very diverse. One is synthetic (moving MNIST), one is scientific (PDEbench), one is atmospheric (satellite images in CloudCast), one is comprised of natural images (KITTI), one contains real-world videos (KTH Action), and the last contains GPS spatio-temporal information (TaxiBJ). Thus, we have experimented with datasets that have very different properties. 

**In addition**, in order to further demonstrate our ADRNet on scientific data, as has been we have run an additional data set from PDEbench, the Navier-Stokes equations. This is a large dataset, comprised of 21,000 images of size 512x512, and our ADRNet achieves the 2nd place among multiple methods, casting it among the SOTA methods, as shown in the Table below:

| Method | UNET | FNO   | MPP-AVIT-TI | MPP-AVIT-S | MPP-AVIT-B | MPP-AVIT-L | ADRNet (Ours) |
|--------|------|-------|-------------|------------|------------|------------|---------------|
| nMSE (lower is better)   | 1.67 | 0.243 | 0.0312      | 0.0213     | 0.0172     | 0.0142     | 0.0168        |


We visualize the results in Figure 1 in the attached PDF rebuttal. All the results and discussions were also added to our revised paper.

***

Lastly, we hope that the reviewers find that we have adequately addressed their concerns, and we would be grateful if they would consider reassessing their rating. We are also open to further discussion and welcome any additional input.

Meta-review: The paper proposed a CNN with non-local operators and demonstrated on Reaction-Advection-Diffusion equation. The paper initially received mixed rating with one reject, one borderline accept, two weak accepts. The raised concerns are the clarity of writing, lack of baselines & visualizations. During rebuttal phase, the authors resolved all previous concerns and got the raised scores converged to acceptance(accept, weak accept, 2 borderline accept). Given such positive consensus from reviewers, I agreed that the paper has its merits and recommend acceptance of the paper.
Decision: Accept (poster)