CALM-PDE: Continuous and Adaptive Convolutions for Latent Space Modeling of Time-dependent PDEs
Authors: Jan Hagnberger
Daniel Musekamp
Mathias Niepert
Venue: NeurIPS 2025 spotlight ()
Keywords: Partial Differential Equations
PDEs
Physical Simulations
Convolution
Discretization-Agnostic
Encoder-Decoder
Neural PDE Solvers
Neural Operator
Latent Space
Primary area: machine_learning_for_sciences
TL;DR: We introduce CALM-PDE, a model based on continuous convolution for discretization-agnostic solving of time-dependent partial differential equations, which achieves strong performance in fluid dynamics simulations.

Abstract: Solving time-dependent Partial Differential Equations (PDEs) using a densely discretized spatial domain is a fundamental problem in various scientific and engineering disciplines, including modeling climate phenomena and fluid dynamics. However, performing these computations directly in the physical space often incurs significant computational costs. To address this issue, several neural surrogate models have been developed that operate in a compressed latent space to solve the PDE. While these approaches reduce computational complexity, they often use Transformer-based attention mechanisms to handle irregularly sampled domains, resulting in increased memory consumption. In contrast, convolutional neural networks allow memory-efficient encoding and decoding but are limited to regular discretizations. Motivated by these considerations, we propose CALM-PDE, a model class that efficiently solves arbitrarily discretized PDEs in a compressed latent space. We introduce a novel continuous convolution-based encoder-decoder architecture that uses an epsilon-neighborhood-constrained kernel and learns to apply the convolution operator to adaptive and optimized query points. We demonstrate the effectiveness of CALM-PDE on a diverse set of PDEs with both regularly and irregularly sampled spatial domains. CALM-PDE is competitive with or outperforms existing baseline methods while offering significant improvements in memory and inference time efficiency compared to Transformer-based methods.

CALM-PDE: Continuous and Adaptive Convolutions for Latent Space Modeling of Time-dependent PDEs
Jan Hagnberger" Daniel Musekamp'
Mathias Niepert 1.2
Machine Learning and Simulation Lab, Institute for Artificial Intelligence, University of Stuttgart International Max Planck Research School for Intelligent Systems (IMPRS-IS) Stuttgart Center for Simulation Science (Sim'Tech)
Abstract
Solving time-dependent Partial Differential Equations (PDEs) using a densely discretized spatial domain is fundamental problem in various scientific and engi- neering disciplines, including modeling climate phenomena and fluid dynamics. However, performing these computations directly in the physical space often incurs significant computational costs. To address this issue, several neural surrogate models have been developed that operate a compressed latent space solve the PDE. While these approaches reduce computational complexity, they often use Transformer-based attention mechanisms handle irregularly sampled domains, resulting in increased memory consumption. In contrast, convolutional neural networks allow memory-efficient encoding and decoding but are limited to regular discretizations. Motivated by these considerations, we propose CALM-PDE, a model class that efficiently solves arbitrarily discretized PDEs in compressed latent space. We introduce novel continuous convolution- -based encoder-decoder architecture that uses an nagillnnneighboohod--cooo apply the convolution operator to adaptive and optimized query points. We demon- strate the effectiveness CALM-PDE on a diverse set of PDEs with both regularly and irregularly sampled spatial domains. CALM-PDE is competitive with or out- performs existing baseline methods while offering significant improvements memory and inference time efficiency compared to Transformer- beased methods.
kernel and learns
Introduction
Many scientific problems, such as climate modeling and fluid mechanics, rely on simulating physical systems, often involving solving ppatio-temporal Partial Differential Equations (PDEs). In recent years, Machine Learning (ML) models have been successfully used to approximate the solution of PDES (Lu al., 2019; Li al., 2020; Brandstetter al., 2022: Chen & Wu, 2024), offering several advantages over classical numerical PDE solvers. For instance, ML models offer data-driven approach that applicable even if the underlying physics is (partially) unknown, can generate solutions more efficiently (Li et al., b021b; Tompson et al., 2017), and are inherently differentiable by design, which is often not the case for numerical solvers (Takamoto al., 2022).
Practical applications usually require a densely discretized spatial domain Ω leading to more than IM spatial points per timestep. For example, ML-based weather forecasting models typically operate on a spatial domain of 720 x 1440 points or pixels (Pathak et al., 2022). Learning the solution of
Corresponding author; j. h.agnberger @ gmail.com
The source code CALM-PDE is available on GitHub: httpss//gghuubccom//ggghhhgggggg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a(0, (z)
z' Riid
Eute, (z. (z))))))
CALM- PDE Encodall es
CALM PDE Decoder de
Z(0) + Vo(2(r), P)dr
Inout Positions ((r,)))) 1
Query Oulput Positions ((0, vl. fm
Latent Time-Stepping in Fixed Latent Space
Figure 1: CALM-PDE encodes the arbitrarily discretized PDE solution into fixed latent space Rixd computes the dynamics in the latent space, and decodes the solution for the given query points.
dieendepeddent PDE directly in the physical domain Ω rather than in a compressed latent space can result in high memory and computational costs. Consequently, several architectures for reduced- order PDE-solving have been introduced (Wu et al., 2022; Alkin et al., 2024; Serrano et al., 2024; Wang & Wang. 2024). These models adopt the pecode-process-eeood paradigm "Sanchez-Gonzalle et al., 2020), wherein the physical domain is encoded into a compact latent space, its dynamics are evolved via a processor, and the output is decoded back to the original domain. Many real-worle applications involve geometries with irregularly sampled spatial domains, necessitating discretization- agnostic models to effectively process such data. Unfortunately, existing approaches mainly utilize Convolutional Neural Networks (CNNs), which necessitate regular spatial discretization (Wu et al., 2022), or memory-intensive attention- baased mechanisms ( (Vaswani et al., 2017) 7) to enable the encoding and decoding of arbitrarily discretized spatial domains (Alkin et al., 2024; Serrano al., 2024; Wang & Wang, 2024).
Motivated by these considerations, we introduce CALM-PDE (Continuous and Adaptive Convolutions for Latent Space Modeling of Time-dependent PDEs), an architecture featuring a novel encoder and decoder for reduced-order modeling of arbitrarily discretized PDEs. CALM-PDE compresses the spatial domain into a fixed latent representation with an encoder that builds on parametric continuous convolutional neural networks (Wang et al., 2018) and adaptively learns where to apply the convolution operator (learnable query points). This enables CALM-PDE to selectively sample more densely in important regions of the spatial domain, such as complex solid boundaries, while allocating fewer points to smoother regions. Furthermore, we incorporate locality inductive bias into the kernel function, consisting of an epsilon neighborhood and distance- weieeeewwiiin.... to enhance computational efficiency and to facilitate the learning of local patterns. An autoregressive model computes the temporal evolution completely in the latent space (latent time-stepping), which can be interpreted as Neural Ordinary Differential Equation (NODE; Chen et al. (2018)). The discretization-agnenn
decoder, which builds on similar layers as the encoder, enables querying the output solution at arbitrary spatial locations. An overview of the CALM- AAE framework is shown in Figure 1.
We demonstrate the effectiveness of our approach through a broad set of experiments, solving Initial Value Problems (IVPs) in fluid dynamics with regularly and irregularly sampled spatial domains.
Our main contributions are summarized as follows:
We propose model featuring novel encoder and decoder that solves arbitrarily discretized PDEs in fixed latent space.
A novel encoder-decoder approach based on continuous convolutions that learns where to apply the convolution operation (query points) to effectively sample the spatial domain.
A kernel function with locality inductive bias (epsilon neighborhood and distance weight- ing) to enhance efficiency and encourage learning local patterns, and a modulation that allows query points with similar positions to consider different spatial features.

---

2 Problem Definition
In the following section, we formally introduce the problem of solving PDEs. We refer to Appendix B for information about the dataset and training objective train neural surrogates for PDE-solving.
Partial Differential Equations. Similar to Brandstetter et al. (2022), we consider time-dependent PDEs over the time dimension E [0, 7'] and multiple spatial dimensions RN where Na denotes the spatial dimension of the PDE. Thus, a PDE is defined as
(z, Mr 2,
E R C
0.u Ftt, w, u, Uw Uw...
(1,w) E [0. T] Ω
u0,ww) = u (w) =u: Blu] w) =0,
Ω
(1)
(t,w) E [O, x 0
where u [0, x Ω + R No represents the solution function of the PDE that satisfies the Initial Condition (IC) u0, w) for t and the boundary conditions B/u]] (t,, w) if w is the boundary an of the domain Q. Ne denotes the number of output channels or field variables of the PDE. Solving a PDE involves computing (an approximation of) the function that satisfies Equation (1).
3 Background and Preliminaries
We briefly introduce deep parametric continuous convolutional neural networks (Wang et al., 2018). For comparison, discrete convolution and discrete CNNs are explained Appendix
R Deep Parametric Continuous Convolutional Neural Networks. Let f,k: R - IN be two real- valued functions and R. then (ʃ * (a) := - (a) k(aa aodd is the convolution of f and k. Wang al. (2018) propose approximate the function k. which is the learnable kernel, with Multi-Layer Perceptron (MIP) and the integral with Monte Carlo integration, which yields
N
(f ())))(a)=
f f(a) k(a a)da Rs
Σ f(a)) k(a a.))
N
1=1
where input points On are sampled from the domain. The kernel function k is constructed using an MLP ka(a- a)) which spans the entire domain and is parametrized by a finite vector 0 containing the weights and biases. Similar to discrete convolutional layers, deep parametric continuous convolutional layer consists of multiple filter kernels for N; input channels, which leads
No ΣΣ fl(a)) ki,olaj a.,)
(f x ())))))))))
where o denotes the th output channel. The output computed for all output or query points aj and input points a, with the function value f(ax). We denote the set of output or query points as A = | and the set of input points as A = {a,, J.W The number of output points A does not necessarily have to be the same as the input points A, which allows the continuous convolutional layer to reduce or compress information (cf., downsampling) to increase the number of points (cf. upsampling). Similar to discrete CNNs, the receptive field for each query point can limited to M points such that
( ((+)) Σ
fe(am)) kiio(aj am) MERF((3)
Σ
(4)
where RF(ay) outputs the indices for the M points that lie in the receptive field of a query point aj- The receptive field can constructed by only considering the K-nearest neighbors of the point a, or
by considering the points in an epsilon neighborhood. Setting M = N means that the receptive field not limited. Appendix D shows a visualization of continuous convolution. To generalize the layer from ID d-D, only the input dimension of the kernel function has to be adapted.
3

---

is encoded with Random Fourier (utD,(a. sh
(ott, Eo. (o.allllll 2
Fscoder de
CAIM-ROF Proceentr d5
Deconde do
g
Steas kng
Continouus Convolution
@ Locally Industive
50 Leammble Query Points oo
Komol
02
Meedi Rezla: -aa)
Figure Encode-proce pooooddeprocss-decodee architecture of CALM-PDE. The encoder reduces the spatial dimension and increases the channel dimension. It is based on multiple CALM layers, which perform continuous convolution on learnable query points constrained to an epsilon neighborhood.
4 Method
First, we introduce the CALM layer, which enhances continuous convolutional layers and acts as a building block for the architecture. Thereafter, we describe the encoder, processor for latent time-stepping, and decoder of the CALM-PDE model. Finally, we elaborate on the training strategy.
4.1 Learning Continuous Convolutions with Adaptive Query Points
We follow the formulation of deep parametric continuous convolutional layer in Equation (4) introduced by Wang et al. (2018) and propose the following improvements for encoding and decoding PDE solutions, which yield the CALM layer used for the encoder and decoder.
Parametrization of Kernel Function. We use a 2-layer MLP that takes a a as input to parameter- ize the continuous kernel function (a ax). The translation Features (RFF; Li et al. (2021a)) to allow the kernel function to be less smooth (i.e., a small change in the input translation can lead to large output changes in the weight) which could be beneficial for high-frequency content where slightly different translation vectors should result in a large difference. Thus, the parametrized kernel function given la a) MLP (RFF(a a)), where is the 5th input channel and the th output channel.
Epsilon Neighborhood and Distance Weighting. We limit the receptive field by considering only the input points an, within an epsilon neighborhood of the query point aj. similar to Wang et al. (2018). The epsilon is dynamically computed by taking the nth percentile of the Euclidean distances from aj all input points og, similar to the mechanism proposed by Chen & Wu (2024). Thus, the hyperparameter p controls the size of the epsilon neighborhood or receptive field a relative fashion. Consequently, the size of the epsilon neighborhood varies depending on how densely the spatial points are sampled. In densely sampled regions, the epsilon neighborhood is smaller compared to sparsely sampled regions, with a larger epsilon neighborhood. We denote the receptive field based on this mechanism as (a, {a, AA [la an ll S c(a;)) where c(a;) is the pth percentile of the Euclidean distances from a, all input points a, E A. Furthermore, we introduce a distance weighting to the kernel function k. We use sof tmax with temperature T,, similar to Gaussian kernel, represent the distances normalized to [0, within the epsilon neighborhood, which yields
||a ||a-al| - T[max
exp
ki a) -MLP(RFF(a a))
(5)
| (mm((()) T (max(a) minn()
Σ exp
ay (a)

---

for the kernel function with min(a) = min a la-a,11. This emphasizes input points closer o, EBP(a)
to the query point within the epsilon neighborhood. The softmax also serves as normalization factor for the Monte Carlo integration. The combination of the limited receptive field and distance weighting is a locality inductive bias, which is helpful because local patterns are often more important in PDE-solving (Chen & Wu, 2024). Part (i) of Figure 2 shows the receptive field and the distance weighting within the receptive field.
Learnable Query Points. In continuous convolution, the input and output points do not have to be the same. This means that downsampling and upsampling are inherently supported, in contrast to CNNs, where strided convolution and variants are needed for downsampling. This allows to reduce the number of points for encoding and to increase the number of points for decoding. We propose to make the query points for both encoding and decoding learnable. This allows the model to sample more densely in regions with important characteristics (e.g.. region that contains turbulence or an obstacle). The query positions Ω is normalized in a range [0. 1) which yields A = {a = {Ir,, [;)}; with zy By ~ 2( 1) for the initialization of the query points in 2D. Alternatively, the query points can be initialized by sampling K query points from the underlying mesh. We call this initialization method of the query points "mesh prior". During the training, the query points are moved with a variant of stochastic &Cg Vj gradient descent such as Adam (Kingma & Ba, 2017) (xj, 4z) (xz, (z) [aaa6:6)))
initially uniformly sampled from the domain Ω. We assume that
where 1) denotes the learning rate and Le the cost function. The kernel function provides feedback in which direction to move, and the sof smax function in Equation (5) avoids hard cuts caused by the learnable query points and epsilon neighborhood by weighting the input points on the boundary of the epsilon neighborhood lower. The learnable query points are illustrated in part (ii) of Figure 2.
Kernel Modulation. We allow each query point to have a customized filter kernel (i.c., different kernel weights for the same similar translation vectors). This way, there can be multiple query points at the same location, each paying attention to different features. This could be helpful for simulations that contain a geometry or obstacles (e.g., many query points close to the obstacle, but each query learns different characteristics). The kernel modulation is done by scaling and shifting the intermediate representations (Perez al., 2018) of the MLP in Equation (5). Each query point has location as well as scaling and shifting parameters. We denote the modulated kernel function as
(a - W2 a ((W1 RFF - a) + b1) O Ya + +..) + b2
i,0
where we exclude the distance weighting for the sake of simplicity. W denotes weight matrix with a suitable shape, o is a non-linearity, and "Y. 3, are the scale and shift for query point a, respectively. Part (iii) of Figure 2 demonstrates that the kernel modulation enables query points, despite having identical receptive fields and translation vectors of input points, to output different kernel weights.
Periodic Boundaries. PDE solutions often involve periodic boundary conditions. Thus, we adapt the translation vector a - a to account for periodic boundaries for the kernel function if the solution has periodic boundary.
CALM Layer. A non-linearity follows the continuous convolution operation, which completes the CALM layer. Hence, the CALM layer with the previously defined kernel function k is given
No
(f*k))(a,)
Σ
(lam))kiplaa, am +b0) (7)
MER
where f and A refers to the sampled input and A to the query points. The query points are learnable, except for the final decoding layer, where the query points correspond to the queried spatial points. o corresponds to suitable non-linearity such ReLU or GELU, and b, is the bias for the th channel.
4.2 Neural Architecture for Discretization-Agnesti Reduced-Order PDE-Solving
The architecture of the CALM-PDE model follows an prodee-processdeecoee paradigm (Sanchez- Gonzalez et al., 2020). Figure 2 shows overview of the CALM-PDE architecture
5

---

Encoder. We stack multiple CALM layers together for the encoder. Each layer increases the number of channels (the number of output channels larger than the number of input channels, i.e., No > N)) and reduces the number of (spatial) points (the number of query points is much smaller than the number of input points, i.e., |A| < [A]]. This design is similar CNN-based encoders, which reduce the spatial size of the feature maps and increase the number of feature maps to encourage hierarchical feature learning. The output of the encoder eg is a set latent tokens Z' for time t and their corresponding positions P. We denote the encoder as
co( co(tu' (w,) fwn 1o 1. (wn)) fwn 1 A1), A A
Ax). A
(8)
-Z'ER'X
P
where k denotes the number of layers, I is CALM layer, (wn)) describes the solution function at time evaluated at the locations {ai) )N I= Z' and are the final representations of the input solution. Note that the positions P are learned, fixed after the training, and independent of u².
Processor for Latent Time-Stepping The model evolves the dynamics within the latent space through autoregressive prediction. In particular, the processor the predicts the difference from the latent tokens Z' at time t to the future latent tokens Z At Vo P) + Z'. To obtain the solution for a timestep t + At, the processor is applied times iteratively completely evolve the dynamics in the latent space (latent time-stepping). We opt for a Transformer model (Vaswani al., 2017) since attention allows the processor capture global information from local tokens, and latent tokens can dynamically interact with each other. For instance, the processor can learn the dynamics of moving vortex in CFD simulation by adapting the latent tokens on the path of the moving vortex and pushing it from one position to another. We include the positional information of the latent tokens into self-attention using the Euclidean distance, similar Chen & Wu (2024). We refer to Appendix E.2 for details. Using the processor to predict the residual and scaling it with the temporal resolution At can be understood as solving a NODE parametrized by tee which is given as Zt + At) = (t)) + S pf+ 1/o (Z)),,, P)dr with an explicit Euler solver. Thus, the processor implicitly defines a vector field that could be integrated with solvers more advanced than the explicit Euler method, such as Runge-Kutta solver.
Decoder. The decoder de also based on multiple CALM layers that reduce the number of channels and increase the number of spatial points. The decoder takes the latent tokens Z', positions P. and set of query points {wy 1-y" as input and outputs the solution in the physical space for the queried points. The decoder with layers given as
dz (w,, NN' = 1s ( (Z', P,A1), Ak- Ak As-1, fWs (ا)))")
(u' (w;))
which maps from the latent tokens and their position P to the physical domain for the given set of queried spatial points {w,})y
4.3 Training Procedure
We opt for an end-to-end training procedure because we found be more stable compared to two-stage training procedure that first trains the encoder-decoder using selftreconsttuutiion loss and, after that, trains the processor to learn the latent dynamics (Yin et al., 2023; Serrano al., 2024). The end-to-end training consists of curriculum strategy that slowly increases the trajectory length during training (Li et al., 2023a) and a randomized starting points strategy (Brandstetter et al., 2022: Hagnberger et al., 2024) that randomly samples ICs along the trajectory. Besides future timesteps, the model predicts the IC, which we incorporate as self-reeoonttruction into the loss function.
5 Related Work
We give an overview of surrogate models, introduce continuous convolutional neural networks, and present reduced-order PDE-solving models. We refer to Appendix F for details related work.

---

PDE Solving with Neural Surrogate Models. Neural surrogate models are increasingly used for approximating the solutions of PDEs (Lu et al., 2019 Li et al.. 2021b, 2020: Cao, 2021: Li et al.. 2023a; Hagnberger et al., 2024). Neural operators (Kovachki et al., 2023) constitute an important category, and prevalent neural operator models include the Graph Neural Operator (GNO; Li et al. (2020)) based on message passing (Gilmer et al., 2017) and the Fourier Neural Operator (FNO; Li et al. (2021b)), which leverages Fourier transforms, alongside its various variants. Transformers are also increasingly employed for surrogate modeling, typically using the attention mechanism on the spatial dimension of the PDE (Cao, 2021; Li et al., 2023a; Hao et al., 2023; Li et al., 2023b). Another distinct group of models, neural fields, is particularly well-suited for solving PDEs due to their ability to represent spatial functions. These models are gaining traction in Pol-iolving applications (Yin et al., 2023; Chen et al., 2023; Serrano et al., 2023; Hagnberger et al., 2024; Knigge et al., 2024).
Point Clouds and Continuous Convolution. Solving fluid dynamics problems can be interpreted as a dense prediction problem for point clouds. Models that apply convolution to point clouds either use a discrete filter and extend it to continuous domain via interpolation or binning (Hua et al., 201 8) or parameterize the filter with a continuous function (Wang et al., 2018; Xu et al., 2018; Wu et al., 2019). These models were mainly introduced for the segmentation of point clouds. Ummenhofer et al. (2019) adapt continuous convolution to simulate Lagrangian fluids, where the model predicts the movement of the particles. In contrast to method, their model parametrizes the filters in a discrete fashion and the weights are interpolated to extend it to a continuous domain, while our approach follows the method proposed by Wang et al. (2018) and uses an MLP to parametrize the filters. Winchenbach & Thuerey (2024) define the kernel function using separable basis functions and employ the Fourier series the basis with even and odd symmetry for particle-based simulations.
Neural Networks for Reduced-Order PDE-Solving. Neural networks for reduced-order PDE- solving usually compress the spatial domain into a smaller space and solve the dynamics in that smaller space. LE-PDE (Wu et al., 2022) employs CNNs for encoding and decoding and an MLP to compute the dynamics. PIT (Chen & Wu, 2024) compresses the spatial domain into a predefined latent grid with position attention that computes attention weights based on the Euclidean distance. UPT (Alkin et al., 2024) supports simulations in the Eulerian and Lagrangian representations and employs hicrarchical structure to encode the input. The encoder uses message passing to aggregate information in super nodes, and a Transformer and Perceiver (Jaegle et al., 2021) to further distill the information into latent tokens. A Transformer computes the dynamics in the latent space, and the Perceiver-based decoder decodes the processed latent tokens. AROMA (Serrano et al., 2024) utilizes cross-attention and learnable query tokens to distill information from the spatial input into query tokens. uses denoising diffusion to map from one latent representation to the subsequent latent representation by denoising the latent tokens. LNO (Wang & Wang. 2024) encodes the spatial input into a latent space by computing cross-attention between the imput positions and learnable query positions in high-dimensiooal space. Since the query positions in LNO are learnable, they simplify the attention computation, which makes the query positions inaccessible. They use a Transformer to compute the next timestep in the latent space, and decoder decodes the latent tokens back to the physical space.
6 Experiments and Evaluation
We focus solving IVPs and evaluate the performance of CALM-PDE on set of various PDEs in fluid dynamics. The experiments are designed to answer the following Research Questions (RQ); RQI: How effective is CALM-PDE compared to the state-of-the-ar methods for regularly sampled spatial points? RQ2: How well does CALM- MDL--EEEE perform on irregularly sampled points? e RQ3: Does solving the dynamics in compressed latent space yield a lower memory consumption and
provide a speedup? + RQ4: Where does the model place the query points and do they learn local or global information?
RQ5: Which components contribute the model's 's performance?
6.1 Datasets
For regularly sampled spatial domains, we conduct experiments on the ID Burgers' equation dataset from PDEBench (Takamoto et al., 2022), the 2D Navier-Stokes equation datasets introduced by Li et al. (2021b), and the 3D compressible Navier-Stoke dataset of Takamoto et al. (2022). The 2D
7

---

Table 1: Rel. L2 errors of models trained and tested on regular meshes. Values in parentheses indicate the percentage deviation to CALM-PDE and underlined values indicate the second-ber best errors.
Relative L2 Error (4)
Model
ID Burgers' V = le"
2D Navier-Stokes
2D Navier-Stokee V == 1e
3D Navier-Stokes
v= le-
( = 1e-
FNO
0.0358 (+46%)
0.0811 (+169%)
0.0912 (-12%)
0.6898 (+2%)
F-FNO
0.0362 (+47%)
0.0863 (+187%)
0.0844 (-18%)
0.6466 (-4%)
OFormer
0.0575 (+134%)
0.0380 (+26%)
0.1938 (+88%)
0.6719 (-1%)
PIT
0.1209 (4391%)
0.0467 (+55%)
0.1633 (+58%)
0.7423 (+10%)
INO
0.0309 (+26%)
0.0384 (+28%)
0.0789 (-24%)
0.7063 (+4%)
AROMA
0.0937 (+281 %)
0.1061 (+252%)
0.1931 (487%)
1.3328 (+97%)
CALM-PDE
0.0246
0.0301
0.1033
0.6761
Euler equation with an airfoil geometry and the incompressible Navier-Stokes equation with cylinder geometries datasets from Pfaff et al. (2020) are used to evaluate the models on irregularly sampled domains. We refer to Appendix H for additional details on the datasets.
6.2 Baseline Models
We compare the CALM-PDE model against FNO (Li et al., 2021b) and Geo-FNO (Li et al., 2023c) for irregularly sampled meshes, F-FNO (Tran et al., 2023), OFormer (Li et al., 2023a), PIT (Chen & Wu, 2024), LNO (Wang& & Wang. 2024), and AROMA (Serrano et al., 2024). We train the autoregressive models (FNO, F-FNO, Geo-FNO, and PIT) with a curriculum strategy that slowly increases the trajectory length, which improves the error compared to training with full autoregressive rollout (see Appendix I). OFormer uses a similar strategy, LNO a one-step training, and AROMA a denoising diffusion training to learn the temporal dynamies of the PDE as proposed by the authors.
6.3 Results
We report the mean values and percentage deviations of the relative L2 error of multiple runs with different initializations. We opt for the relative L2 error as an evaluation metric because weights channels with small and large magnitudes equally and does not ignore time-dependent decay effects like in the ID Burgers' equation. We to Appendix N.1 for the full results with standard deviations and Appendix N.2 for qualitative results.
We train and evaluate the models on regularly sampled spatial domains. Regularly sampled spatial domains refer to meshes with equidistant nodes or points. We select datasets that span the entire spatial dimension from ID to 3D. Table 1 shows the relative L2 errors of the baselines and CALM-PDE model. On 2 out of benchmark problems, CALM-PDE outperforms all baselines. On one problem, CALM-PDE achieves the third-lowest errors and on the remaining benchmark problem, CALM-PDE is outperformed by F-FNO and LNO, and outperforms PIT and OFormer.
RQ1.
RQ2. CALM-PDE is also designed to support irregularly sampled spatial domains. Irregularly sampled domains are characterized by mesh with varying distances between the nodes. Geometries mainly cause this irregularity because the mesh is usually denser in regions of interest, such as the tip of an airfoil, where a higher accuracy is required. We train and evaluate the models on two datasets, namely the airfoil and cylinder datasets. The first dataset simulates the airflow around a static airfoil geometry. The geometry can be considered static because it does not change between different training and test samples. In the second dataset, the water flow in a channel with cylinder as an obstacle is simulated. In contrast to the airfoil geometry, the cylinder geometry changes, which means that each training and test sample has a different cylinder geometry with a different diameter and position. Table 2 shows the relative L2 errors for both benchmark problems. On the airfoil dataset, AROMA achieves the lowest error overall, while our model outperforms Transformer-basse baselines such as OFormer and LNO, and on the cylinder dataset, CALM-PDE delivers the second- best performance, closely trailing AROMA. Thus, CALM-PDE not only supports irregularly sampled domains but also generalizes across different geometries.
8

---

Query Positions CALM-PDE
2D Cylinder Dataset
Model
Relative L2 Error (J)
Decader Layer 2 Query Positions
2D Airfoil
2D Cylinder
Output Positions
1.
1.1
Geo-FNO
0.0388 (-25%)
0.1383 (+17%)
F-FNO
0.1081 (+110%)
0.1490 (+26%)
0.7
0.7
OFormer
0,0520 (+1%)
0.2264 (+91%)
PIT
0.0894 (+74%)
0.1400 (+18%)
0.3
0.3
LNO
0.0582 (+ (+13))
0,1654 (+39%)
AROMA
0.0372 (-28%)
0.1139 (-4%)
-0.1
CALM-PDE
0.0515
0.1186
-0.1 0.3
0.7 1.1
-0.1 0.3 D.7 1.
Figure 3: Learned query positions of CALM-PDE.
Table 2: Relative L2 errors of models trained and
The model samples more query points in the re- gion where the cylinders are located (red rectan- gle) by moving query points to this region.
tested on the irregular meshes with geometries in the fluid flow. The values in parentheses indicate the percentage deviation to CALM-PDE.
RQ3. We measure the inference times NVIDIA A100 GPU to evaluate the efficiency of CALM-PDE. Figure shows the inference times measured on the 2D Navier-Stokes dataset with a resolution of 64 x 64 and 200 trajectories. We increase the trajectory length to measure the scaling behavior. CALM-PDE is significantly faster than OFormer and LNO and achieves a competitive inference time that of FNO and PIT, which are fast baselines due to the use of the Fast Fourier Transform (FFT) and position attention, respectively. Table shows the time per epoch and memory consumption for the forward and backward pass during training on 2D Havier-Stokes with batch size of 32. CALM-PDE outperforms OFormer and LNO in terms of time and memory consumption.
RQ4. Further, we investigate the learned positions of the query points and the represented infor- mation (local or global information). The results show that the model places more query points in regions that can intuitively be considered important and that the tokens primarily learn local information. Figure illustrates the learned query positions for the decoder on the cylinder dataset, showing that the model samples more query points in areas where the cylinders are located. We refer to Appendix L for more details. Thus, learnable query points allow the model higher information density in important regions and allow learning and discovering unknown important regions.
RQ5. Finally, we analyze the impact of the model's s components the ablation study presented in Appendix M. The results indicate that learnable query points reduce the error more effectively than both fixed and randomly sampled query points, as well as fixed query points sampled from the underlying mesh. Additionally, the kernel modulation improves the error for regularly and irregularly sampled spatial domains. Furthermore, the distance weighting, which is part the locality inductive bias, also helps the model to further reduce the error and stabilize the training.
7 Limitations
CALM-PDE compresses the input into smaller latent space, which implies that information such as fine- fineggainee details is lost. This could be an issue for PDEs such as Kolmogorov flow that have fine details that must be accurately captured. Another limitation, that also applies to other neural PDE solvers, are the required computational resources for real-world applications with large spatial domains with millions of spatial points. Compared to the Transformer-bant based methods, CALM-PDI more efficient but not efficient enough for practical applications. For the 3D Navier-Stokes equation with 21 timesteps and a spatial resolution of 64 x 64 x 64, which corresponds to 262k points, CALM-PDE requires one A100 80GB GPU for the training, while LNO requires four A100 80GB GPUs with a batch size of 4, respectively. However, practical applications would require spatial resolutions larger than 64 x 64 x 64.
8 Conclusion and Future Work
With CALM-PDE, we propose an efficient framework for reduced-order modeling of arbitrarily discretized PDEs. The experiments demonstrate that CALM-PDE achieves low errors regularly

---

2D Navier Stokes, 64 64, 200 Trajectories
2D Ntvier-Stokes
20
Model
Time/Batch [ms]
Memory [MB]
OFormer
FNO
126. 17
5453
Inference
10
PIT
OFormer
786.36 #.867
43807
5
CALM-PDE
PIT
124.23 19.16
16968
INO
1196.57=0.57
61191
20
100
CALM-PDE
138.25#4.73
14147
Prediction Steps
Figure 4: Comparison of inference times 2D Navier-Stokes with 200 trajectories. Prediction steps increased to evaluate the scaling. FNO, PIT. and CALM-PDE achieve similar times.
Table 3: Time and memory consumption needed
for the forward and backward pass during training the 2D Navier-Stokes dataset for batch size of 32 on an NVIDIA A100 GPU.
and irregularly discretized PDEs, as well as that the model can generalize different geometries. While it does not consistently surpass all baselines, it delivers competitive errors across various problems. This makes it viable alternative established approaches such as FNOs and transformer- based architectures and demonstrates that convolution can also be applied to irregular meshes. Beyond PDEs, CALM layers have broader applicability, including potential use in fields like chemistry. For future work, we aim to improve CALM-PDE: for problems with high frequencies, which are important in real-world applications. Additionally, future work could investigate "dynamic query points" that depend on the IC and geometry move during the rollout to track important regions. Another interesting direction for future research is improving the model for a two-stage training procedure.
Acknowledgments and Disclosure of Funding
Funded by Deutsche Forschungsgemeinnchatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2075 390740016. We acknowledge the support of the Stuttgart Center for Simulation Science (SimTeec)). The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Daniel Musekamp and Mathias Niepert. Additionally, acknowledge the support of the German Federal Ministry of Research, Technology and Space (BMFTR) as part of InnoPhase (funding code: 02NUK078). Lastly, we acknowledge the support of the European Laboratory for Learning and Intelligent Systems (ELLIS) Unit Stuttgart.
References
Benedikt Alkin, Andreas Füistt, Simon Schmid, Lukas Gruber, Markus Holzleitner, and Johannes Brandstetter. Universal physics transformers: A framework for efficiently scaling neural operators. 2024.
Advances in Neural Information Processing Systems,
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL attps://arxxe http:///w/www./oo/aabaaaaee 1607. 064500.
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. In International Conference Learning Representations, 2022.
Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924-24940, 2021.
Honglin Chen, Rundi Wu, Eitan Grinspun, Changxi Zheng, and Peter Yichen Chen. Implicit neural spatial representations for time-dependent pdes. In International Conference Machine Learning, pp. 5162-5177. PMLR, 2023.
Junfeng Chen and Kailiang Wu. Positional knowledge is all you need: Position-induced transformer (pit) for operator learning. In International Conference Machine Learning, pp. 7526-7552.
PMLR, 2024.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
10

---

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. Pmir, 2017.
Jan Hagnberger, Marimuthu Kalimuthu, Daniel Musekamp, and Mathias Niepert. Vectorized con- ditional neural fields: A framework for solving time-dependent parametric partial differential equations. In International Conference on Machine Learning, pp. 17189-17223, PMLR, 2024.
Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pp. 12556-122 12569. PMLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Binh- -on Hua, Minh-Khof Tran, and Sai-Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 984-993, 2018.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651 15544. PMLR, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https:/ bttapp://arxi..oo /arx/axxvvvv.r// orw./ogg//a...o..o...o...... /bs///bs/uaa (412.69800
www.ivv.org/bbi
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https:
// //arxiv. org/ abs//312. 6114.
David M Knigge, David R Wessels, Riccardo Valperga, Samuele Papa, Jan-Jakob Sonke, Efstratios Gavves, and Erik J Bekkers. Space-time continuous pde forecasting using equivariant neural fields. Advances in Neural Information Processing Systems, 37:76553-76577 2024.
Nikola Kovachki. Zongyi Li. Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-97, 2023.
Yang Li, Si Si, Gang Li, Cho-Jui Hsich, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. Advances Neural Information Processing Systems, 34: 15816-15829, 2021a.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. Transactions Machine Learning Research, 2023, 2023a. URL https:
Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable transformer for pde surrogate modeling. Advances in Neural Information Processing Systems, 36:28010-28039, 2023b.
Zongyi Li, Nikola B. Kovachki, Kamyar Azizzadeneshelli Burigede Liu, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. CoRR, abs/2003.03485, 2020. URL attps://arxiis wwwpxiiooo/r/oooooooo htgg//b///000// 03485.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar, et al. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021b.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388):1 2023c.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv: arXiv:1910.08 1910.03193, 2019.
11

---

Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay. Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators, 2022. URL https: //arxiv. org/ /2022. 11214.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, pp. 3994-3995 2018.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzaler and Peter Battaglia. Learning mesh-based simulation with graph networks. In International conference on learning representations, 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660) 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference machine learning, pp. 8459-8468. PMLR, 2020.
Louis Serrano, Lise Le Boudec, Armand Kassaï Koupaï, Thomas X Wang, Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari. Operator learning with neural fields: Tackling pdes on general geometries. Advances in Neural Information Processing Systems, 16.70581-70611 2023.
Louis Serrano, Thomas X Wang, Etienne Le Naour, Jean-Noël Vittaut, and Patrick Gallinari. Aroma: Preserving spatial structure for latent pde modeling with local neural fields. Advances in Neural Information Processing Systems, 37:13489-13521 2024.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüiger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1611, 2022.
Makoto Takamoto, Francesco Alesiani, and Mathias Niepert. Learning neural pde solvers with parameter-guider channel attention. In International Conference on Machine Learning, pp. 33448- 33467. PMLR, 2023.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. In International conference on machine learning, pp. 3424-3433. PMLR, 2017.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International conference on learning representations, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition. IEFE, 2018. doi: 10.1 109/cvpr.2008..ooooo 0..02477 URL http:/ h/tp://dx... doi. org/10. 1109 / /CtR 2018 2008000774.
12

---

Tian Wang and Chuang Wang, Latent neural operator for solving forward and inverse pde problems. Advances in Neural Information Processing Systems, 37:33085-33107, 2024.
Rene Winchenbach and Nils Thuerey. Symmetric basis convolutions for learning lagrangian fluid mechanics. In The Twelfth International Conference Learning Representations, 2024.
Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries. In International Conference on Machine Learning, pp. 53681-537777 PMLR, 2024.
Tailin Wu, Takashi Maruyama, and Jure Leskovee. Learning to accelerate partial differential equations via latent global evolution. Advances in Neural Information Processing Systems, 35:2240-2253, 2022.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9613-9622, 2019.
Zipeng Xiao, Zhongkai Hao, Bokai Lin, Zhijie Deng, and Hang Su. Improved operator learning by orthogonal attention. In International Conference Machine Learning, pp. 54288-54299 PMLR, 2024.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European conference on computer vision (ECCV), pp. 87-102, 2018.
Yuan Yin, Matthieu Kirchmeyer, Jean- Yves Franceschi, Alain Rakotomamonjy. et al. Continuous pde dynamics forecasting with implicit neural representations. In The Eleventh International Conference on Learning Representations, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have conducted extensive experiments PDEs with regularly and irregu- larly sampled spatial domains to evaluate the model.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper contains limitation section (Section 7) to discuss potential limita- tions of the proposed model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
The paper has no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The used datasets and hyperparameters outlined in detail in Appendix H and Appendix J, respectively. Furthermore, the proposed method fully explained.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The code reproduce the results will be made publicly available upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
wwwssssssslllii imPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpp.ccpppblliiepppp wwtpcccpppcc/ccpcccccccccccccclt wwtpc///g///eeeeeeceeeeee
ceabbmmmsssooPPe ssininolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: detailed description the experimental setting is provided in Appendix J. The data splits are introduced in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the mean values and standard deviations in Table 21 and Table 22 in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided information about the used GPUs as well as detailed information about the memory and execution time in Figure and Table 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ h/tprrip/////ii...... cc cctpublic//ettecccccce hptpblic/ttiiccuuddeeieeeee
Answer: [Yes]
Justification: We have examined the NeurlPS Code of Ethics and ensured that our research fully conforms to its principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impact of the work in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators of all datasets, models, and code are credited properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Comprehensive documentation and practical examples demonstrating the use of the new model architecture will be made available.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The work does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /tapp///////rss.///ee / for what should or should be described.
20

---

APPENDIX FOR CALM-PDE
Impact Statement
23
Details on the Problem Definition
23
C Discrete Convolution and Cross-Correlation
23
D Deep Parametric Continuous Convolutional Neural Networks
24
Additional Details on the CALM-PDE Framework
25
E.1 CALM Layer
25
E.2 Neural Architecture
26
Continuation of Related Work
27
Comparison Related Models
28
H Additional Details the Datasets
29
H.I Regularly Sampled Spatial Domains
30
H.2 Irregularly Sampled Spatial Domains
I Additional Details on the Baselines
31
J Additional Details the Experiments
32
J.1 Hardware
32
J.2 Loss Function
33
J.3 Evaluation Metric
33
J.4 Hyperparameters
K Hyperparameter Study
36
Model Analysis
37
M Ablation Study
44
M.1 Learnable Query Positions
44
M.2 Kernel Modulation
44
M.3 Distance Weighting
45
M.4 Distance-based Kernel Function
45
M.5 Latent Dimension
46
Additional Results
47
N.1
Quantitative Results
47
21

---

N.2 Qualitative Results
47
Additional Experiments
58
0.1 Time-independent Problems
58
0.2 Encoding and Decoding Capabilities
59
0.3 Scaling of Output Query Mesh Size
59
Notation
59
22

---

A Impact Statement
Advanced and efficient neural surrogate models significantly lower the expenses running otherwise cost-intensive simulations, such as those used in weather forecasting and fluid mechanics. This improvement helps to speed up simulations and to reduce energy consumption, costs, and CO2 emissions. However, potential drawback is the risk of misuse by bad actors, fluid dynamic simulations play a role in designing military equipment.
B Details on the Problem Definition
Discretized Dataset. We use discretized data generated by numerical solvers to train and test the surrogate models. The temporal domain [0, 7) is discretized into N, timesteps yielding sequence (u", n², "u.,...,""' u²) with N₁ elements which describes the evolution of the PDE. At == f,+1 denotes the temporal step size or resolution. Similarly, the spatial domain Ω is also discretized into finite set of N points [wn)e=) by discretizing each spatial dimension which yields a discretized representation fu' (w,, 1 of the function. Figure visualizes the discretization process. A dataset D = {(11, Yi)..... (XN., YN,)} for each PDE consists of N, samples. ) target sequence of timesteps.
Xj {u² (w,,) n= denotes the IC and Y, (fu't (wn)))== {u² (wi)) denotes the
Discretized Function {u(, (x, y)m))
Function u(: x. y)
Mesh ((x. y)
1.0
1.0
0.8
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 5: The continuous solution function ut-, xr y) has to be discretized along the spatial dimension with suitable mesh or grid into discrete representation {u(-, (z. W of the function.
Training Objective. The training objective aims to optimize the parameter vector that contains all weights and biases of the model So that best approximates the true function u by minimizing the empirical risk over the dataset D as
N. Σ (1O((L))
1
argmin La arginin DEE
(w5)) X J 1)m Yo
(10)
where Le denotes the overall cost function and L denotes suitable loss function such as the Mean Squared Error (MSE) or relative L2 norm. fe ( [wr} predicted trajectory of the neural network queried with the set of times {4 and the set of spatial points { J.y given the initial condition X, evaluated at the points {w,
Xa, {w>))=)) represents the No
C Discrete Convolution and Cross-Correlation
We briefly explain discrete convolution and elaborate how discrete convolution is used in CNNs.
Discrete Convolution and Cross-Correlatiin In the discrete case (i.e., the functions f and k are only defined for integers), the convolution operator simplifies
(f k)]aa|]
Σ
fla a) k[aa]
(11)
23

---

with a E Z. The points usually have equal spacing because discrete convolution operates on sequences or discrete signals with regularly indexed data points. Similarly the continuous case, k [[]] has be reflected to k to compute cross-correlation.
Convolutional Neural Networks. CNNs implement cross-correlation where f is the finite and discrete input signal (i.e., the input features) with length of N, and k represents a finite and learnable kernel with the length M. This leads to
(f k)[a] reflect k (f+h)|a)= Σ fla a - 1]-
(12)
with N M + 1]. J.. As mentioned previously, cross-correlation is equivalent to convolution with reflected kernel. Since the weights of the kernel are learned during training, it does not matter whether the layer implements cross-correlation convolution. Usually, the input contains multiple input features or channels (i.e., is a vector) and multiple convoluted output features or channels are desired. N, and N, denote the number of input and output channels respectively which yields
(ʃ [()aaa]]
Σ
Σ
Siaa + a [is,oo(())
(13)
where corresponds to the the tth output feature or channel and the 7th input channel and kfa] E R N.*.N contains the filter kemelss k[a{] is usually implemented using a finite, multi-
dimensional array. The convolution layer introduced above can extended from ID d-D.
D Deep Parametric Continuous Convolutional Neural Networks
Figure 6 visualizes continuous convolution with and without receptive field for 4 input points a,, and one query point a1 As proposed by Wang et al. (2018), the kernel function k: is parametrized by MLP. However, continuous convolution can be optimized further for the encoding and decoding of PDE solutions. For instance, CALM layers learn the position of the query points ay to effectively sample the spatial domain, use a locality inductive bias including an explicit weighting with the Euclidean distance to emphasize the input points closer to the query point and an epsilon neighborhood improve efficiency, and each query point modulates the MLP to support different kernels for query points that have similar translation vectors.
(f + k))(a))
Σ
fann)s ksp - an)
(f+
Σ
fam)) kiolas am)
F= mcN
finel
Oo
org
flog!
Crg
flogg
(a) Without receptive field
(b) With receptive field (blue circle)
Figure 6: Continuous convolution without receptive field (a) and with a receptive field (b) that limits the number of considered input points. The kernel function & takes input the translation vector a - a and outputs a weight for an imput channel and output channel o. An MLP parametrizes the kernel function.
24

---

[A Input Points with Festuree f it*
| Ourput Points Features (f Rj E 1d
Cli
CALM Layer
((o)
Cts
Continucus Convolution
@ Localiy inductive Bias
60 Learrable Query Points dj
(t) Kerne Modulation
tf+ k/m
82
Figure 7: A CALM layer takes a point cloud with |AA] points as input and outputs a new point cloud with |A| points and new feature dimension d. CALM layers compute continuous convolution with a locality inductive bias to emphasize closer points, learnable query points (output positions), and kernel modulation to allow different weights for different query points with similar translation vectors.
E
Additional Details on the CALM-PDE Framework
E.1 CALM Layer
Overview. Figure 7 shows CALM layer that performs continuous convolution with three distinct features: (i) locality inductive bias, consisting of a limited receptive field and distance weighting, to emphasize closer points, (ii) Iearnable query points to let the model learn where to compute convolution, and (ii) kernel modulation to allow different query points to have different weights for similar translation vectors. Depending on the purpose (e.g., encoder final layer of the decoder), the query points can be learned or provided as an external input query. The kernel function l is given as
exp(
I--a min(())
ka,la- - a) = W22 VVaao((1) ((W RFF(( - a) -WFFWWWFF(aa----- T +..) + b2
Tmaa a min [a)]
1.
Σ CRF(((
exp(
T [mmax minaa)
l a I| min(a)
Modulated
Weighting
(14)
where Ya and B, denote the modulation parameters which depend on the query point a, and min(a)= ciF(a) ||a - a,1l computes the minimum distance within the epsilon neighborhood. Similarly, max(a) computes the maximum distance within the epsilon neighborhood. The distance normalization ensures that small distances are amplified and the effect of large distances is reduced.
Implementation of Kernel Function. In practice, the kernel function is is parametrized by a 2-layer MLP that outputs large vector in N;-N followed by reshape operation to get kernel matrix in NNN. N₂ denotes the number of input channels and N,, is the number of output channels. The bias term b2 is initialized with Kaiming uniform initialization (He et al., 2015) where the number of input features corresponds to the number of input channels N; and not to the hidden dimension of the MLP. We opt for this initialization to ensure proper initialization of the weights used in the continuous convolution operation. This approach can also be understood as initializing learnable weight hi... which is independent of the translation a and shared between all points, with Kaiming uniform initialization and using 2-layer MI.P without a bias the last layer to learn an additive correction or residual depending the input
parameterize the continuous kernel function oa a).
25

---

(wit (st, s)a)
(u(0, (a.p))))3
(DA M-PDE Encoderee
CALM-PDE Processor fs
CAI M-PER Decoder do
((e, shal
RM
3 Mh
Steoping
((r,
Figure 8: The CALM-PDE model utilizes multiple CALM layers with learnable query positions to encode arbitrarily discretized PDE solutions into fixed latent space. A Transformer processes the compressed latent representation and outputs the latent representations of future timesteps which will be decoded by the decoder. The decoder also uses multiple CALM layers with learnable query positions, except for the last decoder layer which uses the mesh as an external input query.
Receptive Field and Epsilon Neighborhood. The receptive field of query point aj is given as RF(a,) {an A ||a, - Oz2l <(a,)} where e(a,) is the tdh percentile of the Euclidean distances from query point a, to all input points on E A similar to Chen & Wu (2024). The hyperparameter p controls the size of the receptive field. For instance, p 0.1 means that the epsilon neighborhoods are constructed so that each query point aj aggregates at least 10% of all input points. Consequently, the epsilon neighborhood smaller in densely sampled regions than in sparsely sampled regions which avoids convolution aggregation with too many or few input points.
Pointwise Operations. Before and after applying the continuous convolution operation, pointwise linear operation and pointwise MLP are applied, respectively. The pointwise operations enable the model to combine the input output channels of continuous convolution to richer feature representation.
Computational Complexity. The complexity of CALM layer is given follows. Let N be arbitrary number input points. The output of the layer are features for N' output or query points, where N' is constant for all layers except for the final decoder layer. The percentile, which determines the size of the receptive field, and the feature or channel dimension of the CALM layer are denoted as p and d, respectively. This results in Opp N N d) computations in total. In contrast, models such as AROMA (Serrano et al., 2024) or LNO (Wang & Wang, 2024) do not consider a receptive field, which requires O(N N d) computations for N' query tokens. Since p usually small (e.g.. 0.01), the complexity is reduced by two orders of magnitude compared to models without receptive field.
E.2 Neural Architecture
In this section, we provide additional details on the neural architecture of CALM-PDE. Figure shows the architecture of the model with the encoder, processor, and decoder.
Processor for Latent Time-Stepping. The processor network (Z' P) is Transformer with combined attention that combines scaled dot-product attention (Vaswani et al., 2017) and position
26

---

attention (Chen & Wu, 2024). It takes the latent representation Z' for time and the positions P as input and computes the change between the latent representations Zt and Z++1 follows
H1 = (z' [REF
H2 CombinedAttn [H1, P) + Hi
H3 MLP(H2, P) + H2
H4 CombinedAtta H3, P) + H3
H5 MLP(H4, P) H4
(15)
vo Z Vo P) H5Wou
1 = At A Vo P) +
CombinedAttn(H, P) softmax
L d(P, d)) HWV
where d(P, P) outputs a distance matrix with the pairwise distances of the positions LN denotes layer normalization (Ba et al., 2016) and At is the temporal resolution. We only normalize the queries and keys to allow the propagation of the scaling of the features through the processor similar to Cao (2021). The processor is applied iteratively to compute the latent representation of timesteps further into the future. Due to the residual connection and the scaling of the predicted change with the temporal step size At, the prediction Z At tha (Z' P) Z" can be interpreted as solving a neural ordinary differential equation parametrized by we with an explicit Euler solver. The exact solution is given
Z + AI) 2(1)
(Z)),,); P)dT
(16)
which can be approximated with an explicit Euler solver as follows
++
7(1 + At) = 7(1) +
(b((Z()) P)dr
(17)
= [t) + At wh (Z())))
= Z + At (7',
where At denotes the temporal step size.
F
Continuation of Related Work
Latent Neural Surrogates and Latent Time-Stepping. We introduce the term Latent Neural Surrogates (LNS) to refer to models that internally use smaller spatial latent representation instead of the original spatial representation, such as Position-Induum and Latent Neural Operator (LNO; Wang & Wang (2024)). We use the term Latent Time-Stere Steeping (LTS) to describe models that solve the dynamics completely in the latent space without decoding it in every timestep. In contrast, models like PIT and LNO follow physical space, meaning they decode the solution in every timestep and do not belong to the LTS category. Prevalent LTS models include LE-PDE (Wu et al., 2022), which employs a CNN for both encoding and decoding and an MLP as the latent processor. OFormer (Li et al., 2023a) utilizes a Transformer as encoder and decoder and a pointwise MLP the spatial dimension processor. UPT (Alkin et al., 2024) applies attention for encoding and decoding and a Transformer for latent time-stepping. AROMA (Serrano et al., 2024) diffusion Transformer as latent processor. The proposed CALM-PDE model belongs to both model categories. It an LNS as it uses the proposed CALM layers for the encoder and decoder compress the solution and it is an LTS model since it solves the dynamics completely in the latent space with a Transformer.
Transformer (PIT; Chen & Wu (2024))
autoregressive rollout in the
cross-attention encoder and decoder and
Discretization-Agnn Agnostic Architectures. Discretization-agnosss architectures for PDE-solving decouple the input and output spatial domains and allow solving arbitrarily discretized PDEs. Transformer-basee models such as OForme, UPT, AROMA, and LNO are discretization-agnosiic
27

---

Variants such as Geo-FNO (Li et al., 2023c) of the Fourier Neural Operator (Li et al., 2021b) are also pntretizatiin-agnnotte and support arbitrarily discretized inputs. Neural fields are also suitable for discretization-agnosi models and have been used in DINo (Yin et al., 2023), CORAL (Serrano et al., 2023), and Equivariant Neural Fields (Knigge et al., 2024) for PDE-solving. CALM-PDE is based on continuous convolution and also supports the encoding and decoding of arbitrarily discretized and irregularly sampled solutions. Furthermore, CALM-PDE decouples the input from the output domain.
Point Clouds. Solving fluid mechanics problems in the Eulerian or Lagrangian representation can be interpreted as a dense prediction problem for point clouds. Models such as PointNet (Qi et al., 2017a) and PointNet++ (Qi et al., 2017b) have shown great success in the classification and semantic segmentation of point clouds. PointNet is based on pointwise transformations and max pooling operation to extract a global representation of the point cloud. They concatenate the point features with the global representation to obtain rich feature vector with local and global information for semantic segmentation. PointNet++ extends PointNet with hierarchical structure to gradually combine regions into larger ones.
G Comparison to Related Models
We compare related models such as OFormer (Li et al., 2023a), PIT (Chen & Wu, 2024), UPT (Alkin ct al., 2024), AROMA (Serrano et al., 2024), and LNO (Wang & Wang, 2024) with the proposed CALM-PDE model. We briefly introduce the models and elaborate similarities and differences.
PIT (Chen & Wu, 2024) compresses the spatial domain into a smaller latent representation with position attention, computes the dynamics for one timestep, and decodes it back with position attention. They propose an attention mechanism called position attention that computes the attention weights of the input points and the latent points based on the Euclidean distance instead of the scaled-dot product (Vaswani et al., 2017). A predefined set of points is used in the latent space as a latent mesh. UPT (Alkin et al., 2024) supports simulations in the Eulerian and Lagrangian representations and employs hicrarchical structure to encode the input. The encoder uses message passing to aggregate information in super nodes, and a Transformer and Perceiver (Jacgle et al., 2021) to further distill the information into latent tokens. A Transformer computes the dynamics completely in the latent space (latent time-stepping). and the Perceiver- based decoder decodes the processed latent tokens. AROMA (Serrano et al., 2024) utilizes tokens to distill information from the spatial input into the query tokens. The decoding is also realized with cross-attention. The model is trained in a two-stage training. First, the encoder-decoder model is trained as variational autoencoder (Kingma & Welling, 2022). After that, a denoising diffusion Transformer (Peebles & Xie, 2023) is trained to map from one latent representation to the subsequent latent representation by denoising the latent tokens. LNO (Wang Wang, 2024) encodes the spatial input into fixed latent space by computing cross-attention between the input positions and learnable query positions in high-dimensional space. Thus, the attention weights depend only on the positions, similar to PIT and AROMA. Since the query positions in LNO are learnable, they simplify the attention computation, which makes the query positions inaccessible. They use Transformer to compute the next timestep in the latent space, and a decoder decodes the latent tokens back to the physical space. The LNO model is trained with teacher foreing and used for inference in an autoregressive fashion, both in the physical space.
and learnable query
CALM-PDE learns query points effectively sample the spatial domain, where each point has specific position, giving the learned query points tangible physical meaning. In contrast, AROMA and LNO also implicitly learn query points for cross-an cross-attention, but the query points are inaccessible and lack physical relevance, as they are defined in a nigh-diimnsional space. The physical meaning of CALM-PDE's query points enables the incorporation of a mesh prior. For example, the query points can be initialized to be denser in predefined critical regions and sparse in less significant regions. This concept is similarly applied in PIT, where the latent space is structured on predefined latent mesh. However, predefined query points are not always optimal, and the ability to learn query positions provides greater flexibility. This fundamental distinction sets CALM-PDE apart from PIT, which operates without learnable query points. The key distinctions from PIT include the use of continuous convolution, learnable query positions, a kernel MI.P for computing convolution weights, and multiple filters. AROMA and CALM-PDE differ in the encoder and decoder (scaled dot-product
28

---

attention compared to continuous convolution), latent processor, and training procedure. UPT and CALM-PDE differ in the encoder and decoder architecture (message passing and attention compared to continuous convolution). Similarly, LNO and CALM-PDE differ in the use of attention compared continuous convolution in the encoder and decoder. PIT, UPT, and CALM-PDE share the approach of restricting the receptive field to enforce locality and optimizing computational efficiency. AROMA and LNO do not restrict the receptive field of the query tokens. In contrast to the mentioned methods, CALM-PDI LLEM does not lift the input channels into higher-dimensional space, which is required for the Transformer-based models. Due to multiple filters, our model can still extract enough features without the need to operate in such high-dimn gigh--imensional space.
Table provides an overview of similarities and differences between related work and the CALM-PDE framework. We compare the following properties:
Supports Compression: Whether the model supports compressing the solution into a smaller
latent space.
Latent Time-Stepping: Whether the models compute the dynamics completely in the latent space.
Hierarchical Encoder and Decoder: Whether the models use hierarchical encoder and or decoder.
Attention or Kernel Weights solely based on Positions: Whether the models compute attention kernel weights solely based the position of the sampled points.
Large Number of Filters Heads: Whether the models can support large number of filters or heads. For attention-based models, the number of filters is limited by the dimension the query, key, and values, while convolution-based models do suffer from this limitation.
Learnable Query Points: Whether the model has learnable query points query tokens.
Physical Meaning of Query Points: Whether the query points or tokens have a physical meaning, such position that provides information about where the model samples more densely.
No Lifting or Embedding Required: Whether it is required to lift or embed the input channels (e.g.. to 5 channels) into higher-dimensionall space (e.g., 96 to 256) at the beginning, which results in a higher computational complexity.
Table 4: Overview properties of related models and the proposed CALM-PDE model.
Property
OFormer
PIT
UPT
AROMA
LNO
CALM-PDE
Latent Time-Stepping
Supports Compression
X
V
XXXX
XXX
Hierarchical Encoder
X
X
Hierarchical Decoder
X
X
X
Attention or Kernel Weights
X
J
solely based Positions
Large Number of Filters or Heads
X
AX
Learnable Query Points
AXX
Physical Meaning of Query Points
X
No Lifting Embedding Required
X
X
H
Additional Details on the Datasets
We benchmark the baselines and CALM-PDE model on the following datasets with regularly and irregularly sampled spatial domains. We use term regularly sampled spatial domain to refer to spatial points sampled from uniform grid, in contrast to the term irregularly sampled domain, which refers to grid or mesh with non-equidistant points. Meshes are mainly irregular due to geometries or obstacles in the fluid flow. Table 5 shows an overview of the used datasets.
29

---

Table 5: Overview of the used datasets. Static geometry means that the geometry is the same for each sample. In contrast, changing geometry means that the geometry changes with each sample.
PDE
Parameter
Timesteps
Spatial Resolution
Mesh
Geometry
Channels N:
ID Burgers'
le"
41
1024
Regular
No
2D Stvker-Stokes
le
20
64 64
Regular
No
2D Navier-Stokes
le
20
64 64
Regular
No
3D Bavier-Stoker
11 { le-s
21
64 64 64
Regular
No
5
2D Airfoil
N/A
5233 points 1885 points (avg)
Iregular
Yes, static
2D Cylinder
N/A
Irregular
Yes, changing
3
H.1 Regularly Sampled Spatial Domains
ID Burgers' Equation. The ID Burgers' equation models the non-linear behavior and diffusion process of ID flows in fluid dynamics and is defined as
@ru(t, x) + u(t, x),, V
Ozs u(,
E (0. (-1,1)
(18)
where the parameter v denotes the diffusion coefficient. We use the dataset provided by PDEBench (Takamoto et al., 2022) and select a diffusion coefficient is under periodic boundary conditions and the goal is to learn neural surrogates to approximate the function u. We use spatial resolution of 1024 for æ E (-1, and subsample the temporal resolution to 41 timesteps for E (0, 2). The dataset contains 10k trajectories, with the first 1k samples used for testing and the remaining samples for training.
1e to encourage shocks. The equation
2D Incompressible Navier-Stokes Equation. The Computational Fluid Dynamies (CFD) applications. We consider the 2D Navier-Stokes equation for viscous, incompressible fluid in vorticity form on the unit torus which is given
equations are important for
w,)( + ult, w) Vu(l, w) vAu(1, f(w),
w E (0, 1)², (0,
V - ult, w) 0, v00 w) = vo(w),
We (0,1) .t E [0,
(19)
w (0,
where V x denotes the vorticity, u the velocity, and y E R, the viscosity coefficient. vo is the initial vorticity and f is the forcing term. The task is to learn a neural surrogate to approximate the function v. We use the dataset proposed by Li et al. (2021b) with periodic boundary conditions and consider the viscosities v = le -4 and v = le 55 We drop the first 10 timesteps due to less complex dynamics and consider temporal horizon of t (10, 30) (20 timesteps) for v 1e and use temporal horizon of t E (0, 20 (20 timesteps) for y = le The spatial domain is discretized into a grid of 64 x 64. We use the last 200 samples for testing and the remaining ones for training in both cases.
3D Compressible Navier-Stokes Equation. Additionally, we consider compressible version of the Navier Stokes equations (Equations 20a to 20c). The equations describe the flow of fluid and defined as
0e + V (pv) = 0,
(20a)
p(o.v vv. Vv) = -vp+ +p (5 ?)
V(V v).
(20b)
a,(c+ 2v² ) V (p+cc 2v² jv- v. 0' 0,
(20c)
2
p: p= p(t,w) v : p(t,w),
WERt [0,
(20d)
where is the density, v the velocity, and p the pressure of the fluid. denotes the internal energy and o' the viscous stress tensor. The parameters v and ( are the shear and bulk viscosity. Equation (20a) represents the conservation of mass, Equation (20b) is the equation of the conservation of momentum, and Equation (20c) is the energy conservation. The goal is to learn a neural surrogate approximating density, velocity, and pressure. We use the dataset from PDEBench (Takamoto et al., 2023) with
30

---

= <= 1e spatial resolution of 64 x 64 x 64, and the full temporal resolution of 21 timesteps. The equation the dataset under periodic boundary conditions. We use the first 10 samples for testing and the last 90 samples for training.
H.2 Irregularly Sampled Spatial Domains
2D Compressible Euler Equation with Airfoil Geometry. The 2D Fuler equation describes the flow of an inviscid fluid and corresponds to the Navier-Stokes equations with zero viscosity and no heat conduction. The 2D Euler equation for compressible fluid is defined as
Oep V (pv) 81;
p(d.v Vv) + Vp= $2
0c(c + p
pc ]vl |v|= + V 1p + c+
(21)
2
2
p= p(t, W), v wl,,w), p= :=
w WE t [0.2]
where p is the density, is the velocity, and p the pressure of the fluid. denotes the internal energy and $1, $2, $3 are the source terms. The task is to learn a neural surrogate model that approximates the functions p. and p. We use the dataset of Pfaff et al. (2020) which contains the geometry of an airfoil with a no- seoppenttration condition imposed on the airfoil. The distances between the points the meshes range from 2e m to 3.5 m, which makes it dataset with highly irregular sampled spatial domain. We consider a subsampled temporal resolution of 16 timesteps for the experiments.
2D Incompressible Navier-Stokes with Cylinder Geometries. The 2D incompressible Navier- Stokes equation with constant density is defined
0,v 0,
Po(O.v +V Vv) + Vp= up=
(22)
:: v((,,), w(l,w)
W WERIEE [0,
where Po is the constant density, the velocity, and p the pressure. We use the dataset introduced by Pfaff et al. (2020) that models the flow of water in a channel with a cylinder as an obstacle in the fluid flow. With each sample, the diameter and position of the cylinder change. The neural surrogates trained to predict the velocity and pressure. We subsample the temporal resolution to 15 timesteps for the experiments.
I Additional Details on the Baselines
Fourier Neural Operator (FNO) and Geo-ENO. The Fourier Neural Operator is a neural operator based Fast Fourier Transforms (FFTs). We the implementation of FNO from PDEBench (Takamoto et al., 2022) and use the model in an autoregressive fashion as proposed by Li et al. (2021 b) "FNO with RNN structure". However, we curriculum strategy that combines autoregressive training with teacher-forcing training by slowly increasing the rollout length of the autoregressive rollout (Li al., 2023a) and by doing a teacher-forcing prediction with the remaining timesteps of the trajectory (Takamoto et al., 2023). The strategy improves the performance compared to full autoregressive training. We use the Geo-FNO (Li al., 2023c), variant of the FNO that supports irregularly sampled domains, for the irregularly sampled spatial domains experiments. The Geo-FNO also uses the curriculum training strategy for improved training.
Factorized Fourier Neural Operator (F-FNO). The Factorized Fourier Neural Operator (Tran et al., 2023) is an improved version of the Fourier Neural Operator that uses separable spectral convo- lution layers, improved residual connections, and pointwise MLPs. Separable spectral convolution layers factorize the Fourier transforms over the spatial dimension, which significantly decreases the number of model parameters. The improved residual connections, which are applied after the non-linearity, and the pointwise MIPs, akin to the pointwise MIPss in Transformers, further improve the performance. We use the original implementation of F-FNO and train the model in similar fashion as FNO and FNO (i.e., autoregressive rollout with curriculum strategy).
31

---

Operator Transformer (OFormer). OFormer (Li et al., 2023a) leverages an attention-base encoder and decoder to decouple the imput from the output spatial domain. The attention mechanism enables its direct application to irregularly sampled spatial domains. The model uses latent time- stepping to compute the dynamics of time-dependent PDEs. We follow the original implementation of OFormer and train it using a curriculum strategy that increases the trajectory length during training.
Dosition-Inducee Transformer (PIT). PIT (Chen & Wu, 2024) uses an attention mechanism based on the Euclidean distances instead of the scaled dot-product. The model computes latent representation that is based on a latent grid. The latent grid has to be fixed prior to the training and has the same dimension as the physical grid (e.g., 2D latent grid for 2D PDEs). For PDEs with regularly sampled domains, we follow the authors and use a uniform mesh with fewer points or resolution as latent grid which is a suitable choice. Similar to OFormer, PIT directly supports irregularly sampled domains. However, there are several suitable methods for generating the latent mesh for irregularly sampled domains (c.g., uniform mesh or sampling the latent mesh from the input mesh), and the authors did not experiment with irregularly sampled meshes. Thus, we generate the latent mesh for the airfoil dataset by randomly sampling from the training mesh and we use a uniform latent mesh for the cylinder dataset. Sampling mesh for the cylinder dataset is not a good choice since there are huge differences in the meshes of different training samples and sampling from one mesh does not optimally cover the spatial domain. The authors propose to train the PIT model in an autoregressive fashion. We have conducted experiments training the model in an autoregressive fashion and noticed that the currieulum strategy, which significantly improved the performance of PIT. Thus, we train PIT with the same strategy to obtain lower errors.
train the FNO and Geo-FNO, also
Latent Neural Operator (LNO). LNO (Wang & Wang. 2024) compresses the input solution into fixed latent representation with attention. The attention is solely computed using the input positions. A Transformer computes the next latent representation for the next timestep and decoder maps the latent representation back to the physical space. Similar to OFormer and PIT, LNO directly supports irregularly sampled domains. The authors propose to train the model for one-step predictions (teacher forcing) and use it to do an autoregressive rollout. We follow their original setup and keep the training strategy.
Attentive Reduced Order Model with Attention (AROMA). AROMA (Serrano et al., 2024) is reduced-order model that compresses the input solution into a fixed latent representation by applying cross-attention in a Perceiver-lie like fashion (Jaegle et al., 2021). A Transformer computes the latent representation for the next timestep with denoising diffusion, and a decoder maps the latent represen- tation back to the physical space. Similar LNO. AROMA directly supports irregularly sampled spatial domains. The authors use two-stage training procedure that consists of an autoencoder training that trains the encoder-decoder model with selftreconstruction loss and a dynamics training that trains the denoising diffusion model to predict the latent representation of the next timestep. We use their original implementation and keep the training strategy.
Full Autoregressive Training vs Curriculum Strategy. The errors of the autoregressive models (FNO, Geo-FNO and PIT) can be reduced by using a curriculum strategy, that slowly increases the rollout length (L.i et al., 2023a) and performs feacher-forcing training with the remaining timesteps (Takamoto et al., 2023), compared to fully autoregressive training. Table 6 shows that the curriculum strategy performs significantly better compared to fully autoregressive training. Thus, we train the autoregressive models with the curriculum strategy.
Additional Details on the Experiments
J.1 Hardware
We conduct all experiments on NVIDIA A 100 SXM4 GPUs. The training of CALM-PDE takes 2 to hours on an A100 GPU, depending on the dataset. contrast, takes up to 2 days to train the baselines multiple A100 GPUs.
32

---

Table Rel. L2 errors of FNO and PIT trained fully autoregressive fashion and with curriculum learning. The values in parentheses indicate the percentage deviation to the autoregressive model
Relative L2 Error (L)
Model
Training Type
2D Navier-Stokes
V 1e-1
Autoregressive
0.1335 0.00006
FNO
Curriculum
0.0811 00.004 (-39%)
Autoregressive
0.1038=
PIT
Curriculum
0.0467=0.0000
(-55%)
J.2 Loss Function
We train the models with a relative L2 loss (Li et al., 2021b, 2023a,b; Chen & Wu, 2024), which offers the advantage of treating channels with small magnitudes on par with those having large magnitudes, ensuring balanced weighting. while also providing feedback when the PDE solution decays (i.c., the magnitudes decay) over time. Let Y E R. xXNN, be the ground truth and Y E IR N, Xxe the model's prediction. N₂ is the number of timesteps or trajectory length, N the number of spatial points (e.g., 4096 = 64 64 for resolution of 64 x 64 for 2D), and Ne the number of channels. Then, the loss is defined as
- Yuno
L I(ÝY))
1
ΣΣ
(23)
N N
where ||-|| denotes the 1.2 norm.
J.3
Evaluation Metric
We use the relative 1.2 error defined previously an evaluation metric. In addition to the advantages mentioned in the previous section, the relative 1.2 error can be interpreted as a percentage error.
J.4 Hyperparameters
Fourier Neural Operator (FNO) and Geo-FNO. Table 7 shows the hyperparameters for FNO and Geo-FNO used in the experiments. We adopt comparable hyperparameters to those in Li et al. (2021b) for the Burgers' equation and the 2D Navier-Stokes equations. For the 3D Navier-Stokes equations, we follow the hyperparameters outlined in Takamoto et al. (2022), with the exception of reducing the batch size from 5 to For Geo-FNO, we utilize the hyperparameters proposed by Li et al. (2023c) which prove to be effective.
Table 7: Hyperparameters for FNO and Geo-FNO used in the experiments.
Parameter
ID Burgers'
Navier-Stokes
3D Navier-Stokee
2D Airfoil
2D Cylinder
Width
64
24
20
Modes
16
14
12
14
14
Layers
4
Learning Rate
le.3
Batch Size
32
64
2
lipachs
Parameters
549,569
1,812,161
22,123,753
2,327,710
2,327.669
Factorized Fourier Neural Operator (F-FNO). Table 8 summarizes the hyperparameters used for F-FNO our experiments. For the 1D Burgers' equation, we adopt hyperparameters similar to those in Li et al. (2021b), which prove to be effective. In the case of the 2D Navier-Stokes equations, we follow the configuration from Tran et (2023), but reduce the number of modes and the width to align with FNO, necessary adjustment due to GPU memory limitations encountered during training.
33

---

The 3D Stvike--tokes configuration also mirrors FNO's hyperparameters. For the airfoil and cylinder datasets, we adopt the hyperparameters proposed by Li et al. (2023c).
Table 8: Hyperparameters for F-FNO used in the experiments.
PDE
Parameter
ID Burgers'
2D Navier-Stokes
3D Navier-Stokes
2D Airfoil
2D Cylinder
Width Modes
64
24
20
24
24
14
12
Layers
4
4
Learning Rate
le-3
Batch Size Epochs
32
64
2
500
Parameters
665,281
151,361
131913
625,270
625,229
OFormer. Table 9 shows the hyperparameters for OFormer used in the experiments. We employ similar hyperparameters for the 1D) Burgers' equation and adopt the same parameters for the 21) Navier-Stokes equations as presented in Li et al. (2023a). For the 3D Navier-Stokes equations. we reduce the embedding dimensions to ensure the model fits on an A100 GPU. Increasing the model for 3D Navier-Stokes is infeasible due excessive GPU memory usage and memory constraints in the experiments. For the 2D airfoil dataset, we utilize the hyperparameters specified in Li et al. (2023a) and adopt them for the 2D cylinder dataset.
Table Hyperparameters for OFormer used in the experiments. 'Icreasing the model is infeasible due to memory constraints in the experiments.
PDB
Parameter
1D Burgers'
2D Navier-Stokes
3D Navier-Stokes?
2D Airfoil
2D Cylinder
Encoder
Embedding Dimension
96
64
128
128
Encoder
96
192
128
128
128
Out Dimemsion
Encoder Layers
44
Decoder
Embedding Dimension
96
384
256
Propagator Layers
3
Learning Rate
Se-4
5e-4
Se-4
6c-4
6e-4
Batch Size
64
16
8
Iterations
128k
128k
22k
48k
48k
Parameters
660,719
1,850,497
825,157
1,367,812
1,370,371
Position-Inducee Transformer (PIT), Table 10 shows the hyperparameters for PIT used in the experiments. We adopt the hyperparameters introduced in Chen & Wu (2024) for the 1D Burgers' equation, with the modification of increasing the batch size from to 64. For the 2D Navier-Stokes equations, we retain the same hyperparameters as specified in Chen & Wu (2024). To train the 31) Navier-Stokes model on an 100 GPU, we reduce the latent mesh, quantiles, and the number of heads. Increasing the model for 3D Navier-Stoker is impractical due to GPU memory limitations encountered during the experiments. For the 2D airfoil and 2D cylinder datasets, we apply the perparaameters used for the 2D Navier-Stokes equations which prove to be effective. The latent mesh for the 2D airfoil dataset is sampled from the training mesh (cf. mesh prior initialization of CALM-PDE) while the model employs uniform latent mesh for the 2D cylinder dataset.
Latent Neural Operator (LNO). Table 11 shows the hyperparameters for LNO used in the experi- ments. For the ID Burgers' equation, the listed hyperparameters prove to be effective. Compared to the hyperparameters introduced in Wang & Wang (2024) for the 2D Navier-Stokes equations, we decrease the embedding dimension, number of modes, and number of projector layers to better match the other models terms of the number of parameters and required compute. For 3D Savier-Stokes, the embedding dimension is reduced further to train the model on an A100 GPU. Increasing the model infeasible due to memory constraints the experiments. We adopt the hyperparameters from 2D Navier-Stokes for the 2D airfoil dataset which prove to be effective. For the 2D cylinder dataset, we use similar hyperparameters as for the 2D cylinder dataset but reduce the embedding dimension from 192 to 176.
34

---

Table 10: Hyperparameters for PIT used in the experiments. 'Increasing the model is infeasible due memory constraints in the experiments.
PDE
Parumeter
ID Burgers'
2D Navier-Stokes
3D Navier-Stokes'
2D Airfoil
2D Cylinder
Embedding Dimension
64
256
256
256
256
Heads
2
2
2
Layers
Latent Mesh
1024
16 16
8 x
256
16 16
Encoder Quantile
0.01
0.02
0.002
0.02
0.02
Decoder Quantile
0.08
0.02
0.01
0.02
0.02
Learning Rate
6c-4
le-3
Se-4
6c-4
6e-4
Batch Size
64
16
Epochs
500
500
500
500
500
Parameters
78,861
1,249,805
923,659
1,251,088
1,252,383
Table 11: Hyperparameters for LNO used in the experiments. 'Inercasing the model is infeasible due to memory constraints in the experiments.
PDE
Parumeter
1D Burgers'
2D Navier-Stokes
3D Navier-Stokes'
2D Airfoil
2D Cylinder
Embedding Dimension
96
192
128
192
176
Modes
128
Projector Layers
3
2
2
2
Propagator Layers
4
Learning Rute
le-3
Batch Size
64
32
2
8
8 2,397,267
Epochs
Parameters
461,121
2,847,105
1,276,805
2,848,260
AROMA. Table 12 summarizes the hyperparameters used for AROMA in our experiments. We adopt the hyperparameters reported in Serrano et al. (2024), but reduce the number of epochs finish each training within 24 hours.
Table 12: Hyperparameters for AROMA used in the experiments.
PDE
Parameter
1D Burgers' 2D Stokes
3D Navier- Stokkr-Stooks
2D Airfoil 2D Cylinder
Hidden Dimension
128
Encoder- Decnder
Number Lutents
256
256
Latent Dimension
16
16
16
Depth
Encode Geometry
X
Learning Rate
le-3
Batch Size
128
10
32
64 10000
Epochs
1000
500
Dynamies Model
Hidden Dimension Depth
128
Minimum Noise
1e-2
le-3
le-3
Denoising Steps
Learning Rate
le-3
Batch
Epochs
1000
Parameters
665,281
1,845,505
1,852.773
1.950,148
1,949.891
CALM-PDE. Table 13 shows the hyperparameters for CALM-PDE used in the experiments. The notation [16, 32, 64] for the channels means that the first layer has 16 channels, the 2nd 32 channels, and the 3rd 64 channels. Similarly, for different hyperparameters in the table. The dimensionality of the latent space is defined by the number of latent variables or query points and their channel dimension. We determine the dimensionality of the latent space based on the principle that the number query points, receptive field, and channel dimension must cover the entire spatial domain. Similar to discrete CNNs, small receptive field with a large stride (e.g., only a few query points) misses information, while large receptive field with a small stride could have an averaging effect. With this principle, obtain the number of query points (c.g., 1024, 256, 16 for the encoder layers),
35

---

which we use for all 2D experiments. Thus, a fixed number of query points works across different problems.
Table 13: Hyperparameters for CALM-PDE used in the experiments.
PDE
Parameter
ID Burgers'
2D Stvker-Stokes
3D Navier-Stokes
2D Airfoil
2D Cylinder
Layers
Encoder
Channels
[16. 32, 64]
[32. 64, 128]
[64, 128. 256] [1024, 256, 64]
[32. 64, 128]
[32. 64, 128]
Query Points
[256, 64, 8]
[1024, 256, 16]
[1024, 256, 16]
[1224, 256, 16]
10.05, 0.5]
[0.01, 0.01, 0.2]
[0.001 0.01, 0.2)
[0.01, 0.01, 0.5]
10.01, 0.01,
Temperature
[1.0. 1.0, 1.0]
[1.0, 1.0, 0.1.
[1.0. 1.0. 0.1]
[100, 1.0. 0.1]
[1.0. 1.0, 0.10
Layers
3 [128, 64, [256. 1024, -
Decoder
Channels
[32, 16. (64, 256, 1
[64, 32, 1] [256, 1024, -]
[64, 32, 4] [256, 1024, -] [0.5, 0.01, 0.01] [0.1. 1.0, 1.0]
[64, 32, 3]
Query Points
[256, 1024,
Percentile
[1.0. 0.5.
[0.2. 0.01, 0.01] [0.1, 1.0, 1.0] /
[0.2. 0.01. 0.001]
[0.5, 0.01, 0.01] 10.1, 1.0, 1.0]
Temperature
[1.0, 1.0, 1.0]
[0.1. 1.0. 1.0]
Periodic Boundary
Mesh Prior
x
X
X
Processor Layers
Rundom Starting Points
/
V
X
Learning Rate
1e-3
le-3
le-4
6e-4
1e-3
Batch Size
32
Epochs
500
500
250
500
500
Parameters
568.331
2,230,627
8,367,267
2,237,240
2,239,403
K
Hyperparameter Study
This section provides additional information about the hyperparameters of CALM-PDE and their robustness, In particular, we show the similarities of the hyperparameters of CALM-PDE and discrete CNNs, and demonstrate the robustness of the softmax temperature.
Comparison to discrete CNNs. CALM-PDE introduces the softmax temperature, receptive field radius (percentile), channel dimension, and number of query points hyperparameters. The receptive field radius (percentiles) is equivalent to the kernel size in discrete CNNs, and the number of query points to downsampling (e.g., stride and pooling) CNNs. The channel dimension is equivalent to the channel dimension in CNNs. Alternatively, the number of query points can be compared to the number latent query tokens used in other models, such as AROMA (Serrano al., 2024) or LNO (Wang & Wang, 2024). Consequently, CALM-PDE maintains conventional hyperparameter footprint without introducing excessive complexity.
Hyperparameter Selection and Robustness.
Selecting hyperparameters for CALM-PDE does not require an extensive hyperparameter search. As shown Table 13, we use the same tempera- tures, receptive field sizes (percentiles), and number of query points for all 2D experiments, which demonstrates that the hyperparameters are robust and work across different problems.
Hyperparameter Study of Softmax Temperature. The softmax temperature is new hyperparam- eter that controls the distance weighting within the receptive field. We also use the same temperatures across all experiments, demonstrating its robustness. Additionally, study by sweeping the temperature from a high value in model (a) to a low value model (f). As shown in Table 14, temperatures in the range of T E [0.1, 5.0) result in only minor changes in test error, confirming the robustness of the model to this parameter. All experiments are conducted on the 2D Navier-Stokes dataset with 14-
perform hyperparameter
36

---

Table 14: Relative L2 errors of CALM-PDE models with different temperatures trained and tested on
the 2D Navier-Stokes dataset.
Model Configuration
Relative L2 Error (L)
2D Navier-Stoker
Temperature T in Encoder and Decoder
V 1e
(a)
10
0.0398
(b)
5
0.0336
(c)
2
0,0316
(d)
0.0304
(e)
0.1
0.0341
(f)
0.01
0.1640
L Model Analysis
Visualization of Learned Positions. We visualize the learned positions of the query points of CALM-PDE. We conduct the experiment on models trained on the 2D Navier-Stokes equation, 2D Euler equation airfoil, and 2D Navier-Stokes cylinder datasets. Figure 9 shows the latent positions for a regular mesh where CALM-PDE samples more regularly. Figure 10 shows the input and output positions as well learned query positions for the airfoil dataset. The results show that CALM-PDE samples more densely in important regions such as the boundary of the airfoil (see query points of decoder layer 2). A similar, but less emphasized effect be observed for the encoder layer 1. Encoder layer 2 and decoder layer do not show such an effect. Figure shows the positions for the cylinder dataset. Similar the airfoil dataset, CALM-PDE samples where the cylinders are located. This effect can be observed in encoder layer 1 and decoder layer 2. Thus, the learned positions correspond intuitively important regions and learnable query points help the model to have a higher information density in these important regions.
densely in the regions
Query Positions of CALM-PDE 2D Navier-Stokes Dataset
Encoder Layer Query Positions
Layer Query Positions
Input Positions
Latent Positions
0.7
0.7
0.7
0.7
0.3
0.3
0.3
0.3
-0.1
0.3
0.7
-0.1
0.3
0.7
1.1
-0.1
0.7
11
-0.1
0.3
1.1
Decoder Layer Query Positions
Layer Query Positions
Latent Positions
Output Positions
0.7
0.7
0.7
0.7
0.3
0.3
0.3
0.3
0.1
0.3
0.7
-0.1
0.3
0.7
1.1
-0.1
0.3
0.7
1.
-0.1
0.3
0.7
1.1
Figure 9: Input, output, and latent positions of CALM-PDE trained on the 2D Navier-Stokes dataset. CALM-PDE samples more regularly from the domain since the input and output positions are also regularly sampled.
37

---

Query Positions of CALM-PDE 2D Euler Airfoil Dataset
Encoder Layer Query Positions
Encoder Layer Query Positions
Input Positions ns
Latent Positions
1.1
11
1.1
1.1
0.7
0.7
0.7
0.7
0..1
-0.1
0.1
-0.1
0.1
0.3
0.7
1.1
0.1
0.3
0.7
1.1
0.1
0.3
0.7
1.1
0.1
0.3
0.7
1.1
Decoder Layer Query Positions
Decader Layer Query Positions
Latent Positions
Output Positions
11
1.1
0.7
0.7
0.7
0.7
-0.1
-0.1
-0.1
-0.1
-0.1
0.3
0.7
11
-0.1
0.3
0.7
1.1
-0.1
0.3
0.7
1.1
-0.1
0.3
0.7
1.1
Figure 10: Input, output, and latent positions of CALM-PDE trained on the 2D Euler equation airfoil dataset. CALM-PDE samples more densely the boundary of the airfoil (indicated by the red rectangle) by moving more query points to these areas.
Query Positions of CALM-PDE 2D Navier-Stokes Cylinder Dataset
Encoder Layer 1 Query Positions
Encoder Layer 2 Query Positions
Input Positione ns
Latent Positions
1.1
11
1.1
1.1
0.7
0.7
0.7
0.7
-0.1
-0.1
-0.1
-0.1
0.1
0.3
0.7
11
0.1
0.3
0.7
1.1
0.1
0.3
0.7
1.1
0.1
0.3
0.7
1.1
Decoder Layer Query Positions
Decoder Layer 2 Query Positions
Latent Positions
Output Positions
11
0.7
0.7
0.7
0.7
-0.1
-0.1
-0.1
-0.1
-0.1
0.3
0.7
11
-0.1
0.3
0.7
1.1
-0.1
0.3
0.7
1.1
-0.1
0.3
0.7
1.1
Figure 11: Input, output, and latent positions of CALM-PDE trained on the 2D incompressible Navier-Stokes cylinder dataset. CALM-PDE samples more densely in regions where the cylinders are located (indicated by the red rectangle) by moving more query points to these areas.
Information Content of Query Points. Next, we investigate the information content of the query points. We are mainly interested in whether query points represent local global information. Local
38

---

information means that each query point represents the solution of the nearby surrunddings while global information means that a single query point influences the global spatial domain. Due the constrained receptive field, we assume that each token only represents local information. We add Gaussian noise token and investigate how the solution affected validate the hypothesis. Figure 12 shows that only the surrounding of the token is affected which matches our hypothesis that each token represents local information.
2D Navier-Stokes 1e
Prediction Taken Noised
Latent Tokens
Prediction
Absolute Error
1.00
16
5
0.75
6
11
3
0.75
0.75
8
12
0.50
0,50
0.50
0.50
0.25
13
15
1410
0.25
0.25
0.25
7
0.00
0.00
0.00
0.25
0.50 0.75
1.00
0.00 0.25 0.50 0.75 1.00
0.00
0.25 0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
Figure 12: The latent tokens represent local information since adding Gaussian noise token (here:
Token 2) changes only the solution of the token's 's surroundings (here: indicated by a red rectangle).
Visualization of Kernel Weights. Additionally, visualize the learned kernel weights. Figure 13 shows the absolute values and Figure 14 the weights with the sign for the first encoder layer of CALM-PDE trained on the 2D Navier-Stokes dataset. The filters have similarities to the filters learned by discrete CNN and some filters are similar to an edge detection filter (c.g., filter 4 in Figure 14). Figure 15 and Figure 16 show the learned filters for the second encoder layer of CALM-PDE. We only visualize the 64 filters for the first input channel. The filters are less regular and have similarities attention weights Transformers. It is important to note that filters complement each other by having one filter that focuses on subset of tokens and another filter that focuses on the remaining subset of tokens (e.g., filters 31 and 38 in Figure 15).
39

---

Absolute Values of Kernel Weights Encoder Layer Channel
Filer
Filter 2
Fiter
Filter
Filter
Riter
0.60
0.02
0.60
0.05
0.60
-0 025
0.60
0.01
0.60
0.60
0.05
5.00
0.00
0,000
0.35
0.30
0.30
0.20
0.16
0.20
0.20
Filter
Filter
Filter 9
Riter 10
Filter
Filter
0.80
0.60
0.05
0,60
0,60
0.05
0,60
0.60
0.55
0.025
0.55
0.55
0.55
0.5
0.20
6.7
Filter
Filter
Filter
Filter
Filter
Filter
0.60
0.60
0.60
0.60
0.60
0.60
0.58
0.55
0.55
0.025
0.55
0.55
0,025
0.55
0,005
0.00
0.
0.00
00.000
0,000
6.20
0.24
0.05
0.55
0.025
0.35
0.55
0..01
0.58
0.055
0.55
0.02
0.55
0.000
0.00
0.00
0,000
0.00
0,00
0.66
9.25
0.15
620
616
9.20
0.16
0.20
Filter
Filter
Filter
Filter
Filter
Filter
0.60
0.60
0.55
0.025
0.60
0.55
0.60
0.55
D.65
0.60
0.02
0.55
0.60
0.00
0.000
0 1000
0.00
0,00
0,00
0.16
0.50
0.35
0.20
0.24
0.16
0.20
0.34
0.16
0.20
8.44
0.16
0.16
Riter 31
Fiter 32
0.80
0.80
0.025
036
0.20
0.24
eJ6
0.20
0.24
Figure 13: Absolute values of the kernel weights for the first layer of CALM-PDE's encoder trained on 2D Navier-Stokes. The black cross (x) represents the query point.
40

---

Loa med Kernel Weights Encoder Layer Channel
Fitee
Filter
Filter
Riter 4
Riter 5
Filter 6
0.60
0.025
0.60
0.05
0.60
0.025
0.60
0.01
0.60
0.60
0..05
0.025
-0025
0001
- 82825
0.24
036
0.24
Filter
Filter
Filter
Filter 10
Filter 11
Riter 12
0.80
0.60
0.05
0,60
0,60
0,60
0.05
0.60
-0.005
0.55
0.55
0.55
0.55
0.05
0.05
-. 0.025
67 +
0.34
0.20
0.24
6
Filter 13
Filter 14
Filter 15
Fllter 16
Filter 17
Filter 18
0.60
0.05
0.60
0.05
0.60
0.025
0.60
0.60
0.025
0.60
00000
0.58
0,00
0.55
0,00
0.55
-202
0.55
0.00
0.55
0,000
0.55
-0.000
00.5
0.05
8
0.2
0.20
0.24
0,70
0.05
0.005
0.55
0.,00
00.02
0.55
0.00
0.55
0.00
0.58
0,000
0.025
0.55
0,000
0.025
0.55
-0.00
0.05
0.5
0.20
6.5 8 2
0 0.4
0.2
0.20
024
Filter
Filter 26
Filter 27
Filter 28
Filter 29
30
0.60
0.025
0.60
0.60
0.025
0.60
0.05
0.60
0.025
0.60
0.005
0.000
0,000
00025
0.000
0.55
0.00
-005
0,000
0.000
D-16
0.20
615
0.20
6.16
0.50
0.24
0.16
0.20
0.24
0.16
020
0.34
6.1
0.0
0.24
Filter 31
Filter 32
0.05
0.80
0.025
0.60
-8,25
016
024
0.66
Figure 14: Kernel weights for the first layer of CALM-PDE's encoder trained on 2D Navier-Stokes. The black cross (x) represents the query point.
41

---

Absolute Values of Kernel Weights Encoder Layer Channel
Filter
Fiiter 2
Filter
Filter
Fitte
Ritter
0.78
0.81
X
-.005
0.01
0.78
X
0.81
0.70
X
0.78
0.81
X
-0.05
0.11
0.70
X
0.025
0.70
0.81
0.05
0.75
0.75
0.75
0.75
0.75
0.75
0.00
0.00
0.000
0.00
0.000
0,00
0.27
0.90
0.03
0.57
6.90
0.03
0.27
0.90
0.93
0.99
0.57
0.00
0.93
0.87
0.50
0:93
Filter
Fikte
Filter
Filter 10
Filter 11
Filter 12
0.81
0.81
0.81
0.81
0.81
0.81
X
X
0.025
X
X
X
0.025
x
0.02
0.75
0.75
0.75
0.75
0.75
0.75
0.00
0.000
0.000
0.00
0.000
0.00
0.81
0.30
0.93
8.90
0.93
0.90
0.95
0.93
0.99
0.33
0.99
0.95
Filter
Filter
Filter
Filter
Filter
Filter
0.05
0.78
0.78
0.78
00.05
0.78
-0.01
0.78
X
0.05
0.78
x
-0.02
0.75
0.00
0.75
0.000
0.75
-0.00
0.75
0.00
0.75
0.00
0:75
0..00
0.90
0.85
0.02
0.78
0.05
0.78
0.02
0.78
0.78
0.05
0.78
0.05
0.78
0.05
0.75
0.00
0.75
0.00
0.75
0.75
0.75
0.75
Filter
Filter 26
Filter
Filter
Filter
Filter
0.81
0.81
0.81
0.81
0.81
0.78
X
0.02
0.78
X
0.05
0.78
0.025
0.78
0.05
0.78
X
0.05
0.78
-0.55
0.83
0.93
0.83
5.59
0.93
Fiter 31
Fiiter 32
Filter 33
Filter 34
Filter 35
Filter 36
0.81
0.05
0.81
0.51
0.00
0.81
- 0505
0.81
0.1
0.81
0.78
X
0.78
X
0.025
0.78
X
0.78
X
0.78
X
0.78
-0.05
0.75
0.00
0.75
0.000
0.75
0.00
0.75
0.00
0.75
0.0
0.75
0.00
0.87
0.90
0.57
0:90
0.83
0.27
0.90
0.93
0.59
0.57
0.30
0.93
0.50
0.93
0.5
Riter 37
Filter 38
Filter 39
Fitter 40
Filter 41
Filter 42
0.81
0.81
0.81
0.81
0.05
0.81
0.81
0.05
0.78
X
X
D.05
X
0.,0
X
X
0.025
0.75
0.000
0.75
0.00
0.75
0.00
0.75
0.00
0.75
0.000
0.75
0.00
8.87
0.99
0.99
0.87
0.90
0.92
0.87
0.90
0.93
0.87
0.99
0.93
0.87
0.59
0.93
0.87
0.30
0.95
Filter
Filter
Filter
Fitter
Filter
Filter
0.81
0.05
0.81
0.91
0.81
0.81
0.05
0.78
0.78
X
0.78
00.5
0.78
X
0.78
X
0.78
0.75
0.00
0.75
0.00
0.75
0.000
0.75
0.75
0.00
0:75
0.00
0.87
0.90
0.33
0.90
0.33
0.30
0.95
Filter
Filter
0.05
0.005
0.78
0.05
0.78
0.78
0000
0.78
0.78
0.05
0.78
0.75
0.75
0.75
0.75
0.75
0.75
0.87
Filter 55
Filter 56
Filter 57
Filter 58
Filter 59
Filter 60
0.82
0.05
0.81
0.81
-0.1
0.81
- 0.05
0.81
0.05
0.81
0.78
X
0.78
X
0.05
0.78
X
0.78
X
0.78
X
0.78
X
0.00
0.00
0.0
0.90
0.33
0.93
0,57
0,30
0.33
0.57
0.30
0393
Filter 61
Filter 62
Filter 63
Filter 64
0.81
0.05
0.81
0.81
0.055
0.81
0.78
X
0.78
X
0.78
x
0.78
X
0.025
0.75
0.75
0.75
0.75
0.00
0.000
0.00
0.000
0.27
0.00
0.3
0.67
0.90
0.83
0.27
0.90
0.93
0.67
0:30
0.93
Figure 15: Absolute values of the kernel weights for the second layer of CALM-PDE's encoder
trained on 2D Bavier-Stokes. The black cross (x) represents the query point.
42

---

Learned Kernel Weights Encoder Layer Channel
Filte1
Filter 2
Filter 3
Fiter 4
Riter 5
Filter 6
0.81
c.05
0.01
0.05
0.81
0.025
0.
0.05
0.11
0.025
0.81
- 0.05
0.78
x
0.00
0.78
X
0.00
0.78
x
0.78
0.70
0.000
0.70
0.75
-0.05
0.75
-0.05
0.75
0.025
0.75
-0.05
0.75
0.025
0.75
0.05
0.33
0.57
9.00
0.93
Filter 7
Filter 8
Filter 9
Filter 10
Filter 11
Filter 12
0.81
0.81
0.025
0.81
0.025
0.81
0.05
0.81
0.025
0.81
+ 0.025
X
0.000
X
0.000
x
-0,000
X
0.00
X
0.000
X
0,000
0.75
-0.025
0.75
0.025
0.75
0.025
0.75
-0.05
0.75
-0.025
0.75
Filter 13
Filter 14
Fiker 15
Filter 16
Filter 17
Filter 18
0.81
0.05
0.025
0.05
8.01
0.05
n,025
0.78
X
0.00
0.78
0.000
0.78
X
-0.00
0.78
0.00
0.78
0.00
0.78
0,000
0.75
0.05
0.75
0.75
-0.05
0.75
0.75
-0.05
0:75
0.025
67
0
0
21
22
23
0.05
0.025
0.02
0.05
0.05
0.005
0.78
0.00
0.78
0.000
0.78
-0.00
0.78
0.00
0.78
0.00
0.78
0.00
0.75
0.75
0.025
0.75
0.75
2.05
0.75
-0.05
0.75
-0.05
60
+0
+0
Filter 25
Filter 26
Fiter 27
Filter 28
Filter 29
Filter 30
0.81
0.81
0.05
0.81
0.025
0.81
0.05
0.81
0.05
0.81
00.05
0.78
0.78
0.00
0.78
0.78
0.00
0.78
0.78
0.00
-0.025
-0.05
-0.05
-0.05
-0.05
40
Filter 31
Filter 32
Fiter 33
Filter 34
Filter 35
Fitter 36
0.81
0.05
0.81
-0025
0.51
0:05
0.81
0.05
0.81
0.81
0.05
0.78
X
0.00
0.78
X
0.78
x
0.00
0.78
X
0.00
0.78
X
0.78
0.00
0.75
-..55
0.75
825
0.75
0.05
0.75
-0.0
0.75
0.75
-0.05
0.90
0.57
0:50
0.93
Filter 37
Filter 38
Riter 39
Filter 40
Filter 41
Filter 42
0.81
0.81
0..05
0.81
6.025
0.81
0.05
0.81
0.025
0.81
0.05
0.78
0.000
x
-..00
x
0,000
0.78
X
0.00
X
0.000
0.78
0.00
0.75
0.025
0.75
-0.05
0.75
-0.05
0.75
-005
0.75
0.025
0.75
-0.08
0:97
0.99
0.35
0.97
0.99
0.23
0.97
0.90
0.95
0.59
9.97
0.90
0.83
0.87
0.90
0.93
Filter 43
Filter 44
45
46
47
48
0.81
0.05
0.81
0.91
0.05
0.81
0.05
0.81
0.05
0.81
0.05
0.78
0.00
0.78
-0.00
0.78
0.00
0.78
0.00
0.78
0.00
0,00
0.75
0.05
0.75
0.75
-0.05
0.75
-0.5
0.75
0:75
0.05
60
0
Filter
Filter 50
51
52
53
54
0.05
-0025
0.05
0.81
- 0.025
0.78
0.00
0.78
-0.00
0.78
-0,000
0.78
0.00
0.78
0.00
0.78
0,000
0.75
0.75
0005
0.75
0.75
-0.0
0.75
-0.08
0.75
0.025
60
50
0.30
Filter 55
Filter 56
fikter 57
Filter 58
Filter 59
Filter 60
0.82
0.05
0.81
0005
0.81
0.81
0.05
0.81
+ 0.05
0.81
0.0t
0.78
X
0.00
0.78
0.000
0.78
0.78
0.00
0.78
0.00
0.78
0.00
0.1
-0.05
0.75
-0.55
-0.01
0,87
0.90
0.33
0.90
Filter 61
Filter 62
Fiker 63
Filter 64
0.82
0.05
0.81
0.025
0.81
0.05
0.81
0.025
0.78
x
6.00
0.78
x
0.000
0.78
x
0.00
0.78
X
0.000
0.75
0.75
0.025
0.75
-0.05
0.75
0.025
Figure 16: Kernel weights for the second layer of CALM-PDE's encoder trained on 2D Navier-Stokes. The black cross (x () represents the query point.
43

---

Size of Latent Space. We compare the sizes required for one timestep of a PDE in different representations. We define the representation size as the number spatial points tokens times the number of channels per token (e.g., 64 64 64 z 1,3M for a 3D PDE with a spatial resolution of 64 x 64 x 64 and channels). A smaller representation size or latent space can result in reduced model complexity (memory consumption and or inference time). However, if a model has smaller latent space but the computations are computationally expensive, it might be possible that it is less efficient compared to model with larger latent space but simpler computations. Furthermore, small latent space (i.e., dimensionality reduction) can eliminate redundant features and noise. Our observations show that the baselines often have a larger latent space than the physical space while CALM-PDE consistently performs a compression and has a smaller latent space compared to the representation in the physical space.
Table 15: Representation sizes for one timestep of different PDEs in the physical and model's s latent spaces. "A larger latent space infeasible for these models due to memory constraints in the experiments.
Representation Size (1)
Model
ID Burgers'
2D Navier-Stokee
Navier-Stokes
2D Airfoil
2D Cylinder
Physical Space
1024
4096
1.3M
20,932
5655 (avg)
FNO
2048
18,816
276k
32,256
32,256
OFormer
98,304
786k
67M
670k
241k (avg)
PIT
65,536
65,536
131k
65,536
65.536
LNO
12,288
24,576
16,384
24.576
22.528
CALM-PDE
512
2048
16,384
2048
2048
M
Ablation Study
We conduct an ablation study to investigate the effect of (i) learnable query positions, (ii) kernel modulation, and (iii) distance weighting. Furthermore, we replace the proposed kernel function with function that outputs weights only based on the Fuclidean distance. We also investigate the influence of different latent dimensions on the model's performance. We test the modified models on the 2D) Fuler equation airfoil dataset and the 2D Navier-Stokes dataset.
M.1 Learnable Query Positions
In the first experiment, we investigate the effect of learnable query positions. We conduct the experiment on the 2D Euler equation airfoil dataset with an irregularly sampled spatial domain. We examine three different configurations of CALM-PDE with (a) learnable query points and mesh prior initialization enabled (i.e., we initially sample the query points from the mesh and allow the model to change their positions). (b) no learnable query points but with mesh prior (i.e., we sample from the mesh to obtain the query points and their positions are fix), and (c) no learnable query points and no mesh prior (i.e., the query points are randomly sampled from the entire spatial domain and fixed). Table 16 shows that the model with learnable query points and mesh prior achieves the lowest errors. Soley sampling from the mesh to obtain the query points (mesh prior and no learnable query points) yields a significantly higher error. Using random query points with fixed positions results in the highest errors which is intuitive since the model does not necessarily have query points at important regions and cannot move query points to these regions. Thus, it is forced learn the underlying mesh only with the kernel function.
M.2 Kernel Modulation
In this experiment, we disable the query modulation. We conduct the experiment on the 2D Navier- Stokes dataset with a regularly sampled spatial domain and the 2D Fuler airfoil dataset with an irregularly sampled spatial domain. We compare the CALM-PDE model with (a) enabled kernel modulation against model (b) without kernel modulation. Table 17 shows the relative L2 errors for
44

---

Table 16: Relative L2 errors of CALM-PDE models trained and tested on the 2D Euler equation airfoil dataset. The values in parentheses indicate the percentage deviation to CALM-PDE with enabled learnable query points and mesh prior. Underlined values indicate the second-best errors.
Model Configuration
Relative L2 Error (1)
Learnable Query Points
Mesh Prior
2D Euler Airfoil
(a)
0.0515 10.0007
/
0.0709 20.0016 (+38%)
X
0.1627=0.0044 (+216%)
Table 17: Relative 1.2 errors of CALM-PDE models trained and tested on the 2D Navier-Stokes and 2D Euler airfoil datasets. The value parentheses indicates the percentage deviation to CALM-PDE with enabled kernel modulation.
Model Configuration
Relative 1.2 Error (L)
2D Navier-Stokes
Kernel Modulation
2D Euler Airfoil
le
(a)
0.0301 =0.0014
0.0515#0.0000 0.0632 (+23%)
(b)
x
0.0358-10.0014 10.0014 (+19%)
both model configurations. The proposed kernel modulation, which allows each query point to have different kernel function, results in lower error than the models without the kernel modulation.
M.3 Distance Weighting
Furthermore, we investigate the effect of the distance weighting term, which includes the Euclidean distance directly into the kernel function to give more weight to closer points. The mechanism ensures that the model puts more emphasis on closer points within the epsilon neighborhood, provides feedback that points on the boundary of the epsilon neighborhood are less important to avoid hard cuts caused by the displacement of query points, and serves as a normalization factor for the Monte Carlo integration. We compare three models with (a) distance weighting, (b) without distance weighting but with a normalization term 1/M (/)F(a)) and (c) without distance weighting and without normalization term. The kernel function of the model without distance weighting but with normalization term is given as
ki - MLP (RFF a)) RF(())
(24)
where .M |RF( denotes the number of input points the receptive field RF(a) of query point a. The kernel function of the model without distance weighting and without normalization term is given
(a o.) MMPP(FFF(a -a))..
(25)
where MLP denotes the modulated two-layer multi-layer perceptron and RFF random Fourier features. Table 18 shows that the models without distance weighting and without normalization term diverge during training. Including a normalization term 1/M stabilizes the training but still yields a higher error than the model with distance weighting. Thus, distance weighting, which emphasizes input points closer to the query point and serves as normalization, crucial for reducing the error.
M.4 Distance-based Kernel Function
We replace the proposed kernel function, which computes the weights based on translation vectors and the Fuclidean distances, with a kernel function that computes the weights solely based on the Euclidean distances. The temperature in the omax function is learnable parameter, and each
45

---

Table 18: Relative L2 errors of CALM-PDE models trained and tested on the 2D Navier-Stokes dataset. "The value is unavailable since the models diverge during training.
Model Configuration
Relative L2 Error (4)
2D Navier-Stokes V 1e
Distance Weighting
Normalization
(a)
/
softmax 1/M
0.0301 #0.0014
(b)
X
0.059040.0122 (+96%)
(c)
X
X
N/A
Table 19: Relative 1.2 errors of CALM-PDE models trained and tested on the 2D Navier-Stokes dataset. The value in parentheses indicates the percentage deviation to CALM-PDE with the proposed kernel function.
Model Configuration
Relative 1.2 Error (4)
2D Navier-Stokes
Kernel Function
1e-4
(a)
CALM-PDE
0.0301 #0,0014
(b)
Distance-based
0.1783+0.0062 (+492%)
filter has its own temperature parameter. The kernel function is given as
ll--all-miu(a exp(
ki a)
T, (max((a))))) mint(a)
min(a) min (3
|a ay
(26)
exp( aa-a, -mi((a) (max((((( min)())
(RF(F(a)
where T., represents learnable temperature parameter and RF(()) the receptive field for the query point We compare the CALM-PDE model with (a) the proposed CALM-PDE kernel function against a model with (b) the distance-based kernel function. Table shows that the kernel function proposed by CALM-PDE: achieves lower error than distance-based kernel function.
M.5 Latent Dimension
We investigate the effect of different latent dimensions by increasing the number of query points. Starting from model (a), which uses small number of query points, we incrementally scale up to model (c), characterized by a large number of query points. As shown in Table 20, increasing the latent dimension initially leads to reduction in test eтror. However, beyond a certain threshold, further increases of the latent dimension result in an increasing test error, suggesting overfitting or diminishing returns. The table also reports training time, which consistently increases with the number of query points, reflecting the computational cost of higher latent dimension. The notation [256, 64, 4] denotes 256 query points for the first layer, 64 query points in the 2nd layer, and 4 for the last layer.
46

---

Table 20: Relative L2 errors and training times of CALM-PDE models with different numbers of query points trained and tested on the 2D Navier-Stoker dataset.
Model Configuration
Relative L2 Error ())
Training time (L)
Number of Query Points
2D Navier-Stokes
Encoder
Decoder
V == le
(a) [256, 64, 4]
[64, 256, -]
0.0533
4:18h
(b)
[512, 128, 8]
[128, 512,
0.0425
4:40h
(c)
[1024, 256, 16]
[256, 1024,
0.0304
5:25h
(d) [2048, 512, 32]
[512, 2048,
0.0251
6:21h
(e) [4096, 1024, 64]
[1024, 4096, -]
0.0267
8:19h
N Additional Results
N.1 Quantitative Results
This section provides the full benchmark results with the standard deviations of multiple runs, which we omit for a more compact representation in the tables in the main paper. Table 21 shows the errors for the experiments with regular meshes and Table 22 for the irregular meshes.
Table 21: Relative L2 errors of models trained and tested on regular meshes. The values in parentheses indicate the percentage deviation CALM-PDE and underlined values indicate the second-best errors.
Relative L2 Eroor (L)
Model
ID Burgers'
2D Ntvier-Stokes
2D Navier-Stokes
3D Navier-Stokes
y le-3
v= le-4
le
1= ( le
FNO
L.0358
(+46%)
0.0811 10.00000000
(+169%)
0.091240.00m
(-12%)
0.6898 10.0002
(+2%)
F-ENO
0.0362#0.000
(+47%)
0.0863#0.001
(+187%)
0.0844 0008
(-18%)
0.0064
(-4%)
OFormer
0.0875+0.0015
(+134%)
0.0380
(426%)
0.193g 0.193g+0.0016
00.010
(+88%)
0.6719=0.010
(-1%)
PIT
0.1209
(+391%)
0,0467#0.00
(+55%)
0.1633
(+58%)
0.7423
0.7423=0.00m
#0.0036
(+10%)
INO
0.0309 10.0011 (+26%)
0.0384
(+28%)
0,0789
(-24%)
0.7063
1000027
(+4%)
AROMA
0.0937
(+281%)
0.1061
+0.0455
(+252%)
0,1931
(+87%)
1.3328
(+97%)
CALM-PDE
0.0246
0,0301 0010
0.1033
0.6761 20.0020
Table 22: Relative L. errors of models trained and tested on the irregular meshes with geometries in the fluid flow. The values in parentheses indicate the percentage deviation to CAEE and underlined values indicate the second-best errors.
Relative L.2 Error (4)
Model
2D Euler Airfoil
2D Navier-Stokes Cylinder
Geo-FNO
0.038g 10000
(-25%
0. 0.1383=000001 (+17%)
F-FNO
0.1081 00.01995
(+1 (10%)
0. 0.1490+000000
(+26%)
OFormer
0.0520 10.5001..000 10.0003
(+1%)
0.2264 10.01009 (+91%)
PIT
0.089440.00m7
(+74%)
0.1400=40.004
(+18%)
LNO
0.0582 0026
:000006
(+13%)
0.16544t0.0mm
(+39%)
AROMA
0.0372 $0.72-0.000 00.000
(-28%)
0.1139: -000027
(-4%)
CALM-PDE
0.0515 10.0007
0.1186
N.2 Qualitative Results
We visualize the predictions and ground truth of randomly selected trajectories for the benchmark problems.
ID Burgers' Equation: Figure 17
47

---

2D Navier-Stokes with Parameters y/ = 1ee and v = 1e=5: Figure 18
2D Euler Equation with Airfoil Geometry: Figure 19, Figure 20, Figure 22, Figure 21
2D Incompressible Navier-Stokes with Cylinder Geometries: Figure 23, Figure 24, Figure 25
48

---

Burgers' Equation
Prediction
Absolute Error
0.0
0.2
0.4
0.6
0.8
1.0
u10.05 62)
0.02
0.00
0.0
0.2
0.6
1.0
0.0
0.2
0.8
0.8
u00-10.u
0.2
u10.35 k.
0.2
0.0
0.0
0.2
0.4
0.6
0.0
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0,0.20 wt
0.5
0.0
uI0 10.25.u ()
0.5
0.2
0.0
U/0.30 31.30 a)
0.0
0.1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.5
w/0. 35
0.0
0.5
U10 10040,00
0.1
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
u(0.45.60
0.0
0.0
v/0.50.ul
0.5
Prediction
0.1
0.0
Figure 17: Ground truth, prediction, and absolute error for the first 11 timesteps of a randomly selected ID Burgers' equation trajectory.
49

---

20 Navier- Stokes le
20 Nevier-Stokee 1e
Ground Truth
Prediction
Abso
Ground Truth
Prediction
Absolute
10b. al
-05
0.05
-90
111
-05
0.05
0.15
-110
0.05
010
8
0.15
0.10
210
0.05
02
21
02
022
01
21
03
.E10.u0
02
32
01
(a) Parameter 1e-4
(b) Parameter
le
Figure 18: Ground truth, prediction, and absolute error for the first 11 timesteps of two randomly selected 2D Navier-Stokes 1 and 1e trajectories.
50

---

Equation
Velocity
Ground Tuth
Prediction
Absolute Error
u10 al
v10.32..0 oal
200
44
200
100
-50
50
0
200
1000
Mi 2012000,00 sal
50
-0
10 10.0000 oa
150
9
550
U1 992 vol
100
-30
200
V12 NG2.8
100
-200
200
100
-20
300
200
-200
300
U13-20000 Ael
-200
Figure 19: Ground truth, prediction, and absolute error of the velocity for the first 11 timesteps of a randomly selected 2D Euler equation with airfoil geometry trajectory.
51

---

Velocity Vy
Ground
Prediction
Absolute
40
100 000.322..m ub
-100
-2
1000000 u.
50
200
25
300
0
75
-25
111280 ool
75
V1t 100 A0
-200
200
100
NAC 92 vall
50
M12 11224.000 vol
100
200
01223 51.5% 60)
100
-200
v122 200 vol
200
200
200
300
MSS20..u wal
9
Figure 20: Ground truth, prediction, and absolute error of the velocity vy for the first 11 timesteps of a randomly selected 2D Euler equation with airfoil geometry trajectory.
52

---

Equation
Density
Ground Truth
Prediction
Absialute Error
010 a)
U10: 0003222 ua
1.0
A1
0.5
1.00
0.10
0.5
0.15
0.10
0.05
0.5
1.25
0.4
0.75
0.50
0.5
v22 21229.00 oll
1.0
0.2
005
0122 200 ool
0.3
1.0
0.5
663.20 ml
0.4
1.0
005
Figure 21: Ground truth, prediction, and absolute error of the density for the first 11 timesteps of randomly selected 2D Euler equation with airfoil geometry trajectory.
53

---

Equation
Ground Truth
Prediction
Absolute Error
u10 al)
UIo 02-3222.0 vol
100000
50000
125000
110 00, um
1000000
75000
20000
15000
10000
50000
5000
100000
10000
5000
UF v00 wll
100000
50000
20000
50000
30000
V/22 ol
20000
vE2356.u A2
20000
10000
v12 100 val
1.00000
40000
50000
20000
100000
20000
Figure 22: Ground truth, prediction, and absolute error of the pressure for the first 11 timesteps of randomly selected 2D Euler equation with airfoil geometry trajectory.
54

---

Velecity
Gound
Prodiciion
un a.
0475
039
000 ml
04
-04
Figure 23: Ground truth, prediction, and absolute error of the velocity us for the first 11 timesteps of a randomly selected 2D incompressible Navier-Stokes with cylinder geometries trajectory.
55

---

Velecity
1434 al
25
0.25
15
30
645
v0.2
30
30
0.2
-5
04
02
10
04
40
44
25
ME.0 ml
Figure 24: Ground truth, prediction, and absolute error of the velocity by for the first 11 timesteps of a randomly selected 2D incompressible Navier-Stokes with cylinder geometries trajectory.
56

---

Pressure
Gound Tuut
Prodictoon
Absokaia Bra
viti ud
0,500
40.2 al
0.590
0.55
4p.5 uf
00
00
Figure 25: Ground truth, prediction, and absolute error of the pressure for the first 11 timesteps of randomly selected 2D incompressible Navier-Stokes with cylinder geometries trajectory.
57

---

Additional Experiments
Beyond the main experiments, we explore time-independent PDEs with complex and varying ge- ometries. Additionally, we evaluate an rrooder-decode-oon variant of CALM-PDE trained with reffsrrcontttuuttion loss to test its encoding and decoding capabilities. Finally, we analyze the scalability of the model with respect the output query mesh size.
0.1 Time-independent Problems
We conduct additional experiments on time-independent PDE problems. In particular, we experiment on the -nneinndeenddnt airfoil and elasticity datasets from Geo-FNO (Li et al., 2023c). The task is to map an input geometry (e.g.. changing airfoil shape) to a physical quantity. We follow the setup from LNO (Wang& & Wang. 2024) and take the errors for the baselines from their work. The baselines include Geo-FNO (Li et al., 2023c), F-FNO (Tran et al., 2023), Galerkin Transformer (Cao, 2021), OFormer (Li al., 2023a). GNOT (Hao al., 2023), ONO (Xiao al., 2024), Transolver (Wu et al., 2024), and LNO (Wang & Wang, 2024). As shown in Table 23, CALM-PDE consistently outperforms out of 8 baselines and ranks among the top models alongside LNO and Transolver. Additionally, the results demonstrate that CALM-PDE: can generalize across complex, varying geometries. The hyperparameter for CALM-PDE are denoted in Table 24.
Table 23: Relative 1.2 errors of models trained and tested on the time-independent airfoil and elasticity datasets. The values in parentheses indicate the percentage deviation to CALM-PDE and underlined values indicate the second-bes, errors.
Relative 1.2 Error x10 (4)
Model
Time-indepdent Airfoil
Elasticity
Geo-FNO
1.38 (+138%)
2.29 (+332%)
F-FNO
0.60 (+3%)
1.85 (+249%)
Galerkin Transformer
1.18 (+103%)
2.40 (+353%)
OFormer
1.83 (+216%)
1.83 (+245%)
GNOT
0.75 (+29%)
0.88 (+66%)
ONO
0.61 (+5%)
1.18 (+123%)
Transolver
0.47 (-19%)
0.62 (+17%)
LNO
0.51 (-12%)
0,52 (-2%)
CALM-PDE
0.58
0.53
Table 24: Hyperparameters for CALM-PDE used in the time- cone-independent experiments.
PDE
Parumeter
Time-independent Airfoil
Elasticity
Layers
Encoder
Channels
[64, 96, 128]
Query Points
[128, 64, 64]
[1024, 256, 16]
Percentile
[0.25, 0.5]
[0.02,
Temperature
[1.0, 1.0, 1.0J
[1.0. 1.0, 0.1]
Layers
3
Decoder
Channels
196, 64, 11
Query Points
[512, 2048,
[256. 1024,
Percentile
[0.25, 0.01. 0.01]
Temperature
1.0, 1.0]
1.0,
Periodic Boundary
X
Mesh Prior
X
Processor Layers
Random Sturting Points
Leamiig Rate
lc-4
2e-4
Size
Epochs
500
Parameters
4,378,273
4,353,313
58

---

0.2 Encoding and Decoding Capabilities
We train an encoder-decoder-onle variant of CALM-PDE with a self-reconstruction loss (i.e., an autoencoder) to investigate the encoding and decoding capabilities of the model. We train CALM-PDE the self-reconstruucionn
as an autoencoder on the 2D Navier-Stokes v 1e dataset and compare error reported in AROMA (Serrano et al., 2024). The results in Table 25 show that CALM-PDE achieves a significantly lower reconstruction relative 1.2 error compared to AROMA.
Table 25: Relative L2 errors of AROMA and CALM-PDE models trained and tested on the 2D Navier-Stokes dataset. The value in parentheses indicates the percentage deviation CALM-PDE.
Reconstruction Relative L2 Error x10 2 (L)
Model
2D Navier-Stokes
== le-1
AROMA
1.049 (+141 %)
CALM-PDE
0.435
0.3 Scaling of Output Query Mesh Size
To investigate the scaling behavior of the model for large meshes, we increase the number of output query points during inference. The experiment is conducted on the 2D Navier-Stokes v le-4 dataset, where the maximum resolution of the ground truth is limited to 64 x 64. Thus, we only provide the inference time and GPU memory consumption. The experiment is done on an NVIDIA A100 GPU with batch size of 32 and predicting timestep. Table 26 shows the scaling behavior of the model for increased output query mesh sizes.
Table 26: Inference time and memory consumption of CALM-PDE for different output query grid sizes on the 2D
dataset.
2D Navier-Stoker
Qutput Query Mesh Size
Inference Time [ms]
Inference Memory [GB]
64 64 (4096) 128 x 128 (16,384)
10.27
0.84
14.83
1.80
256 256 (65,536)
33.89
5.64
512 512 (262,144)
107.52
21.00
P Notation
Scalars, Vectors, and Multi-dimensional Arrays. We follow the convention and represent scalars with a small letter (c.g.. a), vectors with small boldfaced letter (e.g., a), and matrices and N- dimensional arrays (with N 3) with capital boldfaced letter (e.g., A).
Partial Derivatives.
The notation Ow Oww ut ... is short for the ith order (where i E {1,2,.., n))
partial derivative
Su
8"u 8" Twn
Dis
59


Reviews:
[{"review_id": "HC0tqAB6Ui", "reviewer": "Reviewer_2MgG", "summary": "This paper introduces CALM-PDE, a neural surrogate model for PDE modeling which compresses the spatial domain into a latent representation using continuous convolutional layers with adaptive query points. The method handles arbitrarily discretized (irregular) spatial meshes and evolves the solution entirely in latent space, achieving competitive accuracy on several benchmark PDEs while significantly reducing memory usage and inference time compared to various baselines.", "questions": "1) Can the authors show a comparison of performance and efficiency to Factorized FNO? it is a variant of FNO which performs competitively to other baselines described in the paper on similar tasks. Given its relevance and efficiency, particularly on similar problems, it seems like an important missing baseline.\n2) I also think another recent architecture like Transolver should be discussed and compared to. The best comparisons would be on the irregular grid problems like airfoil, elasticity, plasticity or pipe problems commonly used as baselines for these problems.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1) The method seems to work effectively for both regular and irregular grids. \n2) It also achieves better inference time scaling wrt recent models like LNO (latent neural operator).\n3) The paper is written clearly and is easy to read. \n\nWeaknesses: \n\n1) The accuracy improvements are not universal. In some benchmarks CALM-PDE is outperformed by other models. For example, on one or two regular-grid PDE task the model fell behind FNO and LNO, and on an irregular airfoil flow the Fourier-based Geo-FNO achieved better accuracy. This suggests CALM-PDE’s advantages (for reducing the error) are context-dependent rather than across-the-board. This also suggests that more baselines and datasets are needed for establishing a clearer gain in errors. Other papers in this field compare to various other irregularly spaced datasets such as elasticity, plasticity etc. See the LNO paper for example.\n\n2) Some important baselines such as factorized FNO (FFNO) and transolver, which are quite competitive in this domain are missing. \n\n3) By compressing the solution into a smaller latent space, the method inevitably discards some high-frequency or fine-grained information.", "was_revised": true, "final_justification": "The authors have provided some new results. One sticking point is I really think this method will not work that well had the authors used more complex datasets, such as Rayleigh Benard simulations (or more complex MHD simulations in Physics), as I mentioned during discussions. \n\nSo I am unsure this architecture will actually become useful in the future. But I appreciate the author's efforts during the rebuttal and some people at this conference might still be interested to hear about this; so I'm fine with increasing my score by 1 point. But I am not going to champion this paper.", "rebuttal": "Thank you for your insightful feedback and thoughtful questions, which helped us to enhance our submission. We appreciate acknowledging the versatility of our approach in handling both regular and irregular grids effectively. We've carefully reviewed each of your points and provide our responses below.\n\n> **(W1)** The accuracy improvements are not universal. [...] Other papers in this field compare to various other irregularly spaced datasets such as elasticity, plasticity etc. See the LNO paper for example.\n\nAs you already pointed out with strength 1, CALM-PDE works effectively for both regular and irregular grids. We would like to emphasize that we do not claim CALM-PDE to be the best model across all problems, as no single model can perform best on all PDE tasks. Rather, our goal is to demonstrate that CALM-PDE is a strong and competitive alternative, particularly in challenging settings such as irregular spatial domains. Indeed, CALM-PDE outperforms all baselines in 3 out of 6 benchmarks and achieves second-best errors in 2 out of 6 benchmarks. Even on the remaining benchmark, CALM-PDE outperforms two transformer-based baselines, reducing the error by 58%. Overall, these results provide evidence that CALM-PDE is a viable and competitive alternative to commonly used FNOs and Transformer architectures for solving PDEs. Notably, the experiments demonstrate the effectiveness of convolutional models on irregularly sampled spatial domains and the benefit of sampling the spatial domain via adaptive query points. \n\nMoreover, we use strong baselines and incorporate additional techniques to further improve them. For instance, we train the autoregressive models (e.g., FNO and Geo-FNO) with a curriculum strategy, which significantly reduces their errors by 39% compared to full autoregressive training (see Table 6 in Appendix I, line 799). Taken together, these results position CALM-PDE as a novel and competitive approach for neural PDE solving.\n\nFollowing your suggestion, we’ve incorporated the airfoil and elasticity datasets into our analysis. While these datasets are time-independent, our primary focus remains on time-dependent PDEs. We follow the benchmark setup of the LNO paper and provide the baseline errors reported in that work  (see table below).\n\n| Model | Time-independent Airfoil ($\\times 10^{-2}$ Rel. L2 Error) | Elasticity ($\\times 10^{-2}$ Rel. L2 Error)\n| ----------- | ---------------------- | ---------- |\n| Geo-FNO | 1.38 | 2.29 |\n| F-FNO | 0.60 | 1.85 |\n| Galerkin | 1.18 |  2.40 |\n| OFormer  | 1.83 | 1.83 |\n| GNOT |  0.75 | 0.88 |\n| ONO | 0.61 | 1.18 | \n| Transolver | 0.47 | 0.62 | \n| LNO | 0.51 | 0.52 |\n| CALM-PDE | 0.58 | 0.53 |\n\nThe results show that CALM-PDE outperforms 6 out of 8 baselines and ranks among the top models alongside LNO and Transolver, demonstrating strong performance on both time-dependent and time-independent problems. While Geo-FNO underperforms in this setting, it remains among the top models on time-dependent tasks, highlighting that model performance can vary with problem type. We do not claim universal superiority, but CALM-PDE offers a competitive, convolution-based alternative to Fourier and transformer models.\n\n> **(W2)** Some important baselines such as factorized FNO (FFNO) and transolver, which are quite competitive in this domain are missing.\n\nWe’ve added F-FNO as an additional baseline and benchmarked it on the 2D Navier-Stokes datasets as well as the irregularly sampled cylinder dataset. Please see the following table. For comparison with Transolver and a further comparison with F-FNO, we added the airfoil and elasticity datasets. Please see the table in our response to (W1).\n\n| Model    | 2D Navier-Stokes $\\nu=1e^{-4}$ (Rel. L2 Error) | 2D Navier-Stokes $\\nu=1e^{-5}$ (Rel. L2 Error) | 2D Cylinder (Rel. L2 Error) |\n| -------- | --------------------- | --------------------- |  ----------- |\n| F-FNO | 0.0877 (+191%) | 0.08573 (-17%) | 0.1420 (+20%) |\n| CALM-PDE | 0.0301 | 0.1033 | 0.1186 |\n\n> **(W3)** By compressing the solution into a smaller latent space, the method inevitably discards some high-frequency or fine-grained information.\n\nCompressing the solution into a smaller latent space inevitably discards some high-frequency details, a trade-off we accept for improved efficiency. We acknowledge this limitation explicitly in the paper (see line 321 in the Limitations section), and it is a common challenge in reduced-order models.\n\n> **(Q1)** Can the authors show a comparison of performance and efficiency to Factorized FNO? [...] Given its relevance and efficiency, particularly on similar problems, it seems like an important missing baseline.\n\nWe added F-FNO as an additional baseline. Please see our response for (W1) and (W2).\n\n> **(Q2)** I also think another recent architecture like Transolver should be discussed and compared to. The best comparisons would be on the irregular grid problems like airfoil, elasticity, plasticity or pipe problems commonly used as baselines for these problems.\n\nWe benchmarked our model on the mentioned airfoil and elasticity datasets and included the errors for Transolver. Please see our response for (W1)."}, {"review_id": "5HCh3W47J1", "reviewer": "Reviewer_dQvB", "summary": "The paper proposes a novel architecture (CALM-PDE) for reduced-order modeling of arbitrarily discretized PDEs. The building block CALM layer uses continuous convolution and adaptively learn query points. Numerical experiments on initial value problems in fluid dynamics show the effectiveness of the method.", "questions": "Q1) What values of positions $P$ of latent tokens in Equation (8) can take? How are they initialized and learned? \n\nQ2) How to determine the appropriate dimensionality of the latent variables? Will we observe a saturation effect or diminishing returns in prediction accuracy as the latent dimension increases? Should we use the same hyperparameters for different latent dimensions?\n\nQ3) Can the model handle cases where the physical domain geometry itself is treated as a random input parameter to the neural operator? That is, can it generalize across varying domain shapes or boundary configurations as part of the input space?", "limitations": "Yes, adequately described in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n\nS1) The paper is clearly written and well-motivated (particularly in Section 6, which presents the experiments and evaluation through Q\\&A effectively). \n\nS2) The appendix is thorough and of high quality, offering essential details that enhance understanding and reproducibility of the main text. I found the visualizations of the learned positions in Section K quite informative. \n\nS3) Learning parametric PDEs on irregular meshes is challenging but highly meaningful for real-world applications. The proposed use of a discretization-agnostic decoder, combined with adaptive learning of query points, presents a general strategy that could be broadly applicable to other neural operator frameworks.\n\n[Weaknesses]\n\nW1) The results on the 2D Navier--Stokes equation with high Reynolds number seem not strong enough compared to FNO in terms of prediction accuracy, training time (and memory), and inference time. \n\nW2) There could be more hyperparameters (such as temperature $T$, receptive field radius $\\varepsilon$, number of query points, etc) to tune compared to other baseline models. And it is unclear how robust the model is to these hyperparameters' change.\n\nW3) There has been a lot of follow-up work on reducing the memory consumption of the transformer-based methods. It is possible that using a more efficient way of attention mechanism, transformer-based methods still perform equally well or even better.", "was_revised": true, "final_justification": "The presentation and experiments in the paper are solid and of high quality. In the rebuttal, the authors address my concern about the generalizability and robustness of the method to different PDE setups. Although its current version does not outperform all baselines in terms of accuracy, I think it is still competitive. The ideas the paper offers are inspiring, together with its fruitful experimental results, could be worthy to the research community.", "rebuttal": "Thank you for carefully reviewing our work and asking insightful questions, which helped us to improve our submission. We are grateful that you recognize our approach as a strategy with the potential for wide applicability across various neural operator frameworks. We address your concerns and questions separately as follows.\n\n> **(W1)** The results on the 2D Navier--Stokes equation with high Reynolds number seem not strong enough compared to FNO [...]\n\nWe developed the architecture for irregularly sampled spatial domains, which are important for many real-world applications, as you already pointed out in strength 3. We included the benchmark on regular grids to demonstrate that our approach is also directly applicable to such settings and yields competitive errors. Thus, CALM-PDE is not specially optimized for regular grids. Since the grid is uniformly structured, the inference time, training time, and memory consumption could be improved by exploiting the uniform structure, which we consider as interesting future work. However, we specifically target PDEs with irregularly sampled spatial domains with our method.\n\n> **(W2)** There could be more hyperparameters [...] to tune compared to other baseline models. And it is unclear how robust the model is to these hyperparameters' change.\n\nThe model introduces the softmax temperature, receptive field radius, channel dimension, and number of query points as hyperparameters. The receptive field radius is equivalent to the kernel size in conventional CNNs and the number of query points to downsampling (e.g., stride and pooling) in CNNs. The channel dimension is equivalent to the channel dimension in CNNs. Alternatively, the number of query points can be compared to the query points used in other models, such as AROMA [1] or LNO [2]. Thus, our architecture does not introduce more hyperparameters.\n\nSelecting good hyperparameters for our model does not require an extensive hyperparameter search. As shown in Table 11 in Appendix J.4 (line 856), we use the same temperatures, receptive field sizes (percentiles), and number of query points for all 2D experiments, which demonstrates that the hyperparameters are robust and work across different problems.\n\nOnly the temperature is a new hyperparameter that controls the distance weighting within the receptive field. We also use the same temperatures across all experiments, demonstrating its robustness. However, we’ve conducted a hyperparameter study for the temperature, which shows that it is also robust to changes ($T \\in [0.1, 5]$ yields small changes in error). We ran the experiments on the 2D Navier-Stokes $\\nu=1e^{-4}$ dataset.\n\n| Temperature T in Encoder and Decoder | Rel. L2 Error |\n| ----------- | ---------------------- |\n| 10 | 0.0398 |\n| 5 | 0.0336 |\n| 2 | 0.0316 |\n| 1 | 0.0304 |\n| 0.1 | 0.0341 |\n| 0.01 | 0.1640 |\n\nThe table shows that T=1 yields the lowest errors and that a slightly smaller (e.g., T=0.1) or larger (e.g., T=5) temperature results only in small changes in error. This demonstrates the robustness of the temperature parameter.\n\n\n> **(W3)** There has been a lot of follow-up work on reducing the memory consumption of the transformer-based methods. [...]\n\nIn contrast to (efficient) transformer-based methods, CALM-PDE introduces a fundamentally different and efficient architecture grounded in continuous convolution. The model demonstrates that convolution can be applied to irregularly sampled spatial domains, where usually transformer-based methods dominate [1, 2, 3], and that restricting the receptive field inherently reduces the computational and memory complexity. Thus, CALM-PDE is a competitive alternative to (efficient) transformer-based methods. To address concerns around efficiency, we compare our method with OFormer, which leverages linear attention to improve memory and runtime of the attention mechanism (see Figure 4 and Table 3 on page 9). Importantly, the implementation of CALM-PDE is not heavily optimized for GPUs, similar to the implementations of the baselines, which we consider a fair comparison. However, with optimized implementations, the efficiency of CALM-PDE can be improved further, which we aim to investigate as interesting future work.\n\n\n> **(Q1)**  What values of positions $P$ of latent tokens in Equation (8) can take? How are they initialized and learned?\n\nThe positions $P$ of the latent tokens are the learned query positions of the last encoder layer. We did not constrain the positions that the latent tokens can take. Thus, they could potentially take an arbitrary position. For periodic domains, this is not an issue because the positions are corrected and will be periodically mapped to $[0,1]^n$. For non-periodic domains, we noticed that the positions do not move too far away and stay close to the spatial domain. We opt for this design to keep the method simple. However, additional regularization to force the positions to stay in $[0,1]^n$ could be tested for future work. The plots with the title “Latent Position” in Appendix K (pages 27 and 28) show the learned positions $P$.\n\nWe have two ways of initializing the query positions. a) They are uniformly sampled from $[0,1]^n$ to achieve a uniform cover of the spatial domain. b) They are sampled from a training mesh to have a better initialization and to capture the underlying mesh structure. We use b) only for the airfoil experiment, where we have one irregular mesh for all samples, and a) for the remaining experiments.\n\nThe query points are learned similarly to the weights and biases of a neural network. The query points influence how the input features are weighted. Thus, we can compute the gradient of the loss function with respect to the query positions. The gradient is used to optimize the query positions (i.e., move the query points) with a variant of stochastic gradient descent (we use AdamW [4]).\n\nDetails about the initialization and learning of the query positions are explained in the paragraph “Learnable Query Points\" in line 132 in the paper.\n\n\n> **(Q2)** How to determine the appropriate dimensionality of the latent variables?\n\nThe dimensionality of the latent variables is defined by the number of latent variables or query points and their channel dimension. We determine the dimensionality of the latent variables based on the principle that the number of query points, receptive field, and channel dimension must cover the entire spatial domain. Similar to conventional CNNs, a small receptive field with a large stride (e.g., only a few query points) misses information, while a large receptive field with a small stride could have an averaging effect. With this principle, we obtain the number of query points (1024, 256, 16 for the encoder layers), which we use for all 2D experiments. Thus, a fixed number of query points works across different problems.\n\n\n> Will we observe a saturation effect or diminishing returns in prediction accuracy as the latent dimension increases?\n\nWe’ve conducted an experiment where we change the latent dimensionality by changing the number of latent points, and the results show that we will observe a saturation effect. We measure the relative L2 error on the test set and total training time on the 2D Navier-Stokes dataset with $\\nu=1e^{-4}$. The bold row shows the configuration used in the paper.\n\n| Encoder Query Points [Layer 1, Layer 2, Layer 3] | Test Rel. L2 Error | Training Time |\n| ------------------------------------------------ | ---------------------- | ------------- |\n| [256, 64, 4]  | 0.0533 | 4:18h |\n| [512, 128, 8] | 0.0425 | 4:40h |\n| **[1024, 256, 16]** | **0.0304** | **5:25h** |\n| [2048, 512, 32] | 0.0251 | 6:21h |\n| [4096, 1024, 64] | 0.0267 | 8:19h |\n\n\n> Should we use the same hyperparameters for different latent dimensions?\n\nThe number of query points, receptive field, and channel dimension are related to each other. Thus, it is recommended to adapt them jointly. When decreasing the number of latent points, the receptive field and channel dimension have to be increased and vice versa.\n\n\n> **(Q3)** [...] That is, can it generalize across varying domain shapes or boundary configurations as part of the input space?\n\nYes, the model can also handle varying domain shapes and boundary conditions. The cylinder dataset covers exactly this case. Each sample in the train and test set has a different domain shape and boundary condition (i.e., different diameter and position of the cylinder). The results show that CALM-PDE can also generalize across different domain shapes and boundary conditions.\n\nWe have also conducted an additional experiment on the time-independent airfoil and elasticity datasets from Geo-FNO [5]. The task is to map an input geometry (e.g., changing airfoil shape) to a physical quantity. We follow the setup from the LNO [2] paper and take the errors for the baselines from their paper. The results also demonstrate that CALM-PDE can generalize across complex, varying geometries.\n\n| Model | Time-indepdent Airfoil ($\\times 10^{-2}$ Rel. L2 Error) | Elasticity ($\\times 10^{-2}$ Rel. L2 Error)\n| ----------- | ---------------------- | ---------- |\n| Geo-FNO | 1.38 | 2.29 |\n| F-FNO | 0.60 | 1.85 |\n| Galerkin | 1.18 |  2.40 |\n| OFormer  | 1.83 | 1.83 |\n| GNOT |  0.75 | 0.88 |\n| ONO | 0.61 | 1.18 | \n| Transolver | 0.47 | 0.62 | \n| LNO | 0.51 | 0.52 |\n| CALM-PDE | 0.58 | 0.53 |\n\nThe results show that CALM-PDE outperforms 6 out of 8 baselines and that it ranks among the top models alongside LNO [2] and Transolver [6].\n\n**References** \\\n[1] AROMA: Preserving spatial structure for latent PDE modeling with local neural fields \\\n[2] Latent Neural Operator for solving forward and inverse PDE problems \\\n[3] Transformer for partial differential equations’ operator learning \\\n[4] Decoupled weight decay regularization \\\n[5] Fourier neural operator with learned deformations for PDEs on general geometries \\\n[6] Transolver: A fast transformer solver for PDEs on general geometries"}, {"review_id": "WDozVWPxQm", "reviewer": "Reviewer_27Kq", "summary": "CALM-PDE proposes continuous convolutionnal layers as encoder/decoder components. These layers have the abilities to be used on irregular grids while being based on convolutions. The layer are explained in details and then evaluated on regular and irregular grids a studied through ablations in the appendices.", "questions": "-\tWhat are the differences between LNS/LTS and RO discussed in the related work?\n-\tCan the number of point N differs between samples or must it be fixed ? \n-\tWhat happens if there are no points in a considered neighboorhood ? \n-\tLines 40-42, « mainly utilises CNNs ... or transformers » but no example are given here, I think this sentence should be illustrated. You could add some references (some of them are already cited in the paper such as AROMA or UPT), this point is only for clarity when reading the paper\n- what makes LNO 3x heavier in terms of memory consumption as FNO? \n-\tHow do evolve the performances wrt to query grid size ? in term of computational cost/accuracy?\n-\tHave you considered other architecture for the process ? \n- Why not considering AROMA or UPT as baselines? could CALM-PDE handle parametric PDEs?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strenght :\n -\tThe paper proposes a precise description of the model\n-\tThe text is well written and I found it easy to follow\n-\tA lot of additional experiments and details are available in appendices for in depth comprehension\n-\tI found the idea to be Interesting and it could be useful\n### Weaknesses : \n-\tThe contribution of the method seems to be the encoding/decoding since the process is a Neural ODE. I think a deeper study of encoding/decoding performances/effects would help the reader clearly identify the benefits of the model. \n-\tI would lower the first claim \"a new model that solves ... PDEs\", because the major contribution relies in the encoding/decoding architectures. \n- One of the arguments for the model being the computational cost, I think a deeper comparison on this aspect with other model would strengthen the argumentation. What are the computational cost wrt to existing method (mostly transformer based method such as AROMA/UPT)? How performs CALM-PDE in the encoding tasks wrt to other encoding architectures?", "was_revised": true, "final_justification": "CALM-PDE introduces a convolution-based layer designed to operate on irregular geometries. While most existing methods rely on attention mechanisms or graph-based approaches to handle and compress data on irregular grids, CALM-PDE offers a novel alternative for tackling this challenging setting with convolutions.\n\nThe authors have responded to my questions and provided new experiments to support their claims. I particularly appreciate their honesty in acknowledging the limitations of their method (ie performances and computational cost), which makes this work a promising basis for future research in this direction.\n\nAlthough CALM-PDE does not achieve state-of-the-art performance across the presented benchmarks, its results remain competitive and convincing relative to other baselines.", "rebuttal": "Thank you for your feedback and your questions. Your constructive feedback helped us to improve our submission. We appreciate that you consider our method interesting and helpful. Below, we address your concerns and questions in detail.\n\n> **(W1)** The contribution of the method seems to be the encoding/decoding since the process is a Neural ODE. I think a deeper study of encoding/decoding performances/effects would help the reader clearly identify the benefits of the model.\n\nYes, the main contribution of the method is the novel discretization-agnostic encoder and decoder based on continuous and adaptive convolution. The latent time stepping can be interpreted as solving a neural ODE. However, there are different ways of parametrizing the neural ODE, and previous work relied on point-wise MLPs [1, 2], while we utilize a Transformer with a combination of vanilla and position-attention [3] to parametrize the neural ODE. We added additional experiments that investigate the scaling of the latent query points and the output query points to provide more insights into the encoder-decoder architecture. Additionally, we add an explanation of the computational costs for the encoder. Please see (Q6) and (W3).\n\n> **(W2)** I would lower the first claim \"a new model that solves ... PDEs\", because the major contribution relies in the encoding/decoding architectures.\n\nAs mentioned in the response for (W1), we agree that the main contribution is the novel encoder-decoder architecture and not the paradigm itself. Thus, we’ve updated the paper and changed it from “a new model” to “a model with a new encoder-decoder mechanism” to better highlight the main contribution.\n\n> **(W3)** One of the arguments for the model being the computational cost, I think a deeper comparison on this aspect with other model would strengthen the argumentation. [...] How performs CALM-PDE in the encoding tasks wrt to other encoding architectures?\n\nWe also compare our model against OFormer (linear attention) and LNO (vanilla attention), which are encoder-decoder architectures that utilize attention for encoding and decoding. As shown in Figure 4 and Table 3 on page 9, CALM-PDE achieves a lower inference time and memory consumption compared to these models. Furthermore, we added an explanation to the paper to clarify the computational cost of the models. Let N be an arbitrary number of input points. For simplicity, we only consider one CALM encoder layer. The output of the layer are features for N’ output or query points, where N’ is a constant number. We also have to consider the percentile p, which determines the size of the receptive field and the feature or channel dimension d. This yields that $\\mathcal{O}(p \\cdot N \\cdot N’ \\cdot d)$ computations are needed. Models such as AROMA or LNO do not consider a receptive field, which requires $\\mathcal{O}(N \\cdot N’ \\cdot d)$ computations for N’ query tokens. Since p is usually small (we use p=0.01), the complexity is reduced by two orders of magnitude compared to models without a receptive field.\n\n> **(Q1)** What are the differences between LNS/LTS and RO discussed in the related work?\n\nWe use the term Reduced Order Modeling (RO) to generally refer to models that reduce the model complexity in some way. We further introduce the terms Latent Neural Surrogates (LNS) and Latent Time Stepping (LTS) as concrete ways of reducing the model complexity. For instance, LNS reduces the model complexity by reducing the spatial resolution, and LTS reduces the model complexity by computing the PDE trajectory completely in the latent space. LNS and LTS can be combined (e.g., AROMA), but they can also be used separately (only LNS for LNO and only LTS for OFormer).\n\n> **(Q2)** Can the number of point N differs between samples or must it be fixed ?\n\nYes, the number of input points N can be different between samples, which is the case in the cylinder dataset. Due to continuous convolution, each query point can take an arbitrary and changing number of input points. \n\n> **(Q3)** What happens if there are no points in a considered neighboorhood ?\n\nThis problem can only happen if the epsilon is fixed, which is not the case for our model. We adopt the mechanism from PiT [3], which computes the epsilon dynamically based on the p-th percentile of the Euclidean distances. This simply means that each query point has at least N*p points as input. If the input points are far away, the epsilon will be increased, and vice versa. Please see Appendix E.1, line 574 for details.\n\n> **(Q4)** Lines 40-42, « mainly utilises CNNs ... or transformers » but no example are given here, I think this sentence should be illustrated. You could add some references [...], this point is only for clarity when reading the paper\n\nThanks for pointing that out. We changed the corresponding lines and added LE-PDE [4] as an example for a model using CNNs as well as AROMA and UPT for transformer-based models.\n\n> **(Q5)** what makes LNO 3x heavier in terms of memory consumption as FNO?\n\nThe key difference that makes LNO heavier in terms of memory consumption and time is the use of attention compared to the use of the fast Fourier Transform (FFT). The encoder and decoder of LNO are based on cross-attention, which makes it expensive. In particular, cross-attention between a fixed number of query tokens (queries in attention) and all input points (values, keys in attention) is computed to compress the input into a smaller latent representation. In the 2D Navier-Stokes experiments, LNO uses 128 query tokens and the input has 4096 input points, which requires $\\mathcal{O}(128\\cdot4096)$ computations. For decoding, this process is reversed by using the latent tokens as key, value in attention, and the query positions as queries. The latent representation is processed with a Transformer that computes self-attention, which requires $\\mathcal{O}(128\\cdot128)$ computations. In contrast, FNO makes use of the FFT in each layer with a complexity of $\\mathcal{O}(N \\cdot log(N))$, which yields $\\mathcal{O}(4096 \\cdot log(4069))$ for the 2D Navier-Stokes dataset, which is more efficient.\n\n> **(Q6)** How do evolve the performances wrt to query grid size ? in term of computational cost/accuracy?\n\nWe’ve conducted an experiment on the 2D Navier-Stokes $\\nu=1e^{-4}$ dataset (max. resolution of ground truth is 64x64) where we increase the **output** query grid size during inference. The numbers are obtained on an A100 40GB GPU with a batch size of 32 and predicting one timestep.\n\n| Output Query Grid Size | Inference Time (ms) | Inference Memory Consumption (GB)\n| ----------- | ---------------------- | ---------- |\n| 64x64 (4096) | 10.27 | 0.84\n| 128x128 (16.384) | 14.83 | 1.80\n| 256x256 (65.536) | 33.89 | 5.64\n| 512x512 (262.144) | 107.52 | 21.00\n\nAdditionally, we’ve conducted an experiment where we change the **latent** query grid size (number of latent points). We measure the relative L2 error on the test set and total training time on the 2D Navier-Stokes dataset with $\\nu=1e^{-4}$. The bold row shows the configuration used in the paper. The results show that we can expect a saturation effect in the error.\n\n| Encoder Query Points [Layer 1, Layer 2, Layer 3] | Test Relative L2 Error | Training Time |\n| ------------------------------------------------ | ---------------------- | ------------- |\n| [256, 64, 4]  | 0.0533 | 4:18h |\n| [512, 128, 8] | 0.0425 | 4:40h |\n| **[1024, 256, 16]** | **0.0304** | **5:25h** |\n| [2048, 512, 32] | 0.0251 | 6:21h |\n| [4096, 1024, 64] | 0.0267 | 8:19h |\n\n\n> **(Q7)** Have you considered other architecture for the process ?\n\nBesides a Transformer, we’ve considered a pointwise MLP as the latent processor. However, we decided to use a Transformer instead of an MLP to allow the processor to have a global view, which would not be the case for a pointwise MLP, and to allow latent tokens to dynamically interact with each other with attention.\n\n> **(Q8)** Why not considering AROMA or UPT as baselines?\n\nWe've added AROMA as an additional baseline. Please note that we use a different setup (e.g., different rel. L2 error computed on non-normalized data, no in-t/ out-t split, etc.) compared to the AROMA paper.\n\n| Model | 2D Navier-Stokes $\\nu=1e^{-4}$ | 2D Navier-Stokes $\\nu=1e^{-5}$  | 2D Airfoil | 2D Cylinder |\n| ------- | ----------------- | -------------- |--------|-----------|\n| AROMA | 0.1257 (+318%) | 0.1801 (+74%) | 0.0403 (-22%) | 0.1140 (-4%) |\n| CALM-PDE | 0.0301 | 0.1033 | 0.0515 | 0.1186 |\n\nWe've also measured the inference times on the 2D Navier-Stokes $\\nu=1e^{-4}$ dataset, and AROMA needs 5.18s, in comparison to CALM-PDE with 0.37s. Reducing the refinement steps to 1 reduces the inference time to 3.81s.\n\n> could CALM-PDE handle parametric PDEs?\n\nWe did not experiment with parametric PDEs with varying PDE parameters, but CALM-PDE can be easily adapted to handle such cases. For instance, the PDE parameter could be added as an additional input channel, or a more sophisticated conditioning mechanism could be used to incorporate the PDE parameter [5, 6].\n\n\n**References** \\\n[1] Z. Li, K. Meidani, and A. B. Farimani, “Transformer for partial differential equations’ operator learning,” arXiv [cs.LG], 2022. \\\n[2] L. Serrano et al., “Operator learning with neural fields: Tackling PDEs on general geometries,” arXiv [cs.LG], 2023. \\\n[3] J. Chen and K. Wu, “Positional knowledge is all you need: Position-induced Transformer (PiT) for operator learning,” arXiv [cs.LG], 2024. \\\n[4] T. Wu, T. Maruyama, and J. Leskovec, “Learning to accelerate Partial Differential Equations via latent global evolution,” arXiv [cs.LG], 2022. \\\n[5] M. Takamoto, F. Alesiani, and M. Niepert, “Learning neural PDE solvers with parameter-guided Channel Attention,” arXiv [cs.LG], 2023. \\\n[6] J. Hagnberger, M. Kalimuthu, D. Musekamp, and M. Niepert, “Vectorized Conditional Neural Fields: A framework for solving time-dependent parametric Partial Differential Equations,” arXiv [cs.LG], 2024."}, {"review_id": "qwXGSWtYQg", "reviewer": "Reviewer_5iJC", "summary": "The author proposes an encoder-decoder style latent space PDE solver for solving time-dependent PDEs. First, in the encoder, the author sets multiple levels of query points with decreasing quantities, gradually compressing the PDE function information in the spatial domain into a compact latent space through successive convolutions. Then, a Transformer is used to model the mapping of the PDE function. Finally, in the decoder, multiple levels of query points with increasing quantities are set to gradually decompress the PDE function information from the latent space back to the spatial domain. Learnable query points are introduced, enabling the encoding and decoding processes to adapt to the spatial domain.", "questions": "1. How is the number of query points in each layer of the encoder and decoder determined? Is there a quantitative method to calculate how many query points are needed?\n2. The authors mention that their method does not perform well on PDE problems with many fine details. Did they actually conduct experiments on the Kolmogorov Flow problem? If so, what were the results?", "limitations": "As the authors mentioned, although CALM-PDE makes highly effective use of the latent space, its applicability to real-world scenarios involving a larger number of points is still limited due to the substantial computational cost. However, this is a common limitation shared by all current neural operator methods and should not be considered a drawback specific to this work.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The learnable multi-level query point design ensures that patterns of varying complexity across different spatial locations receive appropriate attention, enabling effective compression of spatial domain information into the latent space.\n2. The use of continuous convolution allows the model to handle PDE problems on irregular grids and decouples observation and prediction points. Moreover, the neighborhood constraints enable the model to capture local high-frequency information during operator learning, which pure Transformer-based neural operator models generally lack.\n3. The proposed model demonstrates excellent performance in both accuracy and efficiency across a wide range of 2D and 3D time-dependent PDE benchmarks.\n4. The authors provide a thorough categorization of latent PDE solvers and a well-articulated discussion on the differences and relationships among related works, offering valuable guidance for future research.\n\n## Weaknesses\nExcept for the experiment on the 2D Navier-Stokes problem (ν = 1e−4), the accuracy and efficiency of CALM-PDE are generally comparable to those of FNO [1] or Geo-FNO[2], while FNO appears to have a simpler architectural design. In this regard, the advantages of CALM-PDE merit further investigation.\n\n[1] Zongyi Li et al. Fourier neural operator for parametric partial differential equations. ICLR 2021.\n\n[2] Zongyi Li et al. Fourier neural operator with learned deformations for pdes on general geometries. JMLR 2023.", "was_revised": true, "final_justification": "The multi-level learnable query point design achieves a better balance between feature compression and preservation, and combined with continuous convolution, it further enhances the generality of the proposed method. I consider this an outstanding latent PDE solver paper and recommend accepting it.", "rebuttal": "Thank you for your effort in reviewing our work and your valuable feedback. We appreciate that you recognize the method, featuring the hierarchical query point design and the limited neighborhood, as important to effectively solve PDEs. We are happy to address the mentioned weakness and questions below.\n\n> **(W1)** Except for the experiment on the 2D Navier-Stokes problem (ν = 1e−4), the accuracy and efficiency of CALM-PDE are generally comparable to those of FNO [1] or Geo-FNO[2], while FNO appears to have a simpler architectural design. In this regard, the advantages of CALM-PDE merit further investigation.\n\nWe would like to emphasize that we use a curriculum strategy to train the autoregressive models, including FNO and Geo-FNO, which improves the error compared to a fully autoregressive training (see Appendix I, Table 6, line 799). For instance, the curriculum training of FNO improves the error by 39% on the 2D Navier-Stokes $\\nu=1e^{-4}$ dataset. Thus, we use strong baselines and incorporate additional techniques to improve them further. In general, CALM-PDE is a viable and competitive alternative to commonly used FNOs and Transformer architectures for solving PDEs. The model demonstrates the effectiveness of convolutional models on irregularly sampled spatial domains and the benefit of sampling the spatial domain via adaptive query points.\n\n> **(Q1)** How is the number of query points in each layer of the encoder and decoder determined? Is there a quantitative method to calculate how many query points are needed?\n\nWe determine the number of query points based on the principle that the number of query points, receptive field, and channel dimension must cover the entire spatial domain. Similar to conventional CNNs, a small receptive field with a large stride (e.g., only a few query points) misses information, while a large receptive field with a small stride could have an averaging effect. With this principle, we obtain the number of query points (1024, 256, 16 for the encoder layers), which we use for all 2D experiments. Thus, a selected number of query points works across different problems and does not need individual hyperparameter search.\n\n> **(Q2)** The authors mention that their method does not perform well on PDE problems with many fine details. Did they actually conduct experiments on the Kolmogorov Flow problem? If so, what were the results?\n\nYes, we have experimented with the Kolmogorov Flow problem from [1] by training only the encoder-decoder model with a self-reconstruction loss (autoencoder) and obtained a relative error of 0.0928 for a simple reconstruction, which is already worse compared to the relative error of 0.8156 for a rollout reported for DilResNet in [1]. Thus, we did not continue the experiment.\n\n**References** \\\n[1] Z. Li, D. Shu, and A. B. Farimani, “Scalable Transformer for PDE Surrogate Modeling,” arXiv [cs.LG], 2023."}]

Author rebuttal:

Meta-review: This paper introduces CALM-PDE, which compresses arbitrarily discretized PDEs with CALM layers (based on continuous convolutions) with multi-level learnable query points, evolves the system in latent space with transformer, and decodes back into original space with CALM based decoder. The method achieves competitive performance compared to strong baselines, and places more query points in regions that are intuitively important.

The reviewers recognizes (which I concur) that the paper addresses the important problem of arbitrarily-discretized domain, which is an important problem in solving PDEs. The reviewers also commend that the paper is well-motivated, well-written and easy to read, the experiments are thorough, and the method achieves competitive performance. During the rebuttal, the authors added additional baselines (e.g., F-FNO, AROMA), added additional experiments showing that the method can generalize across complex, varying geometries, and added additional study into the hyperparameters, etc. These improvements addresses the reviewers' concerns. The authors need to incorporate the additional experiments and discussion into the camera-ready version of the paper. Additionally, for generalization to new geometries, the authors need to detail how the query points' location and parameters are initialized.

Overall, I recommend spotlight.
Decision: Accept (spotlight)