CViT: Continuous Vision Transformer for Operator Learning
Authors: Sifan Wang
Jacob H Seidman
Shyam Sankaran
Hanwen Wang
George J. Pappas
Paris Perdikaris
Venue: ICLR 2025 Poster (2025)
Keywords: Scientific Machine Learning
Operator Learning
Neural Operators
Neural Fields
Vision Transformer
Partial Differential Equations
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We present CViT, a neural operator using vision transformers to learn complex physical systems, achieving SOTA performance on PDE benchmarks without extensive pretraining.

Abstract: Operator learning, which aims to approximate maps between infinite-dimensional function spaces, is an important area in scientific machine learning with applications across various physical domains. Here we introduce the Continuous Vision Transformer (CViT), a novel neural operator architecture that leverages advances in computer vision to address challenges in learning complex physical systems.  CViT combines a vision transformer encoder, a novel grid-based coordinate embedding, and a query-wise cross-attention mechanism to effectively capture multi-scale dependencies. This design allows for flexible output representations and consistent evaluation at arbitrary resolutions. We demonstrate CViT's effectiveness across a diverse range of partial differential equation (PDE) systems, including fluid dynamics, climate modeling, and reaction-diffusion processes. Our comprehensive experiments show that CViT achieves state-of-the-art performance on multiple benchmarks, often surpassing larger foundation models, even without extensive pretraining and roll-out fine-tuning. Taken together, CViT exhibits robust handling of discontinuous solutions, multi-scale features, and intricate spatio-temporal dynamics. Our contributions can be viewed as a significant step towards adapting advanced computer vision architectures for building more flexible and accurate machine learning models in the physical sciences.

Published as a conference paper at ICLR 2025
CVIT: CONTINUOUS VISION TRANSFORMER FOR OP- ERATOR LEARNING
Sifan Wang', Jacob H. Seidman 34455 Shyam Sankaran', Hanwen Wang', George J. Pappas*
Paris Perdikaris
Institution for Foundation of Data Science, Yale University
Graduate Program in Applied Mathematics and Computational Science, University Pennsylvania DDepartment of Mechanical Engineering and Applied Mechanics, University of Pennsylvania *Department of Electrical and Systems Engincering, University of Pennsylvania
*Reality Defender
si sifan. wang@yale. edu
{seidj, wangh1 9) H Eaas upenn edu
shyamss,, pappasg, pgp @seas. upenn.edu
ABSTRACT
Operator learning, which aims to approximate maps between infinit-ddimensional function spaces, is an important area in scientific machine learning with applica- tions across various physical domains. Here we introduce the Continuous Vision Transformer (CViT), novel neural operator architecture that leverages advances computer vision to address challenges in learning complex physical systems. CViT combines a vision transformer encoder, a novel grid- coordinate embedding, and query-wise cross attention mechanism to effectively capture multi-scale dependencies. This design allows for flexible output representations and consistent evaluation arbitrary resolutions. We demonstrate CVTT's effectiveness across a diverse range of partial differential equation (PDE) systems, including fluid dynamics, climate modeling, and reaction-diffusion processes. Our comprehensive experiments show that CVIT achieves state-of siite-of-thee of-the-art performance on multiple benchmarks, often surpassing larger foundation models, even without extensive pre- training and roll-out fine-tuning. Taken together, CViT exhibits robust handling of discontinuous solutions, multi-scale features, and intricate spatio-temporal dynam- ics. Our contributions can be viewed significant step towards adapting advanced computer vision architectures for building more flexible and accurate machine learn- ing models in the physical sciences. All data and code are publicly available at ptttps::/githee com http:/reeiicce www.eeewwwwleeeeee/eaaaeo.o.
INTRODUCTION
Neural operators (Kovachki al. 2023) have emerged a powerful class of deep learning models for scientific machine learning applications, particularly in building efficient surrogates for expensive partial differential equation (PDE) solvers. These models aim learn mappings between infinite- dimensional function spaces, enabling rapid acceleration of complex simulations in fields such as fluid dynamics and climate modeling (Azizzadenesheli
2024 Serrano et al. (023). Popular ar- chitectures like Fourier Neural Operators (FNO) (I. et al.] 202 Tran et al. [021] ) and DeepONet (Lu
al. 2021 Wang al. 2002) have shown promising results, supported by universal approximation
guarantees Chen & Chen 1995 Lanthaler al. 2022, Kovachki et al. 2021 De Ryck & Mishra
2022 and interpretations as learning low-dimensional manifolds in function spaces (Scidman et al.
2022
2023
However, the architectural design of current neural operators has largely been motivated by mathe- matical intuition about the structure of PDE solution spaces, rather than leveraging recent advances in deep learning architectures. This approach, while theoretically sound, may limit the performance and flexibility of these models when applied complex, real-world physical systems.

---

Published as a conference paper at ICLR 2025
In parallel, the field of computer vision has seen significant advancements in neural field models, which represent continuous functions parameterized by neural networks (Stanley) (007)) A key inno- vation in this domain has been the development of conditioned neural fields, which allow for auxiliary inputs to modulate the field's behavior (Xie et al. 2022). These models have demonstrated remarkabe kable effectiveness in tasks such representing high- nigh-resolutton images and 3D scenes (Mildenhall et al.. 2021
Müller et al. 2022).
Motivated by the success of conditioned neural fields in computer vision, we introduce the Continuous Vision Transformer (CViT), a novel neural operator architecture that bridges the gap between advanced computer vision techniques and scientific machine learning. Our main contributions can be summarized as follows:
Continuous Vision Transformer (CVIT). We introduce CVTT, neural operator that combines vision transformer encoder with novel, trainable grid-based coordinate embedding its decoder. This design enables CVIT to effectively capture multi-scale spatial dependencies and allows for continuous querying at arbitrary resolutions.
We present novel insights into the impact of coordinate embeddings on the Lipschitz constant of neural fields and neural operators. This analysis supports the choice of the proposed grid-based coordinate embedding and its efficacy in mitigating spectral bias, common limitation in training neural operators
Theoretical Insights into Lipschitz Constant and Spectral Bias.
State-of-the-are Performance. Despite its architectural simplicity, CVIT achieves sfte-of-the-aa results on challenging benchmarks in climate modeling, fluid dynamics and reaction-diffusic processes. It often outperforms larger models while using fewer parameters and without requiring extensive pretraining.
Taken together, we explore the connection between operator learning architectures and conditioned neural fields, offering unifying perspective for examining differences among popular operator learning models (see Appendices B and C). This perspective not only enhances understanding of existing architectures but also provides general framework for designing future neural operators. By bridging the gap between computer vision and scientific machine learning, CVIT paves the way for improved surrogates of complex physical systems, with potential applications ranging from climate modeling to engineering design.
2 BACKGROUND AND RELATED WORK
Background. The emergence of neural operators has driven advancements solving partial differential equations (PDEs) and modeling physical systems. Building upon the foundational architectures mentioned in the introduction, recent developments have focused on enhancing the expressiveness and efficiency of these models. For instance, extensions of the Fourier Neural Operator (FNO) (Li et al.) (021) have explored factorized representations (Tran et al., (021) to reduce computational complexity while maintaining performance. Similarly, wavelet-based approaches (Gupta et al., 2021 Tripura & Chakraborty 2022) have been proposed to capture multi-scale features more effectively. Most existing neural operator architectures can be interpreted as conditioned neural fields (Xic al. 2022) with specific choices of base fields and conditioning mechanisms, Appendices and for a detailed discussion.
Transformer-Basee Operator Learning. Concurrently, the success of vision transformers in computer vision tasks has inspired their adaptation to scientific machine learning problems. The self-attention mechanism central to transformers offers a powerful tool for capturing long-range dependencies in spatial and temporal data, crucial aspect in many PDE-governed systems. For example, OFormer (Li al. 2022) introduced a novel way to embed continuous input functions and output queries into a transformer architecture. This approach demonstrated the potential of transformers in handling the inherent continuity of physical problems. GNOT (Hao et al. 2023) further extended this idea by incorporating graph structures, allowing for more flexible representation of input functions and improved handling of irregular domains. A significant advancement this direction came with DPOT (Hao al.) 2024), which introduced denoising pre-training strategy. This approach enables the model to learn from diverse PDE datasets, significantly enhancing its generalization capabilities.
2

---

Published as a conference paper at ICLR 2025
latent queeLed
Q.K.V
transforner encoder
Trazaformer Block
tokesize
K.V
Lx
time ponition nebedcings
V
3
quezy
isterpolation
Bulti-beat attertica
Continuous
latent
Vision
Layer
Transforner
(CViT)
Q K V
ouEpUE
$(
Function
Figure 1: Continuous Vision Transformer (CVIT) Architecture: CVTT consists of the following components: (1) Spatio-temporal patch embeddings extract localized features. (2) temporal aggregation module based on the Perceiver architecture, which captures temporal correlations to compresses tokens along the time axis. (3) Transformer encoder that captures multi-scale spatial dependencies via self-attention layers. (4) A novel grid-based positional encoding scheme for query coordinates, allowing for flexible output representation and interpolation. (5) A cross-attention decoder that integrates information from the input function with query coordinates.
Open Challenges. Despite these advancements, several key challenges remain in the field of operator learning, particularly in addressing resolution independence, high-dimensional data, long- range spatio-temporal dependencies and trade-offss between model expressiveness and computational efficiency. Transformers typically rely on fixed-resolution data and positional encodings, which hinder their ability generalize across varying scales irregular domains commonly encountered in scientific problems. Efficiently capturing global and local dependencies in high-dimensional systems further compounds computational demands, especially self-attention scales quadratically with input size. These constraints make it difficult to balance scalability, precision, and efficiency. Addressing even subset of these challenges requires novel architectural designs that can leverage the strengths of both neural operators and vision transformers while overcoming their respective limitations. This sets the stage for our proposed Continuous Vision Transformer (CVIT), which aims to tackle specific challenges related to resolution independence and efficient capturing of multi-scale features in regular grid-based problems, combining the continuous nature of neural operators with the powerful representation capabilities of vision transformers.
CONTINUOUS VISION TRANSFORMER (CVIT)
While neural operators have shown promising results in learning mappings between function spaces, their architectural design has not fully leveraged recent advancements in deep learning. Popular architectures like FNO (Li et al. 2021). DeepONet (Lu et al.) 2021), and NoMaD (Seidman al.) 2022) (see Appendix C primarily rely on simple building blocks such as Fourier layers or fully connected networks, which may not effectively capture complex spatial dependencies and multiscale features present in many physical systems.
Motivated by the success of vision transformers in computer vision tasks, we introduce the Continuous Vision Transformer (CViT), novel neural operator architecture that combines the strengths of vision transformers and continuous function representations. CVTT aims to learn more powerful and flexible representations of dime-dependent physical systems by leveraging vision transformer encoder to capture complex spatial dependencies in imput functions, a trainable grid-based positional encoding for flexible representation of query coordinates, and cross-attention mechanism integrate input function information with query coordinates.
Taken together, the overall design of CVTT's architecture draws inspiration from the analysis of existing neural operators presented Appendix C while leveraging the power of vision transformers enhance expressiveness.

---

Published as a conference paper at ICLR 2025
3.1 ARCHITECTURE DESCRIPTION
The CViT architecture consists of three main components: a vision transformer encoder, grid-based positional encoding, and cross-atiention mechanism.
Patch Embeddings. The vision transformer encoder takes as input gridded representation of the input function u, yielding spatio-temporal data tensor u E RRT*HxW* with D channels.
We patchify our inputs into 3D tokens Up E RTx x xC by tokenizing each 2D spatial frame independently, following the process used in standard Vision Transformers (Dosovitskiy et al. 2020).
We then add trainable 1D temporal and 2D spatial positional embeddings to each token:
Upe up + PE, PE., PE E RXXXXIXC
PE. R x
(1)
Temporal Aggregation. To reduce computational cost, we use temporal aggregation layer based on the Perceiver architecture (Jaegle et al., 2021 Alayrac et al. 2022). Specifically, we learn a pre-defined number of latent input queries z E R xC These queries serve as learnable parameters in cross-attention module, which processes the visual features of our input. The Perceiver module operates on flattened visual features u, E R( x P xxTCC obtained by flattening the positionally encoded features Upe. Through this cross-attention mechanism, we aggregate the temporal information into a compact latent representation Zogg E R #) xx""xx""X xC as:
z' 2 + MHA (LN (2), LN (uy) LN Zagg z'' + MLP (TN (z')).
(2)
Here, z is initialized by a unit Gaussian distribution and E R( x xxTx"xxx xC is obtained by tiling the latent query z (# x W) times. Besides compressing tokens over the time axis, this module also enables the model to handle inputs with a variable number of time steps.
Transformer Encoder. We then process the aggregated tokens Zagg using a sequence of I. pre-norm Transformer blocks /Vaswani et al. 2017: Xiong et al. 22200).
Zo LN(agg)),
(4)
2/2 MSA (LN (Z)-1)) + Zt-1,
i = 1=1...44 l 1=
(5)
Ze: = MLP (LN (zz)) + z/.
(6)
Cross-Attention Decoder. To enable continuous evaluation of outputs, we design a novel and efficient coordinate embedding to capture the fine-scale features of the target functions. Specifically,
we create a uniform grid } C [0, 1|2, for = 1,... N₂ and j = I....NN, along with associated trainable latent grid features x RRN, XX, For query point y E R², we compute Nadaraya- Watson interpolant over grid latent features:
exp (-Blly 31)
W W
Wij
Ei exp (-yl - 11)
Here B > 0 is hyperparameter that determines the locality of the interpolated features. It is important for determining the smoothness of the interpolation function. Specifically, larger values of B yield more localized weight distributions Wij. resulting in higher-frequency interpolant that captures finer-scale variations. Conversely, smaller 3 values produce a smoother interpolant by averaging over a broader neighborhood of points.
The interpolated grid feature X, = x' E RIxC is used as query input to transformer decoder. This decoder uses the output of the vision transformer encoder z as keys and values in cross-attention mechanism:
x X + MHA (LN
LN (Zr.) LN (zr.)),
1.... K,
(8)
X X MIP (LN (xx)),
k= 1..... K.
(9)
Finally, small MI.P network projects the output to the desired output dimension.
4

---

Published as a conference paper at ICLR 2025
Theoretical Insights. The theoretical analysis of Lipschitz constants for different coordinate embeddings (as detailed in Appendix E) provides crucial insights that inform the design choices in CVIT and have broader implications for neural operator architectures. Our findings reveal that the choice of coordinate embedding significantly influences the decoder's capacity to learn complex, high-frequency functions.
For conventional MLP decoders with linear coordinate embeddings, the Lipschitz constant is approxi- mately initialization, which does not enhance the network's ability capture fine-scale features, thereby leading to prediction susceptible to spectral bias (Rahaman et al. 2019). In contrast, random Fourier features can increase the Lipschitz constant through two mechanisms: by increasing the standard deviation of the sampling distribution or by increasing the network width. This observation not only explains the effectiveness of Fourier features in addressing spectral bias, but also highlights a potential trade-off between network size and the frequency range that can be captured.
Our proposed grid-based coordinate embedding in CVTT offers a more direct and controllable approach to increasing the network's 's Lipschitz constant. By adjusting the interpolation parameter B, we can tune the model's s ability to capture high-frequency components the target functions. This provides a clear advantage as it allows for more precise control over the model's s spectral properties without necessarily increasing model complexity.
CVIT vs other Transformer-basee Approaches. To further highlight the key innovations of CViT, we compare it with other popular transformer-basee operator learning methods:
OFormer Li et al., 2022): OFormer uses an encoder-decoder structure. Its encoder combines point-wise MLP and stacked self-attention blocks to process input functions, while employing random Fourier Features (Tancik et al., (220)) for query coordinate encoding. The model generates latent embeddings through cross-attention at sampled query locations. In the decoder, OFormer uses recurrent MLPs for dynamic propagation instead of traditional masked attention. Unlike CVIT, it doesn't use ViT's patch embedding to tokenize input functions, potentially limiting its ability to capture complex spatial dependencies.
GNOT (Hao et al., 2023): GNOT is a transformer-based framework for learning operators that addresses the challenges of multiple input functions and irregular meshes. It designs heterogeneous normalized attention layer, providing an encoding interface for various input functions and additional prior information. Additionally, GNOT features a geometric gating mechanism, described as a soft domain decomposition approach to multi-scale problems. Unlike CVIT, GNOT uses an plain MLP to process query coordinates, which may potentially suffer from spectral bias (Rahaman et al. 2019).
DPOT (Hao et al. 2024]: DPOT is a transformer-based architecture designed for pretraining on multiple PDEs. It introduces a new auto-regressive denoising strategy and novel model architecture based on Fourier attention. Unlike OFormer, GNOT, and CViT, DPOT does not explicitly take query coordinates as imputs, which implies that cannot be evaluated at arbitrary locations.
MPP (McCabe et al. 2023): MPP is large-scale PDE surrogate model that directly employs Vision Transformer as its backbone. Similar to DPOT, it does not take query coordinates as inputs, thereby losing flexibility in diverse output representations.
4 EXPERIMENTS
We compare the CVTT against popular neural operators on three challenging benchmarks in physical sciences. In addition to demonstrating CVTT's performance against strong baselines, we also conduct comprehensive ablation studies to probe the sensitivity on its own hyper-parameters.
CViT model setup. We construct CVIT models with different configurations, as summarized in T For all experiments, unless otherwise stated, we use patch size of 8 x for tokenizing inputs. We also employ decoder with a single cross-attention Transformer block for all configurations. While we tested decoders with multiple layers, they did not yield improvements. The grid resolution is set to the spatial resolution of each dataset. The latent dimension of grid features is set to 512, and it does not match the transformer's embedding dimension, we align it using a dense layer. Besides,
use 3 10 to ensure sufficient locality of the interpolated features. These choices are validated
5

---

Published as a conference paper at ICLR 2025
DeepONet (TV=0.23)
NoMaD (TV=0.16)
FNO (TV=0.19)
CVTT (TV=
0.0
0.5
0.5
1.0
0.0
D.5
D.5
1.0
M
M
Figure 2: Advection of discontinuous waveforms. Prediction (red dashed line) versus ground truth (blue line) for the worst-case example the test dataset, for: DeepONet: NoMaD; FNO; CViT. Also reported is the associated Total Variation (TV) error (J).
by extensive ablation studies, as illustrated in Figure Full details of the training and evaluation procedures provided in Appendix F.
Table Details of Continuous Vision Transformer model variants.
Model
Encoder layers
Embedding dim
ML.P width
Heads
Params
CViT-S
384
384
13 M
CVTT-B
10
512
512
8
30 M
CViT-L
15
768
1536
12
92 M
Baselines. We select the following methods as our baselines for comparisons. DeepONet LLu et al. 2021) One of the first neural operators with universal approximation guarantees. NoMaD (Seidman et al. 2022): A recently proposed architecture leveraging nonlinear decoders to achieve enhanced dimensionality reduction. FNO (Li al.) 2021): An efficient framework for operator learning in the frequency domain. FFNO (Tran et al. 202T): An improved version of FNO utilizing
separable Fourier representation, which reduces model complexity and facilitates deeper network structures. UNO (Ashiqur Rahman et al.) 2022): A U-shaped neural operator with FNO layers.
U-Netat (Gupta & Brandstetter y022): modern U-Net architecture using bias weights and group normalization, and wide ResNet-style 2D convolutional blocks, each of which is followed by spatial attention block. U-F2Net (Gupta & Brandstetter 2022): A U-Net variant where the lower blocks in both the downsampling and upsampling paths replaced by Fourier blocks, each consisting of 2 FNO layers with residual connections. Dilated ResNet (Stachenfeld et al.] 202T): ResNet model that adapts filter sizes different layers using dilated convolutions, providing alternative method for aggregating global information. GK- TiK-Transformer (Cao) 2021) / OFormer (Li et al.)
2022) GNOT (Hao et al. 2023): Various wransformer-based architectures for operator learning. with different designs of their encoder/decoder and attention mechanisms. DPOT (Hao et al. 2024) MPP (McCabe al. 2023 ; Transformer- bassed networks with patch embedding as in ViT. and
pre-trained on extensive PDE datasets using rollout loss.
For all baselines, we report evaluation results from the original papers when applicable. When not applicable, we train and evaluate each model following the suggested settings in the respective paper. Details on the implementation baseline models are provided in Appendix F.2
4.1
MAIN RESULTS
Here provide main results across three challenged benchmarks. The full details the under- lying equations, dataset generation and problem setup for each case are provided in the Appendix; and [.5 respectively.
Section
Advection of discontinuous waveforms. The first benchmark involves predicting the transport of discontinuous waveforms governed by a linear advection equation with periodic boundary conditions. We make of the datasets and problem setup established by Hoop al. (de Hoop al. 2022). This benchmark evaluates our model's 's capability handling discontinuous solutions and shocks in comparison popular neural operator models.
Figure 2 illustrates the test sample corresponding to the worst-case prediction of each model we compared. We observe that CVTT is able better capture the discontinuous targets and yield the

---

Published as a conference paper at ICLR 2025
most sharp predictions across all baselines. This is also reflected in both the relative L2 and Total Variation error metrics; see Appendix Table for a more complete summary. including details on model configurations, well average. median and worst case emrors across the entire test dataset.
Shallow-water equations. The second benchmark involves fluid flow system governed by the 2D shallow-water equations. This describes the evolution of a 2D incompressible flow on the surface of the sphere and it is commonly used as a benchmark in climate modeling. Here we adhere the dataset and problem setup established in PDEArena (Gupta & Brandstetter (2022). allowing us to perform fair comparisons with several state-of-the-an deep learning surrogates.
Table 2 presents the results of CViT against several competitive and heavily optimized baselines. Our proposed method achieves the lowest relative L er- ror. It is also worth noting that the reported per- formance gains are achieved using CViT configura- tions with a significantly smaller parameter count compared to the other baselines. This highlights the merits of our approach that enables the design of parameter-efficient models whose outputs can also be continuously queried at any resolution. Additional visualizations of our models are shown in Figure 3 and I0
Table 2: Performance for 5-step roll-out pre- dictions in the shallow-water equations bench- mark.
Model
# Params
Rel. I,² error (1)
DilResNet
M
13.20%
U-Netar
148 M
5.68%
FNO
268 M
3,97%
U-F2Net UNO
344 M
1.89%
440 M
3.79%
CVIT-S CVIT-B CVIT-L
13 M
4.47%
M
2.69%
In addition, here we perform extensive ablation stud- ies on the hyper-parannn prper-parameters of CVTT. The results are summarized in Figure 4
92 M
1.56%
As shown in Figure (a), observe that smaller patch size typically
leads to better accuracy, but at higher computational costs. Moreover, Figure 4 (b) demonstrates that the model performance significantly influenced by the type of coordinate embeddings used in the CViT decoder. We compare our proposed grid-based embedding against small MLP and random Fourier features (Tancik et al. 2020). The grid-based embedding achieves the best accuracy. outperforming other methods by up to an order of magnitude. This result is in agreement with the theoretical insights presented in Appendix
We also investigate the impact of the resolution of the associated grid. Figure (c) reveals that the best results are obtained when the grid resolution matches the highest resolution at which the model is evaluated. As another direct empirical evidence of theoretical results in Appendix E Figure (d) shows the CVTT model's sensitivity to the hyper-parameter 3 used for computing interpolated degrade predictive accuracy. Finally, in Appendix Figure g observe minor improvements by increasing the latent dimension of grid features, or the number of decoder attention heads. Hence the overall model performance remains robust to variations in these hyper-parameters.
features; too small /8 values
Taken together, these findings not only justify our design choices for CVIT but also provide general insights for the development of neural operators. They suggest that future research in this field should pay close attention the interplay between spatio-temporal tokenization, coordinate representation, and the inherent multi-scale nature of physical systems. Such considerations could lead to even more effective architectures for a wide range of scientific machine learning tasks.
Navier-Stokes equations (NS). Our third benchmark focuses on the compressible and incompress- ible Navier-Stokes equations, which underpin variety of systems, ranging from hydromechanical processes to weather forecasting and turbulent dynamics analysis. For fair comparison with various neural operators and other offe-offthh-art deep learning models, we use the dataset generated by PDEArena (Gupta & Brandstetter] (022). and PDEBench Takamoto et al. (2022). and follow the problem setup reported in Hao et al. (Hao et al. 2024).
Our main results are summarized in Table 3 indicating that CVIT outperforms all baselines we have considered. Notably, the previous best result (DPOT-L (Fine-tuned)) is achieved with much larger munsformer-basee architecture, which involves pretraining on massive diverse PDE datasets followed by fine-tuning on the target dataset for 500 epochs. Here we also note that DPOT was pretrained and fine-tuned using a rollout loss, which directly optimizes for the multi-step prediction task on which the models are evaluated. In contrast, CVTT was trained using one-step loss without any rollout
7

---

Published as a conference paper at ICLR 2025
T+1
T-2
T+3
T+4
7+5
-25
0.050
0.025
Figure 3: Shallow water benchmark. Representative CVTT rollout prediction of the vorticity field,
an
100-2
Parch Size 16 Patch
Fourier
Rel. I Emoo
10
10-1
2-
10
2
102
102
10*
10
Step (n)
x10²
Step (b)
x10*
Step (c)
(d)
Figure 4: Ablation studies for CVT on the waallowwwwtter equations benchmark. Convergence of test
errors for: (a) different patch sizes; (b) different coordinate embeddings; (c) different resolutions of latent grids; (d) sensitivity on interpolating the CViT latent grid features, as controlled by the B parameter. Results obtained using CViT-L with 16 16 patch-size, varying each typer-parameter of interest while keeping others fixed.
finetuning. This difference in training objective significantly favors DPOT for rollout evaluations across all benchmarks. Furthermore, CVIT exhibits remarkable parameter efficiency. CViT-S, with only 13M parameters, achieves accuracy comparable the largest pretrained model (1.03B). Additionally, CVIT shows good scalability; the number of parameters increases, performance improves. These findings strongly support the effectiveness our approach in learning evolution operators for complex physical systems. Additional results and sample visualizations provided in Figure 5 as well as Figure 13 and [4] in Appendix.
Table 3. Relative L error of rollout predictions on the incompressible and compressible Navier-
Stokes and inffusion-Reaction benchmark.
Model
Params
NS
CNS
DR
FNO
0.5 M
9.12 %
9.60 %
12.00 %
FFNO
1.3 M
8.39 %
5.20 %
5.71 %
GK-T
1.6 M
9.52%
3.77%
3.59 %
GNOT
1.8 M
17.20 %
4.20 %
3.11 %
Oformer
1.9 M
13.50
6.25 %
1.92 %
DPOT-Ti
7 M
12.50 %
3.97 %
3.21 %
MPP-S
30M
1.12 %
DPOT-S
30 M
9.91 %
3.37 %
3.79 %
MPP-I. (Pre-trained)
400 M
0.98 %
DPOT-L (Pre-trained)
500 M
7.98 %
2.16 %
2.32 %
DPOT-L (Fine-tuned)
500
2.78 %
1.31 %
0.73 %
DPOT-H (Pre-trained)
1.03 B
3.79 %
1.80 %
1.91 %
CVTT-S
13 M
3.75 %
2.71 %
1.13 %
CVIT-B
30 M
3.18 %
1.99 %
1.11 %
CVIT-L
92 M
2.35 %
1.29 %
0.68 %
8

---

Published as a conference paper at ICLR 2025
T
T+3
T+4
Pef. Smoke
1.0
Preet Smoke
1.0
0.5
Error
0.03
0.02
Figure 5; Incompressible Navier-Stokee benchmark (NS). Representative CVTT rollout predictions of the passive scalar field, and point-wise error against the ground truth.
T+1
T+2
T+33
T+44
T+5
T+6
Ref.
4
Pred. rho
6
4
Abs. Error
0.3
0.1
Figure 6: Compressible Navier-Soe Vavier-Stokes Benchmark (CNS). Representative CVIT rollout prediction of the density field p. and point-wise error against the ground truth.
Diflusion-Reaction equations (DR). Our final benchmark focuses on learning the density fields (u, v) governed by coupled 2D diffusion-reaction equation (DR). This PDE, commonly used in chemistry, models substances reacting and diffusing across spatial domain, illustrating chemical reaction processes and resulting in diverse spatial patterns. For a fair comparison, we use the dataset from PDEBench (TTakamoto et al. 2022) and follow the problem setup described by Hao et al. (2024).
Table presents our main results. Consistent with previous findings, CVIT-L. achieves the best performance across all baselines. CVIT models outperform other approaches with comparable or fewer parameters, demonstrating good scalability and parameter efficiency. Representative rollout predictions provided in Figure 18 and 19 (see Appendix). Our predictions show excellent agreement with the corresponding ground truth
Interestingly, note that the solution of Diffusion-Reannn the Navier-Stoker Stokes equation dataset. resulting uniformly lower test errors for various baselines in the DR benchmark. In particular, OFormer outperforms GNOT in both examples, despite having
equations appears simpler than that of

---

Published as a conference paper at ICLR 2025
comparable parameters. We postulate that this performance gain is due to the use of random Fourier features in OFormer, facilitating more efficient learning of complex functions. This observation further demonstrates the importance of coordinate embedding in designing neural operators and indirectly highlights the effectiveness of our proposed grid-based embedding.
5 DISCUSSION
Summary. This work introduces the Continuous Vision Transformer (CViT), novel neural operator architecture that leverages advances in computer vision to address challenges in learning complex physical systems. CVTT combines the strengths of vision transformers and continuous function representations, achieving stale-of-the-art accuracy on challenging benchmarks in climate modeling, fluid dynamics and reaction-diffusinn processes. Our approach demonstrates the potential of adapting advanced computer vision techniques to develop more flexible and accurate machine learning models for physical sciences. The key innovations of CViT include: (a) vision transformer encoder for capturing multi-scale spatial dependencies in input functions, (b) a trainable grid-based positional encoding for flexible representation of query coordinates, and (c) query-wise cross-attention mechanism ensuring consistent evaluation at arbitrary resolutions. These design choices ensure that CVTT maintains the desirable properties of well-defined conditioned neural field (as discussed in Appendix B while offering improved flexibility and performance (as discussed in Appendix (E).
Our empirical results across various PDE benchmarks demonstrate that CVIT often outperforms larger models with fewer parameters and without extensive pretraining. This highlights the merits of approach in designing parameter-efficient models that be continuously queried any resolution. The broader impact of this work lies in its potential to accelerate scientific discovery through more efficient and accurate simulations of complex physical systems, with applications ranging from climate modeling to engineering design.
Limitations. Despite its promising performance, the proposed CVTT architecture can be further improved in multiple areas. Firstly, like most transformer-based models, CVTT's self-attention blocks scale quadratically with the number of tokens, posing computational challenges for high- dimensional or high-resolution data. Recent advancements in computer vision (Liu et al. 202 lb) are tackling these scalability issues, and these ideas can be transferred to our setting to improve CViT's efficiency for larger-scale problems. Secondly, transformer-bast require large training datasets for optimal performance and may struggle in data-scarce scenarios without careful regularization. To address this, future work could explore leveraging CVIT's ability to reconstruct continuous output functions in conjunction with physics-informed loss functions (Wang et al., 202T 2023) for more data-efficient training and improved generalization. Lastly, while our current experiments treat input functions as sequences of images, many practical applications in scientific computing involve complex geometric structures and diverse input modalities such as unstructured meshes or point clouds. Addressing these cases requires developing appropriate tokenization schemes, and recent work in (Alkin al.) [024] Pang et al.) (022) has taken steps in this direction. Extending CVIT to handle such diverse input modalities represents an important avenue for future research.
models like CVIT generally
Future Work. Building the success of CVTT, several promising directions for future research emerge. One key area is the exploration of more efficient attention mechanisms to improve scalability higher-dimensional problems. Additionally, investigating the potential of pre-training strategies, similar to those employed in DPOT (Hao et al. 2024), could further enhance CViT's performance and generalization capabilities across diverse PDE systems. Finally, extending CVIT to handle 3D problems and developing techniques for multt-rresoluion training and evaluation could significantly broaden its applicability scientific computing domains.
Broader Impact. The broader impact of this work lies in its potential to fuel the development of issolution-invarian foundation models for accelerating scientific discovery and enable more efficient and accurate simulations of complex physical systems, with applications ranging from climate modeling to engineering design.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We would like to acknowledge support from the US Department of Energy under the Advanced Scientific Computing Research program (grant DE-SC0024563) and Yale Institute for Foundations of Data Science. We also thank the developers of the software that enabled our research, including JAX Bradbury et (2018). Matplotlib Hunter (2007). and NumPy |Haris al. (2020).
ETHICS STATEMENT
Our contributions enable advances in accelerating the modeling and emulation of complex physical systems. This has the potential to significantly impact fields such as climate science, fluid dynamics, and materials engineering by providing more accurate and efficient simulation tools. While do not anticipate specific negative impacts from this work, as with any powerful predictive tool, there is potential for misuse. We encourage the research community consider the ethical implications and potential dual-use scenarios when applying these technologies in sensitive domains.
REPRODUCIBILITY STATEMENT
The code used to carry out the ablation experiments is provided as a supplementary material for this submission. All data and code publicly available httpa://gittub.ccm Prediittivennntelllggnnneed/b//
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 35:23716-337366 2022.
Benedikt Alkin, Andreas Fürst, Simon Schmid, Lukas Gruber, Markus Holzleimnrr, and Johannes Brandstetter. Universal physics transformers. arXiv preprint arXiv:2402.12365, 2024.
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli. U-no: U-shaped neural operators. arXiv e-prints, pp. 1rXiv-2204 2022.
Kamyar Azizzadenesheli, Nikola Kovachki. Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design. Nature Reviews Physics, pp. 1-9, 2024.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric PDEs. The SMAI journal of computational mathematics, 7:
121-157, 2021.
James Bradbury. Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL www.gooole////o/oooooooooooooo
J Bruna, W Zaremba, A Szlam, and Y LeCun. Spectral networks and deep locally connected networks on graphs. arxiv. arXiv preprint 131.v613 2014.
Qianying Cao. Somdatta Goswami, and George Em Kariaadkii. Lno: Laplace neural operator for solving differential equations. arXiv preprint arXiv:2303. 10528, 2023.
Shuhao Cao. Choose a transformer: Fourier or galerkin. Advances in neural information processing
systems, 34:24924-24944 2021.
Rohan Chabra, Jan E Lenssen, Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In Computer Eision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. Springer, 2020.
11

---

Published as a conference paper at ICLR 2025
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5799-5809, 2021.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16123-16133, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pp. 333-350. Springer, 2022.
Tianping Chen and Hong Chen. Universal approximation nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE transactions on neural networks, 6(4):911-917, 1995.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6970-6981, 2020.
Maarten de Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost- accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203.13188 2022.
Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physics-informed (and) operator learning. Advances Neural Information Processing Systems, 35:10945-10958, 2022.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances in Neural Information Processing Systems, 30, 2017.
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14304-14313, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
VS Fanaskov and Ivan V Oseledets. Spectral neural operators. Doklady Mathematics, volume 108, pp. S226-S232. Springer, 2023.
Herbert Federer. Geometric measure theory. Springer, 2014.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems, 34:24048-24000 2021.
Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporllssee generalized pde modeling, arXiv preprint arXiv:2209. 2022.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypemetworks. Internationnl Conference on Learning Representations, 2017. URL htt.:/ttts///peeeeiiiiiii http://wwwwwwweeeeeerooree/eeeee
12

---

Published as a conference paper at ICLR 2025
Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying. Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference Machine Learning, pp. 14556-12663 PMLR, 2023.
Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandku- mar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. arXiv preprint arXiv:2403.03544 2024.
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sehastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7825):3573362, 2020.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(3)): 90-95, 2007.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651-4664. PMLR, 2021.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12949-12958, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 2021.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1-97, 2023.
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for deeponets: deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):taa0001, 2022.
Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv preprint arXiv:2004.08688, 2020.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning, arXiv preprint arXiv:2205. 1377,, 2022.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Leurning Representations, 2021. URL
net/
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-1566 2020.
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5773-5783, 202
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF international conference computer vision, pp. 10012-10022, 2021b.
13

---

Published as a conference paper at ICLR 2025
Miguel Liu-Schiaffini, Julius Berner, Boris Bonev, Thorsten Kurth, Kamyar Azizzadenesheli, and Anima Anandkumar. Neural operators with localized integral and differential kernels. arXiv preprint arXiv:2402.16645, 2024.
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/608.03990 2016.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05001, 2017.
Lu Lu. Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-23) 2021.
Michael McCabe, Bruno Régaldo-Saii Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprint arXiv:2310.0299. 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460 4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65()) 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primi- tives with multiresolution hash encoding. ACM Trans. Graph., 414)444::111111100:0:100 July 2022. doi: (0.1145/32222233330122 URL https: /tte:://ooi. http://///oo...11111111100//
100 1ttp:/1..1111532288111 3530117
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions Graphics (ToG), 41()): 1-15, 2022.
Yatian Pang. Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision, pp. 604-621. Springer, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part III 16, pp. 523-540. Springer, 2020.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias neural networks. In International conference machine learning. pp. 5301-5310. PMLR, 2019.
Daniel Rebain, Mark Matthews, Kwang Moo Yi, Gopal Sharma, Dmitry Lagun, and Andrea Tagliasacchi. Attention beats concatenation for conditioning neural fields. arXiv preprint mmXiv:2209.1068. 2022.
Jacob H Seidman, Georgios Kissas, Paris Perdikaris, and George J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net net:/oorumzdd-S0ww--- http:///ggaaapeeeeeeeeee/o/orots
https: https:://pennewwe
14

---

Published as a conference paper at ICLR 2025
Jacob H. Seidman, Georgios Kissas, George J. Pappas, and Paris Perdikaris. Variational autoencoding neural operators. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 30491-30522. PMLR, 23-29 Jul
2023. URL htt.h:tps////..m //tpocee/ ngs. ml w.w...l.....ss.p.mmmmmm..m..
press/
saress/v200 v2tp//v22//////// / sei dman23a...m .hmml
Louis Serrano, Lise Le Boudec, Armand Kassaï Koupai, Thomas X Wang, Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari. Operator learning with neural fields: Tackling pdes on general geometries. Advances in Neural Information Processing Systems, 36:70581-7061 2023.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Kimberly Stachenfeld, Drummond B Fielding, Dmitri Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez- Learned coarse models for efficient turbulence simulation. arXiv preprint arXiv:2112.15D7: 2021.
Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8:13-1 2007.
Elias M Stein and Rami Shakarchi. Real analysis: measure theory, integration, and Hilbert spaces. Princeton University Press, 2009.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1611, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara -ridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprint arXiv:2111.13800) 2021.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. Patchnets: Patch-based generalizable deep implicit 3d shape representations. In Computer Vision-ECCV 2020: 16th Europeun Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 293-309 Springer, 2020.
Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15182-15192, 2021.
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator: neural operator for parametric partial differential equations. arXiv preprint (xXi:22255.0299, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40) :eab86055 2021.
Sifan Wang. Hanwen Wang. and Paris Perdikaris. Improved architectures and training algorithms for deep operator networks. Journal of Scientific Computing, 92(2):35, 2022.
Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert's guide to training physics-informed neural networks. arXiv preprint arXiv:2308. 08466888 2023.
15

---

Published as a conference paper at ICLR 2025
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing Wiley Online Library, 2022.
and beyond. In Computer Graphics Forum, volume 41, pp. 641-676
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533 PMLR, 2020.
Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in neural information processing systems, 32, 2019.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4578-4587, 2021.
16

---

Published as a conference paper at ICLR 2025
A
NOMENCLATURE
Table summarriss the main symbols and notation used in this work.
Table 4: Summary of the main symbols and notation used in this work.
Notation
Description
Operator Learning
X'
The input function space
y
The output function space
u X
Input function Output function
8 SEy
V
Query coordinate in the input domain of s
9 X +
The operator mapping between function spaces
8: R
Encoder mapping
D: R y
Decoder mapping
Fourier Neural Operator
F
Fourier transform and inverse
Fa, F-1
Discrete Fourier transform and its inverse truncated on the first n modes
K
Linear transformation applied to the n leading Fourier modes
W
Linear transformation (bias term) applied to the layer inputs
k:
Fourier modes/ / wave numbers
Activation function
Continuous Vision Transformer
PE,
Temporal positional embedding
PE
Spatial positional embedding
MSA
Multi-head self-attention
MHA
Multi-head attention
LN
Layer normalization
Patch size of Vision Transformer
C
Embedding dimension of Vision Transformer
XN,xx
Latent grid features
B
Locality of interpolated latent grid features
Partial Differential Equations
Vorticity of shallow water equations
1]
Height of shallow water equations
u
Velocity field
P
Pressure field
Passive scalar (smoke)
V
Viscosity
Hyperparameters
B
Batch size
Q
Number of query coordinates in each batch
Number of latent variables interest
T
Number of previous time-steps
HxW
Resolution of spatial discretization
17

---

Published as a conference paper at ICLR 2025
B NEURAL FIELDS& & CONDITIONED NEURAL FIELDS
A neural field is function over a continuous domain param- eterized by a neural network (Stanley 2007). Recently, the computer vision community has adopted neural fields to repre- sent 3D spatial fields, such as neural radiance fields (NeRFs) (Mildenhall et al. (02T), spatial occupancy fields (Mescheder et 2019), and signed distance functions (Park et al. 2019)) These fields represent functions defined over continuous spa- tial domains, with NeRFs additionally taking an element of a continuous ray space as input. In this paper, a point in the domain of neural field is referred to as a query point, here denoted by y. Conditioned neural fields allow these fields to change based on auxiliary inputs u, often represented as latent codes 2, without the need to retrain the entire field; see Figure 7 The latent code affects the forward pass by modifying the parameters of the base field through either global conditioning, where the forward pass is affected uniformly across all query points, or local conditioning, where the modifications depend on the specific query point.
conditioning
glu,
mechanism am
(())
field
Figure 7: The structure of a condi- tioned neural field with both global and local conditioning.
Global Conditioning. Global conditioning in a neural field refers to modifying the forward pass only by the input u and in a uniform manner over the entire query domain. This can be achieved through concatenation, where global latent code is appended to the query vector and passed together through an MLP (Chen & Zhang 2019 Liu et al.] 2021a Jang & Agapito 202I). Alternatively, hypemetwork could accept the auxiliary input and determine the neural field's parameter values based on this input (Ha et al. [2017] Rebain et al.) 2022). However, modifying every parameter of the base field for different auxiliary inputs can be computationally expensive. For example, feature-wise I Linear Modul lation (FiL.) (Perez et al. 2018), which originates from conditional batch normalization (De Vries et al., (017). has been proposed modify only the sine activation function parameters in a SIREN network (Sitzmann et al. (2200). This technique has been used for global conditioning in methods such as r-GAN (Chan et al.) 2021).
Local Conditioning. Local conditioning occurs when the latent code z, which modifies the base field's parameters, is function of the input query y, i.e., 2 = g(y). This results in query-dependent field parameters that can vary across the query domain. A common approach to generate local latent codes is to process the image or scene with convolutional network and extract features from activations with receptive fields containing the query point. This can be done using 2D convolutions
of projected 3D scenes (Xu et al. [019 Yu al. 2021 Trevithick & Yang 2021) 3D convolutions (Peng et al.) 2020)) Liu et al. 2020 Chibane et al. 2020). Muller et al. (Müller et al., 2022) learn features directly on multiresolution grid vertices and create local features via nearest grid point interpolation for each grid. Chan et al. (Chan et al. 2022) create features along three 2D planes and
interpolate features from projections onto these planes. DeVries et al. (De et al. 2021 learn latent codes over patches on a 2D "floorplan" of scene. Other methods divide the query domain into patches or voxels and learn a latent code z for each patch or voxel (Chabra et al. 2020 Tretschk et al. 2020 Chen et al. 2022).
C NEURAL OPERATORS AS CONDITIONED NEURAL FIELDS
Here we demonstrate that many popular operator learning architectures are in fact neural fields with local and/or global conditioning. While connections between architectures such as the DeepONet and the Fourier Neural Operator have been described previously (Kovachki et al. 2021). they are typically formulated under certain architecture choices which allow these architectures to approximate one another. In contrast, here show that, by viewing these architectures conditioned neural fields, they are explicitly related through their respective choices of conditioning mechanisms and base fields.
18

---

Published as a conference paper at ICLR 2025
C.1 ENCODER- DECCDER NEURAL OPERATORS
A large class of operator learning architectures can be described as encoder-decoder architectures according to the following construction. To learn an operator G ++ y an input function u E X are first mapped to a finite dimensional latent representation through the encoding map & X - RR". This latent representation is then mapped to a callable function through a decoding map D RR" + y. The composition of these two maps gives the architecture of the operator approximation scheme,
= D o E. The class of encoder-decoder architectures includes the DeepONet (Lu et al. 202T), the -Csed architecture of (Bhattacharya al. 2021). and NoMaD (Seidman al. 2022).
These encoder-decoder architectures can be seen as neural fields with a global conditioning mecha- nism by viewing the role of the decoder as operator which performs the conditioning of a base field, with a latent code derived from parameterized encoder, = (u)
For example, in DeepONet, the base field is a neural network to (y)) (trunk network) appended with a last linear layer, and the decoder map conditions the weights of this last layer via the output of the encoding map (u) (branch network); see Figure 8 (a) for a visual illustration. In other words, the decoder map a weighted linear combination of basis elements to(y), where the weights are modulated by global encoding of the input function (u) yielding outputs of the form s(y) = (u), to (y)). The PCA-based architecture of (Bhattacharya et al. 2021) can be viewed analogously using as basis elements the leading n eigenfunctions of the empirical covariance operator computed over the output functions in the training set.
DeepONet.
NoMaD, Seidman et al. (Seidman et al. 2022) demonstrated how encoder- encoder-decoder architectures
can employ nonlinear decoders learn low-dimensional manifolds in an ambient function space. These so-called NoMaD architectures typically perform their conditioning by concatenating the input function encoding with the query input, i.e., syy) fo &ss (u)): see Figure 8 (b) for an illustration.
C.2 INTEGRAL KERNEL NEURAL OPERATORS
Integral Kernel Neural Operators form another general class of neural operators that learn mappings between function spaces by approximating the integral kernel of operator using neural networks. They were introduced as generalization of the Neural Operator framework proposed by Li et al. (Li al. 2020). The key idea behind here is to represent the operator G X + y as an integral kernel, g (y) = Sa (Iz, y, u(z))dz, where K is a kernel function. In practice, the kernel function is parameterized by neural network, and the integral is then approximated using quadrature rule, such as Monte Carlo integration, Gaussian quadrature, or via spectral convolution in the frequency domain.
Graph Neural Operator (GNO). A single layer of the GNO Li et al. 2020 is given by
syy) = o (Wu(v)= [V(2]
Ka v, u(x))u(x)).
(10)
mEN(y)
where W E Rd, is trainable weight matrix, and (.).) is local message passing kernel parameterized by & (Li et al. (2000). As illustrated in Figure c), the GNO layer is a conditioned neural field with local and global conditioning the input function The base field uses fixed positional encoding y(v) that maps query coordinates y to input function values at neighboring nodes Nw), i.c., 7()) u(z)/[N followed by linear layer. The encoding width equals the maximum number of neighboring nodes, with zeros filling tailing entries for queries with fewer neighbors. The GNO layer is conditioned globally via the parameterized kernel (.)()) modulating the linear layer weights, and locally via a position-depended bias term Wu(y) added as a skip connection. The layer outputs sy) are finally obtained by summing the linear layer outputs and the skip connection, then applying nonlinear activation function a.
Fourier Neural Operator (FNO). The FNO introduced computationally efficient approach for performing global message passing in GNO layers assuming a stationary kernel (.)() (Li al. 2021 Bruna et al. 2014). Leveraging the Fourier transform, single layer of FNO (Li al. 2021 given
19

---

Published as a conference paper at ICLR 2025
DeepOlket
RoMaD
Graph Neural Operator
Fourier Neural Operator
(u)
(3)
global conditioning
local conditioning
7(2)=
u(e),
N(a)
r(0) (2o (R,, M)), sin (2m 20)
[Nari
Figure 8: Viewing common neural operators conditioned neural fields. Base field and conditioning mechanisms for: (a) DeepONet; (b) NoMaD; (c) the GNO layer; (d) the FNO layer.
by
sty) = (F
(11)
where W E Rdoxd, and K E Re*n are trainable weight matrices, Fn I,2 C" is the Fourier transform truncated to the first n modes, and F.- C - I is the left inverse of this operator. We show that the FNO layer can be viewed as a conditioned neural field employing both local and global conditioning
Without loss of generality, we demonstrate this for one-dimensional query domain y E RR. To this end, note that the convolutional part of the FNO layer (equation IT) can be re- re-written (see Appendix for derivation)
the input function
Fn (KFF(()))
R(Kū), cos (ky, 8)) Σ
Σ ((K)) sin (k; vt),
(12)
where û - Fr, u denotes the truncated Fourier transform of u, and R(z) and 3(2) denote the real and imaginary parts of a complex vector z E C". We may interpret this transformation of u(y) as a linear transformation (which depends on ú) acting the positional encoding,
(1) = [cos(2 (k1, v\), (in(22((11 ())..,.))....................... co.(.....o222((( (k v)), sin (2 (k.. v/)
(13)
The resulting expression (equation 12) is then acted on with a position dependant bias term W Wu(y). followed by a pointwise nonlinearity σ.. In this manner, we see that an FNO layer is a neural field with fixed positional encoding (first Fourier features), a global conditioning of the weights by the Fourier transform of u, and a local conditioning acting as a bias term by Wu(y); see Figure 8)d) for an illustration. This neural field interpretation also suggests an alternative way for implementing FNO layers by explicitly using equation [2 to compute the inverse Fourier transform, thereby allowing them to be evaluated at arbitrary query points instead of fixed regular grid.
Extending to other neural operators. In practice, GNO and FNO layers stacked form deeper architectures with a compositional structure. These can also be interpreted as conditioned neural fields, where the base field corresponds the last GNO FNO layer, while all previous layers are absorbed into the definition of the conditioning mechanism. Analogously, one can examine broader collection of models that fall under the class of Integral Kernel Neural Operators, such as
Gupta et al. 2021 Tripura & Chakraborty Liu-Schiaffimi al. 2024).
2022 Cao et al. 2023
Fanaskov & Oseledets 2023
D THE FNO LAYER AS A CONDITIONED NEURAL FIELD
Proposition D.1. Suppose that single layer of the Fourier Neural Operator (FNO) (Li et al. given by
2027
s(y) a (Wu(y) +F. (KF,((())
(14)
Then the integral kernel term can be expressed
F (KF, (x) Σ R(KK), cos 27 (k3, 9)
Σ
(Ki), sin (2n (kg, 1)),
(15)
20

---

Published as a conference paper at ICLR 2025
Proof. Recall that the map Fn L - C is the Fourier transform truncated to the first n modes and F : + L2 is the left inverse of this operator,
Fr = Σ
2j 2
(16)
1=1
Recall Fuler's formula,
2 c2s (k: y)) + i ssnn (2 (k. y))-
(17)
When the (complex) Fourier coefficients ž1 21,..., 2n come from a real valued signal, the imaginary components in the reconstruction equation cancel and we are left with only real values. In this
case, we decompose each 2j into real and imaginary parts, 2j = aj + 1B8.. then using equation [7 rewrite equation as
n
(=)())
Σ oy (2x(
Σ
Bg sin (2n(2 u))
(18)
This allows us to view the convolutional part of the FNO layer equation as
Fr (w)) = Σ R(K cos (k,, 8)) - Σ 3 (K ú), sin (k), v\),
(19)
where û = Fr u denotes the truncated Fourier transform of u, and R(z) and 3(z) denote the real and imaginary parts of complex vector
C".
E THEORETICAL INSIGHTS ON COORDINATES EMBEDDINGS
Without loss of generality, we consider an MLP with scalar inputs and outputs. Specifically, let e be the input coordinate, g (z) = x and do d1t1 1. An MLP fo(x) is recursively defined as follows
(0) (x) w()) (x) bb
gd oF. (z)), 1= 1.2....,4
(20)
with final linear layer defined by
fe(x) = W(++1) g( + ((L-1)
(21)
where w E Rdexdi- the weight matrix in 1-th layer and a is an element-wise activation function. Here, = (w()) b(),..., ((++1) ([++),((++))))) (L++)) represents all trainable parameters of the network. In particular, all weight matrices are initialized by the Glorot initialization scheme (Glorot & Bengio 2010), i.e.,
suppose that the network is equipped with Tanh activation functions and
W(!) N(O, 2.,2.-,) for I = 1, ......., L 1. Moreover, all bias parameters are initialized to zeros.
Definition E.1. A fffunction f RT + 1RT is called Lipschitz continuous if there exists a constant such that
lf(x) f(v)l2 S L11 v112, Vz, R.
(22)
The smallest L. for which the previous inequality is true is called the Lipschitz, constant of f and will be denoted Lip(S)-
Theorem E.2. (eeeerrrr 2014), Theorem 3.1.6] If f: R" + +RR" R" is a locally Lipschitz continuous
function, then ff is differentiable almost everywhere. Moreover, if f Lipschitz continuous, then
Lip(J) sup |ID.//1l.
(23)
ZERR
where ||[[| (=](=:]]=)==1) ||Mal|| is the operator norm of the matrix M E R
21

---

Published as a conference paper at ICLR 2025
In particular, if is real valued (i.e. m 1), its Lipschitz constant is the maximum norm of its gradient Lip(f) = sup, [V) on its domain set.
By Theorem 1 and the chain rule, the Lipschitz constant of MLPs have an explicit formula (Virmaux & Scaman 2018: Latorre et al., 2020)
2 II L+1
(wl). -diag (o(E)- (z)) w
Lip (fo) = sup ZER"
(24)
In Eq. equation 24 the diagonal matrices diag (x))
difficult to evaluate, they may depend on the input value x and previous layers. Fortunately, most major activation functions are 1-Lipschitz. More specifically, these activation functions have a derivative a(-) [0, 1j. Hence, we may replace the supremum on the input vector x by supremum over all possible values:
2
Lip (fo) v
sup
II i=++1
(WO) diag (2(--1)) w
(25)
(0.1)
L+1
w
(26)
From the above derivation, we immediately obtain that the Lipschitz constant of MLP is inherently tied to the weight matrices of its hidden layers. MLPs are known to suffer from spectral bias, struggling learn nigh-frequency functions efficiently. One reasonable explanation is that Lip(f) is bounded by the product of its parameter norms, which can only increase gradually during training by gradient descent. Consequently, higher frequency components of the target function are learned later in the optimization process (Rahaman et al.] 2019). Random Fourier features (Tancik et al.) 2020) have been proposed as a potential solution this spectral bias issue. In the following theorem. we demonstrate that both random Fourier features and our proposed position embedding can substantially increase the network's Lipschitz constant at initialization. This enhancement enables more efficient learning of complex, fine-scale signals from the early stages of training.
Theorem E.3. Given the definition of MLP in equation 20 and equation the Lipschitz, constant of the following coordinates embeddings can be bounded as follows
(a) For conventional MLPs, the coordinating embedding is trival and can be viewed the first linear layer &1 (I) = Wx + b, where W E Rd b E R, and is the width of the network. Each element of W is independently sampled from NO, 2). The expected Lipschitz constant of &n satisfies
(2d((++1)) =1 1
Lip (01)
(27)
Va r(d/2)
2d'
(b) The coordinate embedding of random Fourier features is given by d2(x) coos W.r)
sin W x)
where W E Rd/2 and each element of W is independently sampled from N s²). with
s > 0 being a hyperparameter. The expected Lipschitz constant of &2 satisfies
d/2-11 2
x svd (1-
1
Lip (#2) 2 T(d/4)
(28)
8dl
(c) The proposed coordinate embedding is defined as:
(5(x) Σ wi(x)
Σ #(e-2x)² f;,
(29)
i=1
where {xa)" is uniform grid in [0, 1}. and f, RRh are bounded latent features indepen- dent of coorlinates, satisfying [|f, < M for some m. M > O. Let ht denote the spacing of the grid, i.e., h Zi+1 x4 for any i O, O..,..., 1. For sufficiently large such that V23 <. h, the Lipschitz constant of dgg is bounded as:
M 2 x Lip (da) S 2M 23e
(30)
22

---

Published as a conference paper at ICLR 2025
Proof. (a) Let W (Wo) Then have
Lip(d1)= = [|]
Σ
W?
Σ
Σ
7
1
(31)
iml
Since Ex(d) 22 ()d11)))
T(d/2)
1 Ex(d) /2I((d+ 1)/2)
E [ii((en)= = Vd Vd T(d/2)
(32)
where r(-) is the Gamma function.
For large d, this can be approximated by:
1 vd
Vd 1- 4d
E Lip(01)
= 1
(33)
4d
(b) We observe that
|[[
W cos( Wx) Wiin(Wxx
4/2
Lip (62) = sup
|
5
2
W? = v2 W/
(34)
#ER
1=1
Similar to the proof of (a), we have
r
++1
(Ti(()2) S xxV2F((d/2)
=
(35)
T ((d)
For large d, this can be approximated by:
d
1
svd (1-
1
E Lip == sv sV2
(36)
2
2d
2d
(c) To estimate the Lipschitz constant of o, we apply Theorem estimating its first-order derivative:
E.2 which reduces the problem to
62((x) = (()()).
(37)
For any given v, there exists unique such that [1+1) We can then decompose d'(x) as follows:
d=)()) (z) + W +
Σ
w(xxf, = 11 Is.
(38)
Let us first estimate I1, I2. Since all f, are bounded, consider g(z) e -88² each wx(r) is translation of g by x,, By differentiation, we obtain:
g'(x) = -2,xre -x²"
(39)
g"(x) = 24 2 -. Bu² -B²
(40)
Setting (²(x) = 0 yields x = + /23 Consequently, g' (x) achieves its maximum value of /2
at V23 and its minimum value of -2BBe at V Therefore, we can bound w/(x: as follows:
2B3 < (x( I 2/8 3
(41)
To estimate Ia. recall that we assume V28 h. For i # k. + and fixed z, we have mh <
xx 24| < (m + 1)h for a unique integer and (x( decreases moves away from 24. Then:
w/(xx) 2.3hh e- 3mm
(42)
23

---

Published as a conference paper at ICLR 2025
Therefore, have
n 2 Σ
[<(())
28mh e Sm2h
(43)
2 2 Σ
28hr e =8h2x² dx
(44)
m=
m-1
28h a2 dr
2
(45)
Ahr
(46)
h
(n-12)
2
e
(47)
2
$ h"
(48)
Since || S M,
| s M
Iun [,u((z)] + (x)++
Σ
(49)
$ M 2 2//2000 hn
(50)
Therefore, conclude that
Lip (6o) S 2M /28e-
(51)
To obtain lower bound for Lip(øs), we proceed follows: Without loss of generality, assume [|22l22 M. Let xt h -23 where (wg] achieves its maximum. Then
(*)(22)l2 2 1| (x" )62l2 (2"())))
J1 J2 J33
(52)
123
We can bound each term as follows:
JI = M /BB 1
(53)
and
3 ( )
J2 s M(3)) h
(54)
22
Note that 1/ goes to 8 goes infinity. So J2 bounded with respect to 8. For J3. we similarly estimate:
(1-e e 8hh (m-12
Jg S M Σ 3mh Bm2
I h
(55)
h
M==1
Thus, combining these estimates, we conclude that
Lip(0s) Z M MMM2 28e
(56)
Our theorem yields key insights into the Lipschitz constants of various coordinate embedding methods. For trivial coordinate embeddings (linear layer MIP),, the Lipschitz constant is approximately 1.
24

---

Published as a conference paper at ICLR 2025
This has impact the network's global Lipschitz constant, resulting in the network remaining susceptible to spectral bias.
In the case of random Fourier features, the Lipschitz constant be improved through two main approaches. First, by increasing the standard deviation of the Gaussian used for sampling the frequency matrix. Second, by increasing the network width. This observation implies trade-off between network width and the frequencies used in the Fourier features.
For our proposed coordinate embeddings, the Lipschitz constant primarily depends the parameter 3 in the interpolation process. This dependence on a single parameter potentially offers more straightforward control over the Lipschitz constant compared to Fourier features.
These findings significantly impact neural field design and optimization using different coordinate embedding techniques. They provide a quantitative basis for comparing embedding methods' capa- bilities, particularly in addressing the spectral bias inherent in standard MLPs. The straightforward control of the Lipschitz constant in our proposed method (via () may offer advantages in model tuning and optimization.
F EXPERIMENTAL DETAILS
F.1 TRAINING AND EVALUATION
Training recipe. We use unified training recipe for all CVTT experiments. We employ Adam optimizer (Kingma & Ba 2014 Loshchilov & Hutter 2017) with weight decay 10-5 Our learning rate schedule includes an initial linear warm-up phase of 5, 000 steps, starting from zero and gradually increasing 10 followed by exponential decay rate of 0.9 for every 5, 000 steps. The loss function a one-step mean squared error (MSE) between the model predictions and the corresponding targets at the next time-step, evaluated at randomly sampled query coordinates:
MSE 111 Q
ΣEΣ
(k) 8
(ys)
(k) 8,
(y:) (x3)
(57)
BQD j=
2
where 8) (y)) denotes the k-th variable of the i-th sample in the training dataset, evaluated at a query coordinate y,, and denotes the corresponding model prediction. It is worth noting that our objective function is a one-step loss, which differs significantly from the rollout loss used in training DPOTs All models are trained for x 10 iterations with a batch size B = 64. Within each batch, we randomly sample Q = 1, 024 query coordinates from the grid and corresponding output labels.
Training instabilities We observe certain training instabilities in CVIT models, where the loss function occasionally blows up, causing the model to collapse. To stabilize training, we clip all gradients maximum norm of 1. If loss blowup is detected, we restart training from the last saved checkpoint.
After training, we obtain the predicted trajectory by performing an auto-regressive rollout the test dataset. We evaluate model accuracy using the relative L norm, commonly used in Li et al. et al. 2021):
Evaluation.
(k)
Rel. L Nest
(58)
||
112
Am
where the norm is computed over the rollout prediction at all grid points, averaged over each variable of interest.
F.2
MODEL DETAILS
For the ID advection equation, we create a tiny version of CVIT to ensure the model size is comparable to other baselines. Specifically, consists of transformer layers with an embedding dimension of 256 and 16 attention heads. A single transformer decoder layer with 16 attention heads is used.
CVIT.
patch size of 4 to generate tokens. The encoder
25

---

Published as a conference paper at ICLR 2025
For the shallow water equation, compressible and incompressible Ntvier-Stokes equations and diffusion reaction equations, the CViT configurations are detailed in [ While we use a patch size of 8 8 by default, Stokes problem. We find that using a smaller patch size better captures the small-scale features in flow motion, leading to improved accuracy.
a smaller patch size of 4 x when training CViT-L for the Navier-
Neep For the DeepONet, the branch network (encoder) is an MLP with 4 hidden layers of 512 neurons each. The trunk-net (decoder) also employs MLP with 4 hidden layers of 512 neurons.
The encoder consists of 4 hidden layers with 512 neurons each, mapping the input to a latent representation of dimension 512. The decoder, also an MLP with 4 hidden layers of 512 neurons, takes the latent representation and maps the output function.
NoMaD.
Dilated ResNet. Following the comprehensive experiments and ablation studies in PDEArena (Gupta Brandstetter 2022). we adopt the optimal Dilated ResNet configuration reported in their work. This model has 128 channels and employs group normalization. It consists of four residual blocks, each comprising seven dilated CNN layers with dilation rates of [1. 2, 4, 4, 2, 1].
U-Nets. Based on comprchensive experiments and ablation studies in PDEArena (Gupta & Brand stetter 2022), we select the two strongest U-Net variants reported in their work: U-Netatt and U-F2Net. For both architectures, we use one embedding layer and one output layer with kernel sizes 3 x 3. Besides, we use 64 channels and channel multipliers of (1, 2, 2, 4) and incorporate residual connections in each downsampling and upsampling block. treenormalization and pre-activations applied, along with GELU activations. For U-Nelatt attention mechanisms are used in the middle blocks after downsampling. For U-F2Net, we replace the lower blocks in both the downsampling and upsampling paths of the U-Net architecture with Fourier blocks. Each Fourier block consists FNO layers with modes 16 and residual connections.
UNO. Following the comprehensive experiments and ablation studies in PDEArena (Gupta & adopt the optimal UNO of 128 channels.
Brandstetter 2022)
FNO. For the ID advection equation, our FNO model employs Fourier neural blocks, each with 256 channels and 12 Fourier modes. The output is projected back to 256-dimensional space before passing through linear layer that produces the solution over the defined grid.
For the shallow water equation, employ the best FNO model from comprehensive experiments and ablation studies in PDEArena (Gupta & Brandstetter 2022). Specifically, we use the FNO consisting of 4 FNO layers, each with 128 channels and 32 modes. Additionally, and two output layers with kernel sizes of 1 x 1, as suggested in Li et al.(Li et al. 202T). We use GeLU activation functions and no normalization scheme.
embedding layers
For the Navicr-Stoker equation, we follow the problem setup in DPOT (Hao et al. 2024), which differs from the setup in PDEArena (Gupta & Brandstetter 2022). Therefore, we directly report the results from DPOT (Hao et al. 2024).
FFNO GK-T OFormer GNOT DPOT MPP. We directly report the results from DPOT HHo et al. 2024). Detailed implementations can be found in their paper.
F.3
LINEAR ADVECTION EQUATION
We consider the one-dimensional linear advection equation
Ou
du
O, E 10.
a1.
dx
with periodic boundary conditions u(0. t) u(1, t) given an initial condition u(x, 0) un(x:) with the advection velocity c = 1. To generate discontinuous initial conditions, the initial condition uo is assumed be
ti = -1+ 20}
26

---

Published as a conference paper at ICLR 2025
Table 5:
Performance of neural operator architectures under the relative L2 and Total Variation metrics for the linear advection with discontinuous initial conditions benchmark.
Rel. L2 error (%, 4)
Total Variation error (L)
Model
# Params
Mean
Median
Worst-case
Mean
Median
Worst-case
FNO
4.98M
13.94
12.03
56.52
0.0406
0.0309
0.1910
DeepONet
2.20M
13.77
12.20
39.64
0.0572
0.0453
0.2300
NoMaD
2.21M
13.64
11.01
69.00
0.0326
0.0245
0.1732
CVTT
3.03M
11.70
10.68
35,44
0.0334
0.0274
0.1516
where zig a centered Gaussian,
Mo N1 C) and, C =
Here - denotes the Laplacian on D subject to periodic conditions on the space of spatial mean zero functions. Morevoer, denotes the inverse length scale the random field and d determines the regularity of ug. For this example, we set T = 3 and d = 2.
Dataset. We make use of the datasets released by Hoop el al. de Hoop et al. 2022). This dataset consists of 40,000 samples of discretized initial conditions on grid of N 200 points.
Problem setup. We follow the problem setup by Hoop al. (de Hoop et al. 2022). Our objective is to predict the solution profile at time = 0.5. We compare the CViT model's performance against several strong baseline neural operators: DeepONet (Lu al. 202T NoMaD (Seidman al. 2022 and Fourier Neural Operators (Li et al. 2021). p. For the training and evaluation, we considered split of 20,000 samples used for training, 10,000 for validation, and 10,000 for testing.
Training. For all models, we use batch size of 256 with a grid sampling size of 128, except for the FNO, which uses the full grid size of 200. The models are trained using the AdamW optimizer (Loshchilov Hutter 2017) with an exponential decay learning rate scheduler. The models are trained for 200,000 iterations, and the best model state is selected based on the validation set performance to avoid overfitting. The final performance evaluated on the test set of 10,000 samples.
Evaluation. Due the discontinuous nature the problem, the relative I.2 norm may not properly reflect the performance of the models. Instead, more suitable metric for this task is the total variation (Stein & Shakarchi 2009), defined as
TV(f.9) = / || - Vgl|dx.
The total variation quantifies the integrated difference hetween the magnitudes of the gradients the predicted solution and the ground truth, providing better indicator of model's ability to capture and preserve discontinuities in the solution.
Table presents a summary of our results. We notice that the CVTT model achieves the lowest mean, median and worst-case relative L errors, outperforming the other architectures on this metric. Furthermore, when considering the total variation metric, the CViT model exhibits competitive performance for the mean and median total variation values, and achieves the lowest prediction emor the worst-case sample in the test dataset.
27

---

Published as a conference paper at ICLR 2025
F.4 SHALLOW- WATER EQUATIONS
Let u denote the velocity and yy interface height which conceptually represents pressure. We define
Ç V x and divergence D V.u. The vorticity formulation is given by:
o (u(S + ()) =
Öt
8D V (u(S ff) = --> (2 + (g7)
Öt
On + (uh)
subject to the periodic boundary conditions. Here f is Coriolis parameter and g 9.81 is the gravitational acceleration, and h is the dynamic layer thickness.
Data generation. The dataset is generated by PDEArena (Gupta & Brandster fer 2022) using SpeedyWeather.J 96 (h 1.875, Ay = 3.75%), and temporal resolution of At 48 h. The resulting
on a regular grid with spatial resolution of
192
dataset contains 6,600 trajectories with a temporal spatial resolution 11 x 96 x 192.
Problem setup. We follow the problem setup reported in PDEArena (Gupta & Brandstetter 2022). Our goal is to learn the operator that maps the vorticity and pressure fields from the previous 2 two time-steps the time-step. All models trained with 5,600 trajectories and evaluated on the remain 1,000 trajectories. We report the relative L errors over 5-steps rollout predictions.
Training. For training CVTT models, please refer section F.1 For training other baselines Table 2. we follow the recommended training procedure and hyper-parameterr presented in PDEArena (Gptaa & Brandstetter 2022) Specifically, we use the AdamW optimizer with a learning rate of 2 x 10 and weight decay of 10-5, Training is conducted for 50 epochs, minimizing the summed mean squared error. We use cosine annealing as the learning rate scheduler (Loshchilov Hutter
2016) with a linear warm-up phase of epochs. An effective batch size of 32 used for training.
Impact of Grid Resolution. our experiments, we matched the resolution the embedding grid to the dataset for convenience. However, model's performance is not significantly affected as long as the embedding grid roughly matches the dataset resolution.
To validate this, different grid resolutions: 95 x 191, 97x 193, and 128 x 200. We performed these experiments using the shallow water equation example under the same hyper-parameter settings. The resulting relative 2 errors are summarized below:
CViT-B
backbone architecture, conduct experiments with three
Table 6: Swallow Water equation: Relative 2 error for different grid resolutions.
Grid Resolution
96 x 192
95 x 191
97 193
128 x 200
Rel. 2,2 Error
1.563 x 10-2 1.572 x 10-2 1.568 x 10-2 1.562 10-2
The results demonstrate negligible variation in error across different resolutions. This empirical evidence supports our claim that CVIT's performance is robust small variations in grid resolution. suggesting that precise matching between the embedding grid and dataset resolution is not critical for model efficacy.
Impact of random Fourier feature frequencies. We conduct an ablation study to investigate the impact of the scaling parameter s random Fourier feature embeddings. Using CViT-B as our backbone architecture, we trained models with different values of the SWE benchmark. As shown in Table model accuracy improves with moderately large values but deteriorates when becomes too large (e.g., 167)) reverting to performance levels similar to 4s. Notably, proposed grid-base based embedding consistently outperforms the random Fourier feature approach across all tested values of
28

---

Published as a conference paper at ICLR 2025
Figure 9: Shallow water benchmark. Ablation studies on the latent dimension of grid features and number of attention heads in decoder. Here we use CViT-B as our backbone architecture.
10
1.6
1.6
1.5
1.5
1.4
1.4
1.3
1.3
2
2
2
2
2
Latent Dim
# Decoder Attention Heads
Table Shallow water benchmark. Ablation studies on the hyper-parameter s of random Fourier features. Here we use CViT-B the backone.
Random Fourier Features
Grid-based embedding
2m
4m
8m
16
8 10
Rel. 2 error
4.94 x 10-2
4.16 x 10-2
3.25 x 10-2
10
2.69 10
F,5
INCOMPRESSIBLE HAVIER-STOKES EQUATION
We consider the two-dimensional incompressible Navier-Stokes equations in the velocity- pressure formulation, coupled with a scalar field representing a transported particle concentration via the velocity field. The governing equations given by:
dc
11 vc= = 0, Øt
Ou Ōt
+ u Vu+ Vp v/ v
u uu
vu 0,
where is the scalar field, u is the velocity field, and and v the kinematic viscosity. addition, the velocity field is affected through an external buoyancy force term f in the y direction, represented as f f)
Data generation. The 2D Navier-Stokes data is generated by PDEArena (Gupta & Brandstetteri 2022), which is obtained grid with spatial resolution of 128 x 128(h 0.25, Ay 0.25) and a temporal resolution of At = 1.5 s. The simulation uses a viscosity parameter = 0.01 and
T+1
T+3
Food Pressure Peszura
2.5
0.0
-2.5
2.5
0.0
2.5
0.64
Figure 10: Shallow water benchmark. Representative CVIT rollout prediction the pressure field, and point-wic wise against the ground truth.
29

---

Published as a conference paper at ICLR 2025
buoyancy factor of (0, 0.5"" The equation is solved on closed domain with Dirichlet boundary conditions (v 0) for the velocity, and Neumann boundaries ( 0) for the scalar field. The simulation runs for 21 s, sampling every 1.5 s. Each trajectory contains scalar and vector fields at 14 different time frames, resulting in dataset with 7,800 trajectories.
Problem setup. We follow the problem setup reported in Hao al. (Hao et al. 2202). We aim to learn the solution operator that maps the passive scalar and velocity fields from the previous 10 time-steps to the next time-step. The models are trained with 6,500 trajectories and tested on the remaining 1,300 trajectories. We report the resulting relative L errors over 4-steps rollout predictions.
Evaluation. We directly report the results as presented in DPOT (Hao et 2024 For comprehen- sive details on the training procedures, hyper-yaranenn parameters, please refer to their original paper.
Impact of Latent Queries in the Time Aggregation Layer. In our implementation, we use single latent query in the Perceiver module. This query condenses information from T time steps into a single time step representation, effectively reducing the length of spatial-temporal tokens and computational costs.
To investigate the effect of this hyperparameter model performance, we conduct an ablation study by training CViT-L model with a patch size of 16, varying the number of latent queries (n). As shown in Table 8 the results demonstrate that increasing the number of queries significantly enhances model performance, though also increases the effective token length proportionally to the number of queries. This improvement likely stems from the fact that n 1 discards too much useful information from the input trajectory during the time aggregation step, highlighting the potential for further optimization in this architecture.
Table 8: Navier-Stokes benchmark. Relative T,2 error for CViT-L models trained with a patch size of 16 and varying numbers of latent queries. Increasing the number of latent queries significantly improves accuracy. Note that these results differ from Table where the best performance is achieved with patch size of 4.
Latent queries
n
1
7
2
4
Rel. 2 error
6.56 %
4.32 %
3.81 %
3.18 %
Analysis of cross-attention patterns. we analyze the attention patterns that emerge in CVTT when trained on the Navier-Stokes equation. We examine both the encoder and decoder cross-attention mechanisms understand how the model learns to process temporal and spatial information.
The encoder's cross-attention mechanism is responsible for temporal aggregation. Figure I shows the averaged attention weights across different samples and attention heads, visualizing how the model attends to different timesteps in the input sequence. We observe that the attention weights systematically favor more recent timesteps, with the final timestep receiving notably higher attention. This temporal bias aligns with the physical intuition that recent states are more informative for predicting future dynamics while still maintaining some attention on earlier states for capturing longer-term dependencies.
The decoder's cross- crts-nattnniion patterns reveal how the model processes spatial relationships. Figure visualizes the attention weights for various query points in the spatial domain. A clear pattern emerges where attention weights decay with distance from the query point, indicating that the model has learned to focus primarily locally relevant information. This locality- behavior is particularly evident in regions of high fluid activity, suggesting the model adapts its attention patterns based on the underlying physics.
These attention patterns demonstrate that CVIT learns physically meaningful representations through its cross-attention mechanisms. The encoder effectively prioritizes recent temporal information while the decoder captures spatially local dependencies, both of which are crucial for accurate fluid dynamics prediction.
30

---

Published as a conference paper at ICLR 2025
10a
Avg. attn weights
10-1
10
10-3
0.00
0.25
0.50
0.75
1.00
Figure 11: Incompressible Navier-Stokes benchmark (NS). Visualization of encoder cross- cross-attention weights in CVíT. It shows averaged attention weights across different samples and attention heads, where the x-axis represents input timesteps. The model learns assign higher attention weights to more recent timesteps, particularly the final timestep.
100
1.0
1.0
1.0
0.25
08
0.25
0.25
0.20
0.15
0.20
0.20
0.15
0.35
0.15
0.10
0.10
0.30
0.10
02
0.05
0.2
0.2
0.2
0.05
0.
0.0
6.0
0.0
0.0
1.0
0.0
1.0
0.0
0.5
1.0
0.5
1.0
Figure 12: Incompressible Navier-Stokes benchmark (NS). Visualization of decoder cross-altention patterns CViT. Spatial attention weights for selected query points (marked by red stars), showing how attention naturally decays with distance from each query location. The attention patterns demonstrate that the model has learned to focus on spatially proximate regions when learning fluid dynamics.
T+1
Ref Velcoity
0.25
0.00
-0.50
Pres: Velcoity
0.25
0.00
0.25
-0.50
0.025
0.020
Abs, Error
0.015
0.010
0.005
Figure 13: Incompressible Navier-Stokes benchmark (NS). Representative CVIT rollout prediction of the velocity field in z direction, and point-wiss wise error against the ground truth.
31

---

Published as a conference paper at ICLR 2025
T+1
T +
T+
T+4
0.5
Ref Velcolty
0.0
-0.5
0.5
Pred. Velcolty
0.0
0.5
0.025
0.020
0.015
Abs.
0.010
0.005
Figure 14: Incompressible Saoker-Stokes benchmark (NS). Representative CViT rollout prediction of the velocity field in y direction, and point-wise error against the ground truth.
F.6 COMPRESSIBLE NAVIER-STOKES EQUATION
The 2D compressible Navier-Stokes equations govern fluid flow with variable density:
3.p + V (pv) = 0, p(0. v v Vv)
(59)
-pp + (S + //3)+/33VVV v),
(60)
pv² V.
<++ pr v v = 0.
0
E +
(61)
2
Here, denotes mass density, v velocity, gas pressure, r 5/3, a' is the viscous stress tensor, and internal energy (determined by the equation of state), The viscous stress tensor a' incorporates both shear (7)) and bulk (() viscosity coefficients. This system captures complex fluid dynamics phenomena, including shock wave dynamics, and finds applications in aerospace engineering and astrophysical fluid dynamics.
Data generation. We use the dataset generated by PDEBench (Takamoto et al. 2022 with M = 0.1, 0.01, where M /ol/e, /e, is the Mach number, c, /Tp/p the sound velocity.
Problem setup. We follow the problem setup reported in Hao et al. (Hao et al. 2024h. We aim to learn the solution operator that maps the activator and inhibitor fields from the previous time-steps to the next time-step. The models are trained with 9,000 trajectories and tested on the remaining 1,000 trajectories. We report the resulting relative I errors over rollout predictions.
For training CVTT models, please refer to section F.I It is worth noting that for this specific task, we employ an initial learning rate 5 x 10-4, different from the 10-3 used other experiments.
Training.
We directly report the results as presented in DPOT (Hao et al. 2024). For comprehen- sive details on the training procedures, hyper-parameters, please refer to their original paper.
Evaluation.
32

---

Published as a conference paper at ICLR 2025
T+1
T+2
T+3
T+4
T+5
T+6
Ref.
1
Pred.
0.03
Abs. Error
0.02
0.01
Figure 15: Compressible Navier-Stokes Benchmark. Representative CVTT rollout prediction of the
velocity field u, and point-wise against the ground truth.
T+1
T+2
T+3
T+4
T+5
T+6
1
Ref.
Pred.
0
Abs. Error
0.02
0.01
Figure 16: Compressible Kavier-Stoker Benchmark. Representative CVTT rollout prediction of the velocity field u, and point-wise against the ground truth.
33

---

Published as a conference paper at ICLR 2025
T+1
T+2
T+3
T+4
T+5
T+6
120
Ref. p
100
80
120
Pred.
100
80
Abs. Error
1.0
0.5
0.0
Figure 17: Compressible Navier-Stoke Benchmark. Representative CVTT rollout prediction of the pressure field p, and point-wise enror against the ground truth.
F.7 DIFFUSION-REATIONN EQUATION
The 2D diffusion-reaction equation is given by
0,u = Du + Duy0y - Ru:
0,0 D2O2zu + D,00yu Ru:
where uf,, is the the activator and v(t, y) denotes the inhibitor. Here D, and D. are the diffusion coefficient for the activator and inhibitor, respectively, Rr = [u, v) and R. (u, v) are the activator and inhibitor reaction function, respectively. The domain of the simulation includes x E 1), (-1,,))) (0, 5]. The reaction functions for the activator and inhibitor are defined by:
Ra u) u v,
(u v)
where & 10 and the diffusion coefficients for the activator and inhibitor are D,, 1 10-3 and D. = 5 x 10 10-3, respectively. The initial condition is generated as standard normal random noise u00, x. y) ~ (0, 1.0) for x E (-1,1) and y 1-)
Data generation. We use the dataset generated by PDEBench (Takamoto al. 2022). where discretized into N₂ = 512, Ny 512 and N, 501, as well as the downsampled version for the models training with N₂ 128, N2. 128, and N, 101, the spatial discretization is performed using the finite volume method, and the time integration is performed using the built-in fourth order Runge- Kutta method the scipy package.
Problem setup. We follow the problem setup reported in Hao al. (Hao al. 2024). We aim learn the solution operator that maps the activator and inhibitor fields from the previous time-steps the next time-step. The models are trained with 900 trajectories and tested on the remaining 100 trajectories. We report the resulting relative I,2 errors over rollout predictions.
is worth noting that for this specific task, we employ an initial learning rate of 5 10-4, different from the 10- used in other experiments.
Training. For training CVTT models, please refer to section F.
We directly report the results as presented in DPOT (Hao et al. 2024). For comprehen- sive details on the training procedures, hyper-parameters, please refer to their original paper.
Evaluation.
34

---

Published as a conference paper at ICLR 2025
7
T
T
T+4
7
0.25
Ref.
0.00
- 00.55
0.25
Pred.
0.00
-0.25
0.002
Abs.
0.001
Figure 18: Diffusion-reactn reacttion Benchmark. Representative CViT rollout prediction of the density field u, and point-wise eπrоr against the ground truth.
T+1
T-2
T T+3 3
T
T+5
T r+C 6
Ref. V
0.0
-0.2
0.0
Pred.
-0.2
0.00075
Abs. Error
0.00050
0.00025
Figure 19: Diffusion-reactin Benchmark. Representative CViT rollout prediction of the density field W. and point-wise error against the ground truth.
F.8 COMPUTATIONAL COST
All experiments performed on single Nvidia RTX A6000 GPU. Average training times varied between 5 hours and 60 hours, depending on the task, input resolution, model size, and patch size. The following table summarizes the training times for different models. We use brief notation to indicate the model size and the input patch size: for instance, CViT-L/16 refers to the "Large" variant with 16 16 input patch size.
35

---

Published as a conference paper at ICLR 2025
Table Training times (hours) on a single Nvidia RTX A6000 GPU for different models and patch sizes (8 x 8, 16 x 16, 32 x 32) on benchmarks of Shallow water equation and Navier-Stoker equation.
Method
Shallow Water
Navier-Stokes
CViT-S/8
15
CViT-B/8
25
14
CViT-L/8
57
28
CViT-L/16
17
10
CViT-L/32
9
6
Table 10: Comparison of parameter count and inference time. All measurements are performed on single NVIDIA A6000 GPU with batch size 32, averaged over 100 runs.
Model
Params
Inference time (ms)
CViT-L
93 M
79.09
ViT
87 M
45.49
FNO
67 M
34.96
UNet
124 M
44.25
36


Reviews:
[{"review_id": "PFqbFkzfTy", "reviewer": "Reviewer_1Did", "summary": "This paper proposes a continuous operator learning method using the vision transformer with a newly introduced coordinate embedding for the query value. The proposed method is based on the previous continuous operator learning method. The authors have provided analysis including comparisons with recent operator learning methods on learning different operator equations, ablation studies for different structure choices, discussions of the proposed method compared to recent arts, theoretical insights of the proposed query embeddings, etc.", "questions": "- I wondered how will the proposed method be applied for more general vision tasks, especially for 3d vision. I wondered what are the authors’ opinions on this? What are the difficulties, such as computations, that might be difficult to apply? \n\nPlease see the above section for detailed comments, too.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed coordinate embedding for the query is interesting, and is effective in the continuous ViT learning, compared to MLP and RFF. Also, it is simple and easy to control through the interpolation parameter $\\beta$.\n- Discussions and analysis in the paper are extensive and interesting. Overall, the paper is well-organized and easy to follow.\n- The proposed coordinate embedding could have the potential to apply to more general continuous learning beyond physical domains. Especially for the vision tasks, such as 3d geometries including point cloud representations, it is worth exploring.", "weaknesses": "- Why not do coordinate embedding for all query, key, value? Only doing coordinate embedding for query preferable in which way? Maybe some further analysis on this could be included to further highlight the effectiveness of the proposed coordinate embedding.\n- The first question brought to the second one, that in the paper (appendix), the authors discussed the Lipschitz constant for different embeddings from linear embedding to random Fourier features (RFF), and to the proposed coordinate embedding, which is interesting and worth discussing. The authors mentioned that the proposed coordinate embedding is easier to tune since it is mainly based on one parameter $\\beta$. However, for RFF, there is also only one parameter to control the spectral bias, and thus control the Lipschitz constant. I wondered if the authors could also provide an analysis on only using RFF for the proposed framework. \n- Figures 3, 5, and 6 lack notations to distinguish columns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ooWSS2r0In", "reviewer": "Reviewer_TxkX", "summary": "This paper introduces a new neural operator architecture that tackles recent computer vision techniques (vision transformers, cross-attention). In particular, the CViT model -- for Continuous Vision Transformer for operator learning -- leverages the perceiver architecture to retrieve the representation of a query at a given continuous coordinate from spatiotemporal data. The relevance of the representations learned by CViT is evaluated on a diverse range of PDE such as fluid dynamics, climate modeling, and reaction-diffusion processes.", "questions": "* \"2+1 dimensional spatio-temporal data tensor\" but it is then written that $\\boldsymbol u \\in \\mathbb R^{T\\times H \\times W \\times D} *i.e.* four coordinates. \n* How separating the positional encoding (PE) between 1D temporal and 2D spatial is different than having one spatio-temporal (thus 3D) PE?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written, easy to read, and technically sound. The proposed approach appears to scale more efficiently than the current baseline thanks to the perceiver-inspired architecture. The results seem also promising.", "weaknesses": "* No clear structure of the related work section. It would help the reader to add more justifications on why transformer approaches struggle with various resolutions. Even if I tend to agree with the conclusion, it is also not clear, why a novel architecture design is required to solve the limitations related to high dimensional data and long-range dependencies.  \n* The transformer architecture in Fig. 1. does not give any insight and is well known. IMO, Authors could either remove it or replace it with the perceiver one. \n* It is unclear how many visual latent queries are used. From the figure and text, it seems that there is only one query which is then duplicated. In the perceiver architectures, these latent queries are learned, does this mean that only one query is learned? The stability of the model with respect to the number of queries could be presented in the ablations.  \n* The ablation with respect to parameter $\\beta$ is difficult to understand as, in Fig. 4, $\\epsilon$ is not introduced. Is $\\epsilon = \\beta$? The value of $\\beta$ seems extremely high ($10^5$) and should be compared to the input's resolution. Here, does eq. (1) reduces to a naive averaging with $\\beta=10^5$? \n* Results on standard benchmarks would help judge the method's efficiency. Here, it seems that the authors show results only on selected equations from PDEBench or PDEArena.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WQPbU9R4XT", "reviewer": "Reviewer_6yDe", "summary": "This paper presents a variant of Vision transformers which allows handling of continuous outputs. The model works by first encoding an input image sequence into a set of latent vectors using cross attention (a la Perceiver style encoding). The latents compress the input over time. The resulting latents are fed through several transformer layers. The readout is the interesting part - a learnable grid of spatial queries is interpolated using Nadarya-Watson interpolation to produce queries which are used to cross attend into the output of the vision transformer. The resulting attention weighted value is used as the outputs - this is done for every output coordinate.\nThe method is shown to work on several interesting physics oriented datasets with several ablation experiments. More analysis is shown in the appendix.", "questions": "My main point of inclarity is the initialization of latent queries in the encoder. The paper states they are initialized with a unit Gaussian - is this a learned initialization, or are the queries sampled every time the model is learned? The latter would mean, for example, that we would expect to see slightly different outputs every time the model is run - is that the case? (I am not saying this is a bad thing btw, just want to understand). Also - how many query vectors are used? the paper mentions it's more than one, but the number is not supplied in the paper.\n\nRelated to the above - I am not sure I understood why the query vector z is tiled across space? if it's the same vector across all of space then the output of the query would be the same for every spatial location, am I wrong? or is there more positional encodings added there? It would be good to clarify this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found this paper very interesting.\n\nOriginality:\nI think this is a very good variation of Vision Transformers which significantly expands their capabilities. I found the interpolated readout mechanism of particular novelty.\n\nQuality:\nThis is a well executed paper. There is ample experimental validation using several different datasets. The baselines chosen (as far as I can tell) are diverse and strong. There is a good set of ablation experiments and deep analysis of the method in the appendix.\n\nClarity:\nThe paper is well structures, well written and easy to read. All the nomenclature is well defined and I had no problem following the paper.\n\nSignificance:\nWhile I'm not an expert in the field to me this seems like a significant contribution to the field which greatly expands the expressive power of vision transformers.", "weaknesses": "All in all I think this is a strong paper, however:\n\n* Other domains? I would have loved to see if the resulting method is applicable to other domain or training setups beyond L2 prediction of physical systems. Does it work, for example, on natural video? Would it be useful as a diffusion model back-bone?\n\n* More analysis of cross attention - one of the more interesting parts is the use of cross attention, both in the encoding and decoding. It would be nice to see visualizations of cross attention patterns emerging in both directions. Do specific latents take on specific roles when decoding? are they responsible over different parts of space? time? both?\n\nWhile I do not expect these to be addressed in the rebuttal I really do think these would maker the paper stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5eKRXAE9aO", "reviewer": "Reviewer_Etk6", "summary": "This paper presents a neural operator architecture inspired by computer vision to address spatio-temporal scientific problems. The proposed model is structured with a transformer encoder featuring spatio-temporal embedding and time-aggregation layers, a grid-based positional embedding, and a cross-attention decoder. The author employs multiple benchmark datasets and model architectures for comparative experiments.", "questions": "- Does the grid resolution similarly impact computational efficiency as the patch size?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The author conducts extensive experiments, comparing several recent model architectures to demonstrate performance outcomes. The results are presented with additional clarity by including the number of parameters.\n\n- The detailed discussion of limitations and future directions is appreciated. Notably, the computational inefficiency for high-resolution data should be addressed in future work.", "weaknesses": "- Since this method draws on the vision transformer, have comparisons been considered with other vision-based extensions for spatio-temporal applications, such as ViViT[1] and VMAE[2]?\n\n- Regarding the datasets selected for experimentation, incorporating more complex, real-world cases could further substantiate performance, as some baseline architectures still yield results similar to those of the proposed model, despite the model size.\n\n[1] Arnab, Anurag, et al. \"Vivit: A video vision transformer.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[2] Tong, Zhan, et al. \"Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.\" Advances in neural information processing systems 35 (2022): 10078-10093.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xem7Jmcle7", "reviewer": "Reviewer_TXUj", "summary": "This paper explores the architectural design of transformers for operator learning methods. To efficiently implement continuous vision transformer architecture, they developed (or adopted) two techniques: perceiver resampler and coordinate embedding. The perceiver modules effectively aggregate temporal dimension, and the coordinate embedding enables the model to handle continuous input domains and improve accuracy. The proposed architecture achieved state-of-the-art results on various challenging PDE benchmarks while maintaining higher parameter efficiency.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It achieved state-of-the-art performance on various benchmark datasets with the network's great parameter efficiency.\n2. Coordinate embedding is an interesting approach to injecting the coordinate information into the network.\n3. The theoretical result related to coordinate embedding is quite informative and useful for the community.", "weaknesses": "1. The proposed grid-based positional encoding may not be a desirable solution for high-dimensional PDEs. As the authors said, it requires high-resolution grids to achieve competitive performance, and this approach would not scale up to more than 3-d time-dependent PDEs, e.g., (3+1) NS equation. \n\n2. The proposed Nadaraya-Watson interpolant would not scale with high-resolution grids since it has to look at all grid features. I am curious the inference cost of the proposed approach according to grid resolutions. Is the inference speed as fast as the previous methods?\nThe grid resolution 128 (N_x) x 128 (N_y) x 512 (feature dim) is already > 9M, and CViT-S model size is 13M. Does it mean that most of the parameters are used for grids?\n\n3. Does this scale well for high-resolution solutions? The authors mentioned that the grid resolution should be similar to the solution resolution. Then, 512 resolution means 512^3, which is roughly 134M. Could you show me how the proposed method can make good predictions for high-resolution solutions? e.g., testing 128 grid resolution model for predicting 512 solution resolution.\n\n4. This is a rather suggestion than a question. Why not coordinate embedding for temporal dimension? currently, y \\in R^2, but why not y \\in R^3? Temporal resolution can be coarse in this case to maintain parameter efficiency.\n\n5. How about other datasets? e.g., other PDEs in PDEBench and PDEArena?\n\n[minor comments]\n1. Numbering for all equations would be desirable during the review process.\n2. How do you determine \\beta = 10^5?\n3. L171: is u_f a flattened version of what? u_pe? not mentioned in the paper.\n4. L174: ‘tilling the latent query’, I am not sure “tiling” is a widely accepted term, I think it is a pytorch’s terminology.\n5. Eq (1): why not just using linear interpolation?\n6. Figure 3, 5, 6: what do different columns mean? Time evolution?\n7. Figure 4: (d) \\epsilon -> \\beta?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)