Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulators
Authors: Payel Mukhopadhyay
Michael McCabe
Ruben Ohana
Miles Cranmer
Venue: ICLR 2026 Poster (2026)
Keywords: autoregressive models
video modeling
compute-adaptive inference
tokenization based artifacts
physics based PDEs
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Overtone enables inference time compute adaptivity for PDE surrogate models, without losing accuracy. It enables novel inference time cyclic rollout strategies suppressing tokenization artifacts.

Abstract: Transformer-based PDE surrogates achieve remarkable performance but face two key challenges: fixed patch sizes cause systematic error accumulation at harmonic frequencies, and computational costs remain inflexible regardless of problem complexity or available resources. We introduce Overtone, a unified solution through dynamic patch size control at inference. Overtone's key insight is that cyclically modulating patch sizes during autoregressive rollouts distributes errors across the frequency spectrum, mitigating the systematic harmonic artifact accumulation that plague fixed-patch models. We implement this through two architecture-agnostic modules—CSM (Convolutional Stride Modulation, using dynamic stride modulation) and CKM (Convolutional Kernel Modulation, using dynamic kernel resizing)—that together provide both harmonic mitigation and compute-adaptive deployment. This flexible tokenization lets users trade accuracy for speed dynamically based on computational constraints, and the cyclic rollout strategy yields up to 40% lower long rollout error in variance-normalised RMSE (VRMSE) compared to conventional, static-patch surrogates. Across challenging 2D and 3D PDE benchmarks, one Overtone model matches or exceeds fixed-patch baselines across inference compute budgets, when trained under a fixed total training budget setting.

Published as a conference paper at ICLR 2026
OVERTONE: CYCLIC PATCH MODULATION FOR CLEAN, EFFICIENT, AND FLEXIBLE PHYSICS EMULATORS
Payel Mukhopadhyay 1.3, Michael McCabe Ruben Ohana Miles Cranmer Unniversity Cambridge, UK FFlatiron Institute, New York, USA Pollyathicc AI
ABSTRACT
Transformer- bassed PDE surrogates achieve remarkable performance but face two key challenges: fixed patch sizes cause systematic error accumulation at harmonic frequencies, and computational costs remain inflexible regardless of problem com- plexity or available resources. We introduce Overtone, a unified solution through dynamic patch size control at inference. Overtone's ' key insight is that cyclically modulating patch sizes during autoregressive rollouts distributes errors across the frequency spectrum, mitigating the systematic harmonic artifact accumulation that plague iked-patch models. We implement this through two architecture-agnee aggnostic modules CSM (using dynamic stride modulation) and CKM (using dynamic kernel resizing)-thhe together provide both harmonic mitigation and compute- adaptive deployment. This flexible tokenization lets users trade accuracy for speed dynamically based on computational constraints, and the cyclic rollout strategy yields up to 40% lower long rollout error in aariance-normalle compared to conventional, static-patch surrogates. Across challenging 2D and 3D PDE benchmarks, one Overtone model matches or exceeds exee-patch base- lines across inference compute budgets, when trained under fixed total training budget setting. Code is made available at: hutps://eituub..om/aa///////// I55ppaa moodulaoor
RMSE (VRMSE)
INTRODUCTION
Numerical solvers have long been the gold standard for simulating spatio-temporal dynamics in systems governed by partial differential equations (PDEs). Numerical solvers offer convergence guarantees, exploit problem sparsity, and provide fine-grained control of the accuracy-compute trade-off through configurable resolution (Morton & Mayers 2005 Malalasekera 2007), driven by
both scientific necessity and computational availability (Berger & Colella 1989 Wedi 2014).
Deep learning has established itself as powerful framework for training surrogate models of physical simulations (Cao) 2021 Li et al. 2023 McCabe al. 2023 Herde et al., 2024). Though expensive
to train, relative cheap inference costs result in overall compute savings, which has led deep learning surrogates become popular in forecasting (Bi al. 2023). PDE-constrained optimization (Li et al.) 2022), and parameter inference (Cranmer et al. 2020) Lemos et al. (023). Recent approaches are often derived from computer vision methods such as Vision Transtormers (ViT (Dosovitskiy et al.. 2020)), which segment discretized fields into patches and then tokens. Non-overlapping patches are used as a means to reduce token count and the associated quadratic attention cost.
However, these surrogates face two major limitations. First, we discover that in autoregressive rollouts, fixed patch sizes cause systematic eтror accumulation at harmonic frequencies through temporal coherence. While patch artifacts themselves well-known, we show that repeatedly hitting the same harmonics at every timestep creates a distinct accumulation pattern. When patches of size are used consistently during autoregressive rollouts, the artificial boundaries inject errors wavenumbers n//k where n is an integer. These errors constructively interfere over time, creating pronounced spectral spikes (Figure 2 and visible grid-like artifacts. Second, fixed-patch models
"Corresponding Email: pm858 cam ac uk, paye wayeemukkoopaeeeoo mukkhopadyyay 800gma com

---

Published as a conference paper at ICLR 2026
have inflexible computational cost. In physical modeling, different applications have hard resolution thresholds for resolving features like shocks or wave-fronts (Boyd 2001), yet smaller patches that
improve accuracy (Dosovitskiy et al. 2020 Wang et al. 2025a) cannot be selected post-training.
t+1
t+2
t+3
base 16x patch
cycle repeats throughout rollout
Convolutional
loe4
koe8
k-16
Kerna
Convolutional
Stride Modulator
K-16 5m4
k- s
k=16 5-16
Figure 1: Illustration of Overtone's CSM (stride modulation) and CKM (kernel modulation). Cyclic modulation distributes errors across frequencies, preventing error accumulation at harmonics and enabling securacy-copput trade-offs at inference time.
We address both challenges through dynamic patch size control during rollout- so far unexplored in autoregressive modeling. Cyclically alternating patch sizes between ki, k2, and ka at consecutive timesteps can mitigate artifacts from accumulating coherently at single harmonics, instead distributing them across the frequency spectrum. This same capability enables compute-adeee adaptive deployment without retraining, allowing users to balance accuracy and computational demands based on available resources.
Overtone implements this through two modules: Convolutional Stride Modulation (CSM) maintains static kernel but dynamically modulates stride, while Convolutional Kernel Modulation (CKM) uses
bicubic interpolation to resize single kernel (as in (Xu et al.) [014] Beyer et al., (023)) for both encoder and decoder, all while cyclically modulating these during rollout (Figure We empirically demonstrate that this cyclic modulation fundamentally alters artifact accumulation in autoregressive rollouts, also with motivation of the technique spectral analysis (Figure 2}. to show how temporal
coherence in tokenization has been unaddressed source of error in spatiotemporal models.
0.060
Restdhal power spectrrum
Fixed
ZordFutt MENZ
CSM
0.065
0.060
0.030
CIN DOSSE
0.025
0.029
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Spatial frequency
Figure 2: Left: Averaged ID residual power spectrum at rollout step 20 revealing harmonic error accumulation fixed-patch models. The fixed patch model shows pronounced spikes harmonic frequencies k/16, while CSM and CKM with cyclic modulation distribute spectral errors across the frequency range, supporting our theoretical motivation (Section 2). Note that this is residual spectrum the ground truth, which is trivially zero for the simulation, and is therefore not shown. Right: Spatial manifestation of harmonic artifacts at rollout step 20 on Turbulent Radiative Layer 2D. Cyclic patch modulation eliminates the grid-like distortions in the fixed model, showing the practical impact of Overtone's 's rollout strategy.
2

---

Published as a conference paper at ICLR 2026
Contributions. We present Overtone, framework that identifies and addresses fundamental issue in patch-based surrogates while providing practical compute flexibility:
Harmonic Artifact Diagnosis: We identify that fixed-patch tokenization causes systematic spectral error accumulation at harmonic frequencies k/p, manifesting as spectral spikes and spatial artifacts that degrade long-term predictions autoregressive surrogates.
2. Cyclic Rollout Strategy: We introduce eyclic stride/patch modulation during inference for autore- gressive rollouts. While adaptive patching exists in vision, we are not aware of prior work that cyclically modulates patches during autoregressive rollouts computer vision, video modeling, or PDE applications. The rollout cycles tokenization scale (c.g., 4 + 8 + 16, repeated), which (i) prevents errors frequency space errors from repeatedly reinforcing at a single patch scale- -ttereyyy reducing patch-lattice harmonic artifacts and (ii) periodically leverages finer tokenization (4 and 8) for higher-fideliee steps while retaining the efficiency of coarser patches (16). Together, these effects reduce VRMSE by up to 40% relative to fixed-patch baselines (patch size 16), without retraining.
3. Architecture Agnostic Controllable Tokenization Modules: We develop CSM (using dynamic stride adjustment) and CKM (using kernel interpolation (Xu et al. 2014 Beyer et al. 2023) auchitecture-agnoste methods for controllable tokenization. These modules operate at the tokenization level and are compatible with backbones like vanilla ViT Dosovitskiy et al. (2020).
axial ViT Ho al. (2020), CViT Wang et al. (2025b), making them flexible plug-ins for PDE surrogates.
4. Compute-Adaptive Deployment: Beyond strong accuracy gains, Overtone provides practical trade- offs between speed and precision at inference. A single model can adjust patch sizes based on available resources, matching or exceeding multiple fixed-patch baselines across all compute budgets on challenging 2D/3D PDE benchmarks from the Well (Ohana al., 2024). Overtone is also competitive with a range of SOTA non-patch baselines, underscoring its broader significance.
We focus on autoregressive surrogate models for time-evolving PDE systems, where the goal is to predict the next state of a discretized physical field from short temporal context.
1.1 BACKGROUND AND RELATED WORK
Architecture-agnosti tokenization for PDE surrogates. CSM and CKM are architecture-agnosii tokenization strategies applied at the encoder and decoder stages. They control the patch/stride configuration for spatial downsampling while remaining independent of the transformer architecture. We show compatibility with standard vanilla ViTs (full attention) (Dosovitskiy et al.) (020) and axial ViT' (Ho et al. 2019). This modularity makes CSM and CKM flexible plug-ins for diverse transformer-basee based PDE surrogates, allowing adaptive deployment without redesign.
Surrogate modeling of PDEs with deep learning.
PINNs (Raissi et al. 2019 Cai et al. 2021 Han
cet al. 2018. Hao et al. 2023a Karniadakis et al. 2022 Lu et al.] 2021 Pang et al. 2019
Sun et al.
2020 Zhu et al. (222)) embed physics in loss functions, while CNN- and GNN- based surrogates
operate on structured or unstructured meshes, supporting spatiotemporal tasks such as climate and
fluid modeling (Brandstetter al. 2022c; Gupta & Brandstetter 2022 Janny al. 2023 Li &
Farimani, [022]
2002| Lötzsch et al.
[022
[0022]PParl Pfaff et al. 2021 Prantl et al. 2022 Sanchez-Gonzzleez et al.
2020; Stachenfeld al. 2022
Thuerey al. 2020 Ummenhofer al.
2020 Wang et al. 2020
Lam et al.] 2022 Nguyen et al.
Pathak Pathak et al. (022). Neural operators (Li et al. [200cc]
ct al. 2019 2022 Ovadia et al.
2023] Brandstetter al. 2022a
Cao 2021 Gupta et al. [202] Hao
et al. 2023b Jin et al. 2022 Kissas et 2022; Kovachki et al. 2021 aim for resolution-invarian
learning, with FNOs (Li al.
2020a) using Fourier transforms for global context.
VIT' and compute adaptive models for PDE surrogate modeling. Initially developed for image recognition tasks, ViTs (Dosovitskiy et al., 2020) have seen increasing adoption in the PDE surrogate modeling literature (Cao) 2021; Li et al. 2023 Liu et al. 2023). Despite their strengths, traditional ViTs rely on fixed patch sizes, which strongly limits their flexibility and adaptability when modeling PDE-based systems discussed in the introduction. Examples of kernel interpolation for adaptive sizing include SiCNN (Xu et al. 2014), which applied bicubic interpolation to CNN kernels and trained multiple scales, and FlexiViT (Beyer al. 2023), which applies bicubic interpolation to
3

---

Published as a conference paper at ICLR 2026
VIT kernels for image classification. Although explored in computer vision, adaptive patching for PDE surrogate modeling has only been explored in (Zhang et al. 2024a), the difference with our work being that their patchification is data-driven, while ours can manually controlled based on compute requirements.
METHODS
Overview. CSM and CKM operate as encoder-decoder modules within a VIT based architecture for autoregressive PDE prediction. The transformer applies sequence of temporal attention, spatial attention, and MLP blocks (Appendix CC.2). Crucially, Overtone's ' framework is architecture- agnostic test it with axial attention and vanilla ViTs (and also recent model- Coneil-Cn ViT (Wang et al. 2025b) in Appendix E).
Utility of flexible models. As illustrated in Figure increasing the number of tokens-equivallee to using smaller patch sizes in models with fixed patches- enhance-e the accuracy of autoregressive prediction tasks, but at a higher computational cost. The cost for models trained with fixed patch sizes cannot be controlled at inference time. This limits the utility of these models budget- aware production settings and motivate the exploration of flexible models for PDE modeling with ViT-inspired architectures.
Research in the broader machine learning community has extensively demonstrated the value of scale (Zhai et al. 2022 Hoffmann et al. 2022). As transformer-based surrogate models similarly scale in parameter count, ensuring inference-tints time scalability crucial, particularly to ensure these models are able to be used in
environments. To avoid any confusion, by inference scalability, we mean patching/striding scalability or tunability, and not scalability in mesh resolution. Training and maintaining multiple fixed-patch models is burden especially as large pretrained or foundation models are being built (McCabe et al.) 2023] Herde et al.] 2024] Morel et al., (025). In such scenario, if Overtone's 's flexible strategies can enable training of single model without retraining for multiple patch resolutions without any loss accuracy, these strategies can form the building blocks for larger scale foundation models for PDEs.
Patching. Given an input image with height И and width W, ViTs will divide the image into non-overiapping patches of size k x k, producing N = H/k] [W/k] tokens. For global attention, transformer cost scales as O(N²), and for axial attention, as O(N /N) (Ho et al. 2019). In convolutional tokenization, patches are extracted via kernels size A: and stride s, resulting in token count N N₁, Nu where N₁, [(H- /s] + 1 and N = L(W - k)/s + 1. While most ViT based architectures use = s, our CSM and CKM modules decouple these parameters to ficxibly control token count inference.
Convolutional Stride Modulator (CSM), CSM provides flexible downsampling by modulating the stride s E {4, 16) at each forward pass. Consider imput E X for some input space C M xTxC with M, T and C being the batch size, time context and input channels respectively. We pad the input with learned tokens (informed by boundary conditions) to avoid edge artifacts. During training, the model is trained with random stride sizes sampled from 4, 8, 16 during the forward pass (Appendix K.5 At inference, starting from input context f = (x ,..,, x²') CSM uses fixed kernel base and computes:
y = Conv (i', w hase)
ii Processor(u"me),
#+ Conv (#fa hass), stide
strde #t
where we cycle 8; (4, 8, 16) mod 3)41 and slide -1--11 26.T-
Here, Processor denotes a transformer-basee processor that operates on the encoded tokens, Cony indicates transposed convolution, and the output a X R Bx #xxxxxxxx is the next-step prediction. CSM is compatible with both single-stage and multi-stage encoder-decoder modules. Implementation details and code provided in the supplementary materials.
4

---

Published as a conference paper at ICLR 2026
Convolutional Kernel Modulator (CKM). CKM introduces dynamic patch size selection within convolutional encoder and decoder blocks, enabling models to select patch size E {4. 8, 16) for each forward pass pass-powerrs powers-of-foeee of-two that reflect standard discretizations in PDE data. Unlike most ViT-based surrogates that fix patch size at 16 (McCabe et al. 2023) Morel et al. 2025). CKM provides flexible tokenization essential for harmonic error mitigation (analyzed in Section This is achieved using kernel interpolation (Xu et al. 2014 Beyer et al. 2023).
At training time, the model sees random kernel size sampled from 4, 8, 16 (Appendix K. At inference, starting from 3" = (x',..,, x,,,,"),,, CKM resizes a base convolutional kernel w base TR. Rluse X Rass xCxC' using Pl-resize (detailed in Appendix K and computes:
2
Cony (i', To m stride ha
Diat Processor(cenc): T
Conv T (I B Srww"" )
enc
stride
where we cycle k; = (4, 8, 16) mod 3 and slide 11:1-11 22.77
Here, BR, E X is a bicubic interpolation matrix and B its pseudoinverse transpose.
Examples of kernel interpolation for adaptive sizing include SiCNN (Xu al. 2014), which applied bicubic interpolation to CNN kernels and trained at multiple scales, and FlexiViT (Beyer et al. 2023), which applies bicubic interpolation to VIT kernels for image classification. Using kernel interpolation, we introduce cyclic modulation during vollouts-transfomeett transforming it from a compute-flexibiliy tool into a method for mitigating harmonic eπror accumulation in honghooizonn predictions. In our vanilla VIT and Axial ViT models, we adopt two-stage convolutional encoders and decoders following the hMLP architecture (Touvron et al.] 2022), which has recently been adopted for large-scale PDE surrogates (Morel et al. 2025 McCabe et al. 2023), and apply CKM independently each stage.
Together, the Overtone framework offers general strategies for flexifying convolutional tokenization in ViT-based autoregressive schemes, as well mitigating accumulating patch artifacts.
Cyclic rollout strategy. In standard autoregressive forecasting, patch sizes remain fixed across the rollout. By contrast, Overtone's flexible modules enable patch sizes or strides alternate across timesteps, introducing new temporal patterns tokenization. This test-time capability, decoupled from training, leads striking empirical effect: alternating rollouts consistently suppress patch artifacts periodic errors that appear as harmonics in spectral residuals (Figure ()
These artifacts are ubiquitous in fixed-pe fixed-patch VIT based models, regardless of attention type (vanilla, axial, Swin), and cannot be removed through training alone (Appendix [F). Alternating rollouts yield cleaner, more stable long-horizon predictions. Our findings primarily empirical, though provide mathematical motivation for this phenomenon below. This discovery was only possible due the test-time flexibility of Overtone-a capability entirely absent from prior PDE surrogates.
Why eycling mitigates artifacts. Under the simplifying assumption of lincarized error model, the evolution can be written as en+1 (w) = e,)e,, (w) + a. (w), where propagates existing errors and a, injects fresh errors patch boundaries. With a fixed patch size these injections align harmonic frequencies m/k, remaining phase-locked across timesteps and leading to rapidly compounding error eirolevisible as spectral spikes Figure By contrast, varying across timesteps disrupts this temporal coherence, spreading errors more evenly and mitigating growth. This should be understood as heuristic explanation consistent with our empirical findings. See Appendix for more extended mathematical sketch and physical intuition.
3 RESULTS
To evaluate the practical advantages of compute- adaptive inference and validate the harmonic mitiga- tion benefits, we conduct experiments on 2D and 3D datasets from The Well (Ohana al.) 2024). Our evaluation shows how a single flexible model can serve multiple deployment scenarios while also achieving superior accuracy through harmonic emor distribution.
5

---

Published as a conference paper at ICLR 2026
Experimental setup. We compare training single flexible model versus multiple fixed-patch models. We train three fixed-patch models (patch sizes 4, 8, 16) for a fixed number of optimizer steps each, while training CSM and CKM models once with the same total compute budget. See Appendix C for details.
This setup reflects practical question for enabling adaptive compute: given fixed training budget, it better to train one flexible model or several static ones? This corresponds to deployment settings where inference budgets may vary (or are unknown a priori) and single model must support multiple compute points; when only single tokenization is required, training a specialized fixed-patch model remains a natural choice. All evaluations are performed on models trained under this constraint. After training. we compare the performance of CSM and CKM models against individual models trained at static patch sizes. If the compute-elastic models can enable flexibility without incurring significant accuracy loss, this supports deploying single flexible PDE surrogate across multiple inference-time compute settings without training and maintaining separate fixed-patch models.
We test both axial ViT (Ho al., 2019) and vanilla VIT (Dosvviskkiy et al., 2020) architectures to measure generality. The input consists of six time frames, with loss optimized for next-step prediction. We use this teacher forced training setup throughout, which matches The Well setup and the majority of PDE surrogates we compare against. This ensures a fair comparison across all baselines. Alterating training strategies, such rollout training, is complementary to work and can additionally help to improve prediction accuracy further for all models. Incorporating them alongside Overtone is an interesting direction for future work. See Appendix C. for dataset details and Appendix for ablations.
3.1
FLEXIBLE INFERENCE
Stear Flow
Turbukent Radiative
2D
Active Mutter
Rayleigh Bénard Convection
0.045
CSM
That VRMS
0.20
0.08
CKM
Fixed
0.03
0.035
0.16
0.02
0.025
0.13
3072
1024
1024
at inference
tokens inference
tolenss inference
at infereecce
Figure 3: Next-step prediction test VRMSEs v.s. number of tokens at inference of a 100M parameter model trained on four 2D datasets. Lower VRMSE is better. Token count is proxy for required compute. Note that there three separate fixed-patch models (green), as each needs be trained from scratch. This plot shows the compute-accuracy trade-off, which is also explored more in Section 3.
We choose the VRMSE metric (Appendix C.4) measure of accuracy. In Table I and Figure 3 we report VRMSE across a range of 2D and 3D PDE datasets from the Well (Ohana et al. Each model is evaluated at multiple token counts, which correspond to varying stride sizes in CSM patch sizes in CKM-revealls Importantly, we train the fixed patch size (p.s.) models as separate networks at each patch size, while we train CSM/CKM only once using random patch/stride schedules and evaluated without retraining. We show results for two different transformer architectures, Axial ViT (Ho et al. 2019) and vanilla full attention ViT (Dosovitskiy et al. 2020). Note that the goal is not to compare the two base architectures against each other, but to show that for both architectures, higher token count implies greater accuracy, and that CSM/CKM can provide token count tunability at inference against the corresponding fixed patch baselines without losing significant accuracy.
2024).
the impact of inference-time compression on predictive accuracy.
Across all datasets and token configurations, CSM and CKM consistently match or outperform their fixed-patch counterparts-desppe being single, flexible models. This holds for both axial and vanilla attention processors. Increasing the number of tokens at inference- using smaller strides (CSM) or patch sizes (iKM)-immrood VRMSE but increases compute (Table T Figure 3 For example, in the Active Matter dataset (2562), reducing patch/stride size from 16 to 4 increases token count from 256 to 4096, triples inference time (0.21s/step to 0.63s/step), and increases compute from to 170 SEFLOPs-ai reduces error by over 30%. Similarly, for 3DD datasets (643), decreasing patch

---

Published as a conference paper at ICLR 2026
size from 16 to 4 increases token count from 64 to 4096, increases inference time 8x (0.11s/step to 0.8s/step), but improves VRMSE by ~2x. Token count directly controls the compute- or outperforms multiple fixed models at every compute budget, without retraining. They excel across all frequency bands (BSNMSE analysis in Appendix D), preserving both coarse and fine features.
trade-off. A single CSM/CKM model matches
Vanilla VIT (100M)
Axial ViT (50M)
Well
Dataset
# Tokens
CSM
CKM
f.p.s.
CSM
CKM
f.p.s.
Baseline
2048
0.00546
0.00549
0.00677
0.0077
0.0070
0.0085
0.1049
Shear Flow
512
0.0088
0.0086
0.0096
0.0104
0.0098
0.0124
128
0.0112
0.0124
0.0140
0.0135
0.0143
0.0173
3072
0.146
0.133
0.143
0.153
0.145
0.153
0.2269
Turbulent Radiative
768
0.179
0.154
0.173
0.179
0.165
0.186
Layer 2D
192
0,204
0.200
0,225
0.211
0.209
0.238
4096
0.0171
0,0192
0,0213
0.0238
0.0244
0.0276
0.0330
Active Matter
1024
0.0214
0.0221
0.0245
0.0282
0.0260
0.0340
256
0.0294
0.0344
0.0315
0.0368
0.0369
0.0435
4096
0.0248
0.0250
0.0310
0.0296
0.0291
0.0361
0.2240
Rayleigh-Bénard
1024
0,0349
0.0328
0,0362
0,0406
0.0388
0.0443
256
0.0459
0.0467
0.0469
0.0537
0.0548
0.0582
4096
0.287
0.267
0.272
0.305
0.289
0.288
0.3063
Supernova
512
0.380
0.362
0.387
0.388
0.371
0.392
Explosion
64
0.417
0.414
0.426
0.421
0.419
0.434
4096
0.121
0.103
0.118
0.138
0.124
0.127
0.2096
Turbulence Gravity
512
0.183
0.164
0.179
0.197
0.181
0.194
Cooling
64
0.199
0.203
0.199
0.215
0.216
0.212
Table 1:
Test VRMSE across multiple 2D and 3D PDE datasets for next-step prediction using CSM, CKM, and fixed patch size (f.p.s.) models. We train CSM/CKM models once and evaluate them across multiple token counts (i.e., patch/stride configurations), whereas we train each fixed p.s. model separately. Results are shown for two transformer backbones: Axial VIT (50M) (Ho et al.) 2019) and Vanilla ViT (100M) Dosovitskiy et al. (2020). We also report the best outcomes from the Well benchmark suite (Ohnaa et al. 2024), although after retraining the Well baselines for accurate results.
Table includes Well benchmarks: FNO Li al. 2021). TENO (Kossaifi et al. 2023). U-Net, and CNext-U-Net (Liu et al.) 2022) (Ohana et al. 2024). re-trained with tuned rates for best accuracy. Our flexible models outperform these baselines across most tasks and budgets. We further demonstrate the modularity of our approach by integrating CKM with CViT (Wang et al., (2025b), a recent hybrid architecture. Results show flexified CViT consistently outperforms fixed-patch variants across four 2D PDE datasets. Full details are in Appendix
3.2 CSM & CKM ENABLE NEW INFERENCE- STRIDE/PATCH SCHEDULES
A key advantage of compute-elastic stride/patch tokenization is that it enables inference-time rollout schedules that are inaccessible to fixed-tokenization patch-based surrogates. standard patch-based models, inference repeatedly applies identical patch at every autoregressive step. In contrast, campute-elastic models allow the stride and patch size to vary dynamically across rollout steps, without retraining. To our knowledge, PDE surrogates rarely treat inference time rollouts controllable knob. Here, compute-elastic tokenization turns rollout inference into a controllable scheduling problem, enabling budget- and horizon-awee aware rollouts without retraining.
Using a single trained compute-elastic model, we vary the inference- time stride/patch schedule during autoregressive prediction. We take the fixed 1 model (trained with the setup described in 3)
7

---

Published as a conference paper at ICLR 2026
natural fixe-tokeeiiaate reference point, and compare it against the CSM/CKM alternating schedule that cycles stride and patch sizes over time at inference. Quantitatively, Table2]shows rollout VRMSE averaged over 10 steps. Across both processor backbones, flexible models consistently outperform the fixed patch models-often by large margins margins-demoneee demonstrating greater rollout stability.
Axial ViT (50M)
Vanilla ViT (100M)
Dataset
CSM
CKM
f.p.s.
CSM
CKM
f.p.s.
Shear Flow
0.0565
0.0579
0.0793
+28.7%
0.0375
0.0385
0.0504
+25.6%
TRL-2D
0.475
0.570
0.571
+ +16.8%
0.373
0.409
0,446
+16.4%
Active Matter
0.384
0.390
0.640
+40.0%
0.359
0.351
0,370
+5.1%
Rayleigh-Bénan Béngrd-
0.166
0.2159
0.217
+23.5%
0.140
0.215
0.2273
+38.4%
Supernova 1.310
1.270
1.905
+33.3%
1.20
1.14
1.75
+34.9%
TGC
0.667
0.680
0.860
+22.4%
0.559
0.527
0.77
+31.6%
Table 2: 10-step averaged rollout VRMSE across datasets. CSM/CKM vs. fixed patch size (f.p.s)
models. A shows percentage improvement of best flexible model f.p.s. Abbreviations: TRL-2D: Turbulent Radiative Layer 2D, TGC: Turbulence Gravity Cooling, Supernova: Supernova Explosion.
Table compares representative inference-time stride schedules that the same set of strides {4, 8, 16} but differ in their ordering over the rollout, and we find that the 4 - 8 + 16 ordering performs best among the tested permutations. All permutations use the same multiset of strides and therefore have the same total token count over the rollout (for the random case, its the same total token count on average throughout the rollout with uniform sampling); they differ only temporal ordering. These results show that beyond the overall compute budget, the timing of when higher-resolution tokenization is applied materially affects the rollout stability.
We also evaluate additional schedule families beyond permutations of the same cycle (Appendix A). These include (i) a two-step dwell periodic schedule that holds each stride/patch value for multiple consecutive steps, (ii) warm-up schedules that use smaller stride/patch values early in the rollout before switching to larger values, and (iii) schedules that use larger stride/patch values for most steps with occasional smaller- value steps. Across representative systems, some of these alternatives achieve rollout error close to the 4- 8 + 16 baseline, while others degrade substantially, reinforcing that schedule choice is novel inference-time control knob enabled by compute-elastie tokenization.
Dataset
4-88+16
116
16 16-4--8
Random
Shear Flow Supernova Explosion
0.0375
0.0442
(-17.8%
0.0437
(-16.5%)
0.0433 (- (-15.5%
1.204
1.290
(-7.1%)
1.325
(- -0.0%)
1.369 (-13.7%
Turbulence Graviry Cooling
0.5592
0.606
(-8.4%)
0.6101
(-9.1%)
0.6136 (-9.7%)
Table 3: Vanilla VIT (CSM): 10-step average rollout VRMSE for representative inference-time stride/patch schedules. Percentage values indicate relative degradation VRMSE compared the 4 4 + 16 baseline.
3.3 HARMONIC ARTIFACT MITIGATION
Our theoretical motivation (Section suggests cyclic patch modulation distributes errors across frequencies. preventing accumulation at harmonics. We validate this through alternating rollout: CSM and CKM models cycle through stride/patch sizes 4, 8, and 16 at each prediction step, avoiding the harmonic artifacts frequencies k k/p that plague all fixed-patch ViT variants (vanilla, axial, Swin) and arise from tokenization mechanics, not training (Appendix Fb
Figures 2 andd illustrate this behavior for representative rollouts from Turbulent Radiative Layer 2D, Active Matter and Shear Flow. In each case, fixed p=16 inference exhibits growing spatial artifacts at long horizons, while scheduled rollouts remain visually smooth and physically coherent. We identify that this effect is prominently reflected in the residual power spectra of each of the figures, where scheduled inference reduces error power at the patch-lattice frequencies. In particular,
8

---

Published as a conference paper at ICLR 2026
Ground Truth
Rood P16
[Residual power spectrumt: conc.
0.025
Feed P16
0.020
CKM traliout 4-+8 -166))
CSM
CKM
0.010
0.005
Frequency
Figure 4: Step 20 rollout for the concentration field of the Active Matter dataset. Left: ground truth, fixed p=16, CSM, and CKM. Right: corresponding ID averaged residual frequency spectrum.
under fixed-patch inference the residual spectra exhibit pronounced peaks patch-lattice harmonic wavenumbers proportional to k/p (for integer k), consistent with periodicity induced by the patch grid, and these peaks are substantially attenuated under cyclic rollout inference. Additionally, our models respect physics constraints over long rollouts, indicating they are not just memorizing patterns (See Appendix(H]iae shear flow long trajectory rollout and physics based validation). See Appendix for additional visualizations. These improvements emerge without additional training, achieved purely through inference-time patch modulation.
Ground Truth
Fixed
|Rediuaal power spectrum: tracer]
0.012
Fixed P16
0.010
CSM (rollout 4 4-+8-16) CKM (rollout 4-8-16)
0.004
CSM
0.002
0.1
0.2
0.3
0.4
Frequency
Figure 5: Step 40 rollout for the tracer field the Shear Flow dataset. Left: ground truth, fixed p=16, CSM, and CKM. Right: corresponding residual frequency spectrum.
3.4 COMPARISON TO NON-PATCHED BASELINES
Although our focus is not non non-patch architectures, Table and Table 11 provides reference comparison to state-of- the art CNN/Neural HWI//eeurl-OOperaorr surrogates. It shows that flexible models are competitive with SOTA PDE surrogate architectures that do not use paseh- tokenization (e.g..
TFNO (Kossaifi et al. 2023 FFNO (Tran et al. 2023). Transolver (Wu et al.) 2024). SineNet (Zhang al. 2024b)). Overtone's flexible models often significantly outperform these baselines. This result reinforces the potential of adaptive patch-based transformers to serve as competitive surrogates for
PDE modeling, without sacrificing accuracy or flexibility. We did this architectural comparison for two complex physical systems, Shear Flow and Turbulent Radiative Layer 2D from the Well (Ohana
al. 2024) (comparisons on two more datasets in Table
3.5
ABLATIONS
Ablations across model sizes (7M-100M) confirm: smaller patches improve accuracy, single flexible models outperform multiple fixed ones (within our total budget matched setting), and inference adapts without retraining (Appendix[L) L1 The approach remains robust to additional patch options (Appendix [.2)). temporal context (Appendix I.4). and underlying base patch size (Appendix (L3). Reducing patch diversity during training (e.g., omitting P 8) degrades performance the excluded

---

Published as a conference paper at ICLR 2026
Table 4: 10-step rollout performance comparison with non-patch based PDE surrogates on Shear Flow and Turbulent Radiative Layer 2D datasets, Our flexible models show competitive performance while offering compute adaptability.
Model-
TENO
FENO
SineNet
Transolver
CSM + VIT (Ours)
CKM + ViT (Ours)
# Parameters >
20M
6.2M
35M
11M
7M
100M
7M
100M
Shear Flow
0.89
0.105
0.17
1
0.076
0.0375
0.096
0.0385
Turbulent Radiative Layer 2D
0.81
0.485
0.65
>
1
0.458
0.373
0.477
0.409
size, underscoring the value of patch variety (Appendix L55 This ablation notes that we do not claim zero-shot generalization to unseen patch/stride configurations.
Limitations. Our approach requires training with multiple patch sizes to enable unference-time wlexibilit--we do claim zero-shot adaptation to unseen patch configurations. This is deliberate design choice rather than a limitation: training with diverse patch sizes ensures robustness across the operating range. Zero-shot patch adaptation remains an open problem for future work. Additionally, as is standard for models at the 50M-1000000 100M parameter scale, our experiments use single seeds due to computational constraints. This practice is consistent with recent large-scale PDE models (MPP, DISCO, CViT, Poseidon) which face similar resource limitations. The breadth of our experiments across multiple datasets and architectures helps compensate for the lack of error bars. Finally, the results are for regular grids, and extension to irregular geometries remains open for future work.
CONCLUSION
We presented Overtone, unified framework for compute- adaptive inference in transformer- basedd PDE surrogates that addresses both practical deployment challenges and fundamental accuracy limitations. Overtone provides dynamic patch size control inference time without retraining, allowing users to balance computational resources and accuracy requirements on demand. This flexibility solves a key problem production deployment where different applications have varying computational budgets and accuracy needs.
Beyond practical compute flexibility, we discovered that cyclic patch modulation fundamentally improves rollout stability by mitigating harmonic artifacts. Fixed patch sizes cause errors that interfere constructively at frequencies k. p.p manifesting as spectral spikes and grid artifacts. By alternating patch sizes during rollouts, we distribute these errors across the frequency spectrum, preventing coherent accumulation reduuci patch based artifacts compared to fixed baselines.
To implement these capabilities, we developed two architecture-agnosii modules for Overtone: CSM, which uses dynamic stride modulation, and CKM, which uses kernel interpolation for dynamic kernel control. Across challenging 2D and 3D benchmarks, single Overtone model matches or exceeds multiple fxxeh-patch baselines across all compute budgets. This dual benefit- benefit-practical deployment flexibility plus core accuracy improvement makes Overtone valuable for both research and production settings where computational efficiency and prediction quality are paramount.
Future directions. The harmonic error distribution idea extends beyond PDE surrogates to any au- toregressive, vision-based model. Future work could explore adaptive (rather than eyclic) modulation strategies, extend these techniques to patch-baser methods beyond transformers (Appendix []. and apply the same insights to video prediction and other spatiotemporal tasks. Additionally, our strategy could be applied to other VIT variants like Swin Liu et al. (2021) or CSWin Dong al. (2022).
Finally, CSM/CKM could be integrated into large foundation models for physical systems (MoCabe et al. 2023) Herde et al. 2024 Hao et al. 2024). In such settings, inference-time stride/patch control may enable flexible adaptation across downstream tasks that naturally occupy different points along the compute-accuracy spectrum. Recent work has, in fact demonstrated the effectiveness of this approach at scale by incorporating CSM into large multiphysics foundation model, Walrus, yielding strong performance across wide range of 2D and 3D systems (MeCabe et al., 2025). Together, these results position flexible tokenization as a key tool for improving both computational efficiency and long-horizon accuracy in autoregressive surrogates.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
Polymathic AI gratefully acknowledges funding from the Simons Foundation and Schmidt Sciences, LLC. Payel Mukhopadhyay thanks the infosy-CCambride AI centre for support. The compute for this project involved multiple sources, and the authors would like to thank the Scientific Computing Core, a division of the Flatiron Institute, a division of the Simons Foundation for extensive computational support. Miles Cranmer is grateful for support from the Schmidt Sciences AI2050 Early Career Fellowship and the Isaac Newton Trust. We thank Geraud Krawezik for help with compute resources, and the broader Polymathic AI team for valuable discussions and feedback.
REFERENCES
Matthew W. Abruzzo, Drummond B. Fielding, and Greg L. Bryan. Taming the TuRMoiL: The Temperature Dependence of Turbulence in Cloud Wind Interactions. The Astrophysical Journal, 966(2):181, May 2024. doi: 10.3347/1538-4355aaleeel
Marsha J Berger and Phillip Colella. Local adaptive mesh refinement for shock hydrodynamics. Journal of computational Physics, 82(1):64-88) 84, 1989.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Komblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen. Ibrahim Alabdulmohsin, and Filip Pavetic. FlexiVIT: One model for all patch sizes. 2022 IEEE/CVF Conference on Computer Vision and Pattern doi: 10. 1w.11099cYpp622292223...1111 URL http:ttts:://et pettpp::/doi..om org
Recognition (CVPR), 6 2023.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 619(7970):533-538 2023.
John P. Boyd. Chebyshev and Fourier Spectral Methods. Dover Publications, Mineola, NY, second edition, 2001. ISBN 9780486411835.
Johannes Brandstetter, Rianne van den Berg. Max Welling, and Jayesh K Gupta. Clifford neural layers for pde modeling. arXiv preprint 2tt.::222000499444 2022a.
Johannes Brandstetter, Max Welling, and Daniel E Worrall. Lie point symmetry data augmentation for neural PDE solvers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Pruceedings of Machine Learning Research, pp. 2241-2256. PMLR, 17- 23 Jul 2022b. URL https: http:::s/pproceedingg miir http:://oo.eeeaaaooocooeeeeeeaaa html
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. arXiv preprint arXiv:2202.03377, 2022c.
Keaton Burns, Geoffrey Vasil, Jeffrey Oishi, Daniel Lecoanet, and Benjamin P Brown. Dedalus: A flexible framework for numerical simulations with spectral methods. Physical Review Research, 2(2):023068, 2020.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (pinns) for fluid mechanics: review. Acta Mechanica Sinica, 37(12): 1727-1738, 2021.
Shuhao Cao. Choose a transformer: Fourier or galerkin. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang and Wortman Vaughan (eds.). Advances in Neural In- formation Processing Systems, volume 34, pp- 24924 24940. Curran Associates, Inc., httppppprrr.lleeppppe//222222220 Pappaaa
2021. URL httpthtttttccc /ttttppp//prooeeeiingssss neurips..... wewurip..ccppppps.....ee Ei e/
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation saauea inference. Proceedings of the National Academy of Sciences, 117(48):0055533006 2020.

---

Published as a conference paper at ICLR 2026
Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Drummond B. Fielding, Eve C. Ostriker, Greg L. Bryan, and Adam S. Jermyn. Multiphase Gas and the Fractal Nature of Radiative Turbulent Mixing Layers. The Astrophysical Journal Letters, 894 (2):L24, May 2020. doi: 10.3847/2041-88 0115772041-813/abbbca 12113//dc
Drummond B Fielding, Eve C Ostriker, Greg L Bryan, and Adam S Jermyn. Multiphase gas and the fractal nature of radiative turbulent mixing layers. The Astrophysical Journal Letters, 894(2):124, 2020.
Max Gronke and S. Peng Oh. The growth and entrainment of cold gas in a hot wind. Monthly Notices of the Royal Astronomical Society: Letters, 480(1):L111-L11, October 2018. doi: 10.1093//mnrs//ssyy133
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587 2021.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwaveelt-based operator learning for differential equations, 2021.
Jayesh K. Gupta and Johannes Brandstetter. Towards caati-spatiitemporalsschle generalized pde modeling, 2022.
Jiequn Han, Aruuff Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 1 115(34)885058851) 2018.
Carl J Hansen, Steven D Kawaler, and Virginia Trimble. Stellar interiors: physical principles, structure, and evolution. Springer Science & Business Media, 2012.
Zhongkai Hao, Songming Liu, Yichi Zhang, Chengyang Ying, Yao Feng, Hang Su, and Jun Zhu. Physics-informed machine learning: A survey on problems, methods and applications, 2023a.
Zhongkai Hao, Chengyang Ying, Zhengyi Wang, Hang Su, Yinpeng Dong, Songming Liu, Ze Cheng, Jun Zhu, and Jian Song. Gnot: A general neural operator transformer for operator learning. arXiv preprint 1332 2023b.
Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandku- mar, Jian Song, and Jun Zhu. DPOT: Aeto-regressive denoising operator transformer for large-scale PDE pre-training. arXiv preprint S0.iv:22003035444 2024.
Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient Foundation Models for PDEs. arXiv preprint arXiv:2405. /9101, 2024.
Keiya Hirashima, Kana Moriwaki, Michiko S Fujii, Yutaka Hirai, Takayuki R Saitoh, and Junichiro Makino. 3d-spatiotemporal forecasting the expansion of supernova shells using deep learning towards high-resolution galaxy simulations. Monthly Notices of the Royal Astronomical Society, 526(3):405444066 2023a.
Keiya Hirashima, Kana Moriwaki. Michiko S Fujii. Yutaka Hirai, Takayuki R Saitoh, Junichiro Makino, and Shirley Ho. Surrogate modeling for computationally expensive simulations of supernovae in high-resolution galaxy simulations. arXiv preprint 1xXi:23311004400 2023b.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn. and Tim Salimans. Axial attention in multidi- mensional transformers, 2019.
12

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimen- sional transformers, 2020. URL htttttsss://ppeeeeeee ottp://ww...o..ooooooooo--------
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203: 15556, 2022.
James R Holton and Gregory J Hakim. An introduction to dynamic meteorology, volume 88. Academic press, 2013.
Steeven Janny, Aurélien Benetteau, Madiha Nadri, Julie Digne, Nicolas Thome, and Christian Wolf. Eagle: Large-scale learning of turbulent fluid dynamics with mesh transformers. In International Conference on Learning Representations (ICLR), 2023.
Pengzhan Jin, Shuai Meng, and Lu Lu. Mionet: Learning multiple-input operators via tensor product. SIAM Journal on Scientific Computing, 44(6):43490-3355 2022.
Sadik Kakaç, Hafit Yüncü, and Kunio Hijikata. Cooling of electronic systems, volume 258. Springer Science & Business Media, 2012.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M. Preciado, George J. Pappas, and Paris Perdikaris. Learning operators with coupled attention, 2022.
Jean Kossaifi, Nikola Kovachki, Kamyar Arizzadenesheli, and Anima Anandkumar. Multi-grid tensorized fourier neural operator for high-resolution pdes. arXiv preprint arXiv:2310.0011 2023.
Nikola Kovachki. Zongyi Li. Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2 108. 08481, 2021.
Pijush Kundu,
Cohen, and David R Dowling. Fluid mechanics. Academic press, 2015.
Remi Lam, Alvaro Sunchez-Gonzaler Matthew Willson, Peter Wirnsberger, Meire Fortunato, Alexan- der Pritzel, Suman Ravuri, Timo Ewalds, Ferran Alet, Zach Eaton Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Jacklynn Stott, Oriol Vinyals, Shakir Mohamed, and Peter Battaglia. Graphcast: Learning skillful medium-range global weather forecasting, 2022.
Pablo Lemos, Liam Parker, ChangHoon Hahn, Shirley Ho. Michael Eickenberg, Jiamin Hou, Elena Massara, Chirag Modi, Azadeh Moradinezhad Dizgah, Bruno Regaldo-Saint Blancard, and David Spergel. Simbig: Field-level simulation-bast inference of galaxy clustering, 2023.
Jichao Li, Xiaosong Du, and Joaquim RRA Martins. Machine learning in aerodynamic shape optimization. Progress in Aerospace Sciences, 134:100849, 2022.
Zijie Li and Amir Barati Farimani. Graph neural network- lagrangian fluid simulation. Computers & Graphics, 103:201-211, 2022. ISSN 0097-8493, doi: https://doi.org/10..u 10.1// cag. 2022.02.004. URL https: www. www.scieecccce rect. com wttp:///////ccceeeeeecci.ccccooe
SSC973493220062066
Zijie Li. Dule Shu, and Amir Barati Farimani. Scalable transformer for PDE surrogate modeling,
In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
/openreview.coo net wtt.//ormmimoooooo 2id do
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations, 2020b.
13

---

Published as a conference paper at ICLR 2026
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations, 2020c.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL
net o /tte//frrrmm110FFPP////mmmm
Xinliang Liu, Bo Xu, and Lei Zhang. HT-net: Hierarchical transformer based operator learn- ing model for multiscale PDEs, 2023. URL https::/oppereviie net forum? ?id=
5Y5z500sK2el
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Winfried Lötzsch, Simon Ohler, and Johannes Otterbach. Learning the solution operator of boundary value problems using graph neural networks. In ICML 2022 2nd Al for Science Workshop. 2022.
URL
net. forum? 1d=4vxxxxxx 7wic
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint (iX:::990000//101 2019.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM review, 63(1):208-22) 2021.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdauta Goswami, Zhonggiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based fair data. Computer Methods in Applied Mechanics and Engineering, 393: 14778, 2022.
Suryanarayana Maddu, Scott Weady, and Michael J Shelley. Learning fast, accurate, and stable closures of a kinetic theory of an active fluid. Journal of Computational Physics, pp. 112869, 2024.
Weeratunge Malalasekera. An introduction to computational fluid dynamics: the finite volume method. Pearson Prentice Hall, 2007.
Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprint arXiv:2310009994 2023.
Michael McCabe, Payel Mukhopadhyay, Tanya Marwah, Bruno Regaldo-Sain Blancard, Francois Rozet, Cristiana Diaconu, Lucas Meyer, Kaze W. K. Wong, Hadi Sotoudeh, Alberto Bietti, Irina Espejo, Rio Fear, Siavash Golkar, Tom Hehir, Keiya Hirashima, Geraud Krawezik, Francois Lanusse, Rudy Morel, Ruben Ohana, Liam Parker, Mariel Pettee, Jeff Shen, Kyunghyun Cho, Miles Cranmer, and Shirley Ho. Walrus: A cross domsin foundation model for continuum dynamics, 2025. URL https:t://
org/ abs abs. 15684
Rudy Morel, Jiequn Han, and Edouard Oyallon. Disco: learning discover evolution operator for multi-phy-ies-agnosii prediction, 2025. http://tss.//aaxiioooo////////// 19499
Keith W Morton and David Francis Mayers. Numerical solution of partial differential equations: an introduction. Cambridge university press, 2005.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta, and Aditya Grover. Climax: A foundation model for weather and climate, 2023.
14

---

Published as a conference paper at ICLR 2026
Ruben Ohana, Michael McCabe, Lucas Meyer, Rudy Morel, Fruzsina J. Agoes, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B. Dalziel, Drummond B. Fielding, Daniel Fortunato, Jared A. Goldberg, Keiya Hirashima. Yan-Fei Jiang, Rich R. Kerswell, Suryanarayama Maddu, Jonah Miller, Payel Mukhopadhyay, Stefan S. Nixon, Jeff Shen, Romain Watteaux, Bruno Régaldo- Saint Blancard, François Rozet, Liam H. Parker, Miles Cranmer, and Shirley Ho. The well: large-scale collection of diverse physics simulations for machine learning, 2024. htthttps: / //arxiv. otg. /b//a////2222/2 00568
Oded Ovadia, Adar Kahana, Panos Stinis, Eli Turkel, and George Em Karniadakis. Vito: Vision transformer-operator arXiv preprint 2xXv:2333.08891, 2023.
Guofei Pang. Lu Lu, and George Em Karniadakis. fpinns: Fractional networks. SIAM Journal on Scientific Computing, 41(4)::2603-42226, 2019.
informee neural
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay. Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, 2022.
Romana Perinajová, Joe F Juffermans, Jonhatan Lorenzo Mercado, Jean-Paul Aben, Leon Ledoux, Jos JM Westenberg, Hildo Lamb, and Saša Kenjere3. Assessment of turbulent blood flow and wall shear stress aortic coarctation using image-based simulations. Biomedical engineering online, 20(1):84, 2021.
Tobias Pfaff, Meire Fortunato, Alvaro Gonzale,, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference Learning Representations, 2021. URL httpthppsp///ppneeeettttt wwwp:wee/ooorroo ? id=roNgY10 _XP
net
David R Poirier and G Geiger. Transport phenomena in materials processing. Springer, 2016.
Lukas Prantl, Benjamin Ummenhofer. Vladlen Koltun, and Nils Thuerey. Guaranteed conservation
of momentum for learning particle-based fluid dynamics, 2022.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informn neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Lord Rayleigh, Lix. on convection currents in horizontal layer of fluid, when the higher temperature is on the under side. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 32(199):529-546 1916.
Arthur Rizzi. Separated and vortical flow aircraft aerodynamics: afd perspective. The Aeronautical Journal, 127(1313) 1065-1103, 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention-MICCAl 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234 184-241. Springer, 2015.
Alvaro Fanchez-Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying. Jure Leskovee, and Peter W. Battaglia. Learning to simulate complex physics with graph networks, 2020.
Gerald Schubert, Donald Lawson Turcotte, and Peter Olson. Mantle convection in the Earth and planets. Cambridge University Press, 2001.
Gerold Siedler, John Gould, and John A Church. Ocean circulation and climate: observing and modelling the global ocean. Elsevier, 2001.
William D Smyth and James N Moum. Ocean mixing by kelvin-helmholtz instability. Oceanography, 2502):140-149, 2012.
Kim Stachenfeld, Drummond Buschman Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Gonzalzz Learned simulators for turbulence. In International Conference on Learning Representations, 2022. URL
15

---

Published as a conference paper at ICLR 2026
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09866 2021.
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based physics-constrainee deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:1 12732, 2020.
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):25-36, 2020.
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Herve Jegou. Three things everyone should know about vision transformers. arXiv preprint arXiv:2203.09795 2022.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023. URL
https::/epentt revi eeww.we. net www.n.eefoorrriiee// forummod=tmmmmmm
MP iMP TPa
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. International Conference on Learning Representations, 2020.
URL
wttpeeewwwwwwwweeeeeeeeeeeeeeeee net/
Prathmesh Vinze and Sebastien Michelin. Self-organization of autophoretic suspensions in confined shear flows. Physical Review Fluids, 9( 2024.
Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Scaling laws in patchification: An image is worth 50,176 tokens and more. arXiv preprint arXiv:2502.03738, 2025a.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp. 1457-1466, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984, doi: 10. 01145/339448.340031999
URLpttps://ddot http:///ooo.rr//////// 11111114334441 0015553394486..8 3403199
Sifan Wang, Jacob H Seidman, Shyam Sankaran, Hanwen Wang, George J. Pappas, and Paris Perdikaris. CVit: Continuous vision transformer for operator learning. The Thirteenth Interna- tional Conference Learning Representations, 2025b. httptt/sss///peeeeeee......
Forum?id=<RnCeuuyy
Nils P Wedi. Increasing horizontal resolution in numerical weather prediction and climate simulations: illusion panacca? Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 372 (0:2000)))211100300 2014.
Baole Wen, David Goluskin, and Charles R Doering. Steady rayleigh-bénard convection between no-slip boundaries. Journal of Fluid Mechanics, 933:R4, 2022.
Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries. In International Conference on Machine Learning, 2024.
Ting Wu and Guowei He. Space-time energy spectra in turbulent shear flows. Physical Review Fluids, 100544 2021.
Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang. Scale-invariant convolutional neural networks. CoRR, abs/1411 1,669, 2014. Utt.tt.p://arxiivoog/// 1411. 6369
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104-12113, 2022.
Pei Zhang, M. Paul Laiu, Matthew Norman, Doug Stefanski, and John Gounley. Matey: multiscale adaptive foundation models for spatiotemporal physical systems, 2024a. org/abs/ /212. 20601
//arxivv
16

---

Published as a conference paper at ICLR 2026
Xuan Zhang, Jacob Helwig, Yuchao Lin, Yaochen Xie, Cong Fu, Stephan Wojtowytsch, and Shuiwang Ji. Sinenet: Learning temporal dynamics in time-dependent partial differential equations. In The Twelfth International Conference on Learning Representations, 2024b. URL https://
Min Zhu, Handi Zhang, Anran Jiao, George Em Karniadakis, and Lu Lu. Reliable extrapolation of deep neural operators informed by physics or sparse observations. Computer Methods in Applied Mechanics and Engineering, 412:1 16064, 2023.
17

---

Published as a conference paper at ICLR 2026
Dataset
4 4-+ 16
coarsc+bursts
two-step dwell
wam-up-Pcoans.
reverse warm-up
Shear Flow
0.0375
0.0514
(- 37.1%%)
0.0380
(-1.3%)
0.03837
(-23%)
0.05407
( (-4.2%)
Supernow Explosion
1.204
1.533
(-27.3%)
1.208
(-0.3%)
1.212
(-0.7%)
1.5999
(-32.9%)
Turbulence Graviry Cooling
0.5592
0.689
(- (-2.2%%)
0.5506
(+1.5%)
0.5577 (+0.3%)
0.724
29.5%)
Table 5: Vanilla VIT (CSM): 10-step average rollout VRMSE for additional inference- time schedule families. Percentages indicate relative change compared to the 4 8 + 16 baseline (positive is better / lower VRMSE compared to 4-> + 16 baseline ). All results use the same trained model as Table 3 only the inference- stride/patch schedule differs.
A ADDITIONAL INFERENCE-TIME SCHEDULE FAMILIES
To complement the permutation study in Table we evaluate several additional families of inference- time stride/patch schedules designed to probe qualitatively different behaviors beyond cyclic or random orderings. The goal of this appendix experiment is not to optimize schedules, but to check whether the improvements reported for the 4 4 8 + 16 cycle are specific to that particular periodic pattern, and illustrate how different schedule "shapes" (e.g.. bursty compute allocation or horizon- dependent policies) affect rollout accuracy.
All results this section use the same trained Vanilla VIT (CSM) model as in Table we vary only the inference-tin time stride/patch sequence. For clarity, we list the 10-step schedules explicitly (written as stride/patch per rollout step):
Baseline periodic (used in the main text): 4
8
16 (repeat),
[4, 8, 16, 4, 8, 16, 4, 16, 4, .......
Mostly coarse bursts: a bursty schedule that uses coarse tokenization for most steps with occasional fine "bursts", [16, 16, 16, 16, 4, 16, 16, 16, 16, -J.
Two-step dwell periodic: periodic schedule with longer dwell time at each tokenization level (two consecutive steps per stride value), + s 8 + 16 + 16 (repeat), [4, 4, 8, 8, 16, 16, 4, 4,4,& 4,8,8.....
Warm-up then coarse (early fine): a horizon-dependent schedule that allocates finer strides early 4488 in the rollout and then switches to larger strides of 16 for the rest of the rollout, [4, 4, 8,8, 16, 16, 16, 16, 16, 16....
Reverse warm-up (late fine): the reverse horizon-dependent schedule, which delays smaller strides until late in the rollout, [16, 16, 16, 16, 16, 16, 8, 8, 4,
Table supports two main observations. First, the rollout improvements from scheduling not confined to a single periodic pattern: both the two-step dwell periodic schedule and the warm-up then coarse schedule achieve performance close to the 4 16 baseline on these representative systems. This suggests that multiple simple schedule families can yield similar rollout VRMSEs when allocating tokenization resolution across rollout steps.
Second, schedule choice still matters: not all ways of varying tokenization across steps are beneficial. In particular, the mostly coarse + bursts schedule and the reverse warm-up schedule substantially degrade rollout accuracy on these systems. Together with the permutation results in Table 3] these results indicate that rollout-time tokenization is a meaningful inference- time control knob, while exploration of more sophisticated schedules like dataset dependent schedules, or choosing optimal scheduling automatically as rollout progresses is left for future research.
On the artifacts side, we find that although the warm-up then coarse schedule can achieve rollout error close to the 4 4 8 + 16 baseline (Table after the initial warm-up it uses a constant stride value (16) for the remainder of the rollout. Figure 6 shows that, in our experiments. schedules that alternate stride/patch values throughout the rollout (c.g., 4 + 16 repeat and 4-+44- + - 16 + 16 repeat) were more effective at attenuating latch-lattice harmonic peaks in the residual spectrum than warm-up schedules that become constant later in the rollout. This highlights that schedule form affects artifact behavior beyond aggregate rollout VRMSE error.
18

---

Published as a conference paper at ICLR 2026
|Residual power spectrum: tracer]
0.012
rollout 4-+8+16 repeat
0.010
rollout 8-44-8-88-11--0
warmup then coarse early fine
0.008
0.006
0.004
0.002
0.1
0.2
0.3
0.4
Frequency
Figure 6: Averaged ID residual power spectrum (tracer field, Shear Flow) for long-horizon rollouts under different inference-tingg time stride/patch schedules: 4 8 - 16 repeat, 4 + 16 16 repeat, and a warm-up then coarse (early fine) schedule that transitions from 4 and 8 to 16.
B HARMONIC ERROR ANALYSIS
Consider the error between predicted and true fields during autoregressive rollouts. Working in the Fourier domain under first-order linearization, let e, (w) C denote the Fourier coefficient of error at step n. frequency w E TT, x|2 The eπror evolves as:
Cn+1(w) (w) = ((w)en(W) + a,((w,
(1)
the linearized propagation factor (how the model propagates existing errors) and a, (w) is the fresh error injected at step 2 0 by patch boundaries, which we assume is zero-mean with finite variance. Throughout this section, E[|] denotes expectation over stochastic realizations of the injected errors, while (-), denotes averaging over the rollout step index n.
where ((w)
fixed patch grid of period k acts as a spatial Dirac comb, concentrating error injection at the harmonic lattice:
(2Tm, 24 w= k
k 2Tmy
mm, my Z
(2)
When cycling through patch sizes (ki, k2; k3), errors injected Ω. U, Ur (the union of all harmonic lattices). However, only frequencies in 52, = N Say, (the intersection crrrsection-ffequenncess common to all patch sizes) maintain consistent spatial alignment across all steps.
Solving the linear recurrence yields (w) Ei= (w)) The key question is: do the injected errors (uww) add up constructively or destructively? To quantify this, let ) = ([la,(())) and define the normalized coherence between errors separated by 7 steps:
(E (w)nn(())n+++uu)))) )²(
7- (w)
ln+(w] S 1,
which is the time-averaged correlation coefficient. When & 1, errors align constructively; when 7 Rs 0, they are uncorrelated. Regrouping by time-lag gives the exact identity:
R-1
nー1-r \xw)) 3R[A(W) 74 (w)).
E e (w) (w)
2 20
Σ
When the model neither amplifies nor dampens frequencies significantly ((A(w) se 1), this simplifies to:
(e)e no ? + [w)(w) Σ (n - T)
(5)
Two regimes emerge: (i) Phase-locked injection (yr ze 1) yields O(z²) growth the harmonic spikes
observe. (ii) Decorrelated injection (7r 0) yields only O(n) growth.
19

---

Published as a conference paper at ICLR 2026
With cyeling, only frequencies in Ω2 see consistent alignment every step. aor cycle k {4. 8, 16). we have 22n = S216 (the least common multiple lattice). On most frequencies w E Ru \ 2n, the changing patch grid temporally thins and phase-misaligns injections, reducing #NAT Yr (w)) in (
This increased decorrelation shifts a portion of the error growth from quadratic to linear, which might explain the spectral spike suppression.
In our linearized analysis, the expected error energy at a frequency omega after n steps decomposes into: (i) term that is always linear in n, and (ii) a correlation term that depends how ahase-aligned error injections are across timesteps. For fixed patch grid, on problematic harmonics the correlation term behaves like sum +n, i.e. O(n²). and dominates. With temporal cycling, the patch grid changes so that injections are typically not phase-aligned, driving the correlation term toward zero and leaving the linear term dominant. This is the sense in which cycling shifts part of the growth from quadratic to linear, consistent with the observed rollout error reduction. We emphasize that this analysis is approximate: the contribution is primarily empirical, and the theory is meant as a physically motivated explanation rather than a full proof.
We do not assume independence of ae, claim exact rates for specific w, or assert optimality of our cycle. We chose powers of two in our patch sizes for computational efficiency and to avoid excess padding at image boundaries. While coprime sizes and a longer cycle might minimize 2n further (reducing the common harmonics), we leave exploration of such potential trade-offs to future work.
Intuitively, the key effect of alternating patch/stride sizes during rollout can be understood cleanly in the spatial domain. When model is rolled out autoregressively with fixed patch size, local errors tend to accumulate at the patch boundaries. Since the patch grid is fixed, the same locations see repeated errors step after step, causing these errors reinforce and appear as grid-aligned patterns or "checkerboards" over long horizons. By contrast, when the patch size changes from one step to the next, the patch boundaries shift. This redistributes where those local errors are introduced: boundaries from one step do not align with those in the next step. As a result, eπror no longer accumulates coherently in the same locations and instead spread more evenly across the spatial domain, thereby mitigating the structured buildup that causes grid artifacts.
Implementation-wee wise, the embedding dimension is fixed as in a standard ViT. Changing patch size only alters (1) the number of tokens and (2) the spatial footprint per token. During training we randomize over patch/stride sizes 4, 8, 16, so the network explicitly learns to interpret tokens from all these grids within single shared embedding space. Small patches capture fine-scale structure; larger patches emphasize coarse/global structure. Alternating them over time lets the model repeatedly "refresh" fine and coarse scales, which empirically reduces checkerboard artifacts and improves long-horizon accuracy. So while a deterministic schedule may look "physically odd"' at first glance, it is simply structured way of varying the receptive field of an already tettiipptch-traate model, not arbitrarily oscillating the physical representation.
Additionally, alternating patch sizes does not inject random noise, but it breaks the deterministic repe- tition of errors by varying the discretization pattern. Because the boundaries shift deterministically, the same spatial locations are longer repeatedly exposed boundary errors. This prevents errors from reinforcing in fixed positions and instead disperses them across the domain, reducing structured checkerboard artifacts over long rollouts. This leads to a more uniform error distribution (and reduces checkerboards) without requiring explicit noise.
C EXPERIMENT DETAILS
C.I DATASETS
The datasets that we benchmarked taken from The Well collection Ohana al. (2024). We selected 2D and 3D datasets with complex dynamics, ranging from biology (Active Matter Maddu al. 2024)) to astrophysies (Supernova Explosion & Turbulence Gravity Cooling Hirashima et al. (2023a b); Turbulent Radiative Layer 2D Fielding et al. (2020)) and fluid dynamics (Rayleigh-
Bénard & Shear Flow Burns et al. (2020). Here is information about their resolution and physical fields (equivalent to number of channels).
Active Matter: Active matter systems are composed of agents, such as particles or macro- molecules, that transform chemical energy into mechanical work, generating active forces
20

---

Published as a conference paper at ICLR 2026
Dataset
Resolution
of fields/channels
Active Matter
256 256
11
Rayleigh-Bénan Bérigh--Bénrddd-
512 x 128
Shear Flow
128 256
Supernova Explosion
64 x 64 x 64
Turbulence Gravity Cooling
64 64 64
Turbulent Radiative Layer 2D
128 x 384
Table 6: Specifics of the datasets benchmarked.
or stresses. These forces are transmitted throughout the system via direct steric interactions, cross-linking proteins, or long-range hydrodynamic interactions, leading to complex spa- tiotemporal dynamics. These simulations specifically focus on active particles suspended in viscous fluid leading to orientation-dependent viscosity with significant long-range hydrodynamic and steric interactions.
Rayleigh-Bénard: Ehyleigh-Bénarte convection Rayleigh (916)); Wen al. (2022) is
phenomenon in fluid dynamics encountered in geophysics (mantle convection Schubert et (2001), ocean circulation (Siedler al. (2001), atmospheric dynamics Holton & Hakim (2013). in engineering (cooling systems Kakaç et al. (2012), material processing Poirier & Geiger (2166)), in astrophysics (interior of stars and planets Hansen et al., (2012). ). occurs in a horizontal layer of fluid heated from below and cooled from above. This temperature difference creates density gradient that can lead to the formation of convection currents, where warmer, less dense fluid rises, and cooler, denser fluid sinks.
(2021; Vinze & Shear Flow: Shear flow phenomena Kundu et al. (2015;; Wu & He
Michelin (2024) occurs when layers of fluid move parallel to each other at different velocities, creating a velocity gradient perpendicular to the flow direction. This can lead to various instabilities and turbulence, which are fundamental to many applications in engincering (c.g., aerodynamics Rizzi (2023)). geophysics (e.g., oceanography |Syyth & Moum 22012). and biomedicine (e.g. biomechanics Perinajová et al. (202T). I. Code and software used to generate the data: https://github. wwwm//ud//eaeeoeoooeooe RudyMorel/thee wtwel//te-welllrbe--- We used this codebase to generate the dataset resolution of 1024 2048, and then downsampled that 128 x 256 for memory purposes.
Supermova Explosion:
Supernova explosions happen at the end of the lives of some massive stars. These explosions release high energy into the interstellar medium (ISM) and create blastwaves. The blastwaves accumulate in the ISM and form dense, sharp shells, which quickly cool down and be star-forming regions. These small explosions have significant impact on the entire galaxy's 's evolution.
Turbulent Radiative Layer 2D: astrophysical environments, cold dense gas clumps move through surrounding hotter gas, mixing due to turbulence at their interface. This mixing creates an intermediate temperature phase that cools rapidly by radiative cooling, causing the mixed gas to join the cold phase as photons escape and energy is lost. Simulations and theories show that if cooling is faster (slower) than mixing, the cold clumps will grow (shrink) (Gronke & Oh 2018 Abruzzo et al. (022).. These simulations Fielding al. (0000) describe the competition between turbulent mixing and radiative cooling at a mixing layer.
Turbulence Gravity Cooling: Within the interstellar medium (ISM), turbulence, star forma- tion, supernova explosions, radiation, and other complex physics significantly impact galaxy evolution. This ISM is modeled by turbulent fluid with gravity. These fluids make dense filaments, leading to the formation of new stars. The timescale and frequency of making new filaments vary with the mass and length of the system.
C.2 MODEL CONFIGURATION
The core transformer processor consists of multiple stacked processing blocks, each composed of three key operations: (1) temporal self-attention, which captures dependencies across time steps, (2) spatial self- self-aattetion, which extracts spatial correlations within each frame, and (3) multi-layer perceptron (MLP) for feature transformation. The number of such processing blocks varies across model scales, as detailed in Table The full configuration is visualized in Figure 7
21

---

Published as a conference paper at ICLR 2026
Table 7: Model architecture configurations across different scales.
Model
Embed Dim
MLP Dim
Heads
# Blocks
Base Patch Size
7.6M
192
768
3
12
[16, 16]
25M
384
1536
6
12
[16, 16]
100M
768
3072
12
12
[16, 16]
CKM/CSM/Fixed Patch
Decoder
xN
spatiotemporal
MLP
transformer
block
Spatial
ay2aA
self-attention (vanilla OR axial)
Temporal
t-A
self-attention
(causal)
CKM/CSM/Fixed Patch Encoder
Figure 7: This visualization represents the overall architecture of our models. The CSM/CKM/Fixed patch are various strategies for the patch encoding and decoding. The patch encoder block followed by N transformer based spatio-tempora blocks containing sequential time attention, followed by full spatial attention, followed by MLP. The embedding dimension column in Table represents the hidden dimension that the CSM/CKM/Fixed patch strategies embed the real space patches into.
C.3 TRAINING CONFIGURATION AXIAL VIT AND VANILLA VIT EXPERIMENTS
We use the following training configuration for the Axial VIT and Vanilla VIT experiments:
Hardware and parallelism. We train with a global batch size of 16 on 8 NVIDIA A100 80GB GPUs using Py Py'occ FSDP. No gradient accumulation.
2. Optimizer and regularization. We use Adam with learning rate 10 and weight decay 10-4, and set drop path to 0.1. We tuned the learning rate over {10-5, 10-4, 10-33.
3. Training loss. All models are trained with Normalized Mean Squared Error (NMSE), averaged over spatial dimensions and fields.
4. Prediction target. We use delta prediction, i.e., the model predicts the change in the field from the previous timestep. This affects training only; validation and test metrics are computed on reconstructed fields.
5. Positional encoding. We use RoPE (Su et al. 2021
6. Training details: Each fixed-patch baseline (p {4,8, 16)) is trained for 20,000 optimizer steps (200 epochs with epoch size of 100). The compute-elastic CSM/CKM model is trained
22

---

Published as a conference paper at ICLR 2026
with matched total number of optimizer updates equal to the sum of the three fixed-patch runs, sampling the stride/patch setting uniformly from {4, 8, 16) at each update (same global batch size and hardware). Under uniform sampling, this results in approximately equal numbers of updates at each setting, matching the per-setting training budget of the fxxe--pach suite.
7. Data splits. We use the official train/validationntes split provided by The Well dataset (Ohana et al. 2024). Accordingly, within each dataset, for each set of simulation parameters, split initial conditions 80/10/10 across train/validationnlest.. For example, if there are 100 initial conditions for each parameter settings (each trajectory with 200 timesteps), we use 80 trajectories per parameter setting for training, 10 for validation, and 10 for testing.
C.4 VRMSE METRIC
The variance scaled mean squared error (VMSE): it is the MSE normalized by the variance of the truth. has been used for the benchmarking of the Well (Ohana et al. 2024).
VMSE(u, v) (/u v/
(3/u /|2) +
We chose report its square root variant, the VRMSE:
(Iu - w/ 2/2/1/2
VRMSE(U, v)
((lu ú|?) + +
Note that, since VRMSE(u, ū) R 1, having VRMSE > 1 indicates worse results than an accurate estimation of the spatial
23

---

Published as a conference paper at ICLR 2026
D BINNED SPECTRAL ERROR ANALYSIS
D.1 DEFINITIONS
We assess robustness using the binned spectral normalized mean squared error (BSNMSE) Ohana et al. 2024). BSNMSE measures the mean squared emror between prediction and target after restricting both fields a spatial frequency band and then normalizing by the target energy that band. Concretely, we first define the (unnormalized) binned spectral mean squared error (BSMSE) as
BSMSE u) = uu uB u=l²)
(6)
where the band-limited field ug is obtained by applying a bandpass filter in Fourier space,
"g = [F[u] 1g]
Here, F denotes the discrete Fourier transform and 1s is indicator function selecting frequencies within 13
For each dataset, define three disjoint frequency bands B1. B2, and B3, corresponding to low, intermediate, and high spatial frequencies. We construct these bands by partitioning wavenumber magnitudes evenly logarithmic scale.
Finally, the BSNMSE is defined by normalizing the BSMSE by the target energy in the same band:
( - UB
BSNMSE,(U, u)
(8)
[175s'
2
A value of BSNMSE& indicates that the error that scale is at least as large as the energy of the target signal, i.e., worse than predicting zero coefficients within band B.
D.2 BSNMSE FOR DIFFERENT DATASETS
Fig. showw the sext prediction BSNMSE of the test for Rayleigh-Bénard, Bénard, Turbulent Radiative Layer 2D, Shear Flow and Supernova Explosion. The spectrum is divided into three frequency bins: low, mid and high being the smallest, intermediate and largest frequency bins whose boundaries evenly distributed in log space. Crucially, also emphasized in the main text, note that the fixed patch models (green) are separately trained models, while CKM (orange) and CSM (blue) (here shown is the flexified vanilla VIT model) are single models trained at randomized strides/patches.
Following results in Tables 1 and 2 of the main text, story is reflected in the BSNMSE metric as well. other words, it is clear from the BSNMSE plot that CSM/CKM (coupled with vanilla ViT for this plot) enable the creation of single flexible model while maintaining comparable accuracy the fixed-patch baselines (and often improving it) across these cases. This supports training a single compute-clastic model that can be deployed at multiple inference-tine time tokenizations.
24

---

Published as a conference paper at ICLR 2026
100M model: Rayleigh Benard
Low frequencies
Mid frequencies
High frequencies
4x 10
CSM
2x10
3 10
2x10-
Rxed
BSNMSE
2 10
10
10
6x
10
100M model: TRL 2D
6x
BSNMSE
xx
4x
3x10-
Inference patch size
Inference patch size
Inference patch size
100M: shear flow
2x10-
-x10
10-1
18 10-
BSNMSE
1.6xx 10
60
1.4x 10-5
4x
3
-0-
x0055
2x
4
10-1
3x10-
Inference patch size
Inference patch size
Inference patch size
7.6M model: 3D SN64 frequency analysis
3x
-0-
5x
B5NNSE
2x
-x
4x
2x10-2
16
16
16
Inference
Inference
Figure 8: BSNMSE of the next-step prediction of the test in the Rayleigh-Benard, Turbulent Radiative Layer 21), Shear Flow and Supernova Explosion datasets. The spectrum is divided into three frequency bins: low, mid and high being the smallest, intermediate and largest frequency bins whose boundaries are evenly distributed in log space. Crucially, as also emphasized in the main text, we note that the fixed patch models (green) are separately trained models, while CKM (orange) and CSM (blue) (here shown is the flexified vanilla ViT model) are single models trained at randomized stride/patch sizes.
25

---

Published as a conference paper at ICLR 2026
Table 8: CViT next-step prediction performance across datasets and token counts, comparing CKM and Fixed patch size (f.p.s.). As before, the f.p.s. models are separately trained models while the flexible models are one single model capable of handling multiple patch resolutions at inference. Similar to the vanilla VIT and axial VIT architectures, we find that CVTT consistently gains from smaller patch sizes, can be flexified for practical deployment and an improvement accuracy.
Dataset
Tokens/Patch
Flexible CViT
Fixed CVIT
2048/4
0.120
0.184
Shear Flow
128/16
0.127
0.319
3072/4
0.302
0.299
Turbulent Radiative Layer 2D
192/16
0.340
0.364
4096/4
0.054
0.110
Active Matter
256/16
0.065
0.127
E
CVIT EXPERIMENTS
CVTT experiments. We further evaluate our flexible patching strategy within the Continuous Vision Transformer (CViT) architecture (Wang et al. 2025b), recent vision transformer-basee model tailored for PDE surrogate modeling. CVTT features a patch- based transformer encoder coupled with novel, grid- grid-based continuous decoder that directly queries spatiotemporal locations. As the decoder operates without patch-basee tokenization, we apply our kernel modulation (CKM) only to the encoder, leaving the rest of the architecture unchanged.
flexified CVTT therefore refers to the encoder being a CKM. For CViT (Wang et al. 2025b) whose original design features a single-stage encoder (instead of multi-stage hMLP we used for our vanilla ViT and axial ViT experiments) and decoder based on grid-based querying, CKM is applied only to the encoder, for the single stage; because the idea to see if flexification on top of the base fixed patch architecture helps or not. These use cases show CKM's modularity and adaptability: it can flexify arbitrary convolutional patching pipelines, without requiring architectural changes or any overengineering to the base convolutional or attention mechanism or task head.
Since CVTT is designed for 2D problems, we restrict our evaluation to 2D datasets, ensuring fair and focused test of patch flexibility without altering any other model components. This experiment shows that our flexible tokenization strategy is architecture-agnotic and can be modularly integrated into advanced PDE models like CViT.
Table X shows that a flexified CVTT consistently outperforms fixed patch size CViT models across four diverse 2D PDE datasets. As in earlier experiments, reducing patch size (e.g., from 16 to 4) increases token count and compute, but yields significantly improved accuracy. Importantly, a flexified CVIT provides this securacy-coppus trade-off inference without retraining, demonstrating the modularity and architectural agnosticism of Overtone. This result also shows how adaptive patching be extended to complex hybrid architectures like CVIT, unlocking new modes of inference-time flexibility in scientific surrogate modeling.
In terms of rollouts, since CVTT a query based continuous decoder, instead of patch based decoder, it does not inherently suffer from the issues of patch artifacts. Even for that case, Table 9 shows that flexified CViT performs a lot better than the corresponding static patch version. This again shows that our flexification approach is useful even for cases of hybrid encoder-decoder architectures where only the encoder is patch based, but the decoder not. This reinforces the architecture agnostic-ness and the SOTA performance of our flexible models.
Model details for CVIT. For the base CVIT architecture, we utilized the CViT-S configuration in Wang et al. 2025b) (See Table of this paper), setting: Encoder layers 5, Embedding dim 384, MLP width 384, Heads 6. We set the embedding grid size for CViT to be the same as the dataset size.
26

---

Published as a conference paper at ICLR 2026
Table 9:
10-step rollout VRMSE comparison across datasets for CViT vs. fixed patch size 16 (f.p.s)
models. A indicates the best percentage improvement (i.c., lowest VRMSE vs. fixed patch bascline).
Dataset
Flexible CVIT
Fixed p.s. 16
A
Shear Flow
0.063
0.2598
+75.7%
Turbulent Radiative Layer 2D
0.540
0.568
+4.9%
Active Matter
0.78
3.33
+76.57%
CViT Patch Embeddings. The ViT encoder takes as input gridded representation of the input function yielding a spatio-temporal data tensor u E RTx#xW* x with D channels. The model patchifies the input into tokens Up E R # Y xC by tokenizing each 2D spatial frame indepen- dently, following the process used in standard Vision Transformers (Dosovitskiy et al. 2020). The patching process involves single convolutional downsampling layer. We add trainable ID temporal and 2D spatial positional embeddings to each token. These are absolute position encodings, unlike the RoPE encodings used for vanilla and axial VIT experiments.
PE, E RTXIXIXC PE, E =
Upe = Up + PE, + PE.,
The original paper (Wang et al. 2025b) has already detailed the above description of CVTT. Our main experiments were to flexity this CVIT architecture and show that under given compute budget, training a flexible CViT is much more advantageous than training multiple fixed patch CVIT models. To this end, we adapt CKM flexify the convolutional layer of CVTT, randomizing the patch size at the encoding stage. Additionally, since base CVIT has absolute learned position embedding, we resized the position embedding at each step of the forward pass as well using bilinear interpolation to make it compatible with the randomized patch size.
Goals with flexified CVIT. Our goal with the CVIT experiments is to show that, similar to the axial VIT and vanilla VIT results, CViT also benefits from flexification. In other words, with fixed compute budget, it is much more beneficial to train a flexible CVTT model than to train and maintain multiple static resolution CVTT models. This leads to the creation of flexible CVIT, similar to the flexible versions of axial and vanilla ViTs we created earlier.
First we show that similar to axial and vanilla ViT, CVIT also consistently gains accuracy as the patch size, aka token counts is decreased (Table 8). Additionally, we show that our alternating patch strategy for CViT leads to dramatically better accuracies than the fixed patch models (Table 9). Through these experiments, we achieve the flexification of this recent architecture. This experiment further shows the architecture agnostic-ness of Overtone's methods; meaning we can make flexi- fication compatible with range of base architectures, exemplified by results in the main text, augmented the flexification of this advanced hybrid architecture like CVIT. Flexified CViT is compute-adaptive meaning it can adapt to various downstream compute/accuracy requirements inference after being trained only once, eliminating the need to train and maintain multiple models.
Training details for CVIT experiments. We follow a consistent training protocol to ensure fair comparison between fixed-pater and flexified CVIT models. All models use the CVIT-S architecture from Wang et al. 20255), configured with 5 encoder layers, an embedding dimension of 384, MLP width 384, and 6 attention heads. We match the spatial embedding grid size the input resolution of each dataset.
We fix the learning rate at 10-4, chosen after a hyperparameter search over {10-2, 10-3 10 10-4]. We set weight decay to 10-5, 5 We choose batch size to fully utilize GPU memory for each dataset. We conduct training on a single NVIDIA A100 80GB GPU.
Due to compute limitations, training epochs vary across datasets, but remain consistent within each dataset. Concretely:
For Shear Flow, we train each fxed-pptchh model (patch sizes 4, 8, and 16) for 30 epochs.
* For Active Malter, train each for 40 epochs.
27

---

Published as a conference paper at ICLR 2026
For Turbulent Radiative Layer 2D, train each for 160 epochs.
We train the corresponding flexified CViT models (i.e., with CKM) with the same total compute budget as all three Exed-patch models combined. This allows a direct, compute- compute-equivalen comparison. As described in Section 3.1 this setup addresses practical question: under a fixed training budget, is better train multiple static models or one flexible model? If the flexible model enables training without any loss in validation accuracy, we have flexible model by training only once eliminating the need to train and maintain multiple fixed patch models.
T+11
T45
10
T+20
T+25
G.T.
Fluxfred CAT
ERSS. 16
Figure 9: Rollout for the by Shear Flow dataset for the CViT model. Top: Ground Truth (G.T.); Middle: Flexified CVIT; Bottom: Fixed Patch 16 (F.P.S. 16) models.
T+1
G.T.
Flexitted CVTT
FPP.. 16
Figure 10: Rollout for the density Turbulent Radiative Layer 2D dataset the CVIT model. Top:
Ground Truth (G.T.); Middle: Flexified CVTT; Bottom: Fixed Patch 16 (FPSS. 16) models.
28

---

Published as a conference paper at ICLR 2026
F PATCH ARTIFACTS
Patch artifacts are widely present in patch based PDE surrogates. In this section, we will provide extended evidence for the existence of these artifacts. Note that these patch artifacts are a problem for the architectures where both the encoder and decoder are patch based, so this is not problem for an architecture like CVIT.
F.1 OUR EXPERIMENTS
have investigated and found these artifacts to present range of patch based ViT architectures. Below will show patch artifacts arising in three different types of ViT based architectures, in different kinds of datasets: axial ViT (Ho et al. 2019), full VIT (Dosovitskiy et al. 2020 and Swin (Liu et al. 202T
Figures and 12 reflect the presence these artifacts in rollouts of two example fields of the Active Matter and Turbulence Gravity Cooling datasets respectively. Figure also shows artifacts arising when the base attention is changed to swin. Note that having a shifted window attention does not change the fact that the encoder and decoder still patchify with fixed size patches conventional format this fundamental
ViTs. Therefore even though attention can happen in a process still remains.
limination imposed by the
T+1
T+5
T+20
Ground Truth
Flood Patch
Figure 11: Artifacts arising in fixed patch models of the axial+Vii ViT fixed patch 16 model. Results are shown for the Exr field of Active Matter dataset.
We have additionally tested that these artifacts are a core problem with the standard patch based ViTs, and they do not simply arise due to training details, such different learning rates. Figure I4 shows that patch artifacts are agnostic to learning rates. This is expected because they arise because of the inherent limitation of fixed patch grids.
We have additionally tested that these artifacts also arise in neural operator models like AFNO (Guibas et al., (02T), which also has a patch based encoder and a decoder. We took the AFNO model provided as a baseline comparison in the Well benchmark suite (Ohana et al.) 2024), and trained it on the Turbulent Rudiative Layer 2Ddataset, and as expected due to its patch based encoding and decoding, patch artifacts arise here well as shown in Figure 15
F.2
OTHER PAPERS
These artifacts have also been in models where patch based encoders and decoders used. For example, a recent study on pretraining on the dataset of the Well by Morel et al. (Morel et al., 2025). The artifacts can be Figures 10 (3rd and 4th row), Figure 11 (3rd and 7th row), Figure 12 (3rd
29

---

Published as a conference paper at ICLR 2026
T41 T+5 T+10 T+20
Fatch Truth
Figure 12: Artifacts arising in fixed patch models the vanilla+ fixed patch 16 model. Results shown for the u field of Turbulence Gravity Cooling dataset.
Ground Truth
T+1
T+5
Fixed Patch (16)
Figure 13: Artifacts arising in swin transformer model where the spatial module is replaced by swin transformer, keeping everything else the same. Results are shown for the density field of Turbulent Radiative Layer 2D dataset.
and 7th row)and Figure 13 (3rd, 4th and 7th row). In this paper, these artifacts are most common in the Multiple Physies Pretraining, MPP (McCabe al. 2023) approach, which has both a patch based and decoder.
The broad goal in this section is show that these patch based artifacts are quite commonly seen in models that incorporate patch based encoding and decoding. Researchers can incorporate our flexible patching/striding methods into wide variety of these models, and we anticipate that the rollout strategies we discussed in the main text can be improve the overall stability, accuracy and mitigate patch artifacts in these models. Due to limited compute, experiments with every possible architecture is difficult, but our study opens up possibilities in exploring this direction.
30

---

Published as a conference paper at ICLR 2026
Ground Truth
LR=1E-2
LR=1E-3
LR-1E-4
Fixed Patch (16)
Figure 14: Artifacts in the density field of Turbulent Radiative Layer 2D dataset at rollout step 20. Artifacts appear regardless of the learning rate used for training.
density
True
Predicted
Ertor
X
Figure 15: Artifacts in the density field of Turbulent Radiative Layer 2D dataset at rollout step 20. This is the AFNO (Guibas et al., 2021) model. Patch artifacts also arise here because even though its a neural operator based method, but the encoder and decoder still uses ViT based patch encoding/decoding.
31

---

Fourier neural operators (Li et al. 2021) perform convolutions in the frequency domain and were originally developed for PDE modeling. We compare to a state-of-the- variant, the factorized FNO (Tran et al., (023) which improves over the original FNO by learning features in the Fourier space in each dimension independently. a process called Fourier factorization. We used the open source FFNO codebase provided in the original publication. We used small version of the FENO model with 4 layers, 32 fourier modes and 96 channels. This amounted to a model size of ~ 7M. During training, a learning rate of 10-3 is utilized after parameter tuning in the range 10-4 Training was performed on a single Nvidia A100-80GB GPU, We set batch size to Published as a conference paper at ICLR 2026
G BASELINES USED AND MORE ROLLOUTS
Section 3.4 (Table provided baseline comparisons with respect to a range of SOTA models which are not patch based. On the neural operator side, we included FFNO (Taa et al. 2023). We also test against SOTA convolution based model, SineNet (Zhang et al. 2024b). Additionally, we tested against a recent Physics Attention based scheme, Transolver (Wu et al., 2024). In addition to these comparisons, we also perform comparisons against baseline models provided with the Well dataset module, and the well paper (Ohana et al. 2024). These models were TENO (Kossaifi et al. 2023). FNO (Li et al., 202I. U-Net (Ronneberger et al. 20155) and CNext-U-Net (Liu et al., (022). We reported the best baseline results out of these comparisons against the Well benchmarks. These non-patch baselines covered diverse comparisons. Below we describe these models and the settings used to make comparisons:
SineNet: This model consists of multiple sequentially connected U-shaped network blocks called waves (Zhng al. 2024b). While traditional U-Net architectures use skip connections support multi-scale feature processing, the authors argue that forcing features to evolve across layers leads to temporal misalignment in these skip connections, ultimately limiting model performance. In contrast, SineNet progressively evolves high-resolution features across multiple stages, reducing feature misalignment within each stage and improving temporal coherence. This model has achieved SOTA performance on a range of complex PDE tasks. Refer to the original paper for more details.
We used their open source code from: |attps:/ cww wwtediiieeaaabaeae /IReelaaaAIRR/// /////llo00 / mm mainn main/
Sataalpp7 py We use the SineNet-8-dual model with configurations outlined in their codebase, taken http://loommmiannn
from
fttta:://gittuu..com///
pdearena/ httpa/////aaaaaaaeaoo///////r//e
py (sinenet8-dual model). The network has 64 hidden channels with 8 waves. We set the learning rate to 2 x 10-4, set after doing coarse tuning among 10 -4- 2 x 10 4,5 x 10 10-4,11 10-3 The resulting model has around 35M parameters. We performed training on single Nvidia A100-80GB GPU. We set batch size at 16.
F-FNO:
10 10-3 and 10-2
16.
Transolver: In this model (Wu et al. 2024). the authors propose new Physics- scheme that adaptively splits the discretized domain into series of learnable slices of flexible shapes, where mesh points under similar physical states is ascribed to the same slice. The model achieved SOTA performance in a range of benchmarks including large-scale industrial simulations, including car and airfoil designs. We use their open source codebase and adapt it to train on the Well. We used 8 hidden layers, 8 heads, 256 hidden channels, and 128 slices. This amounted to a model of size 1 11M. This configuration matches the configuration presented in their main experiments. Training was performed on single Nvidia A100-80GB GPU. We set batch size to 16.
As shown in Table 4. transolver generally struggled with the PDE tasks we tested on. We tested by tuning the learning rates varying between 10-3, 1 and 10-2 without any significant improvement in performance. In the original paper, for autoregressive prediction tasks, they tested on 64 x 64 Navier Stokes, and downsampled their Darcy data into 85 x 85 resolution. Our datasets, on the other hand all exceed 128 x 128, and therefore the training was much slower. In order to have fair comparison, do not downsample our datasets for training the transolver model. The transolver code we used, is taken directly from their codebase.
TENO, FNO, U-Net, CNext-U-Net The Well benchmarking suite provides these additional
baselines with default configs together (Ohana et al. 2024h. We use these default baselines. Neural-
32

---

Published as a conference paper at ICLR 2026
operator v0.3.0 provides TFNO. Defaults are 16 modes, blocks and 128 hidden size. U-net classic has default configs of: initial dimension = 48, spatial filter size = 3, block per stage = 1. up/down blocks 4, bottleneck blocks 1. CNext-U-Net has configs: spatial filter size = 7, initial dimension 42, block per stage = 2. up/down blocks = 4 and bottleneck blocks = 1. Finally, the FNO model has 16 modes, 4 blocks and 128 hidden size. As noted before, the Well baseline package provides these model configs and more details can be found there. We showed the best benchmark results from the Well baseline suite and found that Overtone's models consistently outperformed these baselines. Table IO shows the additional baselines.
Table 10: Next-step VRMSE across Well benchmark datasets using baselines provided in the Well.
Dataset
FNO
TFNO
U-Net
CNext-U-Net
Active Matter
0.262
0.257
0.103
0.033
Rayleigh-Bénne Bénerdd
0.355
0,300
0.469
0.224
Shear Flow
0.104
0.110
0.259
0.105
Supernova Explosion
0.378
0.379
0.306
0.318
Turbulence Gravity Cooling
0.243
0.267
0.675
0.209
Turbulent Radiative Layer 2D
0.500
0.501
0.241
0.226
We trained all of the above mentioned baselines such that they the number of observed samples as our flexible models (see Appendix C.3). ensuring that the comparison is fair.
Out of all of these baselines, found that F-FNO and SineNet gave the most competitive results to Overtone's 's models, as shown in Table performance of F-FNO and SineNet the two datasets shown in the text, performed additional comparisons of these baselines on more datasets shown in Table II Clearly, these additional comparisons tell the same story that out flexible archilecture-agnostic models outperform SOTA architectures on complex physics tasks.
4 on two of our complex 2D datasets. Given the competitive
FFNO
SineNet
Vanilla VIT (Ours)
Axial VIT (Ours)
Dataset
7M
35M
CSM (7M)
CKM (7M)
CSM (45M)
CKM (45M)
Boyleigh-Bénarce
0.94
0.137 0.563
0.192
0.1664
0.215
Active Matter
0.567
0.760
0.522
0.384
0.390
Table 11:
10-step rollout VRMSE comparison across patch-free models and our patch-based surrogate variants. We report scores for both 7M and 100M parameter versions CSM and CKM. Results shown for vanilla and axial self-ateention scheme.
Figure 16 visually shows that CSM/CKM + axial ViT models (45M size) are significantly better visually than the SineNet 35M model in long rollout predictions well. These rollouts for highly complex and chaotic Rayleigh- Rayleigh-Bénard Bénard dataset. Our flexible models achieve state-of-the-a performance here.
To prove the effectiveness of Overtone's flexible models further, we show the 44th step rollout of all four fields of the highly complex and chaotic Rayleigh-Bént Bénarddataset Figure 17 for CSM + vanilla VIT, Figure T8 for CKM + vanilla ViT and Figure 19 for FFNO. We chose to show FFNO because it performed best among neural operator models in our baseline suite. All of the models have a comparable size of about 7M paramaters. Along with the metrics noted in Table I] find compelling visual evidence that CSM/CKM perform dramatically better than a competitive model like F-FNO a similar parameter level. This demonstrates Overtone's strong performance.
More rollouts. Additionally, provide some more rollouts for different datasets for our flexible models. Figures 20 and [2]] Consistent with the message of our paper, we find that the CSM and CKM strategies help to stabilize and significantly improve rollouts compared to fixed patch models; along with beating a range of state-of-the art models that we have extensively discussed.
33

---

Published as a conference paper at ICLR 2026
T+10
T+20
T+30
T+35
T+40
G.T.
CSM + Axial ViT CKM + Axial ViT
SineNet
Figure 16: Figure showing a rollout for the Sarétgh-Bénares dataset. From top to bottom: ground truth (G.T.). CSM + axial ViT, CKM order of 40M parameters.
axial ViT, SineNet. They all have comparable sizes in the

---

Published as a conference paper at ICLR 2026
CSM + Vanilla ViT (7M)
rayleigh benard Frame 44 pressure
buoyancy
velocity x
velocity y
0.4
0.4
0.8
0.2
0.2
0.2
0.1
True
0.0
0.4
-0.1
-0.2
-0
0.2
-0.2
-0.4
-0
0.3
0.4
0.4
0.2
0.2
0.2
0.1
0.6
Predicted
X
0.0
0.0
0.0
0.4
00.1
-0.2
-0.
0.2
-02
-0.4
Figure 17: Figure showing the 44th step rollout of the 7M CSM + Vanilla VIT model.
35

---

Published as a conference paper at ICLR 2026
CKM + Vanilla ViT (7M)
rayleigh benard Frame 44 pressure
buoyancy
velocity x
velocity y
0.4
0.4
0.8
0.2
0.2
0.2
0.1
True
0.0
0.4
-0.1
-0.2
-0
0.2
-0.2
-0.4
-0
0.3
0.4
0.2
0.2
0.2
0.1
0.6
Predicted
X
0.0
0.0
0.0
0.4
0.11
-022
-0.
0.2
-02
-0.4
Figure 18: Figure showing the 44th step rollout of the 7M CKM + Vanilla ViT model.
36

---

Published as a conference paper at ICLR 2026
FFNO (7M)
rayleigh_benarc
benard Frame 44
buoyancy
pressure
velocity x
velocity y
0.4
0.4
0.8
0.2
0.2
0.2
0.1
True
0.0
0.0
0.4
-0.1
-0.2
-00
0.2
-0.2
-0.4
-0
0.3
0.4
0.4
0.2
0.2
0.2
0.1
0.6
Predicted x
0.0
0.0
0.0
0.4
00.1
-02
-0.
0.2
-02
-0.4
-0.
Figure 19: Figure showing the 44th step rollout of the 7M F-FN model.
37

---

Published as a conference paper at ICLR 2026
T+1 7+5
T+10
T+15
T+20
G.T
CRN ACS VT
OMM Juta VIT
Flccd P16
Figure 20: Rollouts of the Active Matter dataset.
T+1
T45 T+10
T+15
T+20
G.T.
CSN Von la VIT
CEN Vants NTT
Roo P16
Figure 21: Rollouts of the Turbulence Gravity Cooling dataset.
38

---

Published as a conference paper at ICLR 2026
H PHYSICS INFORMED BEHAVIOR
We investigate whether the models respect physical constraints such as mass and momentum conser- vation. First, we show long (100 steps) trajectory rollout for shear flow for 100 steps in Fig. 22 We find that in accordance with the main text, fixed patch 16 model develops artifacts already by step 40, while CSM/CKM has much greater long horizon stability.
T+10
T+200
T+40
T+660
T+-100
Ground Truth
Patch 16
CSM
CKM
Figure 22: Long horizon rollout for shear flow
To probe physics based constraints, we performed new diagnostics on our CSM and CKM models on the incompressible shear flow dataset and measured the mass conservation,
and momentum conservation,
10,u vÂu + u Vu + Vpl,
over 50 autoregressive rollout steps. Averaged over four randomly chosen test set trajectories at Reynolds number
Re = 5 x 10
(provided in the Well test set), the normalized deviations of these quantities relative to the ground truth numerical simulations approximately 0.5% (CSM) and 0.75% (CKM) for momentum, and 7.6% (CSM) and 6.7% (CKM) for mass.
These results show that momentum conservation is tracked very closely, while conservation errors are somewhat larger (as is typical for autoregressive surrogates) but remain bounded (< 10% error) over long rollouts, which is impressive for long rollouts in emulators.
These results indicate that our models remain stable and preserve conserved quantities (at levels acceptable for measuring accuracy in fields like astrophysics and fluid dynamics) well over long horizons.
39

---

Published as a conference paper at ICLR 2026
7.6M
25M
100M
2x10-2
Test VRMSE
10"2
CSM
CKM
6x 10
Fixed patch
128
512
2048
128
512
2048
128
512
2048
# of tokens at inference
# of tokens at inference
# of tokens at inference
Figure 23: Comparison of next-step test VRMSE as function the number tokens at inference for convolutional stride modulation models (CSM), convolutional kernel modulation models (CKM), and individually trained fixed patch models (7.6M, 25M and 100M parameters) for the Shear Flow dataset.
ABLATION STUDIES
1.1 MODEL SIZE ABLATION
For the Shear Flow dataset, we trained the CSM/CKM and fixed patch (for the vanilla ViT self- attention) models. The next-step interpreted in the same way as Figure 3 i.e. the green points are separately trained static patch models that we evaluate the corresponding training patch/size or token counts at inference. CSM and CKM on the other hand are single model that evaluate different token counts at inference. The plots show that CSM/CKM outperform the corresponding static patch counterparts consistently at a range of parameter counts varying from 7M to 100M. This ablation shows that story is consistent across scaling parameter count.
VRMSE are shown in Figure 23 The results are to be
For further investigation, repeated the ablation experiment for the smallest 7M model for other benchmark datasets, shown in Table 12 We find CSM/CKM to be consistently performing competitively with their fixed patch counterparts across benchmark datasets. This shows, again that flexification is valuable to be incorporated into current patch based PDE surrogates, since provides flexible choice of patching/striding parameter allowing tunability with various downstream tasks and compute adaptivity in PDE surrogates.
Test VRMSE 7M Vanilla VIT
Dataset
Patch size
CSM
CKM
Fixed patch size
3072
0.180
0.187
0.190
Turbulent Radiative Layer 2D
0.210
0.210
0.210
192
0,245
0,254
0.26
4096
0.0400
0.0405
0.0409
Active Matter
1024
0.050
0,046
0.043
256
0.066
0,070
0.050
4096
0.044
0.046
0.061
Rayleigh-Bénac Bénard
1024
0.060
0,060
0.073
256
0.074
0.080
0.083
4096
0.270
0.258
0.261
Supernova Explosion
512
0.343
0.328
0.337
0.370
0.364
0.367
0.102
0.096
0.100
Turbulence Graviry Cooling
0.152
0.138
0.133
64
0.169
0.169
0.164
Table 12: Test VRMSE for different patch sizes (or, token counts) inference time for three different 2D datasets using 7.6M parameter models. These values are for next-step prediction.
40

---

Published as a conference paper at ICLR 2026
1.2 ROBUSTNESS TO ADDING MORE PATCH/STRIDE OPTIONS
We performed additional experiments to explore the effect of adding more patch size options. Specifically, we ran our CKM + 50M axial VIT model with the following patch size sequences for the wurbulent.radian aide seatiieelayye wew..eeee.eee..... _2D dataset (2, 3, 4 and 5 patch size choices). Note that still choose patches as powers 2, following standard practice in this field. We choose the following patch options:
88, 16];
[4, 8, 16]; (4,8, 16, 32]; [4,8, 16, 12,
(also shown the fixed patch 16 reference)
We report the next step VRMSE loss on the validation set for these four models when the inference performed patch size of 16 for consistency across models. As detailed in the paper, note these flexible CKM+axial VIT models which is trained only once, and then can be flexibly deployed at inference with multiple patch sizes.
Patch sequence
VRMSE (Inference p.s.=16)
[16] Fixed patch
0.222
[8, 16]
0.208
[4, 8, 16]
0.204
[4, 8, 16, 32]
0.208
[4, 8, 16, 32, 64]
0.221
Based on the results above, the impact of adding additional patch sizes in this range is marginal, and for all of these models, the key conclusion of our paper holds: that without a loss of accuracy over fixed patch baselines. We did not notice any training instabilities when increasing the number of patch sizes/strides shown during training. The loss curves remained smooth and convergent.
achieve flexibility
Additionally, detailed in Appendix K the Pl-resize mechanism depends only the base patch size and the randomly selected patch size, and the resulting pseudo-inverse matrix is a smooth function of these parameters. Therefore, introducing additional patch options does not introduce any inherent source of instability an expectation that is fully supported by these empirical observations.
We additionally performed the same scaling experiment with the CSM+axial VIT model and obtained similar results. Below we show the next-step scaling VRMSE on the validation set. These new results show that the models are robust to adding more flexibility.
Stride sequence
VRMSE (Inference stride=16)
[16] Fixed patch
0.222
[8, 16]
0.213
[4, 16]
0.207
[4, 8, 16, 32]
0.209
[4, 8, 16, 32, 64]
0.216
Our method is robust to increased flexibility.
I.3 UNDERLYING PATCH SIZE ABLATION IN CKM
We vary the underlying patch size of CKM (coupled with axial ViT), We use sequence of base p.s. between 4, 8, 16 and 32. We tested on the Turbulent Radiative Layer 2Ddataset.
As shown in Table 1 we find that of the base patch sizes perform comparably to each other. Our conclusion on the adaptive compute, i.c., claim that flexible models can provide compute adaptivity in PDE surrogates without compromising on the accuracy hold at all base patch size choices.
41

---

Published as a conference paper at ICLR 2026
Base patch size
Inference patch size
4
8
16
4
0.157
0.181
0.246
8
0.156
0.179
0.244
16
0.160
0.175
0.216
32
0.161
0.179
0.221
Table 13: Ablation study on the effect of the base patch size used in CKM, evaluated on the Turbulent Radiative Layer 2D dataset with an Axial ViT backbone. Values are for next-step prediction VRMSE.
1.4 TIME CONTEXT LENGTH ABLATION
kept the number of input time frames to be fixed at because we wanted isolate the effect of dynamic patching/striding in our model.
The choice of Tiiimeiipputs that produce stable Long-horizon rollouts from a shorter input context stronger, since they learn to predict next steps with less ground-truth information. For reference, competitive PDE surrogate models such CVIT and SineNet use 10 input time steps, neural operator- pperator-based models like FNO, FFNO, and TENO also commonly use 10, models such as MPP (NeurIPS 2024) and DISCO (ICML 2025) use 16. By using 6 input frames, setup requires the model predict longer rollouts from less context while still remaining broadly consistent with common practice in the literature.
consistent with other choices in the literature. We consider models
We performed ablation varying the number of input frames (Nimput steps 3, 6, 12) for our CKM + 50M AVTT model. We kept all other settings and hyperparameters fixed. The results (next-step VRMSE on the validation set) are shown below:
Input time steps
Next-step VRMSE
0.202
6 (default)
0.201
0.213
We find that varying the number of input steps in this range has only marginal impact on accuracy. In practice, smaller input context is also attractive for efficiency, since it reduces both training and inference cost while still providing stable Long-horizon predictions.
1.5 SENSITIVITY TO PATCH DIVERSITY DURING TRAINING
Next performed experiment determine the effect of omitting patch sizes during training. To this end, we train on patch/stride sizes of 4 and 16 for our flexible models, and then during inference, we test on patch/stride of 4, 8 and 16. Clearly, as shown in Table 14 when we perform the evaluation on the omitted patch size of 8, the performance degrades for both CSM and CKM. This behavior is expected, and underlines the importance of training on a diverse patch/stride sizes obtain the best generalization performance at inference.
Inference p.s.=4
Inference p.s.=8
Inference p.s.=16
CKM + axial VIT CSM axial VIT
0.158
0.453
0.230
0.160
0.377
0.222
Table Performance drop when patch size 8 omitted during training used inference. We trained models only patch sizes 4 and 16. Results indicate the importance of patch diversity during training to ensure generalization at inference patch/stride choices.
42

---

Published as a conference paper at ICLR 2026
density
100
TUR
50
Predcted
Erter
25
Figure 24: Flexified AFNO model with eyclic patches reduced artifacts, compared to Fig.
J NEURAL OPERATOR BASED NON-TRANSFORMER METHODS BENEFITING FROM SMALLER PATCH SIZES
In this work we focused on patch-based based transformer architectures because they currently cover broad range of competitive state-of-the-ar PDE surrogate models (e.g., CVTT (ICLR 2025), MPP (NeurIPS 2024), DISCO (ICML 2025), and POSEIDON (NeurIPS 2024)). However, non-transformer models such AFNO (Guibas et al. 202T) share a similar motivation, and patch size turns out to be an important parameter for them as well.
To demonstrate this, trained the AFNO bascline provided with the Well package on the Turbulent Radiative Layer 2D dataset with patch sizes of 16 and 4. Reducing AFNO'S ' patch size from 16 to 4 improved the test VRMSE from 0.298 to 0.211, 1, showing that patch size is critical design choice. We also found patch size to be crucial a recent DPOT (Hao et al.] 2024 model, which performed large-scale pretraining on PDEs. These results highlight that patch size matters not only for transformer frameworks but also for other neural operator architectures.
To motivate this direction further, ran the 20M-parameter fixed-patch-16 AFNO baseline (as provided in the Well package) on three 2D datasets and compared it with our flexible models and other baselines:
Model / Dataset
Shear flow
TRL 2D
Active matter
AFNO (20M, p=16)
0.689
0.486
0.990
FFNO (7M)
0.110
0,485
0.567
SineNet (35M)
0.170
0.650
0.760
CSM VIT (7M)
0.0762
0,458
0.563
CKM + VIT (7M)
0.096
0.477
0.522
Table 15: Performance of AFNO and other baselines on three 2D datasets.
These benchmarks show that fixed-patch AFNO performs significantly worse than our flexified transformer-based models, especially on Shear Flow and Active Matter. We also tested a "flexified" AFNO by adding the CKM module to its patch embedding and de-embedding steps. On Turbulent Radiative Layer 2D, where AFNO was most competitive, the fixed-patch AFNO (p=16) reached 10-step VRMSE of 0.486 with visible checkerboard artifacts. The CKM-flexified AFNO with alternating patch sizes reduced the VRMSE 0.460 and mitigated the artifacts as shown below in Figure. 24 to be compared to artifact filled AFNO model in fig. 15]
Regarding DPOT, we note that CVIT (which we have extensively studied in this paper) already provides detailed comparisons to DPOT in its original publication, and CVTT outperformed DPOT across diverse benchmarks. We therefore chose CVIT as the representative strong baseline for our experiments. These experiments confirm that researchers incorporate flexible methods into architectures beyond transformers in a straightforward manner, enabling task-aware, inference- time control in both pretrained and task-specific surrogate models.
43

---

Published as a conference paper at ICLR 2026
K DERIVATION OF THE PI-RESIZE MATRIX FOR CONVOLUTIONAL KERNEL MODULATORS
K.1 MOTIVATION FOR KERNEL RESIZING
The CKM- based model dynamically adjusts convolutional kernel sizes at each forward pass while maintaining a fixed architecture. This allows variable patch sizes (4, 8, 16), which crucial for handling different spatial resolutions while ensuring compatibility with the VIT.
However, this flexibility introduces problem: we train the CNN encoder, which extracts ViT tokens, with fixed base convolutional kernel. To ensure that features extracted at different patch sizes remain aligned, we use a resize transformation, which projects the base kernel dynamically selected kernel size. Examples of resizing kernels through interpolation include (Xu et al. 2014) and (Beyer) al. 2023).
K.2
MATHEMATICAL FORMULATION
Let:
Whuse E B x XCn be the learned CNN weights for the fixed base kernel size buse,
W RRxkkeexXe be the resized kernel for a dynamically selected kernel size
B be an interpolation matrix (e.g., bilinear, bicubic) that resizes input patches from size k to size h.use.
x E R.. Bxkxkxan be an input patch, where:
B is the batch size.
x k is the local receptive field defined by the kernel.
Gn is the number of input channels.
Intuitively, the goal is find resized patch match the tokens of the original patch
set of patch- emhedding weights W such that the tokens of the
hase W (Br),
(10)
where * denotes convolution. The goal is to solve for W. Mathematically, this is optimization problem.
K.3
LEAST- SQUARES OPTIMIZATION FOR PI- RESIZE
K.3.1 EXPANDING THE OBJECTIVE
We aim to expand the expectation:
Exexx [(x Whuse xT BTW)?]
(11)
Since inner products can be rewritten matrix-vector multiplications, note that:
(x. babasc = x: brbase
Step 1: Expand the Square
Using the identity (a - b)2 = a² 2ab 62, we expand:
Thuse xT BT (x Whtuees 2x T. (TTB"W (xT BT)
(12)
Step 2: Take the Expectation Now, we take expectation over x X:
Exsx [xTTwaen bbssss2 2x W bese (I TTTTTW +
(13)
Using the definition of the covariance matrix:
44

---

Published as a conference paper at ICLR 2026
Σ Exwx(xr
(14)
we apply the linearity of expectation each term:
Eg WDbeey = W/use 1 EW buse
(15)
[x T B w W BEB" W
(16)
Eg[ [e'l"Whor Whare (x B W)] BW] Whhsse T ΣB" W.
(17)
Step 3: Write in Matrix Form
Substituting these into the expectation:
Exsx x B² w) base, bWs 2W base, T WBT + W WT BEB"W.
(18)
The above expression can be re-written as:
base W (Wbase,T - WT B) - -B"W).
(19)
Conclusion This result expresses the squared error between the transformed weight matrix W and the resized weight matrix B Whase weighted by the covariance matrix Σ. The quadratic form measures the deviation terms of feature space aligmment, ensuring that the transformation remains consistent with the learned base kernel.
The optimal solution minimizing is given by
W = (VEBT) base
(20)
where (VEBT) VE is the Moore-Penrose pseudoinverse matrix.
K.4 FINAL TRANSFORMATION AND IMPLEMENTATION
Thus, at each forward pass, we apply the following steps:
1. Randomly select a kernel size k E {4, 16).
2. Compute the interpolation matrix that resizes the input patch from & to h
3. Transform the kernel weights using:
W = (B W
(21)
4. Apply the resized kernel W for convolution.
This ensures that dynamically changing kernel sizes maintain compatibility with the fixed architecture while enabling flexible patch sizes.
K.S CSM PSEUDOCODE
45

---

Published as a conference paper at ICLR 2026
Algorithm 1 Convolutional Stride Modulator (CSM)
Input: x E xB
Output: BX
goaze, fixed kernel size
Step 1: Padding
Pad with learned tokens based on boundary conditions.
Step 2: Stride selection
Sample stride(s) from {4, 8, 16).
If the encoder/decoder is multi-stage, split s across stages.
Step 3: Encoding
1. For each encoder stage with assigned 8:
t- Conv(x, hass sttridde s)
Step 4: Transformer processing
Pass tokens through transformer processor (architecture-agnostt))
Step 5: Decoding
1. For each decoder stage with assigned s:
hase stride =)
return a
46

---

Published as a conference paper at ICLR 2026
K.6 CKM PSEUDOCODE
Algorithm 2 Convolutional Kernel Modulator (CKM)
Input: E XXXx
Output:
IR
wasse base kemnel weights, B: resizing matrix
Step 1: Sample patch size
Sample patch size k E {4,8, 16) once per forward pass.
If the base encoder/decoder be flexified is multi-staged, split across stages.
Step Encoding
1. For each encoder stage with assigned k:
Resize kernel: w f- w
Conv(x w, stride I-)
Step 3:
Transformer processing
Pass tokens through any transformer processor (architecture-agnostic)
Step 4:
Decoding
For each decoder stage with assigned k:
Resize kerne:: 1 base
f ConTIranspose( w, stride- strridd k)
return 2
47

---

Published as a conference paper at ICLR 2026
LLM USAGE
We used large language models (LLMs) during this project. We employed them as general-purpose assistive tools, comparable to IDE or grammar-checking software. Specifically, we used them for (i) code assistance, including generating boilerplate, debugging, and suggesting implementation details, and (ii) language editing, including grammar, clarity, and stylistic improvements drafts. All generated content was reviewed, corrected, and verified by the authors, who take full responsibility for the final text and code.
48


Reviews:
[{"review_id": "aXqRd3Uj9L", "reviewer": "Reviewer_M73p", "summary": "This paper introduces Overtone, a framework for transformer-based PDE surrogate models. The model is trained once using randomly sampled patch sizes or strides. During inference, this single model can be deployed in two modes: (1) at a fixed patch size, which allows users to dynamically trade computational cost for accuracy, or (2) using a cyclic modulation of patch sizes during autoregressive rollouts. The authors evaluate this framework on 2D and 3D PDE benchmarks from *The Well* collection, primarily comparing against multiple, individually trained, fixed-patch-size baselines, as well as other SOTA surrogates such as FFNO and SineNet. The results indicate that the single Overtone model matches or exceeds the performance of other models, and that the cyclic rollout strategy mitigates harmonic error accumulation, leading to improved accuracy in long-horizon predictions.", "questions": "See weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors analyzed the phenomenon in Transformer-based PDE prediction where fixed patch grids lead to error accumulation at specific harmonic frequencies, which manifests as spatial artifacts. To address this, they adopt strategies common in computer vision, incorporating multi-size kernels and multi-strides within a single model, thereby mitigating this problem in ViTs and achieving modest performance improvements.\n\n- The authors conduct evaluations on multiple PDE datasets from *The Well* benchmark and provide comparisons against several non-Transformer-based Neural Operators.", "weaknesses": "- Insufficient Baselines: The paper claims flexibility but lacks critical comparisons against existing ViT variants that also process multi-scale information (e.g., U-ViT or Swin-Unet), making Overtone's relative advantages unclear. The authors are expected to add direct performance comparisons against these relevant baselines in their revision.\n\n- Limited Novelty: The core modules (CKM and CSM) are not novel. CKM is heavily based on prior work such as FlexiViT, and CSM (stride modulation) is a standard technique, limiting its contribution to an application of known methods to autoregressive PDE rollouts.\n\n- Presentation: There are some problems with the layout of the charts in the text, and the authors are expected to refine them during revisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRsjNDwbgN", "reviewer": "Reviewer_wD4o", "summary": "The paper identifies a key failure mode in Transformer-based PDE surrogates: the use of fixed patch sizes in autoregressive rollouts leads to the systematic accumulation of errors at specific harmonic frequencies, manifesting as grid-like visual artifacts. The diagnosis of temporally coherent harmonic error accumulation is a valuable insight. The proposed method, implemented via two architecture-agnostic modules (CSM and CKM), not only improves the stability and accuracy of long-horizon predictions but also offers a practical way to trade computational cost for precision at inference time without retraining.\n\nDespite the good empirical results, the paper is weakened by a lack of deep theoretical grounding and leaves several critical questions about the method's underlying mechanics unanswered.", "questions": "See questions from weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*   The paper's core strength lies in identifying and articulating the problem of harmonic error accumulation due to temporal coherence. This is a subtle but important issue that has been overlooked. The proposed cyclic modulation strategy is a simple and effective solution that directly addresses this root cause, and the empirical results demonstrate its success.\n\n*   The authors conduct a thorough and rigorous experimental evaluation. The use of diverse and challenging 2D/3D datasets from The Well benchmark ensures the findings are relevant. The comparison against not only fixed-patch baselines but also a range of other non-patch-based models (FFNO, SineNet, etc.) provides a robust contextualization of the method's performance.\n\n*   The Overtone framework is designed to be architecture-agnostic, a claim supported by its successful application to vanilla ViT, Axial ViT, and CViT.", "weaknesses": "While the method is empirically successful, there are several areas of concern that detract from the paper's overall quality.\n\n*  The theoretical analysis in Section 2 and Appendix A provides some solid insights. But I don't quite understand how changing patch grid temporally thins and phase-misaligns injections which then shifts a portion of the error growth from quadratic to linear. (line 847~852)\n\n*  A major concern is the physical intuition behind cyclically modulating the input representation over time. I agree on the author's claim, that when the patch grid is fixed, the same locations see repeated errors step after step, causing these errors to reinforce and appear as grid-aligned patterns or “checkerboards” over long horizons. I also believe, at each autoregressive step, changing the patch size effectively changes the model's \"receptive field\", which helps improve the performance. But what doesn't make sense to me is when it's applied temporally like in this paper. This raises a fundamental question: **why should the model's perception of the physical state oscillate in this deterministic, cyclic manner?** This seems physically implausible. Furthermore, it's unclear on an implementation detail: **how are token embeddings handled with the changing patch size?** In a standard ViT, the embedding dimension is fixed to 786 as it's $c \\cdot p^2$. If the embedding dimension is kept fixed while the patch size changes, the model is cyclically fed tokens that represent vastly different amounts of spatial information. The rationale for why this is beneficial *temporally* is not well-explained.\n\n*  Missing Critical Ablations and Comparisons:\n    1.  The authors claim their method is architecture-agnostic and show that Swin Transformers suffer from similar artifacts (Appendix E). However, they do not present results for applying their method to a Swin Transformer. This is a logical omission that weakens the generality claim.\n    2.  The benefit of *temporal cycling* is not de-confounded from the benefit of simply having a model that is purely *spatially cycling*. An alternative model design could process the entire input with three different patch sizes (using three separate but smaller models) in every time step. Without this comparison, it's unclear if the cyclic schedule is the key ingredient, or if the training on diverse patch sizes is doing most of the work.\n     \n\n*   The evaluation relies solely on a single error metric (VRMSE). For physics simulation tasks, it is crucial to analyze whether the model preserves important derived physical quantities. For example, in the fluid dynamics datasets, an analysis of the predicted vorticity field (derived from the velocity) would provide much deeper insight into whether the model is learning the correct physical structures or merely achieving low pixel-wise error.\n\n*  There is no shape information in figure 3. You can just use same shape in green.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lCKWmJCfiR", "reviewer": "Reviewer_u9p5", "summary": "This paper focuses on the systematic error accumulation during the autoregressive prediction of the fixed patch vision transformer. The authors observe that cyclically modulating patch sizes during autoregressive rollouts distributes errors across the frequency spectrum. Specifically, Overtone is presented, which contains dynamic stride modulation (CSM) and dynamic kernel resizing (CKM) to implement the cyclic patch modulation. As a result, Overtone achieves 40% error reduction in long rollouts.", "questions": "(1)\tHow to decide the hyperparameter in CSM and CKM, such as the adaptive patch size or stride size, as well as the cycle size.\n\n(2)\tIt seems that Transolver performs quite badly. Do you follow their official configuration for the Navier-Stokes benchmark?\n\n(3)\tIn Figure 4, although both CSM and CKM do not contain the harmonic artifacts, their predictions are still different from the ground truth. Do you have any interpretation about this? \n\n(4)\tHow much extra computation overload do you need in training CSM and CKM?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThe authors notice and theoretically analyze an interesting problem, that is, systematic harmonic artifacts in the rollout of fixed patch vision transformers.\n\n-\tThe proposed method is verified to be effective in long-term rollout.\n\n-\tSufficient implementation details are included.", "weaknesses": "Despite the above strengths, I think this paper has some implementation errors, which may cause the experimental results to be meaningless.\n\n### (1) Potential implementation error. \n\nIn canonical vision transformer (ViT) and its follow-ups, such as Swin Transformer, patch embedding is implemented as a flattening and linear projection. In contrast, this paper adopts convolutional layers for patch embedding and decoding. I do not think this is a correct implementation. Thus, I think the proposed method may fail in handling canonical vision transformers.\n\n### (2) A missing simple baseline.\n\nIn video prediction, a simple baseline to reduce such autoregressive error is rollout training. I think in the current experiments, all the models are trained in teacher forcing, which means during training, all the model input is ground truth. In such teacher-forced training, it is easy to observe so-called “systematic harmonic artifacts”. However, I doubt that such an accumulation error will be significantly resolved if the authors apply rollout training. Even with rollout training for about 10 steps, the model will be significantly improved.\n\nThus, I suggest that the authors rerun the experiments under a rollout training configuration. At least, provide some investigations into the effect of rollout training.\n\n### (3) More Transformer baselines.\n\nAdditionally, I think some transformer-based models should also be compared, such as [1,2].\n\n[1] Scalable Transformer for PDE Surrogate Modeling, NeurIPS 2023\n\n[2] Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers, ICML 2025\n\n### (4) Missing comparison with the ground truth spectrum.\n\nI appreciate that the authors provide the spectrum comparison in Figure 2 and the visualization comparison in Figures 4 and 5. However, I think Figure 2 should also involve the spectrum of ground truth, and Figures 4 and 5 are also expected to record the spectrum. I think this will make the comparison more direct and intuitive. Additionally, since the authors mentioned that “Overtone distributes errors across the frequency spectrum, preventing the systematic harmonic artifacts that plague fixed-patch models”, I am curious about the gap between the distributed spectrum and the ground truth. \n\n### (5) Limitation in irregular geometries.\n\nAll the patch-based transformers are limited to regular geometries. I think the authors should discuss this in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHrG31ZfnI", "reviewer": "Reviewer_2Q4X", "summary": "This paper, \"Overtone: Cyclic Patch Modulation for Cleaner, Faster Physics Emulators,\" introduces a novel framework named Overtone to address two critical challenges in Transformer-based partial differential equation (PDE) surrogate models during autoregressive rollouts: harmonic error accumulation caused by fixed patch sizes and the inflexibility of computational cost. The authors propose two architecture-agnostic modules—CSM (Convolutional Stride Modulation) and CKM (Convolutional Kernel Modulation)—that dynamically cycle through different patch sizes during inference. This approach effectively suppresses harmonic artifacts while enabling compute-adaptive deployment. Extensive experiments on various 2D and 3D PDE benchmarks demonstrate significant error reduction and flexible accuracy-speed trade-offs.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Well-Defined and Significant Problem:** The paper accurately identifies a fundamental limitation in current Transformer-based PDE surrogates: systematic harmonic error accumulation due to fixed patch sizes. This issue is crucial for the long-term stability of rollouts.\n2. **Novelty of Inference-Time Cyclic Modulation:** The proposed strategy of cyclically modulating patch sizes during inference is pioneering in this field. It offers a fresh perspective on mitigating harmonic artifacts while simultaneously achieving computational adaptability.\n3. **Theoretical Underpinnings:** The mathematical analysis using a linearized error model provides theoretical support for the observed harmonic error accumulation, enhancing the rigor of the work.\n4. **Comprehensive and Solid Experimental Design:** The experimental section is thorough, encompassing multiple PDE datasets and model architectures. Extensive ablation studies robustly validate the method's effectiveness and generalizability.", "weaknesses": "1. **Insufficient Justification for the Training-Inference Strategy Discrepancy:** A notable and somewhat counterintuitive finding is the requirement for a cyclic strategy during inference, despite using random patch size sampling during training. The observation that a random schedule significantly harms inference performance lacks deep theoretical explanation and experimental substantiation:\n   - The current explanation, primarily based on a brief comparison in Appendix H.6, states that random scheduling hurts performance but does not adequately elucidate **why randomness is beneficial during training yet detrimental during inference**.\n   - There is a lack of in-depth investigation into the **optimality of the cyclic strategy**. Why was the specific sequence `[4, 8, 16]` chosen? How would other deterministic sequences—such as non-monotonic patterns (e.g., `[4, 16, 8]`) or sequences with different periodicities (e.g., `[4, 8, 8, 16, 8, 4]`)—perform? A more systematic exploration of alternative cyclic schedules is needed.\n2. **Investigate the Impact of Training Strategy on Artifacts and Method Efficacy**:\n   - Clearly specify the training methodology used (e.g., teacher forcing vs. mixed teacher forcing/rollout training).\n   - If only teacher forcing was used, discuss the potential impact of rollout training on the emergence of harmonic artifacts and verify whether the proposed cyclic modulation remains beneficial in that regime. This would strengthen the claims regarding the method's robustness and practical utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)