UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
Authors: Junhong Shen
Tanya Marwah
Ameet Talwalkar
Venue: Accepted by TMLR ()

Abstract: We present Unified PDE Solvers (UPS), a data- and compute-efficient approach to developing unified neural operators for diverse families of spatiotemporal PDEs from various domains, dimensions, and resolutions. UPS embeds different PDEs into a shared representation space and processes them using a FNO-transformer architecture. Rather than training the network from scratch, which is data-demanding and computationally expensive, we warm-start the transformer from pretrained LLMs and perform explicit alignment to reduce the modality gap while improving data and compute efficiency. The cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data and 26 times less compute. Meanwhile, it is capable of few-shot transfer to unseen PDE families and coefficients.

Published in Transactions on Machine Learning Research (11/2024)
UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
Junhong Shen
junkongs@eendreu.com..d
Machine Leorning Deportment, Carnegie Mellon University
Tanya Marwah
tmarwah Gandrew..ww.....mmmmmmmmm www.aawwwwwwwwaccaacammmmmmmmm.. cmai edu
Machine Learning Deportment, Curnegie Mellon Universily
Ameet Talwalkar
talwalker@cmmu. edu
Machine Learning Deportment, Carnegie Mellon University
Reviewed on OpenReview: https: // openreview. net/ forum? id= Or9mb,
Abstract
We present Unified PDE Solvers (UPS), data- and compute-effiient approach to developing unified neural operators for diverse families of spatiotemporal PDEs from various domains, dimensions, and resolutions. UPS embeds different PDEs into a shared representation space and processes them using a FNO-transformer architecture. Rather than training the network from scratch, which data-demandiing and computationally expensive, we warm-start the transformer from pretrained LLMs and perform explicit alignment to reduce the modality gap while improving data and compute efficiency. The cross-modal UPS achieves state-of-the-art results on wide range of ID and 2D PDE families from PDEBench, oulperforming existing unified models using 4 times less data and 26 times less compute. Meanwhile, it is capable of few-shot transfer to unseen PDE families and coefficients.
1 Introduction
Partial Differential Equations (PDFs) play pivotal role modeling and understanding real-world phenomena, such as fluid dynamics and heat transfer. Although there cxists rich body of classical PDE solvers (Boyd LeVeque 2007 Moukalled al.
2001;
2016) that are effective and mathematically proven, these solvers often incur substantial computational costs when used in practice, they need to be re-run every time coefficient or boundary condition changes. This motivates the development of neural operators (Li et al.
2020a Chen & Chen 1995
Lu et al. 2019), which use neural networks to approximate solution map for a PDE family and can generalize to different initial initial/boumie /bbundary conditions or coefficients. While existing neural
operators (Lippe et al. 2023 Hao et al.) 2023a Marwah et al. 2023) have demonstrated strong performance on various practical benchmarks (Taaamoto al.] 2022 Gupta & Brandstetter 2022), most of them are designed to work with single PDE Jumily. Training separate model for each PDE family remains costly.
Several recent works, such Subramanian et al. 2023), MPP (McCabe et al. 2023 and DPOI (Ho ct al. 2024), have taken initial steps towards developing foundation models for PDE solving, learning unified operators that transfer across PDE families. These models are pretrained from scratch using extensive amounts of data, and compute. For example, MPP trained with over 80,000 PDE trajectories on 8 NVIDIA H100 GPUs for 200,000 steps. Despite the development costs, the resulting models are limited in generalization ability- all existing unified models focus on pretraining with 2D PDEs, Finally, as these
This work was done the time ours.

---

Published in Transactions on Machine Learning Research (11/2024)
Unified Representation
Unified Architecture
2D Navier Stokess
2D Shallow Water
"Burgers Equation v-0.001"
Embedding Network
LLM Body
LLM Embedder
meta
Poshtion
Encoding
mix
Pointwise Conv
hyo
ut (x) E
R Nxnxn
FNO
Predictor
Linear
LLM Embedder
Linear 1 Prediction
Prediction
Alignment
Stage 1
Stage Multi-Task Fine-Tuning
1D Burgers
Navier Stokes
Embedding Pretraining
Figure 1: To adapt pretrained LLMs for PDE solving, UPS first transforms PDE of different dimensions, channels, and resolutions into unified representation (left panel). Then, the data is processed with unified architecture that integrates FNO layers, PDE metadata, and LLMs (right panel). The architecture is trained in two stages. In stage 1, we pretrain the embedding network using a joint loss that simultaneously optimizes (i) the distribution similarity between PDE features and text embeddings to align the modalities, and (ii) the prediction performance of extracted PDE features. stage 2, we fine-tune the entire model on a dataset that combines multiple families of spatiotemporal PDEs with varying domain dimensions, initial/bouddary conditions, and coefficients. UPS achieves competitive results with significantly better sample-efficiency than existing methods.
models are developed using only PDE trajectories, they do not leverage meta information that could help distinguish between various PDE families, such as the name of the family and the coefficients.
We present Unified PDE Solvers (UPS), which learns unified neural operators for complex time-dependent PDEs with improved efficiency and generalization ability. Unlike existing efforts that train models from scratch, we propose a novel method adapt pretrained Large Language Models (LLMs) to PDE solving, This is inspired by the line of work that repurposes LLMs for scientific domains like mathematics (ewkowycz)
et al. 2022), computational biology (Shen et al. 2023 Vinod et al. 2023 Joachimiak et al. 2023). and chemistry (Bran et al.] 2023 Shen et al. [0244). These works not only show how LLMs utilize both text and non-text information to solve scientific problems and transfer effectively to unseen tasks, but also provide strong evidence that the general-purpose pretraining and inductive biases of LLMs could substantially reduce the sample complexity needed for adaptation.
Concretely, UPS adapts pretrained LLMs to time-evolution operators that map the current state of PDE to its future state for general spatiotemporal PDEs (see Section 3 eqquaiin I for definition) using two key designs (see also Figure I and Section 3
1. We propose unified data representation scheme to align PDEs with varying dimensions and physical quantities into the same feature space. Given the space and time discretization {u.(x)){ where x E Rd is the spatial variable and ux(x) is the state variable, UPS homogenizes ux(x) from diverse PDEs into a shared "superspace" I2 Xxn dmay where dmax is the maximum dimension of x among all PDEs considered, N is the superset of the physical quantities, and n is the resolution. This framework of embedding lower-dinensional PDFs in higher dimension enables UPS to model cross-dimensional PDEs simultaneously and distinguishes us from all existing unified operators, which do not consider low dimensional PDEs in 1D.
2. We employ a unified network architecture to predict u,+1 (x) based on (x). To leverage pretrained LLMs, we design three-war way architecture that consists of (i) a FNO (Li et al. 2020a) based embedding network to convert PDE data into resolution-invarian the PDE features and the text embeddings of the PDE metadala; and (iii) a prediction network to generate the final output. Inspired by previous cross- modality adaptation work (Shen et al. 2023), we employ a two-stage align-then-refine process for model training. However, we improve the align stage by using a joint loss that adds feature extraction on top of alignment to pretrain the embedding network. We improve the refine stage by fine-tuning collection of PDE tasks rather than single task. Our enhanced
sequential features; (ii) an LLM body to process
2

---

Published in Transactions on Machine Learning Research (11/2024)
workflow outperforms naive transfer and previous cross- cross-modal approaches, reducing both the data and compute needed for training.
By design, UPS can handle diverse PDE families, data dimensions, channels, and resolutions. More crucially, by warm-starting with pretrained LLM weights and applying explicit. alignment, UPS strikes a balance between effectiveness and efficiency-it it achieves state-of-the-art performance across 9 datasets from PDEBench (Takamoto et al., 2022) (7 in-distribution, 2 out-of-distributien
using about 20,000 training trajectories, single A6000, 60,000 train steps, and under 100 GPU hours. This means that we achieve better results than existing unified models using 4 times less data and 26 times less compute.
Beyond prediction accuracy, we confirı that UPS preserves key properties of neural operators, such as grid- and invatiionnivvaieeeee.. We also show that UPS is compatible with a variety of LLM backbones, including RoBERTa (Liu et al. 2019), T5 (Raffel et al. 2020). and CLIP (Radford et al. 2021), and demonstrates better performance when scaled to larger backbones. We believe that the model-agnostic design of UPS olfers a systematic approach to harnessing the ad advancements in LLMs for PDE solving, and it takes further step towards building generalized foundation models for more complex physical systems efficiently. Code is Saammboongaan/TnffcPPEEESS
available at https:/ pattps//ggihuu..oo
2 Related Work
Recent years has seen a variety of neuraal-network-bbesse methods for approximating PDE solutions. Hybrid solvers (Hsieh et al.) 2019 Bar-Sinai et al.,
Kochkov et al. 2021 apply classical solvers like finite Moukalled et al.) 2016) to a low-rcsolution grid and use neural
2019;
clement/ cettoont/vouuuu /vlumm methods LeVeque 2007
networks to predict the correction terms.
Others directly approximate the PDE solutions with neural
2021 Shen et al. 2022). using variational losses (Yul networks (Sirignano 2017 Raissi et al. 2019 Khoo et al.
et al. 2018 or physical constraints defined by the PDE (Raissi et al. 2019 Bruna et al. 2024). Being mostly
equation-specific, these methods can solve one PDE at a time. The learned models do not apply to other
PDEs in the same family, let alone other families.
A more general approach involves learning neural operators (Lu et al. 2019: Li et al. 2020a b) which approximate an infniteedimeessonaal operator between two functional spaces. For time-dependen PDEs, a neural operator maps the current state of PDE to the next state, with quantities like initial conditions provided as input. Neural operators can be implemented using any architecture. For example, Fourier neural operator (FNO) (Li et al. 2020a) uses convolution-baseer integral kernels evaluated in the Fourier space. Other works also use transformer models (Cao 2021 Li et al. [022 Hao et al., 2023a) or U-Net (Lippe et al.) 2023 Learning noural operators enables solving an entire family of PDE and they can casily adapt to new parameterizations of PDE without fine-tuning However, the learned operators cannot extend to different. PDE families.
To facilitate operator transfer across PDE familics, two recent works develop large pretrained models for multiple physical systems: Subramanian et al. (2023) train FNOs on steady-state linear PDEs with periodic boundary conditions; McCabe et al. (2023) design new transformer architecture based on the axial attention (Ho et al.) 2020) and train it using various 2D non-linear, time-dependent PDEs. While these methods show that a unified operator can outperform single-family operators, they are limited two aspects. First, existing unified methods consider mainly 2D PDEs for pretraining and evaluation. In contrast, UPS leverages a unified representation scheme to tackle both 1D and 2D PDEs. This method can be also extended to any d-dimensional systems in theory. Second, existing methods pretrain large models from scratch and necessitate extensive GPU resources and pretraining data, which can be prohibitive to collect for high-dimensional complex PDEs. However, by adapting from pretrained LLMs and closing the modality gap between text and PDE efficiently, UPS achieves competitive results using 4x less data and 26x less compute.
Beyond the aforementioned works, DPOT (Hao et al., 2024 was developed concureentyy with our work and presents an auto-rogrossive denoising strategy for pretraining. While DPOT has shown better transforability to unseen PDE tasks than MPP, shares the same limitations of focusing on 2D problems for pretraining and requiring large amount of data and compute (8 A800 GPUs for 500,000 steps).
3

---

Published in Transactions on Machine Learning Research (11/2024)
A final work that is related to ours is ORCA (Shen et al. 2023 which proposes general workflow for adapting pretrained inguage/vision transformers to non-text/ ceftexxx/viiiing /isiion inpuis. While ORCA uses PDFBeenh in its evaluation, it is not tailored to PDE solving and requires adapting a separate model for every dataset. The resulting models are not grid- or resohution-mvvriant,, which are key properties of neural operators and achieved by UPS. Moreover, by learning from multiple PDEs and sharing knowledge across families, UPS obtains significantly better empirical results than ORCA.
3 Methodology
Our goal is to train unified neural operators for spatiotemporal PDEs with varying domain dimensions, coefficients, initial and boundary conditions. These PDEs could model range of quantities that evolve over time, from scalars (e.g,, pressure, density) to vectors (e.g.. velocity). To achieve this, we propose UPS, which consists of unified way to represent the PDE and LLM-based network to model them.
3.1 Unified Data Representation
We model PDEs that follow the general form:
@u(L, T. ul(t ou(l,x) @u(t,
)
it
dx Ox?
(1)
u(0,x) = uo(x)) B (u(t, v)) h(y)
where x E Q C IRd is the spatial variable, u [0. 7] x Ω + Rdd is a time-varying function defined over the domain Q for finite time T. Here, L is a (possibly non-linear) operator which acts on u and multiple partial derivatives of u w.r.t the spatial variable æ. (x(x) Q + Rdd denotes PDE's initial condition, and the operator B defines the boundary condition where y E 09 is a point on domain's boundary, and h 02 - de defines the given function over the boundary PDE familics in this form include Navier-Stokes equations, Reaction-Diffusion equations, Burgers equations, and many others that describe phenomena like fluid dynamics and heat flow over time. They also constitute most PDE benchmarks in the field of machine learning (Takamoto et al. 2022 Tu et al. 2022: Roberts et al. 2021
Consider a set of S spatiotemporal PDEs [==)s= Here, each w = (x)) is a solution to a PDE of the form defined in equation [ such that for all t E [T,], we have uī(x) E R and x E D c Rd where d² is the dimension of the PDE s. For each uy, we assume that we have an n-point discretization of the functions {uf)E It=1 at points W {zf, x..... 2. . }, where each of Rdd That is, for each PDE s E S and 1. E T., we have the realization of the function ut on a grid with each dimension divided into n parts, thus giving rise to nd points in the set. We assume that n is constant across PDE families. We note that this value n is a hyperparameter, and ideally should be the minimum number of points that work well for all the PDEs considered, for example, low-viscosity Wavier-Stokes may require more discretization points compared to the high viscosity counterparts. Denote the set of N physical quantitics considered for cach PDE as V- {ui, -...... J. Our goal is learn an operator Go which, for given PDE s, predicts the state of the PDE at time t +1 based on its state at time E [T,], i.e., ( = Go(u(=))) We thus need unified representation for the inputs so model can handle different quantities at once.
Unifying Dimension Let d' denote the dimension of the PDE s and d = max,es d². We want to represent all datasets in Idd. Thus, for PDEs with d² < d, the final d - d' coordinates of x; E W are set to zero. Tn this work, we mainly consider PDEs defined over one- and two-dimensional domains, ie., d" E (1,2) Vs E S. Hence, for PDEs with d² = 1, the point x E Ω is represented with the 2-coordinate (æ, 0). Note that our methodology to unify the total number of dimensions in the PDE is general and can be adapted to PDEs defined in higher-dimensionn function Uy on all the points in Wa, unless stated otherwise.
domains as well. In the following, we will denote ux(x) as the value of the
Unifying Physical Quantities We consider a fixed set V [U1, [ of N physical quantities and train our model on the quantities that belong to V for each PDE. The quantitics we consider in this paper are velocity (in both x and y directions), pressure, and density, and they are the superset of all quantities for
Unless stated otherwise, we assume that the value of the function is the boundary, L.e h(y) 0 for all y E an.
4

---

Published in Transactions on Machine Learning Research (11/2024)
the PDE families we evaluate. This leads to N = 4. If dataset does not use particular quantity, the entire dimension corresponding to it is set to 0.
With the above procedure, we lift every PDE to a unified space so u E RTxxNxx Vss E S. To obtain the datasets for forward prediction, we generate input-output pairs via autoregressive leacher-forcing: for each time step & E [T'], we use u, to predict 1.+1, yielding T. - 1 pairs of data from a single trajectory. We append the coordinates of each x E Wa to the imput and maintain an output mask to mask out the zero-padded dimensions when computing the loss.
3.2 Unified Architecture
Transformer models have demonstrated success in various domains like natural language (e.g., Touvron et al, 2023), vision (c.g., Dosovitskiy et al., 2021), and audio processing (c.g., Lu et al. 2023) In this work, we explore the potential of transformers for PDE solving. We break down the UPS architecture into 3 parts: an embedding network that transforms the unified representation into sequence features; the model body, consisting of the pretrained LLM layers; and a predictor that generates the prediction (Figure D.
FNO Embedding Network The embedding network plays two roles. First, it projects the PDE (x) into the LLM's sequential embedding space Riiee, where I denotes the sequence length of the embedded features and e denotes the LLM's hidden dimension. Second, it should extract key features of the PDE imput to enable subsequent transformer layers to make predictions. Therefore, design PDE-specific embedding network with FNO layers for feature extraction, a linear layer for dimensionality matching, and a concatenation operator for adding metadata (Figure
We use FNO due to its strong empirical performance [Li et al. 2020a Takamoto et al. 2022 and its ability to extract asooution-invariant features. As we consider maximum two-dimensional PDEs in this work, we use a series of 2D FNO layers with | channels to obtain PDE features in gixn". Then, to map the FNO output to the LLM's embedding dimension, we apply a pointwise convolution with input channel nd, output channel e, kernel size 1, stride 1. This yields the desired soquential features hYDE E R
Since UPS is intended to handle diverse data from various generating sources, we leverage the PDE's metadata in addition to the input dynamics. The motivation is that LLMs can use the textual information to better understand the context and characteristics of different PDEs. To implement this, we specify the metadata in which is embedded into sequential features hmeta using the LLM's
the form [PDE family] [coefficients]] tokenizer and text embedding layer. We then concatenate the meta features and the PDE feat to get. Amix == [/mmea: hppee] Finally, we apply positional encoding and layer norm to hmix- This will be the input to the subsequent transformer layers.
we perform various ablation studies on the embedding network. We investigate the effect different hyperparameters, such as the channel dimension I in FNO, and show that incorporating metadata improves both prediction performance and generalization ability of UPS
In Section 5.3
Utilizing Pretrained LLMs The main body of a UPS model consists of pretrained transformer layers from an LLM. Thus, we pass Amix to the pretrained transformer layers, which produce the hidden states h E iixe. Since there is no causal structure in the spatial dimensions of a PDE, we do not apply autoregressive masking to hmix and allow the embedded features to attend to each other.
Our design provides flexibility for using different LLMs the model body. We show experiment results with multiple L.Mss in Section 5.3. While different LLMs have different performance, they are competitive with existing baselines. We also show that adapting from pretrained weights outperforms training the same architecture from scratch, so UPS is especially useful for low-data rogimes.
Finally, we define a prediction head to transform the hidden state of the LLM body h to the predicted next step of the input (z) E Nxn" (we predict all the physical quantitics in the set V). This is achieved by averaging over the sequence dimension of h to get shape R applying linear layer to map it to IR Nw. and reshaping the results to obtain the final prediction (z). The linear predictor is shared for all PDFs.
Linear Predictor
5

---

Published in Transactions on Machine Learning Research (11/2024)
4 Full Workflow and Training
We train UPS in two stages. In the first stage, we train the embedding network to align hmix with the LLM's embedding space. This is because LLMs are trained for the text modality, which has distinct charactoristics and features from physical processes like fluid dynamics and heat flow. Stage 1 reduces the modality gap to prevent the distortion of pretrained weights. Next, we fine-tune the entire model on a datasct of multiple families of spatiotemporal PDEs.
Stage 1: Embedding Pretraining Intuitively, there is modality gap between text data used to train general-purpose LLMs and PDEs. Previous work has also shown that directly fine-tuning pretrained LLMs on non-text inputs can result in suboptimal performance (Lu et al. 2022). To address this, Shen et al. 22033 introduced ORCA, which performs distribution matching before tunint to enable cross-modal adaptation. That is, given a randomly initialized embedding network, we first pretrain to minimize the distribution distance between the embedding network's output- in our case and the text emboddings of a external reference NLP dataset, which we denote as /L.M- This process makes the cross-modal distribution resemble the text distribution that the LLM is pretrained on. Following the ORCA work, use the CoNLL-2003 dataset (Sang & Meulder 2003) as the reference dataset for alignment.
We propose several PDE-specific improvements to the aligmment process. First, unlike ORCA which uses an optimal transport (OT) based metric for measuring the distribution distance, we use the maximum mean discrepancy (MMD) distance for CPS. This is because the OT-based metric requires discrete class labels to compute, making it unsuitable for PDEs. In contrast, MMD acts directly on the features hmix and is more computationally efficient. Thus, we define
Calign = -| PDEMM 11, D [k<a, d')] - 22E DarM [k(a, b)] DD [k(b,b']
(2)
where k(a,, d') = exp(1la - a' 22 denotes the Gaussian kernel; D Drno and Dris denote the distributions of the PDE embeddings Amix and the reference text embeddings hLM
Second, to improve the feature extraction ability of the embedding network in the context of our downstream lask, introduce lask loss for PDE forward prediction, i.e., the normalized root mean squared (nRMSE) loss between the prediction úi-1(en (z) and the ground truth (z))
8
T--1
|u (x) at (x))l2
Ctask =
(3)
T
(x))(2))12
==0
1=
Thus, the final objective for pretraining the embedding network is the joint loss Lemb = Lalign + Lrask- We show in Section 5.3 that both objectives are essential to the overall performance of UPS.
Stage 2: Multi-Task Finc-Tuning In contrast to most existing neural PDE solvers, which train a separate model for each dataset, UPS is trained using one large dataset consisting of PDE data from multiple generating sources (all of 5). Hence, after learning the embedding network, we fine-tune the entire model (the embedding network, the LLM body, and the linear predictor) using Lrask defined in equation 3 We evaluate the performance of UPS in Section 5.1 and find it outperforms existing single-dataset neural operators. We also show that UPS generalizes to unseen PDE families and coefficients (Section
5.2 -the tte zero-shot and few-shot adaptation performance is competitive with models specifically trained on the entire target datasct.
5 Experiments
Data We train and evaluate our method using PDEBench (Takamolo el al. 2022 For training, we combine datasets from different PDE families: Burgers Equation (ID), Advection (ID) ) Diffusion-Sportion (ID), Shallow-Water (2D), compressible Navier-Stokes (1D and 2D), and incompressible Navier-Stokes (2D). We explicitly hold out two families, 1D and 2D The dataset details can be found in Appendix A.1 We autoregressively generate the predictions and use the
to evaluate the generalization ability of UPS.
6

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: nRMSEs (lower is better) for in-distribution PDEBench families, with baseline results taken from Takamoto et al. 2002 Shen al. (002)) : McCabe et al. (2023); Hao al. (2024) O means that the result is not available. On all datasets, UPS with RoBERTa-Paas (UPS-B) achieves the lowest nRMSFs among all smaller models and UPS with
RoBERIa-Large (UPS-L) achieves the lowest nRMSEs among large models. Numbers are bolded for each model
size group.
# Parars Advection Burgers
Diffusion Sorption
Navier Stoless
Shallow Water
Navier Stokes
Incoimp Navier Stoless
(sorted) 11)
11>
11
11>
2D
20>
21)
Singlo= Fammlly
FNO
466K
0.011
0,042
0.0017
0.068
0.0044
0.36
0,0942
GNOT
18M
0.0068
0.0373
OForee
19M
0.0072
0.0321
U-Net
7.M
0.67
0.34
0.15
0.72
0.083
5.1
0.1903
ORCA
125M
0.0008
0.12
0.0016
0.062
0.006
0.3541
0.1529
Unified (Small
Uniteed FND
168K
0.013
0,0501
0.0011
0.0101
0.0033
0,152
0,1061
MPP-B
116
0.0024
0.0281
DPOT-M
122M
0.00229
0.0177
UPS-B (Ours)
199M
0.0027
0.0390
0.0009
0.0056
0.0016
0.0153
0.0931
Unified (Large)
UPS-L (Ours)
357M
0.0022
0.0373
0.0009
0.0045
0.0015
0.015
0.0924
MPP-1.
400M
0.0022
0.0208
DPOT-L
300M
0.0023
0.0158
scale-independent normalized root mean squared error (nRMSE) as the evaluation metric, defined as follows:
1 Sumt
(x) (2)))
mRMSE
(4)
Stest
Iuu (x) 2|
We preprocess all the PDEs by normalizing each dataset along the channel dimension to ensure the scale of uz (3r) across datasets is simila
Baselines We compare against two sets of baselines: (i) single-family models trained on individual PDE datasets, including the widely used U-Net (Ronneberger et al. 2015). FNO Li et al. 2020b), the improved
version FFNO (Tran et al. 2023). the transformer-based GNOT (Hao et al. 2023b) and OFormer (Li et al., 2023). as well as the cross- modal ORCA (Shen et al. ) (ii) unified models trained on multiple datasets, including MPP (McCabe al., 2023), DPOT (Hao et al.) 2024), and a unified FNO trained using data transformed by our unified representation scheme. We note that MPP and DPOT focus on 2D PDEs and are pretrained on 2D Navier-Stokes, Shallow-Water, and Iiffusion-Reacttin from PDEBench. [Subramanian et al. (2023) is not included a baseline because its models are pretrained on different PDE families (e.g., Poisson's 's and Helmholtz equations) not present in PDEBench.
Implementation Details As noted in Section 3. UPS is compatible with any pretrained LLM. We present our main results using RoBERTa (Luu et al. 2019) and study other backbones in ablation studies (Table 4 We set the embedding FNO channel to 32. Since the resolution of the 2D datasets in PDEBench 128, we set the model resolution n to 128 and downsample datasets with higher resolutions. All of our experiments can be run on single NVIDIA A6000 GPU, See Appendix [A.2] for training details. Due to computational constraints, results are based single run per network configuration.
5.1 Achieving State-of-the-AN Art Results on PDEBench with Compute Efficiency
We first study the in di-distribution performance of UPS, i.e., we evaluate UPS on the test splits of the datasets that are used to train UPS, which consists of PDEs that share the same boundary conditions and coefficients with the training samples, but have different initial conditions. The results shown in Table D In general, UPS with RoBERTa ranks first on all 7 datasets and improves the stale-of-the-art by an order of magnitude on many ID datasets. We analyze the results in more details below.
We standardize the data by subtracting the mean and dividing by the standard deviation ensure training stability when using pretrained model weights. For loss computation, we apply an inverse transformation the outputs to revert them to the original scale. Although data normalisation may affect non-linear equations, essential prevent loss explosion during fine-tuning, We leave exploring alternative methods to minimize distortion non-linear dynamics as future work.
7

---

Published in Transactions on Machine Learning Research (11/2024)
Compare with Single-Family Operators We outperform all single-family models like FNO and ORCA, which train a different model for every PDE family. This shows the benefits of learning a versatile neural operator rather than multiple specialized ones, and our model is capable of extracting universal rules when learning to model multiple PDE equations.
Compare with Unified Operators We note that existing unified models like MPP and DPOT do not pretrain or evaluate on ID problems due to the limitation of their data representation. In contrast, UPS embeds low-dimensional PDEs in sigh-dimensional spaces and model all PDEs uniformly despite the dimensionality difference, achieving state-of-the-ar results on all ID datasets in PDEBench. As for the 2D problems considered, UPS with RoBERTa-Bas outperforms MPP-B and DPOT-M, which have similar model sizes. UPS with MoBERTa-Larg' oulperforms MPP-L and DPOT-L. We emphasize that UPS is trained on significantly fewer trajectories per PDE family (<5K) compared to the baselines. Besides, UPS can be run on a single A6000 for less time while maintaining good performance. This shows the data and compute benefits of adapting from pretrained LLMs.
Since MPP and DPOT focus on 2D problems and use different set of pretraining datasets from ours, we train 2D-only UPS on all 2D datasets in PDEBench to provide more direct comparison. The results are shown in Appendix Table 8 Notably, while 2D UPS is still trained with less data (since the other methods use additional datasets like PDFArena. (Gupta & Brandstetter 2022 ), our method ranks first on of 8 datasets, outperforming DPOT on 5 of 8 datasets and outperforming MPP on 3 of datasets.
Recall also that we train 2D unified FNO using the datasets processed by our dimension unification scheme. The unified FNO does not always outperform single-family FNOs, especially on 1D tasks, possibly because the network is 2D, and the relatively simple architecture might not be able to extract shared information across PDE families and leverage it to improve performance. More crucially, UPS outperforıns unified FNO on all datasets, showing the efficacy of our LLM- based architecture.
Scaling Up LLM Backbones To study the scaling behavior of our mothod, we adapt from both RoBERTa- Base (149M parameters) and RoBERTa-Large (387M parameters) and report the results in Table [ The first observation is that the two versions of UPS all outperform baselines of similar sizes, achieving both effectiveness and efficiency. Besides, UPS-Large generally outperforms UPS-Base, which shows that scaling up the backbone has the potential to yield better results.
In addition to prediction errors in Table ! we visualize some of the UPS outputs in Appendix A.4 and show that it is indeed able to capture the key features and dynamics of different PDE families. For efficiency metrics, we report the training compute requirement, FLOPs, and inference time for UPS in Appendix A..22 Compared to existing work, our method has lower FLOOPs and shorter inference time. This shows that our method can be deployed in practical environments where both computational efficiency and speed are critical.
5.2 Generalizing to Unseen PDEs with Data Efficiency
In this section, investigate the gencralization (out- of-distribution) performance of UPS under three sce- narios: (i) unseen PDE families, (ii) PDFs belonging to the training families but with different coefficients, and (ift) PDEs with higher-resolution grids. Unless heherwise specified, UPS-B is used.
Table 2: Zero- and few-shot transfer performance of UPS on unseen PDE familics and coefficients. Our few-shot results are competitive with baselines trained with more data. UPS-B refers to UPS with RoBERTa-Base.
Unseen PDE Facilis
Uneon Cooficcnnts
# Samples Morkl ID Relf-Rect 2D Dii-RRuct
ID Burgers
2D Navier-Stokes
1.0
M 11=1,, = (=
Unseen PDE Families As mentioned earlier, we hold out the Diffusion-Reaction equations from de- veloping UPS. We first directly evaluate UPS these two tasks and report the zero-shot transfer performance. Then, we study few-shot transfer by randomly sampling k e 100} trajectories from the training sets of the held-out tasks and use them to fine-tune UPS. Lastly, we report the fine-tuning results with the full training dataset. The results are
UPS-B 0.0557
1.0593
0.0566
0.103
0
FNG
0,1839
1.2
L,0922
1.4302
ORCA
0.1818
1.0812
1.6316
1.6399
UPS-B 0.0107
0.3327
0.0134
0.0809
10
FNO 0.188
0.8198
0.67
0.537
ORCA 0.104
0,326
0-4899
0.1823
UPS- 0.0034
0.2508
0.0022
0.0543
100
FNO 0.0677
0.1869
0.0123
0.3362
ORCA
0.0051
0.1362
0.027
0.0898
UPS-B 0.0003
0.041
0.0008
0.0191
SK (Full)
FNO 0.0044
0.12
D.0031
0.098
ORCA
0.001
0,62
0.012
0.0287
8

---

Published in Transactions on Machine Learning Research (11/2024)
shown in Table 2 As the number of adaptation samples increases, the prediction error decreases. Notably, the 100-shot result of UPS on 1D datasets is better than the baselines trained on 9,000 data, i.e., we use 90x less data to match the performance of single-family operators. This makes UPS useful for low-resource PDE problems where data collection is costly and training models from scratch is challenging. On 2D Diflusion-Reactionn we are slightly worse than pretrained MPP (0.0292) and DPOT (0.0106) since this dataset is considered as in-distrubution for MPP and DPOT.
Unseen Coefficients UPS also generalizes to PDEs in the same families as the training data but with different coefficients. We verify this by adapting UPS to Burgers Equation with v 1.0 (the model is trained on V = 0.001) and compressible Navier-Stokes with M == 1, "f = ( 0.1 (the model is trained on M - Ç = 0.1). The last two columns in Table 2 shows that while our zero-shot performance is already competitive, the performance after further adaptation outperforms most considered baselines.
Unscen Resolutions Zero-shot resolution refers to training the model on a lower resolution of the input data and evaluating them directly a higher resolution. PDE solvers with this ability are better equipped to handle real-world scenarios where input data may vary in resolution due to practical constraints or sensor-based limitations. Recall that UPS is trained with n-point discretization Wa, and we set n = 128 because most 2D datasets in PDEBench has resolution 128. Now, we evaluate the performance of UPS for (256, 1024), increasing the resolution of the input PDE. This is achieved by downsampling the higler-resolution inputs to make them compatible with UPS and then upsampling the output prediction to the desired resolution. We do not fine-tune the model at all.
Table 3: UPS with resolution 128 has an nRMSE of 0.0033 for Advection and 0.0931 for incompress- ible Navier-Stokes. We directly test UPS on higher resolutions.
Test Resolution
256
512
1024
Advection (uRMSE)
0.0057
0.0064
0.0068
Incomp Navier-Stokes (nRMSF)
0.119
0.126
We report the resolution generalization performance for 1D Advection Equation and 2D incompressible Navier-Stokes in Table Although the uRMSEs for both PDEs slightly increase compared to the nRMSE for the training resolution, they outperform all baselines in Table [ columns, UPS generalizes to higher resolutions in zero-shot manner.
Since the numbers are similar across
5.3 Ablation Studies
We perform five sets of studies to ablate various design decisions in UPS. S1-S4 demonstrate why adapting from pretrained LLMs is beneficial, while S5 is related to the FNO embedding network.
S1: Pretrained LLMs vs. Training From Scratch Compared to existing single-family models like FNO, UPS uses a transformer-basell archilecture with more parameters and reuses the pretrained LLM weights for the model body. To show that our results are not solely attributed to the model size and that modal adaptation is important, we evaluate the model's performance when we train a transformer model from scratch using the same PDE datasets without doing anything more complicated. As shown in Table training from scratch results in much worse performance than UPS, showing the benefits of adapting pretrained LLM.
S2: Cross-Modal Alignment We also test the importance of the two objectives used in stage 1, i.e., alignment loss with MMD, and task loss with nRMSE. We study three settings: (i) using only Lalign for stage as in Shen et al. (2023): (ii) using only Lrask for stage 1; and (iii) removing stage 1 from our workflow entircly. As shown in Table 4 while removing any objective reduces the performance across all datasets, removing the task loss has a more significant negative effect. Meanwhile, removing the entire stage of embedding pretraining hurts prediction accuracy. This shows that simply finc-tuning the LLM without considering the modality gap or learning lo extract PDE features is ineffective.
S3: Incorporating Fert Metadata UPS leverages the PDE's metadata by combining its text emboddings with the Icarned PDE embeddings. To study whether incorporating such metadata is helpful and identify an optimal approach, we compare our workflow with two alternatives: (i) we do not use metadata, so Amix := hYDE; (ii) we use metadata, but instead of concatenating features from two modalities, QK" apply a cross-attention mechanism: hmix softmax( Ve VV, where Q WihhDDE K W Wghhmet:
9

---

Published in Transactions on Machine Learning Research (11/2024)
Table 4: Results for the ablation studies. For each set of experiments, only the specified settings are different; all the other hyperparameters and training configurations are the same. Overall, the full UPS-Base workflow (first row for every study) most effectively leverages the pretrained knowledge of LLMs and obtains the best results.
Study No
Settings
Advection
Burgers Diff-Sorp Navle-Sttlees
Shallow- Water Stoler-Stoles Incceup Navier-Stoke
1D
1D
1D
ID
2D
2D
2D
Presraiaed LLM
0.0027
0.0399
0.0009
0.0056
0.0016
0.0153
0.0931
81
Training From Seratei
0,017
0,0546
0.0036
0.0158
B.0012
0.0461
0.1442
Align and Task
0.0027
0.0390
0.0009
0.0056
0.0016
0.0153
0.0931
Task Only
0.0048
0.0389
0.0009
0.0065
0.002
0.0184
0.1046
Aligu Ouly
0.0039
0.043
0.0011
0.0063
B,0022
0.1187
0.1082
No Embollding Pretraining
0.0049
0.0436
0.0019
0.0072
0.0024
0.0197
0,1079
Comratenaat Protiaiind Text
0.0027
0.0399
0,0009
0.0056
0.0016
0.0153
0.0931
Choss- Atteettioooo Protrained Text
0.003
0.0420
0.0009
0.0065
0.0023
0,0180
0. 1082
83
Concatenate One-Hot
0.0029
0.0447
0.0011
0.000
0.0018
0.0198
0.005
Conostenate Louredd Embesddings
0.0041
0.0474
0.3014
0.0111
0.0036
0.0295
0.1103
No Meta Information
0.0122
0.0433
0.001
0.0031
0.0026
0.0238
0,1171
RoBERT Base
0.0027
0.0800
0.0009
0.0056
0.0016
0.0163
0.0631
81
Flan-TS-Base
0.0091
0.0401
0.0076
0.0038
0.0028
0.037
0.1106
CLIP-Bese
0,0016
0.0321
0,30118
0.0063
B.0019
0,0151
0.0905
1= 32
0,0027
0,0399
0.0009
0.0056
0.0016
0.0153
0.0931
85
= 20
0.0024
0.1422
0,0009
0.0068
B,0022
0.1157
0,1043
1 ==3 8
0.0032
0,0129
0.0009
0.0071
0.0021
0.0195
0,1061
and V = =Wyhmeea- To further investigate whether the pretrained text embeddings contribute beyond merely labeling the PDE type, we also study alternative embedding strategies that do not leverage language pretraining: (iii) we replace the pretrained text embeddings of the PDE meta information with one-hot encoded vectors representing each PDE type. This setting serve as a bascline to assess the impact of meroly labeling the PDE types without any semantic understanding; (iv) we also lest setting where PDE types were embedded using a randomly initialized embedding layer that was trained from scratch along with the rest of the network, i.., each new token represents a PDE family.
The results are shown in Table 4 UPS outperforms the non-metadata baseline, demonstrating the effect of incorporating metadata as a textual form of domain knowledge, which LLMs are able to understand. The results also suggest that feature concatenation is better than cross-modal attention, possibly because the latter is harder to optimize. Lastly, the setting utilizing pretrained text embeddings consistently outperforms the one-hot and maabeedding-foom-srrad pretrained embeddings demonstrated faster convergence compared to the alternative strategies. This suggests that the pretrained semantic knowledge in the LLM indeed contribules lo processing the PDE data, not just in labeling PDE types but might also in understanding the underlying physical phenomena. However, we leave studying the the exact mechanism of cross-modal transfer and the optimal combination of metadata and PDE data as a future direction.
settings. In terms of learning dynamics, we also observe that using
S4: Other LLMs/ / VLMs To study whether UPS applies to other pretrained models, we further investigate Flan-T5 (Chung et al. 2022) and the vision language model CLIP (Radford et al.) 2021]. In particular, for CLIP, we use its text model to encode the metadata and its vision model to process the PDE data. The results are reported in Table 4 Since these models are trained using the same datasets and optimizer configuration as RoBERTa, the results are not fully optimized. Nonetheless, their performance is competitive with existing baselines, and CLIP further outperforms RoBERTa on 3 tasks, This shows the compatibility of UPS with diverse pretrained backbones. future direction is to study whether optimizing the training hyperparameters for cach pretrained cspeccasy VLMs like CLIP that are trained for an additional vision modality could improve downstream performance.
S5: FNO Embedder & Target Sequence Length As discussed in Section 3 the channel I of the FNO layers in the embedding network determines the sequence length of the PDE features that will be fed into the transformer layers. To study how this hyperparameter affects learning outcomes, we vary E (8, 20, 32} and report the results in Table ! In general, increasing imppovess the size and capacity of the embedding network, as well as the expressivity of the PDE features. This leads to lower prodiction error. However, using too many parameters for the embedding network may result in a trade-off between effectiveness and efficiency. For instance, we also experimented with = 64 (Appendix A.3.3p and find that the longer sequence length
10

---

Published in Transactions on Machine Learning Research (11/2024)
leads to slight performance improvements but with much higher computational costs. Thus, opt for 1 = 32 in our main experiments.
6 Conclusion and Future Work
In this paper, we present UPS, a method for adapting pretrained LLMs to unified time-evolution operators that predict the next state of PDE from the current state. UPS applies to a diverse set PDE families defined over onc- and woo-dimensional domains, with varying initial conditions, boundary conditions, coefficients, and resolutions. To train UPS, we develop two-stage cross-modal adaptation protocol that first. pretrains FNO-based embedding network and aligns its hidden representations with the LLM's embedding space, and then fine-tunes the entire model a dataset containing diverse families of PDEs. Since UPS is adapted from pretrained models, it requires fewer training samples and compute than previous approaches for training unified PDE sol vers from scratch. We show that UPS achieves state-of-the-art performance across multiple datasets from PDEBench and is capable of zero- and few-shot transfer to different PDE families, coefficients, and resolutions.
We identify several future directions based
work. First, we can validate method broader range of PDEs with higher-order temporal derivatives threo-dimensional domains. Meanwhile, to seek truly general foundation model for PDE, we aim to exlend the types of tasks that UPS can solve. Currently, UPS is only applicable to forward prediction. It is important to study inverse problems of parameter estimation for different PDEs as well. For an impact statement, see Appendix A.5
Acknowledgement
We thank Mikhail Khodak and Wenduo Chong for providing useful fcodback on the paper. This work was supported in part by the National Science Foundation grants ISS170512,, IIS1838017, ISS2046613, IIS21 12471, and funding from Meta, Morgan Stanley, Amazon, and Google. Any opinions, findings and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of these funding agencies.
References
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15344 1534444
2019.
John P Boyd. Chebyshev and Fourier spectrol methods. Courier Corporation, 2001.
Andres M Bran, Sam Cox, Andrew White, and Philippe Schwaller. Chemcrow: Augmenting largo-language models with chemistry tools. arXiv preprint arXiv:2304.05376 2023.
Joan Bruna, Benjamin Peherstorfer, and Eric Vanden- Eijnden.. Neural galerkin schemes with active learning for high-dinensional evolution equations. Journal of Compulational Physics, 496:112588, 2024.
Shuhao Cao. Choose transformer: Fourier galerkin. Advances in neural information processing systems, 24:44924-24940, 2021.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions Newral Networks, 6(4):911-91 1995.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuczhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instructionffinttume language models, 2022.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Jayesh K Gupta and Johannes Brandstetter. Towards matli--patioleempoaa-callll generalized pde modeling.
arXiv preprint (xXi::220.116166 2022.
Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learnöng, pp. 25556-2556 PMLR, 2023a.
Zhongkai Hao, Chengyang Ying, Zhengyi Wang, Hang Su, Yinpeng Dong, Songming Liu, Ze Cheng, Jun Zhu, and Jian Song. Gnot: A general neural operator transformer for operator learning. orXiv preprint arXiv:2302.1433 2023b.
Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Aulo-regressive denoising operator transformer for large-scale pde pre-training, 2024.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimennsioall transformers. International Conference on Learning Representations, 2020.
Jun-Ting Hsieh, Shengjia Zhao, Stephan Fismann, Lucia Mirabella, and Stefano Ermon. Learning neural pde solvers with convergence guarantees. arXie preprint arXiv:1906. 01200, 2019.
Marcin P Joachimiak, J Harry Caufield, Nomi I, Harris, Hyeongsik Kim, and Christopher J Mungall. Gene set summarization using large language models. ArXiv, 2023.
Yuchaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mothematics, 32(3):421 435, 2021.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118 (21):22001884118, 2021.
Randall LeVeque. Finile difference methods for ordinury and partial differential equalions: steady-stale and time-dependent problems. SIAM, 2007.
Ailor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843 3857, 2022.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. ar preprint arXiv:2205.1366 2022.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: /ttter///operrevier
het/EBorum?id-EEPPPq3uEE
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkunar. Fourier neural operator for parametric partial differential equations. arXiv preprint XrXiv:2010. 08895, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprinl Xrv::2003.000 03485, 2020b.
Phillip Lippe, Bastiaan S Veeling, Paris Perdikaris, Richard E Turner, and Johannes Brandstetter. Pdo-refiner: Achieving accurate long rollouts with neural pde solvers. Xix preprint ar Xrxio:2308. 2000020333,,,, 2023.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xr::::1907.11660 2019.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. ArXiv,
2023. URL hattps://apli semanticschollllt wttpettcccccclaoooorooorrppeee arg/CorpusID
266573555
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers universal computation engines. Proceedings of the AAAI Conference Artificial Intelligence, 36(7):7628-760 7636, Jun. 2022.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identi- fying differential equations based on the universal approximation theorem of operators. arXiv preprinl 191.::1100.03100, 2019.
Tanya Marwah, Ashwini Pokle, Zico Kolter, Zachary C Lipton, Jianfeng Lu, and Andrej Risteski. Deep equilibrium basod neural operators for stcady-state pdes. arXiv preprint Xiv:2911 00234,, 2023.
Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, al. Multiple physics pretraining for physical surrogate models. arXiv preprint OrXiv:2310.029) 2023.
Fadl Moukalled, Luca Mangani, Marwan Darwish, F Moukalled, L Mangani, and M Darwish. The finite volume method. Springer, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Colin Raffel, Noam Shazeer, Adamn Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-ters transformer. The Journal of Machine Learning Reseorch, 21(1):5485 5551, 2020.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Compulational plysics, 378:686-7000 2019.
Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog, Alexander Pfefferle, Sushil Ammanaghatta Shivakumar, Arjun Krishnakumar, Yubo Wang, Rhea Sanjay Sukthanker, Frank Hutter, Fuxhen Hasanaj, Tien-Dung Le, Mikhail Khodak, Yuriy Nevmyvaka, Kashif Rasul, Frederic Sala, Anderson Schneider, Junhong Shen, and Evan R. Sparks. Automl decathlon: Diverse tasks, modern methods, and efficiency al scale. In Neural Information Processing Systems, 2021. URL
https://api www.s/sscanancccccc.ccccm wwt.://rrrrr//oppuuuut00. :051
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, 15059.04997 2015. URL https://apl. semanticscholar. Org/CorpusID/
3719281
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the con1l-2003 shared task: Tanguage-independent named entity recognition. Conference on Computational Natural Language Learning, 2003. URL
https://api.. wemppsiicatnnicchlarrrr http://ccccooaoooorooooorosadaaa
orgg/CorpusID:2277716
Junhong Shen, Mikhail Khodak, and Ameet Talwalkar. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: align then refine. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-Ilm: Repurposing general-purpsse Ihms for specialized domains, 2024.
K Spiliopoulos Sirignano, J and. A deep learning algorithm for solving partial differential equations. ArXiv e-prints, 2017.
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitrry Morozov, Michael Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. arXiv preprint arXix:2306. 00258, 2023.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596 1611, 2022.
Hugo Touvron, Thibaut Lavril, Gantier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodrigues, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 2023. URL https: www.nticscholleeeeeo
atg//orpuusD:2572199000 257219404
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In The Eleventh International Conference on Learning Represenlations, 2023. URL https:///pennrvvier
I
Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. NAS- bench-360: Benchmarking neural architecture search diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datosets and Benchmarks Track, 2022.
Ria Vinod, Pun-Yu Chen, and Payel Das. Reprogramming pretrained language models for protein sequence representation learning. arXiv preprint arXiv:2301.02100 2023.
Luo Yining, Chen Yingfa, and Zhang Zhen. Cidbench: A large-scale benchmark for machine learning methods in fluid dynamics. 2023. URL https: /tttps://arxiv..o ha//aaxxvvvvrr.. wwtprvv.orggbb//221111 05963
Bing Yu et al. The deep ritz method: deep learning-based numerical algorithm for solving variational problems. Communications in Mathemalics und Stalislics, 6(1):1 12:1-12,-40 2018.
14

---

Published in Transactions on Machine Learning Research (11/2024)
A Appendix
A.1 Datasets
As mentioned in Scction we train our models using the datasets provided in the PDEBench (akaamooo et al. 2022). The time-dependent PDE families considered by our models are: Burgers Equation (ID), Diffusion-Spotionn (1D), Shalloo-Waater (2D), compressible Navier-Stokes (1D and 2D), incompressible Navier-Stokes (2D), and Diffusion-Reactionn (1D and 2D). For each E S, the number of points in the n-point discretization W is 128, i.c, n 128. For PDEs where the OOEbench-provideer grid has more than 128 points in each dimension, sample 128 equispaced points.
In this section, we provide few key properties and considerations for the PDEs used in this paper. The initial conditions u(0,z) x) for most of the datasets are sampled from superposition of sinusoidal waves. The set of coefficients and number of trajectories used per PDE are reported in Appendix Table 5 For full details on the data gencration process and the hyperparameters used to generate the PDE dataset, we refer the reader Takamoto et al. (2022).
A.1.1 Burgers Equation (1D)
Burgers equation is commonly used to model the nonlinear dynamics of various fluid dynamics systems. Given the field u(t,z) E (0,2) x (0. 1) + R the PDE is defined as follows:
a ut x) + 0, (t, "
1zzuu(t, (tt,
(5)
Here the diffusion cocfficient the viscosity of the liquid, and æ is the density of the liquid.
A.1.2 Diffusion-Sorption Equation (1D)
Diffusion-Sorption is nonlinear diffusive process slowed down by an external force that is dependent of the state variable R. This PDE is used to model groundwater contamination transport processes. The PDE is defined as the following:
D D Ouoult,z), R(u) R(u)
0,u(t,
(6)
where 1), (0, 500), and D 5 10-4, For more details the initial conditions, boundary conditions and the function R(u), we refer the reader to Takamoto et al. (2022). For our training, we use 4500 trajectories for this PDE generated by varying the initial conditions.
A.1.3 Advection Equation (1D)
Given advection speed 3, the advection equations are expressed as:
@zu(t,z) + 30, I)) 0 u(0, x)) = uz(
(7)
where x E (0, 1) and (0, 2). Various examples in this dataset are generated by sampling multiple initial conditions from super-position of sinusoidal waves used in Takamoto al. (2022).
A.1.4 Compressible Navier-Stoker (1D and 2D)
Given density velocity u, pressure p, internal energy of the system e the compressible Navier-Stokes equations are defined as follows.
0.++ V. (pu)= 0,
(5+ 11
p (0.u + u Vu) -Vp+nAu+
V (V u) 3
(8)
|ul
(p+ct
2²
0,
c
+ V. 2
pp
u u 0¹ =
15

---

Published in Transactions on Machine Learning Research (11/2024)
Here, (-1, for 1D Navier-Stokes and (0,1)² for 2D Navier-Stokes, and E (0,1). 1). Compressible Navier-Stokes stokes are used to model multiple real-world phenomena in aerodynamics and fluid dynamics.
A.1.5 Incompressible Fluid Navier-Stoker (2D)
We define the equations for incompressible fluid Navicr-Stokes where we imposc the condition that the fluid is "incommpresssible That is, the equation follows the following condition:
V u 0
(9)
For density p and pressure the equations used to generate the data in Takamoto et al. (2022) are as follows:
p p + u Vu) = - Vpu + nau + f
(10)
where f an external forcing function, and Dirichlet boundary conditions. Here x E [0, 1]2 and the initial conditions u and the forcing term f are sampled from two-dimensional Gaussian random fields. Please refer to Takamoto et al. (0022) for more details on the data generation process.
A.1.6 Reaction Diffusion (1D and 2D)
Reaction Diffusion are diffusive processes with external force applied to the system that may or may not depend over the field variable u. They are often used model many thermodynamical systems.
ID reaction diffusion is defined as follows:
0,u(t, v v0mu x)( u(t, x))
(11)
for all r E (0,1) and t e (0.
For 2D reaction diffusion, let u(t, x) = (t, 2)., uz(t, Then the equations are defined as:
uuu x) u v1 i1 + 11 v1a + #1 ui u2
(12)
01,1(1,20 v2dagaz u2 v2d U2 + u1 U22
where k 5 10-3 and Vi and vn are diffusion coefficients. Here æı (-1,1) and 22 E (-1,1) and the initial conditions are sampled from Gaussian random field.
A.1.7 Shatlow-Water Equations (2D)
These are derived from Navier-Stoker and are a framework for modelling free-surface flow problems. We denote by in (x), and (z) as the velocities in the horizontal and vertical directions and h as the height of the water and b defining the spatially varying bathymetry (the measurement of the depth of water in oceans, rivers, or lakes). The shallow-water equations defined as follows:
0,h + 8z, hun + 10, =0, O,
0,hu, + Oz,
u + 2 g9hh²
+ 0x, unuhh -grhū b,
(13)
1 h gghh² ) d0 High -grhdrbb
0,huz + On2
where x E [-2.5, 2.5)² and or is the gravitational acceleration.
A.1.8 Summary
The following table summarizes the cocfficients of the datascts used train and test our model (note that ID/2D Diffusion-Reaace Reaction only appear in the test set but not the training set). We also provide the number of training and test. trajectories. We generate the input-output pairs using autoregressive teacher-forcing.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Table For each PDE family, we select one set of coefficienss and use the data for training and testing UPS.
Dineusion
Dataset
Coefficients
Nun Train Trajectorie
Num TTss Trajecturies
Timesteps
Resalution
Advection
8 m 0.4
4500
1000
128
Burgers
ID
Diflusion- Reaction
Retusion-Reecttin
0.5, p 1.0
1500
10000
128
Diftusion Sorption
4050
100
21
128
Compressible Navien-Stokees
1) = c= 0.1, rand periodie
1000
Shallow Water
2D
Diflosion-Resce Reaction
105
10
101
128
Compresible Navier-Stokess
M = -- ( - 0.1, periodic
4500
1000
128
Inconpressbble Navier- Stolles
A.2
Experiment Details
A.2.1 Training Hyperparameters
We use the following training hyperparameters for all of our experiments, unless otherwise specified. Due to time constraint, we have not performed exhausitive hyperparameter search or tailor the hyperparameters to each experiment setting.
Batch size: 32
Gradient accumulation: 1
Gradient clipping:
Dropout: 0
Optimizer: Adam
Learning rate: 5F-5
Weight decay: 1E-5
Training epoch: 20 for stage 1, 100 for stage 2
We use the CoNLL-2003 dataset (Sang & Meulder 2003 as the reference dataset for alignment in stage 1.
A.2.2 Efficiency Analysis
We run all of our experiments on a single NVIDIA A6000. Table shoow the detailed model size, per epoch training time (in seconds), and total training time (in hours) for different network configurations. Note that train the models for 100 epochs.
Table Trainable parameters and training time for each LLM backbone.
RoBERTa-Bass
RoBERTa--aage
Plan-TS T5-Base
CLIP-Base
Num Params
149M
387M
176M
132M
Per Epoch (s)
3200
7600
3500
3000
Total (hrs)
88
211
97
83
We reported additional metrics such as FLOPs and the time required for predicting a single step for PDE instance in Table 7 assuming the input data is 2D with 4 channels and resolution 128. We mainly compared with unified models that have similar model sizes. Compared to these existing work, UPS has lower FLOPs and shorter inference time. This shows that our model is idcal for deployment in practical environments where both computational efficiency and speed are critical.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Table Efficiency comparison for unified neural operators.
UPS-B
MPP-B
DPOT-M
Num Params
116M
Forward FLOPs (G)
72.66
102.12
75.44
Single Step Inference Time (ms)
1.77
2.34
A.3 Detailed Experiment Results
A.3.1 2D-Only UPS
Table 8: Training UPS with all of the 2D datasets in PDEBench and compare with MPP and DPOT. Note that beyond these PDEBench datasets, MPP is also pretrained PDEArena (Gupta & Brandstetter, 2022 and DPOT is pretrained on PDEArena (Gupta & Brandstetter 2022) as well as CFDBench (Yining et al. 2023
Baseline results taken from Hao et al. (2024). means that the result is not available.
Params
PDEBench 2D Navier Stokes-(n, ()
2D Difi-React 2D Shallow Water
(sorted within groups)
1,0.1
1,0.01
0.1,0.1
0.1,0.01
M0.1
FNO
0.5M
0.098
0.096
0.097
0.360
0.170
0.265
0.12
0.0044
Small-
FFNO
1.3M
0.0212
0.052
0.0366
0.162
0.0452
0.104
0.0571
0.0116
Sized
GNOT
188
0.0325
0.0420
0.0373
0.0228
0.0341
0.0285
0.0311
0.00678
Oforner
1.9
0.0417
0.0625
0.0254
0.0205
0.0229
0.0192
0.00717
MPP-Ti
7M
0.0442
0.0312
0.0168
0.0066
DPOT-Ti
7M
0.0173
0.0397
0.0285
0.0132
0.0220
0.0176
0.0321
0.00560
Medium-
MPP-S
30M
0.0319
0.0213
0.0112
0.0024
Sized
DPOT-S
30M
0.0153
0.0337
0.0245
0.0119
0.0187
0.0153
0.0379
0.00657
DPOT-M
122M
0.0116
0.0238
0.0177
0.00866
0.0129
0.0108
0,02992
0,0029
UPS-B (Ours)
149M
0.0112
0.0600
0.0277
0.0085
0.0124
0,0211
0.0243
0,0018
UPS-L (Ours)
387M
0.0102
0.0596
0.024
0.0083
0.0102
0.0209
0.0236
0.0015
Large- Sized
MPP-L
400M
0.0147
0.0098
0.00220
DPOT-L
500M
0.0100
0.0216
0.0158
0.00872
0.0115
0.0101
0.0232
0.00233
DPOT-H
1.03B
0.00961
0.0180
0.0138
0,00847
0.0105
0.00948
0.0191
0.00199
A.3.2 Few-Shot Adaptation
Compared to full fine-tuning stage 2, we lower the learning rate when performing few-shot adaptation to prevent catastrophic forgetting.
Batch size: 32
Gradient accumulation: 1
Gradient clipping: -1
Dropout: 0
Optimizer: Adam
Learning rate: 1E-5
Weight decay: 1E-5
Epoch: 100
The following table reports the time required for few-shot experiments. Note that for Burgers equation, we train the model using 0.001, but the results here for 1.0.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Table 9: Time for few-shot experiments. Our model outperforms most existing baselines on these tasks by using fewer than 500 samples and much shorter adaptation time.
Num Samples
1D Diffusion-Reacee
2D Diflusion-Reactinn
Burgers 1.0
Per Epoch (s)
Total (hrs)
Per Epach (s)
Total (hrs)
Per Epoch (s)
Total (hrs)
0.08
50
0.28
1.33
0.28
100
0.64
3.11
1.11
500
112
3.11
512
14.22
96
2.67
A.3.3 Ablation on Longer Sequence Length
We studied the effect of embedding sequence length in Section 5.3 paragraph S5 of the main paper. The results show that among = {8, 20, 32), larger / indeed leads to better performance. However, since LLMs can support sequence lengths much longer than / = 32, we consider expanding the feature length (the number of "tokens") used to represent PDE data. See results below.
Advection Burgers Diffusion-Sorption Navier-Stokes Shallow-Water Navier-Stokes Incomp Navicr-Stokes
ID
2D
2D
2D
1 - 32 0.0027
0.0399
0.0009
0.0056
0.0016
0.0153
0.0931
- 64 0.0034
0.038
0.0009
0.0054
0.0015
0.0162
0.0988
While 1 = 64 performs slightly better some tasks, increasing the sequence length means that (i) the embedding network is going to be larger (since also corresponds to the width of the FNO layers), and (ii) the training time will increase as each sequence is longer. Both increase the training cost. Hence, we want to select the / that achieves a balance between efficiency and effectiveness. That's why we use ( 32 for our main experiments.
A.3.4 Long-Horizon Prediction
As stated in Section
3 our method mainly focuses on predicting the next step from the current step, i.e., (x) = Go (x)). However, we are also interested in the prediction capacity of our mothod over a longer period of time. Thus, we study an additional setting that predicts (f-10 We show non-aulogressivee evaluation results since otherwise we will only have very few time steps for each PDE trajectory. The table below shows that UPS is still effective for long-horizon prediction compared to baselines like FNO. Even though the prediction interval is longer, the error rates only slightly increase possible because non-autoregressive evaluation, so the errors do not accumulate.
Advection Burgers Diffusion-Sorption Navicr-Stokes Shallow- Navier-Stokes Incomp Navicr-Stokes
1D
ID
ID
ID
2D
2D
2D
At 1
0.0027
0.0399
0.0009
0.0056
0.0016
0.0153
0.0931
10
0.0034
0.04
0.0011
0.0074
0.0026
0.0189
0.134
19

---

Published in Transactions on Machine Learning Research (11/2024)
A.4 Visualization
A.4.1
Burgers Equation
51.
11
11
2:20
2125
t1.30
t135
t1 40
MWW WW
Grourd
A.4.2 Diffusion-Sorption
tx5
t:10
1115
11:20
1125
11.30
t1:35
11.40
Ground
20

---

Published in Transactions on Machine Learning Research (11/2024)
A.4.3
1D Navier Stokes
We show V2, density, and pressure.
5+5
t-10
11
2:20
t:25
tv:30
t1 40
Grund
245
1+100
2::15
1::20
1125
11:30
to.35
t..40
Ground Tr.sh
t:5
t:10
5115
2:20
t:25
11
3:
t: 40
Ground Truhh
A.4.4 Shallow Water
21

---

Published in Transactions on Machine Learning Research (11/2024)
A.4.5 2D Navier Stokes
We show V2, Vgs density, and pressure.
I+0
t+10
H+15
t+220
L+25
1+330
1+35
Ground
I-DD
t+5
1+10
2+15
1120
1::25
14:30
14:35
Ground Tr.sh
t+0
L+5
1+10
++15
L+20
L+25
t+30
:+35
Ground Truth
t+-0
1+5
t+1C
1+:15
1+20
t+25
14:30
14:35
Cround Trush
In the prediction for 2D compressible Navier-Stokes we see few artifacts in our generation. Furthermore, for quantities like pressure, our network often seems to generate an overly smoothened output. This could be because the 2D Navier-Stokes the only PDE in our datasct that requires us model pressure, and therefore the network biased towards predicting uniform value, which in our case 0. We believe this can be avoided by adding more families of PDFs that model pressure, and is fertile ground for future work.
22

---

Published in Transactions on Machine Learning Research (11/2024)
A.5 Broader Impact
This paper calls for ML communityy's 's allention to lake advantage of LLMs and apply them to wider range of real-world problems beyond the NLP domains. This moves towards truly democratizing machine learning in real life. In terms of broader socictal impact, our work can excrt a positive influence as it contributes to reusing existing models and resources, reducing the computational burden of developing new large-scale models on massive data. However, lowering the barrier for applying LLMs to wide range of tasks necessarily comes with the risk of misuse. Hence, it is imperative develop adaptation methods with better privacy, safety, and fairness guarantees.
23


Reviews:
[{"review_id": "DRnCB9T7Xz", "reviewer": "Reviewer_gB39", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Good variety of baseline networks\n- Clear explanation of the approach and the different phases of evaluation and training\n- Good effort to examine generalization capability and to conduct some ablation tests for different network components\n\n**Weaknesses**\n\n- The impact of the LLM embedding is still somewhat unclear, and I think would benefit from further testing\n- I have some other concerns relating to some of the ablation tests\n  (see below)\n- Estimates of computational costs seem somewhat simple", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Unified PDE Solvers (UPS), a network architecture\nand state representation approach designed to permit one network to\nimplement neural operators for solving multiple PDEs. This\narchitecture incorporates a language model as part of an embedding to\nencode the type of PDE and other parameters alongside a Fourier Neural\nOperator to encode the current state. LLM-style transformer layers\nprocess these to produce a final prediction for a future timestep. The\npaper compares performance on several PDEBench tasks and against a\nvariety of baseline neural network operators.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**(Important)** The biggest question/concern I have is about the\ncontribution of the $h_{\\text{meta}}$ from the LLM. I appreciate your\nablation study S3, but I'm still not sure whether the LLM is\ncontributing more than just providing a tag for the particular PDE.\nDoes the LLM's language pretraining help, or is its output just\nhelping label the PDE type for later stages of the network?\n\nDid you try providing the parameters and PDE type in a different way\n(i.e. a one-hot encoding per PDE or another opaque embedding)? Or\nproviding not the true PDE name, but instead some other \"fake\" name so\nlong as it's consistent? Is the specific PDE name and LLM adding some\nadditional capability that couldn't come from a different\nembedding/encoding approach?\n\nI'm just concerned that even with S3 it seems hard to rule that out.\nSome of the results in table 4 for the cross-attention approach look\npretty close in performance to those with the LLM. I just wonder about\nthe impact of the language input.\n\n**(Moderate)** I would be interested in information on any trends you\nobserved during fine-tuning (this is somewhat related to the first\nnote above). Broadly, how much of the final performance comes from the\ntask-specific fine tuning vs. the knowledge the LLM started with? I\nappreciate that your experiments S1 and S2 look at this question. To\nmy eyes, the results in Table 4 seem to suggest that for certain\nsystems the task loss is very important (for example in the experiment\nS2 Diff-Sorp 1D column, the largest drop appears to come with the task\nloss), although it doesn't look like this trend is consistent across\ntasks.\n\nEven with the observed benefits of the language pretraining in S1 with\nthe fine-tuning it seems a bit unclear to me where where exactly the\nimprovements are coming from.\n\nFor the above information in Table 4 (and also for other result\ntables), are the reported numbers averaged across several trials\n(multiple, separately-trained networks)? I imagine they are averaged\nacross the test trajectories. It would help to have some sense of the\nvariance in some of these numbers since in some cases they look pretty\nclose.\n\n**(Low)** When you quantify the computational costs, do you have some\nidea of these beyond the number of training samples and number of\nGPUs? What you report is a useful value to have, but something like a\nFLOP count or time required for testing would also help compare the\ncosts of running these networks once they are trained, especially in\ncases where your approach may be used many times after training.\n\n**(Low)** In A.5.5 you mention having some issues with the pressure term\nsince other systems don't use these quantities and as a result the\nmodel is trained to predict zeros. Would you expect this behavior\nmight get worse if this model were to be generalized to a wider set of\npossible systems (so that more quantities might experience this bias)\nand maybe benefit from some masking to indicate to the network the\nrelevant quantities to compute?", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EH0ZtscS8V", "reviewer": "Reviewer_7gUR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n* S1: The model performance is consistently better than the previous work. \n\n* S2: The ablation study partially supports the effectiveness of each component of the model, in particular, the effectiveness of the pre-trained LLM as the encoder block.  \n\n* S3: Possibly, the model might be the most data-efficient method, though this claim needs more analysis as described below. \n\n\nWeaknesses: \n\nW1: Evaluation: \nAs far as I checked in the code (main.py L196, (function: evaluate)), the evaluation is a so-called “1-step” prediction, not an autoregressive prediction (the previous time-step prediction is used to predict the next time-step), even for the inference. This does not seem properly emphasized in the main body, such as in the experiment section. It could be highly possible that the cited baseline performance would be partly measured in autoregressively. So, the authors should clarify that the other baselines are also measured in a one-step prediction manner. Moreover, the evaluation of the autoregressive prediction is directly related to the real-world application (emulating numerical simulation and evaluating long-time behavior). So, the absence of an autoregressive evaluation reduces the value of this paper. One possibility to make the one-step prediction method more appropriate for a realistic application is to predict a far future step (10 to 100 steps of future information), not one time-step future which is usually a very easy task for ML models.\n\nW2: Efficiency analysis: \nTable 1 and efficiency analysis seem scientifically meaningless. As far as I understood, the total computation cost and the error are not evaluated using the same training and test dataset but citing the original paper’s numeric information. Although one can speculate something from it, it cannot be regarded as a scientific analysis. If the authors want to claim Table 1 as their major result, the comparison should be performed more carefully and correctly (using the same datasets, training setup, and hopefully similar model size). \nBesides, it would be helpful for readers to measure the computational cost in terms of flops of the model. \n\nW3: Using a too-easy dataset to evaluate the performance (resolution & zero/few-shot study):\nFor the resolution study, the 1D advection equation is too easy. It is helpful for readers to evaluate using more difficult datasets, such as 2D NS equations with figures of the predictions. \nFor zero/few-shot studies, diffusion reaction equations are too easy. It would be fruitful to evaluate them with alternative datasets, such as the Kuramoto-Sivashinsky 1D equation and 2D compressible NS (M=1 or larger). \n\nW4: Insufficient explanations: \nThere are several descriptions where only insufficient or no information is provided (though I might miss them during the reviewing process). For example, \n\n* W4.1: for $h_{LM}$, the explanation of the external reference NLP dataset is missing. \n\n* W4.2: Is the linear predictor shared for all the PDEs or prepared for each PDE? \n             \n* W4.3: In section 5, the authors wrote that the dataset is normalized, which results in the solution not satisfying the original PDEs if considering a non-linear equation (e.g., consider $\\partial u/\\partial t + u \\partial u/\\partial x = 0$. Assuming $u_0$ is a solution of the equation, $k u_0$ ($k$ is a constant) does not satisfy the equation:  $k \\partial u_0/\\partial t + k^2 u_0 \\partial u_0/\\partial x = k (\\partial u_0/\\partial t + k u_0 \\partial u_0/\\partial x) \\neq 0)$, where the $k$ can be introduced by the normalization. If the authors claim that their model is a neural operator, they should validate why they are allowed to perform the data normalization which actually alters the considering PDEs.  \n\n* W4.4: There are several undefined symbols, such as $d, d_u, d_s, d^s_u$ (Sec. 3.1), (x, 0) (Sec 3.1 Unifying Dimension, this usage of the open parenthesis appears only here), and UPS-B (Section 5.2). \n  \nW5: Some description needs to be polished:\n\n* W5.1: In section 3, the label and model prediction are unified as $u^s_t$, which is very confusing to distinguish them. (moreover, sometimes $u^s_t$ becomes $u^t_s$, which increases the confusion).  \n\n* W5.2: Equation (1)’s L is not an operator but seems a function. Maybe \n     $$\\hat{L} (I, \\nabla, \\nabla \\nabla, \\cdots) u$$\n    would be more appropriate as a description of an operator, mathematically.  \n   Besides, $B(u) = 0$, in Eq. (1) means only considering the Dirichlet boundary condition. Is it the authors’ intention (if we consistently apply the description of the original L)? \n\n* W5.3: The English of the paper should be polished, e.g., verbs are sometimes used as nouns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new foundation model (FM) for solving partial differential equations (PDEs). Different from the previous FM for PDEs work, this paper claims to achieve better accuracy in a more efficient way on a benchmark dataset by (1) a new unified data representation scheme and (2) a unified network structure. In particular, the unified network structure includes combining several LLM blocks in order to take into account the PDE and its parameter information and feature improvement. The experiment indicates that the usage of the pre-trained LLM improves the performance even for the present PDE tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major: \n\n* W1: Properly describe the evaluation method: 1-step prediction, not autoregressive, with the justification of why not evaluating autoregressively. If possible, evaluation in an autoregressive manner and 1-step prediction but far future time-step are desirable to improve the paper value.  \n \n* W2: Re-assess the computational cost using the same setting for all the models. Assessing the computational cost with flops would also help to improve the paper value. \n\n* W3: Reconsider using a more difficult dataset for resolution and zero/few-shot study to evaluate the model performance more appropriately. \n\nMinor: \n\n* W4: Approach W4.1 – W4.4 appropriately.  \n\n* W5: Approach W5.1 -- W5.3 appropriately. \n\nBesides, \n\n* Eq. 13 (Shallow-Water Equations 2D):  Seemingly, the perpendicular direction’s derivative terms are missing: $\\partial_{x2} (u_1 u_2 h)$ in the second equation and $\\partial_{x1} (u_1 u_2 h)$ in the third equation. \n\n* Please check if using color font (Figure 2 caption) is allowed, which is rarely seen in scientific journals.", "broader_impact_concerns": "The broader impact statement is provided in Section A.6 and there seems little concerns on the ethcal implications because of the treating topic (pure scientific application for the moment).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rRg5IzBCMW", "reviewer": "Reviewer_WV3b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper presents a novel approach to solving PDEs using a transformer-based architecture, which is pretrained on a large language model. This is an interesting approach in re-using pretrained language models in the context of PDE solving.\n\nThe method is shown to be more efficient in terms of data and compute compared to existing methods. This is an important aspect, in particular under the light of the reused LLM backbones. The method is applied across a variety of PDE families and achieves strong results on PDEBench.\n\nThe experiments on unseen PDE families is an interesting aspect of the paper, showing that the method can be transferred to new PDE families with few-shot learning. This is an important feature for practical applications, especially for a generalization to new PDEs with potentially limited data or new conditions.\n\nThe paper is clearly structured and uses established benchmarks. The experiments are well-documented and the results are presented in a clear and understandable way.\n\n### Weaknesses\n\nWhile the method has convincing empirical results, the theoretical foundation of the method is unintuitive and not well studied. As LLMs have been trained on very different data, it is unclear why they should be useful for PDE solving. In particular, this is in contrast to structured Transformer architectures for 2D data, like Vision Transformers, which could be directly applied to the observed data. The paper should provide a more detailed motivation for the usage of LLMs in this context, as well as how these tokens are used in the LLM. For more detailed comments, see requested changes.\n\nFurther, while the method uses language models, the focus is on encoder-based models which is not the most recent development in the field. The paper should discuss the choice of the transformer architecture in more detail, and how the method could work on autoregressive models, which are more commonly used in the context of LLMs nowadays.\n\nAdditionally, the paper highlights the choice of the FNO network for being resolution-invariant. However, the whole backbone with the embedding network is not resolution-invariant and fixes the spatial dimensions in its linear layers. This needs to be clarified in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for solving PDEs using a transformer-based architecture, which is pretrained on a large language model (LLM). The method uses an FNO network for mapping the PDE inputs into a latent space which is aligned with the LLM data distribution. In experiments, the method is trained across multiple PDE families and is presented as being more efficient in terms of data and compute. Evaluated on PDE families from PDEBench, it achieves competitive results with other existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Critical*:\n- Notation in Section 3.1 needs to be carefully checked and made consistent. The spatial variable is defined as $\\mathbf{x}\\in\\Omega^s\\subset \\mathbb{R}^{d_s}$, while shortly after, $x_i^s\\in\\mathbb{R}^{d_s}$ is defined. Why is $\\mathbf{x}$ in a subset of $\\mathbb{R}^{d_s}$, while $x_i^s$ is not? And why is $\\mathbf{x}$ bolded while $x_i^s$ is not, although both have the same dimensionality? This should be clarified. The notation of $u^s_t(\\mathbf{x})$ for all values of $x_i^s$ is also confusing and should be better separated.\n- Section 3.1: \"We assume that n is constant across PDE families.\" - This statement would need an explanation on why this is a reasonable assumption. In particular with varying spatial dimensions, this might not be the case, and some more chaotic PDEs require finer discretizations than others.\n- Section 3.2, FNO Embedding Network and Utilizing Pretrained LLMs: it seems unintuitive to map the channels of the FNO to the sequence length, instead of using a spatial discretization, as for instance in Vision Transformers. Similarly, in the output prediction, a linear layer maps the latent space of the LLM directly to the full spatial output dimensionality. As this design choice is very unintuitive, it needs be motivated in more detail. Further, in terms of number of tokens, what is expected to be represented by each token? How do they differ, and how are attention layers helping in this context?\n\n*Would strengthen the work*:\n- The generalization tests to unseen PDEs is limited. If the model is so training efficient, it would make the paper more convincing by doing a cross-validation style of generalization test, where the model is trained on a subset of PDEs and tested on the rest.\n\n*Minor changes*:\n- Page 1, top: \"20,000 training data,\" -> \"20,000 training data points,\"\n- Page 4, top: \"These PDEs could model a range quantities\" -> \"These PDEs could model a range of quantities\"\n- Page 4, Section 3.1: \"each u[...] is solution to a PDE\" -> \"each u[...] is a solution to a PDE\" \n- Page 4, Section 3.1: \"the value of the the function\" -> \"the value of the function\"", "broader_impact_concerns": "Sufficient impact statement provided in A.6.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting, and even ambitious, cross-domain study of combining LLMs with PDE solvers. The reviews and discussion was insightful and supports acceptance.
Decision: Accept (poster)