Machine Learning with Physics Knowledge for Prediction: A Survey
Authors: Joe Watson
Chen Song
Oliver Weeger
Theo Gruner
An Thai Le
Kay Hansel
Ahmed Hendawy
Oleg Arenz
Will Trojak
Miles Cranmer
Carlo D'Eramo
Fabian Buelow
Tanmay Goyal
Jan Peters
Martin W Hoffmann
Venue: Accepted by TMLR ()

Abstract: This survey examines the broad suite of methods and models for combining machine learning with physics knowledge for prediction and forecast, with a focus on partial differential equations.
These methods have attracted significant interest due to their potential impact on advancing scientific research and industrial practices by improving predictive models with small- or large-scale datasets and expressive predictive models with useful inductive biases.
The survey has two parts.
The first considers incorporating physics knowledge on an architectural level through objective functions, structured predictive models, and data augmentation.
The second considers data as physics knowledge, which motivates looking at multi-task, meta, and contextual learning as an alternative approach to incorporating physics knowledge in a data-driven fashion. 
Finally, we also provide an industrial perspective on the application of these methods and a survey of the open-source ecosystem for physics-informed machine learning.

Published in Transactions on Machine Learning Research (05/2015)
Machine Learning with Physics Knowledge for Prediction: A Survey
Joe Watson 1.2.,, Chen Song", Oliver Weeger".1 Theo Gruner 1.3 An T. Le', Kay Pompetzki', Ahmed Hendawyl Oleg Arenz', Will Trojaks, Miles Cranmer", Carlo D'Eramo 1.344 Fabian Biilow", Tanmay Goyal", Jan Peters-.2.1 Martin W. Hoffman [joe, theo, an, kay, ahmed, oleg, carlo, con/@robot-learnse {chen. song, marten. w.hoffamnn fabian. buelow, tanmay, (eega)@dde.co abb. com weegerdeps. su-ddrrmstadt..
w. trojakkibm.cc com
Department of Computer Science, Technical Universily of Darmstadt, Germany
Systems AT for Robot Learning, German Research Center for AT (DPKT), Germany Hessian Center for Artificial Intelligence (hessian.AT), Germany
Center for Artificial Intelligence and Data Science, University of Würzburg, Germany Centre for Cognitive Science, Technicol University of Dermstadt, Germany "BBB Corporate Research Center, Mannheim, Germany Department of Mechanical Engineering, Technical Universily of Darmsladl, Germarcy
IBM Research UKI, United Kingdom
Data Intensive Science, University of Combridge, United Kingdon
Now the Oxford Robotics Institute, University of Orford
Reviewed on OpenReview: https: // openreview. forun? id- ZiJYohyLU
Abstract
This survey examines the broad suite of methods and models for combining machine learning with physics knowledge for prediction and forecasting, with focus on partial differential equations. These methods have attracted significant interest because of their potential im- pact on the advancement of scientific research and industrial practices, promising improve- ments to using small- or large-scale datasets and expressive predictive models with useful inductive biasos. The survey has two parts. The first considers incorporating physics knowl- edge on an architectural level through objective functions, structured predictive models, and data angmentation. The second considers data physics knowledge, which motivates look- ing at multi-task, meta, and contextual learning physics knowledge in data-driven fashion. Finally, we also provide an industrial perspec- live on the application of these methods and a survey of the open-source ecosystem for physics-informed informed machine learning.
alternative approach to incorporating
1 Introduction
Prediction lies the heart of science and engineering, and many advances involve the discovery of simple patterns often expressed as symbolic expressions that approximate the evolution of our physical world. Still, these mathematical models are only achieved through some degree of simplification and abstraction, and thus rarely capture the complexity of the physical system in its full fidelity. As result, there is significant interest in learning models from only measurements of the real world, spurring the development of machine learning (ML) (Bishop and Nasrabadi, 2006) for the physical sciences. This field relies on using large datasets of measurements to constrain highly flexible parametric models rather than evoking domain knowledge.
Machine learning methods differ greatly in the small- and big-data regimes. The small data regime, popu- larized since the early days of machine learning, covering state estimation, system identification, and kernel

---

Published in Transactions on Machine Learning Research (05/2015)
methods, considers the setting where data is used to estimate few unknown parameters given many assump- tions (Chiuso and Pillonetto, 2019). The big data era, popularized by deep learning, can leverage large-scale datasets to train wer-parameteriieed models that learn their own internal representations of underlying sys- tems (L (L'heureux ct al., 2017). Fueled by the ever-increasing data and compute available, machine learning can now directly learn internal representations, bypassing the need for extensive domain expertise and man- ual feature engincering (Goodfellow et al., 2016). This explosion in data capacity is further complemented by advancements in hardware and software. Development in graphical processor units (GPUs) has increased both the size of models that can be designed and the speed at which they can be trained. Mature libraries optimize linear algebra computations, allowing these complex tasks to be implemented easily and computed efficiently.
However, for applications that require a high degree of reliability, robustness, and trust in their predictions, purely data-driven models could be deemed insufficient, especially when there limited data. A natural compromise is build predictive models that leverage the prior knowledge obtained through centuries of scientific study with the flexibility and scale of modern machine learning (Karniadakis et. al., 2021; Karpaine et al, 2022). Achieving this requires models that can integrate prior knowledge in the form of inductive biases (Baxter, 2000), c.g., additional task-informed structure incorporated into the model, objective, or learning algorithm. This added structure acts as filter, essentially guiding the model to focus on solutions that align with scientific understanding. As a result, the model's search space becomes smaller, with less irrelevant territory to explore. This targeted exploration allows the model to learn effectively even with limited data. The key challenge lies in designing these induetive biases carefully. If the model is too restrictive, the resulting solution might be suboptimal due underfitting. This review surveys such methods to inform machine learning models with prior physics knowledge for prediction.
There are several tasks where such informed models are useful. One is complex forecasting tasks, such as weather prediction, where lots of measurement data are available, and traditional physies models struggle to model such chaotic systems that involve multiple spatial and temporal scales (Schultz et al., 2021; Kurth et al., 2023). Another is solving inverse problems (Ghattas and Willcox, 2021), where realistic physics model can be examined and queried to calculate what input conditions system needs for desired output. These results monitoring is crucial in industry for ensuring the smooth operation of production. By integrating physical models with real-time measurements, these systems assess operational accuracy and predict potential failures, ensuring system reliability (Aldrich and Auret, 2013). This survey covers the diverse ways physics knowledge can be incorporated into ML across datasets D, objectives C, and models ug (Figure 1). models are architectures and prediction techniques informed by domain knowledge, covered sections 3.1,
then inform the design and operation of the system. Additionally, active fault
informed
Section
Solve the Infer the system
Generalize across systems
Inductive bias
simulation
3.1: Differential equations
X
prediction continuous time
3.2: Parameterizing equations
X
physics-inforued model
3.3: Plysics-informed losses
X
physics-informed objective
3.4: Neural operators
X
V
multi-instance learning
3.5: Latent variable model
X
V
X
latent features and dynamics
3.6: System ideutification
x
/
plysics-informed models, objectives
3.7: Model invariances
x
/
physics-inspired invariances
4.1: Multi-task learning
x/x
1/x
learning
4.2: Meta learning
/ x
multi-instance learning multi-instance,, contextual learning
4.3: Neural processes
x
The review is structured to cover the various ways learning can be incorporated into physics- informed learning. Here, learning can refer to the solution (or simulation) of a physical system, inferring the physical system from data, and learning a model that generalizes across several instantiations of the physical system, e.g., different boundary conditions.
Table 1:
2

---

Published in Transactions on Machine Learning Research (05/2015)
D
D
DN
11e
Ulp
11g
U
Informed
C(D,0) 0)
CCD, 6)
LCD, 0)
C 0)
L 0)
Uninformed
Physics-inforeed models
infoice-informed losses
Physics-infrmmed data augmentation
Data-driven generalization
Learning with contextual data
This survey covers the diverse ways physics knowledge can be incorporated into ML across datasets D, objectives L and models ug. Physics-informed models are architectures and prediction tech- niques informed by domain knowledge, covered in sections 3.1, 3.2, and 3.5. Learning arbitrary models with physics-informed losses is discussed sections 3.3 and 3.6. Data angmentation can also incorporate physics knowledge to encode invariances (section 3.7). Data-driven learning across multiple datasets and experi- ments includes neural operators (section 3.4), multi-tash learning (section 4.1) and meta-learning. Finally, contextual data another data-driven approach incorporate additional domain knowledge, as seen in operator learning (section 3.4) and neural processes (section 4.3).
Figure 1:
3.2, and 3.5. Learning arbitrary models with physics-informed losses is discussed in sections 3.3 and 3.6. Various ways encode physics knowledge in the form of model invariances are covered in section 3.7. Data- driven learning across multiple datasets and experiments includes neural operators (section 3.4), multi-task learning (section 4.1) and meta-learning (section 4.2), and has been used for both physics-informed and uninformed models and losses. Finally, cantextual data is another data-driven approach to incorporate additional domain knowledge, as seen in operator learning (section 3.4) and neural processes (section 4.3). This review is structurod cover many these cases, as shown in Table
Contribution. Machine learning that leverages prior knowledge from sciences and engineering a rapidly developing field. Several surveys have already been conducted, focusing on specific techniques such as physics-informed neural networks (PINNs) (Karniadakis al., 2021; Pateras al., 2023), neural operators (Huang al., 2022a), Koopman operators (Brunton al., 2022), or exploring how best to combine scientific knowledge with machine learning methods (Von Rueden et al., 2021; Seyyedi al., 2023). Wang and Yu (2021) recently provided a survey that focuses primarily on deep learning models for dynamical systems with a focus on architectures and objectives, and therefore overlaps with our survey in areas such as physics- based equivariances.
However, our survey takes a broader machine learning perspective, investigating how prior physics knowledge can be incorporated through not only architectures, but also datasets, and the combination of the two. The survey designed to fulfill several key objectives. Firstly, our aim is to identify and communicate the outstanding challenges in physics-informed learning to machine learning researchers, providing a pathway for future research directions. Secondly, the survey offers critical guid- ance, positioning physics-informed learning concepts within the broader landscape of machine learning and connections between various engincering domains. Thirdly, we establish refined method taxonomy, provid- ing a structured framework for calegorizing and distinguishing the different approaches. Additionally, the survey assesses frameworks for evaluating the industrial relevance of these methods and use cases. Lastly, we identify future trends that could shape industrial applications. Through these contributions, the survey seeks bridge the gap between theoretical research and practical implementation, making physics-informed informed machine learning more powerful and useful to tackle tough real-world problems.
3

---

Published in Transactions on Machine Learning Research (05/2015)
2
Background
This section delves into the technical concepts foundational to this review, with particular focus on the physical and mathematical modeling techniques employed in sciences and engineering.
2.1 Ordinary Differential Equations
An ordinary differential equation (ODE) involves functions of one independent variable and their deriva- tives (Hale, 2009). ODEs emerge across broad spectrum of scientific and engineering disciplines, modeling phenomena that exhibit temporal changes in quantities. Examples include the motion of celestial bodics, population dynamics, chemical reactions, and mechanical systems, among others. In many cases, the re- lationship between a (scalar or vector-valued) function and its rate of change f can be expressed (or re-formulated) in terms of the initial value problem of first-order ODE as:
du = flt, T = [0,
(1)
di
(0) Mg-
(2)
Solving ODEs involves finding a function u(t) that satisfies the given equation (1) under specific initial conditions (equation (2)). Those methods often categorized into two families: analytical and numeri- cal. Analytical approaches, such as separation of variables, characteristics equations, integrating factor, eigenfunction methods, aim to find exact solutions under deep insights into the equation structure. When analytical solutions are not possible or intractable, numerical methods become crucial to approximate the solution by breaking down the desired interval into smaller steps. In the numerical solutions of ordinary differential equations, they are commonly categorized into explicit and implicit schemes, each possessing distinct features and applicability. Explicit methods compute the state of the system at future time solely based the current state, it makes them straightforward and computationally efficient. For example, the classic Euler and Mannge-Kutta methods. These methods are generally preferred for problems that require computational speed and do not exhibit stiff behavior. In contrast, implicit methods involve solving one or more equations to find the future state, since the future state itself is also part of the equation. This design makes these methods, such as backward Euler or Crank-Nicon Nicolson, more computationally expensive but significantly more stable compared explicit methods. They are particularly useful for stiff equations where explicit schemes might fail or require very small time steps maintain accuracy and stability.
Furthermore, solving systems of ODEs is crucial for analyzing complex systems, such as when addressing partial differential equations (PDEs), also section 2.2, where the method-of-lines emerges as key tech- nique. This method converts PDEs into ODE systems by discretizing the spatial domain and considering time as a contimuous variable, simplifying their solution. For further details, see Ascher and Petzold (1998).
2.2 Partial Differential Equations
While ODEs provide a robust framework for modeling time dependent processes, many real-world phenomena involve interdependencies across multiple dimensions. PDEs extend the principles of ODEs by incorporating spatial variability and are used to model systems where the state depends on multiple variables. This section covers the general form of PDEs, including boundary and initial conditions, and discussos various numerical methods for solving them, highlighting their critical role in modeling.
The general form of partial differential equation for dynamical system such as heat conduction, fluid dynamics, structure mechanics, electromagnetics, can be defined as (Evans, 2022),
La[u(x)] f(x),
E S
(3)
du
ei (x) + cz(x)
b(x).
E a2.
(4)
dn
La is a differential operator that contains partial derivatives of the function Ω - Rdd with respect spatial coordinates and (potentially also) time, and Ω is the space-domain domain containing the set of all
4

---

Published in Transactions on Machine Learning Research (05/2015)
points in both bounded space and time interval over which equation (3) is defined. f(x) represents source term that may influence the system's 's behavior, and a :2 -> Rda are the parameters of the differential operator, which could vary across æ. Moreover, b is the boundary condition and n is normal to the spatial boundary. Notably, equation (4) addresses the initial conditions at the starting time and accommodates three principal lypes of boundary conditions, i.e. Dirichlet, Neumann, and Robin boundary conditions depending on the value of e₁ and C2-
While the analytical solution of PDEs is restricted to very specific problems, there have been enormous developments in numerical methods for approximating and these methods rely on discretization, which is a process that transforms the continuous nature of space and time into scries of discrete points. Spatially, the common approaches are finite difference (LeVeque, 2007), finite volume (LeVeque, 2002), and finite element methods (Hughes, 2003). Temporally, one breaks the time interval into multiple steps. Here, methods like the method of lines convert the PDE a system of ordinary differential equations solvable by techniques such as Rungo-Kutta. Even a finite difference scheme can be used for temporal discrctization. Beyond these separale approaches, advanced methods like finite element space-time methods tackle both spatial and temporal aspects together, offering increased efficiency and accuracy for complex problems. Needless to say, there are many numerical methods developed in approximating u in a PDE, we only focus on the few common approaches in this review. The finite difference method (FDM) approximates derivatives by differences, essentially replacing continuous changes with discrete steps. Tt then solves the resulting equation on a grid of points (Thomas, 2013). The finite volume method (FVM) similar to FDM, focuses on the conservation laws over discrete volumes, such as fluid dynamics and heat transfer problems (Eymard et al., 2000). The finite element method (FEM) breaks down the domain into smaller, simpler parts called elements. Within cach element, the solution is approximatod using specific basis functions that can accurately represent the behavior of (Szabó and Babuška, 2021). Spectral methods, which employ globally defined high-order basis functions, offer exceptional accuracy for problems with regular geometries. These methods represent the solution as a sum of mathematical functions like Fourier series or polynomials on the domain (Canuto et al., 2007). In contrast, mesh-free methods do not require a pre-defined mesh. They employ scattered points throughout the domain to represent the solution, making them well-suited for problems with complex geometries or discontinuities (Griebel et al., 2005).
2.3 System Identification
Solving ODFs and PDEs is sometimes referred to the forward problem, simulating a physical system to satisfactory fidelity. The inverse problem, inferring the parameters of physical system from data, is also of interest (Colton et al., 1990). This setting has many names. System idenlification is the classical name from the control theory literature (Aström and Eykhoff, 1971), where typically some domain knowledge is assumed and only control input and measurement data is available. In more modern machine learning terminology, model learning or regression can also be used (Bishop and Nasrabadi, 2006), but these terms apply to a wider set of problems that do not necessarily involve timeseries data, domain knowledge or dynamical systems.
System identification involves defining the dynamics of system through sequence of inputs wi, measure- ments Yt: and an initial state Φ1 over finite horizon T as trajectory, i.e. Y = /......, ,rr] (Tangirala, 2018). Typically, Markovian dynamical system is assumed, i.e. the state #++1 = fo wy) and the obser- vation ye- =ge(#;) are both parameterized by 0, and the mapping between states and observations g is often known by design to simplify the learning problem and ensure observability. Moreover, additive noise terms may be added to capture uncertainty in the dynamics and sensors, resulting in likelihoods p(#++1 ot, w. O) and p(ye | az, 0). As the trajectory data is not independently and identically distributed (IID), a typical regression approach on the dataset may not produce satisfactory results, as the forecast may drift due to accumulating prediction errors over time. One possible solution is backpropagation through time,
K T-h h ( (y go (So o...o so (a: (k) , (. (1) ))),
LEPTT (0) = - -E
(R)
A=1
this loss is considered as the prediction error L for K trajectorics over a time horizon h. This captures the prediction problem's sequential nature but requires a more expensive objective that may be harder to opti- mize. A probabilistic approach considers the joint distribution p (Y, X W,0) which, continuing the Markov
5

---

Published in Transactions on Machine Learning Research (05/2015)
assumption, factorizes into p(z=)II(= p(y: xt, ()p(241 Øt, wt 0). The latent states X are inferred using Bayesian filtering and smoothing algorithms (Särkkä and Svensson, 2023), and the model parameters O are optimized to maximize the log marginal likelihood of the measurements Limi( = log p X W,0) dx using algorithms such as expectation maximization (EM). Approximate inference techniques are required if the posterior pl X Y, 0.0) can not be computed in closed form.
2.4 Function Approximation with Neural Networks
We denote a parametric function fo R2 - Rdd parameterized by some O. This review will focus on the deep learning approaches to function approximation (Goodfellow al., 2016), but we will also touch on Gaussian processes (Rasmussen al., 2006).
A (feed-forward) neural network (FFNN) is comprised of I consecutive nonlinear operations g() R Rd++1 where d1 = dæ and d2 = dy, on internal representations 2 (1)
(T--1) fa( =0 0 g ..... 0 (z).
(6)
Each gld) is specified by a nonlinear function a and an affine transform:
gn..0 (z () = ow; + b.),
(7)
where 0, ( b.} are the weights and biases of the layer and 0 = (0....., 01) } is the set of parameters of the FFNN. a is an element- wise nonlinear 'activation' function, such as the sigmoid function, tanh, and rectified linear unit (ReLU). These models are typically referred to as multi-layer perceptions (MLP).
Given a dataset of imput and output data (zz, 3/ supervised machine learning determines parameters 0 of a neural network (NN) such that y, ~ fo(m;)) This training can be performed efficiently using the backpropagation algorithm, which implements the chain rule efficiently for the network structure using dynamic programming. For training these models, it is common to use accelerated first-order methods such as stochastic gradient descent (SGD) with momentum and Adam (Kingma and Ba, 2014). Random mini- batch updates are also deployed to alleviate memory issues and incorporate random exploration. Many deep learning architectures exist with specific inductive biases well suited to certain lasks.
For sequential tasks that require memory, recurrent neural networks (RNNs) (Rumelhart et al., 1985) have a stateful representation state 2 so that (ye, 1) fo( 2²). A mar-Markovian alternative is self-attention, which computes its own atlention weights that dictate which values inform the internal representation. Mathematically, this is written z sequence of representations. The softmax operation and 'query' Q and 'key' K terms compute the self- attention weights that scale the 'value' terms V. The outer product in the softmax means self-attention scales quadratically in the sequence length. Self-attention can be stacked in a scalable fashion to form the transformer architecture, which capable of learning rich correlations between complex sequences of data, such as natural language (Vaswani et al., 2017).
(oftmax(Qo(Z(11)) (Z(((()))) ( Vo where Z is a
For structured data that can be defined using graph with nodes N and edges E, a graph neu- ral network computes its representations by aggregating across the graph using messaging passing, i.e., 2 = (EjEN. f(()) (00(2((-)) (2(111) = (1-1)) Euy)), applying the function approximator to neighboring representa- tions (Bronstein et al., 2021).
Finally, auto-cneoding architectures are used to learn compressed representation z" of data using an "encoder' 'decoder' architecture, minimizing any reconstruction error from the use of this representation è = (=") x" (z)) Throughout this survey, these architectures are seen as means to design function approximators with relevant structural properties for prediction. For further technical details regarding deep learning, please refer to Goodfellow et al. (2016).
3 Knowledge-Driver Priors from Physics
Many machine learning methods are inspired by domain knowledge, such as the families of geometric deep neural networks (Gerken et al., 2023), e.g., convolution neural networks and graph neural networks, to

---

Published in Transactions on Machine Learning Research (05/2015)
capture equivariant features from data efficiently. This section reviews techniques for efficiently learning differential equations, either through data, the model architecture, the learning objective, or the underlying computation.
3.1 Learning Differential Models from Data
A challenge in learning models of physical processes is that our reality exists in continuous time, while the numerical algorithms require discretization. A useful inductive bias in this setting is to define a black-box model contimuuus time by incorporating integration into prediction. The result is a more flexible predictive model that can learn and forecast on irregular time intervals rather than a rigid discrctization. This approach is referred learning the differential equations.
Ordinary differential equations. The connection between ODEs and deep learning, the neural ODE, enjoys long history. Perhaps the carliest predecessor to 'deep' learning of differential equations from data is the ResNet architecture (i.e., residual flow block) proposed by He et al. (2016), although the original motivation was to mitigate gradient attenuation for parameters in early layers. A ResNet block defines a discrete-time residual transformation
D(+1 = Dt (xx),
(8)
where f is a transformation block with parameters 0r, æ, and 0++1 are the input and output at time step t, respectively. Inspired by the ResNet architecture, Weinan (2017) and Haber et al. (2018) both proposed to parameterize the (infinitesinal) continuous-time transformation
dx(()) = fo(x
(9)
dt
which is equivalent to the discrete-time transformation I++1 I = e fa(x)) with an infinitesimal residual block c f-()) as e + O. From the neural network perspective, Chen et al. (2018) shows that equation (9) is equivalent to a 'continouusdepth" neural network with starting input 2g and continuous weights 0. The invertibility of such networks naturally comes from the theorem of the existence and uniqueness of the ODE solution.
Predicting and training with a neural ODE requires choice of ODE solving technique, which in turn requires discretization of the lime variable, see section 2.1. The power of neural ODE comes from its memory efficiency from training without backpropagation through the solver, thus enabling it to learn sequences of morizoo data efficiently.
Training neural ODEs. require a sufficiently large dataset to prevent overfitting, especially for highly nonlinear dynamics. When data is scarce, alternative strategies are necessary. Classical system identification methods (Voss et al., 2004) estimate structured model parameters using statistical techniques. Meanwhile, hybrid neural system identification approaches (Chen et al., 2021; Forgione and Piga, 2021) incorporate inductive biases, such as physics-based constraints or regularized latent representations, to improve generalization. To obtain the gradient the objective w.r.t. the parameters 0, is required to propagate the gradients through the numerical integration. A straightforward approach to store all of the function evaluations in the forward pass of the ODE and backpropagate through the whole numerical integration. Yet, the space complexity increases linearly with time T, which becomes computational burden for long-time series and high-order numerical solvers (Norcliffe and Deisenroth, 2023). Given scalar loss function L, the adjoint of (tt,, a(t)= = dL/dx(t), having the dynamic derived from the chain rule,
The common optimization paradigm learning neural function approximations
da(t) a(t) (fe()))). dt
(10)
dx
provides memory-effiient way to obtain the gradient of the loss function by integrating additional time-reversed ODE (Chen et al., 2018)
dc = =-f" f% a(t) (fo(
00 dt.
(11)
d0
7

---

Published in Transactions on Machine Learning Research (05/2015)
As the gradient is obtained as the solution of the time-reversed ODE, the gradients can be evaluated without storing any additional data. Thus, the adjoint method optimizes computational resources by balancing time against space complexity. Several works (Gholaminejad et al., 2019; Zhuang et al., 2020; Onken and Ruthotto, 2020) showed that the gradient ostimation based on the adjoint method is error-prone due to the numerical inaccuracies of solving the backward ODE, due to the typical high Lyapunov characteristic exponent of highly nonlinear ODEs. Many improved backward integrators have been proposed to mitigate this issue (Zhuang et al., 2020; 2021a; Gholaminejad et al., 2019). Concurrently, Ott al. (2020) studied the validity of neural ODEs representing continuous flows, true contimuous-time analogs of ResNets. They claimed that valid neural ODE is invariant to solver configurations, achievable through critical step-size adaptations for integrators. Krishnapriyan et al. (2023) then generalized neural ODE architecture to RK4 integration scheme, showing generalization and convergence to true continuous flow.
Augmented neural ODEs. In system identification tasks, where a dynamical system is to be learned from measurements, see section 2.3, formulating a (neural) ODE in terms of the observable quantities is typically not sufficient to describe the system dynamics, see also section 3.5. The idea of augmented neural ODEs (Dupont et al., 2019) is thus to augment the space of (obscrvable) state variables x with additional variables h that lift the dimensionality of the problem:
d [t) dt h((
= fo ((x(8) [t)) [((1) :t)
(12)
Augmented neural ODEs allow observed trajectories to intersect and enable the learning of more general dynamics over the observed state space. Furthermore, according to Dupont et al. (2019), augmented neural ODEs can achieve lower losses, better generalization, and lower computational cost than regular neural ODEs, also since time discretizations can be coarser as the learned rate functions fo are smoother. However, challenges are the determination of the size of h and training without knowing the values of h.
Partial differential equations. Learning and solving PDEs have recently emerged as significant areas of interest, reflecting the complexities inherent in accurately capturing spatiotemporal dynamics across ex- tremely fine temporal and spatial scales. In contrast to the ODE operator (equation (9)), PDEs involve working with infiniteddimensione spaces. Specifically, the learning process involves creating function that maps between two function spaces, representing the input data (like initial conditions) and the output solutions of the PDEs.
We highlight that the adjoint method has been used in the context of gradient-based shape optimization (Zahr and Persson, 2016). However, utilizing the adjoint method for learning PDF is unexplored. Farliest works propose to learn a convolutional neural network Zhu and Zabaras, 2018; Bhatnagar et al., 2019). Building these approaches, reto-regrossive methods have been introduced to estimate the solution at discrete time steps u( / + At) = so (1), u( ()), and either, mix with conventional numerical methods in classical PDE solvers (Bar-Sinai et al., 2019; Greenfeld et al., 2019; Hsieh et al., 2019), or propagate the solution through time by recurrent calls of the networks (Brandstetter et al., 2022). All previous learning PDE methods have in common that they rely on a discretization of the domain, yet in many scenarios, it is desirable to evaluate the solution at any point in the domain. For these reasons, learning approaches that directly learn a mapping from a point in the domain to its solution uo have been actively explored in recent years (Yu et al., 2018; Raissi et al., 2019; Bar and Sochen, 2019), which introduces physics constraints losses, is covered in detail in section 3.3. Yet, these models come at the cost that they provide solutions for particular PDE instance and do not generalize to different inputs. Addressing these contemporary issues of learning PDEs, directly learning the solution operator of the PDE as a map between dimensioall spaces has gained traction in recent years (Lu et al., 2019; Li et al., 2020a; 2021a; Gupta al., 2021). These models extend the fundamental work of Chen and Chen (1995) that first showed universal approximation theorem for learning operator networks. These models are explored in more detail in section 3.4.
map between finite Euclidean spaces fe (Guo et al., 2016;
8

---

Published in Transactions on Machine Learning Research (05/2015)
3.2 Learning Models with Algebraic Structures
Rather than explicitly solving the differential equation, parameterizing the structure of the differential equa- tion to guide the learning problem of inferring the differential equations from algebraic expressions is an emergent line of work. In particular, we first focus on learning the dynamical system equation as an ODE derived from Lagrangian or Hamiltonian constraints, which ensure energy conservation. The ODE is then only implicitly described with various neural networks encoding partial derivatives utilizing AD. This achieves sample-efficient model learning and efficient computation within a single forward pass and greatly enhances stability and generalization, benefiting many robotics control applications. Then, we shift the focus recent specific line of work parameterizing differentiallalgebraic equation, which models more general math- ematical laws from observed data. We do not discuss the related but distinct task of using machine learning methods to discover symbolic models that explain observed measurements (Cranmer et al., 2020a).
Parameterizing dynamical system equations. Liu et al. (2005) was the first to introduce Lagrangian constraints into the optimization problem to enforce physics consistency for learning the equation of motion. Contrasting to the physic-informed neural net paradigm, they typically parameterize the general Fuler- Lagrange equation from the Lagrangian C(q. q) =T q) -V(g)= const, for each index i of the coordinate q,
0C OC dq: 0q:
d
dq: T.
(13)
where T is the generalized force. Liu et al. (2005) optimize directly the system trajectory and dynamic parameters, minimizing the violation of the constraint from equation (13). Much later on, by realizing the rigid-body kinctic energy T(q. Q) = T q under gravitational potential field dV/dqq = g(q),
Lutter and Peters (2023) propose deep Lagrangian networks (DeLAN), parameterizing directly the inertia tensor Mo(g) with positivity constraint ensuring dynamical stability property, and the gravity force go (Figure 2). Combining automatic differentiation with the state partial derivatives, the derived derivatives w.r.t. time and, thus, the violation of Lagrangian constraint are efficiently computed in a forward pass (see Figure 2), effectively facilitating robot. model learning with sample elficiency. Cranmer et al. (2020b) then generalizes DeLAN to the Lagrangian Neural Network (LNN), which assumes a more general form of kinetic energy rather than rigid-body kinetics. The LNN works by directly computing the matrix inverse of the Hessian of a learned Lagrangian
0²c ) and computing the implied acceleaation (0 )
L 0q oq
TCc 3q
L 0q 0q
q
(14)
where the inverse is typically approximated with Moore- Penrose pseudoinverse. the Moore-Peuross pseudoinverse may cause numerical instability due to ill-conditioned Hessians and suffers from poor scalability in high dimensions (Golub and Van Loan, 2013). The training is then done via backpropagating through a loss between the true and predicted q.
However, employing
Concurrently to DeLAN, Hamiltonian Neural Networks (HNN) were developed (Greydanus et al., 2019), which enforce energy conservation via the constraint violation of the separable Hamiltonian H(g.p) T(q,p) + V(q)= const. In particular, the violation of the Hamiltonian equation of motion on the phase space represents the training loss
CHNN (0) = oHo oq TMo
0p at
(15)
Op at
ōq
where is now the canonical coordinate, and is the state momentum. HNNs typically paramcterize the Hamiltonian Ho with a neural network and utilize the AD similar to DeLAN to compute the partial derivatives to minimize equation (15).
Toth et al. (2019) applics the same concept of HNN to discover conservative law in pixel observations, effectively learning the Hamiltonian flow mapping between the initial condition distribution to any time state distribution. In the presence of observation noise, the symplectic recurrent neural network (SRNN) (Chen
9

---

Published in Transactions on Machine Learning Research (05/2015)
VoC(D,
((2,0)
go(4)
gn.
En(T 9n M,d. LV,L, L V. 14n +...
Mn
Lo(g)
Vylin
Figure 2: A schematic of deep Lagrangian networks for rigid body physics. Through careful use of auto- matic differentiation, the Lagrangian loss can be constructed from the state variables and rigid body terms, avoiding the need to disferentiate the model with respect to time. The predicted inertial matrix M(q) is also guaranteed to be positive semi-definite through careful parameterization.
et al., 2019) harnesses the symplectic integrator (e.g,, leapfrog integrator) to rollout multiple trajectories from initial states, then performs crckpropagationn-hrongg through-time to train the RNN He w.r.t. the Hamiltonian loss (equation (15)). They found SRNN robustly learns the Hamiltonian dynamics under noisy state observations.
Jin et al. (2020) then generalize previous works for identifying both separable and non-separable Hamilto- nian systems from data, approximating arbitrary symplectic maps based on appropriate activation functions. Notably, all mentioned works assume energy-conservative autonomous systems; however, many real-world dynamical systems depend explicitly time and dissipate internal energy. Pomi-ooamln networks (De- sai et al., 2021; Neary and Topeu, 2023) was proposed to parameterize the phase-space ODE
[
0 ! ] - D(a))
8H
O G(g)
④ -
0g
(16)
dp
with neural networks on the damping D matrix, the Hamiltonian H. the generalized force G(g)u, effec- tively recovering the underlying stationary Hamiltonian, time-dependent force, and dissipative coefficient. Recently, Roth al. (2025) introduced the concepts of Lyapunov and global stability of dynamic systems into Port-Hamiltonian NNs by enforcing the Hamiltonian to be convex and possess strict minimum. This additional physical bias and mathematical structure greatly improve robustness of training and predictions as instabilities are avoided, enhance extrapolation capabilities, and reduce model variance.
Parametcrizing differential algebraic equations. The aforementioned methods explicitly evoke the equation of motion derived from Lagrangian/Hoan Dograngian/Hammilooiaan Hammitooiann frameworks in the forward. However, discovering general symbolic rules from data is challenging. At the time of writing, an emergent line of work under the neural network (ALNN) (Hajij et al., 2020) parameterizes algebraic structures,
name of algebraically-infone modeling a finite number of algebraic objects with a given set of neural networks {ffe : R - RTM
In essence, the algebraic operators of neural networks can be defined straight forwardly given certain condi- tions. Denoting (R") as the set of networks fe 12" R", the algebraic operators are defined for
concatenation:
(So, x (a,) y) = (Jo, Jo (z)), So, EN(R"), So, EN(R)*, (Jo, + (a)) = (Ja, (æ)) + (o,())),, if So So, EN()),
addition:
scalar multiplication:
(a * (x) * fe, (x)) [fe. feal fo, fe. fe. fe..
Lie brackets
where o is the composition operator, assuming ((R)) is closed under the compositions. Typically, given a set of generators [s and algebraic relations {r:) : a neural network defined for each generator, and the loss can be minimized with standard stochastic gradient descent (Bottou, 2012), satisfying all algebraic relations. For example, one can learn the Tang-Baxter equation (Hajij et al., 2020), given R A x A Ax
10

---

Published in Transactions on Machine Learning Research (05/2015)
and some set A, by defining a neural network So == R and minimizing the algebracc relation as the violation of equation (17). Recently, Moya and Lin (2023) parametrize a class of differentiallalgcrraic equations
(R x idA) (idA x R) (R ida) (idA x R) (R ida) (id x R)
(17)
2), q (ta) Q0- (fo) 20:
(18)
0 g)
where (t)) RRd are the dynamic states and xt) E R the algebraic variables. This class of differential- algebraic equations has been used to model various engineering applications, such power network systems with frequency-dependen dynamic loads or nonlinear oscillations. Under the AINN framework, Moya and Lin (2023) model f. as neural networks and treat both statements in equation (18) as algebraic relations. Given training data on q, they perform a model rollout with an implicit Runge-Kutta method (Iserles, 2009) from the initial conditions, then learn ʃ.g by matching the rollout trajectories against the training data and minimizing the violation of the constraint g(g. z), satisfying the defined algebraic relations.
AINN is still in its infancy state with very few works. However, foresce vast applications of paramcter- izing algebraic rules with neural networks in many hybrid systems (i.e., discrete-contimmous) in robotics engineering applications.
3.3 Learning Simulation Solutions with Physics- Informee Losses
Physics-informed neural networks (Raissi et al., 201 7a;b) mesh-free approach for solving PDEs that uses a NN as function approximator for the solution ug. Therefore, the PDE (equation (3)) is transformed into an unconstrained optimization objective by penalizing violations of the differential equation and boundary conditions through and squared penalties:
((00)=/o (Llouo(()) - f(x) dæ, (s(0)
/ (u)(x)) bb(x) dx, (19)
EVINS cy ( Lo(), ()) -
Jox
where of > 0 and c > are weights in the loss function. Cy referred to as the physics- or residual loss and Co as the boundary loss. Note that these two objectives consider different quantities, so careful tuning of the weights is required to balance the multi-objective nature of the problem. Using these loss functions, learning PINNs can be considered unsupervised, as no data is required. However, physical knowledge is embedded in the loss definition through the specification of the PDE operator La in the residual term and provides a supervision signal during training. In section 3.6, we also discuss an extended PINN objective that includes a data loss term in addition, similar to the boundary condition loss. If Levss = 0, then ug satisfies the weak form of the PDE boundary value problem This approach of 'residual minimization' with function approximators can be traced back to Dissanayake and Phan-Thien (1994) and Lagaris et al. (1998), but has grown in popularity recently due to the maturity of deep learning methods.
In practice, the losses in equation (19) must be computed using finite sums rather than exact integrals, this means that collocation or numerical integration must be applied. Morcover, this approach takes advantage of automatic differentiation (AD) (Rall, 1981), which is already used in training neural networks (Goodfellow et al., 2016). During training, reverse-mode AD is used to efficiently compute parameter gradients from the evaluated ono-dimensional objective to the high-dimenssonal parameter space, using a combination of dynamic programming and chain rule. Reverse- or forward-mode AD can be used to compute Lz)) The choice depends whether the output or input is higher dimensional, respectively, AD is dictated by this'. Forward mode is particularly attractive as it enables derivatives to be computed during the forward pass. Note that second-order derivatives require sequence of forward- and reverse-mode AD.
the complexity of the
Reverse-mode is preferred for deep learning as objectives always have outputs.
11

---

Published in Transactions on Machine Learning Research (05/2015)
N Σ (uo(2nn) b(
Cb VoL((O))
(b(0)
0
u (x)
V vx
Ω
1 Σm= #=== (Laus(2nn) f(x)))
V (x)
N
of (O)
1,(0)
Figure 3: A schematic of physies-informed neural networks, detailing the relationship between the approxi- mated model the solution (x) and the plysics-informed loss terms, where the integral over the domain has been replaced with a Monte Carlo approximation. Using forward-mode automatic differentiation, the required gradients of the solution are used to minimize the residual PDE error term (blue), while the bound- ary condition here only uses the solution values (magenta). However, model gradients may be required for the boundary loss, depending on how the boundary condition is defined.
Variational PINNs (Kharazmi et al., 2019) optimize the weak form rather than the strong form in the Petrov-Galerkin setting. The loss function becomes
M
() = Σ |ALouot um (z) mamm
/ f(z) um (x) dx
(20)
where 8m test functions and M is the number of sampling points. This approach is attractive because it can lower the order of differential operators that the neural network approximates, making the computation of loss functions cheaper. However, the analytic tractability puts severe restrictions on the models and settings that can be considered, and the authors only demonstrate the approach on one- and two-dimensional case of Poisson's equation.
Separable PINNs (Cho et al., 2023a) propose different architecture order for forward-mode AD to be more efficient. Each network takes an element of æ as input and has Q outputs. The final prediction is a product over input elements and summation over network outputs, i.e..
ΣI f4 (zx) g=1
(21)
Since each network has a one-dimensional input, forward-mode AD is efficient. The structure also means that evaluating a d-dimensional mesh of N points requires Nd network evaluations rather than Ad, Since the sum-product mixing function is relatively cheap in comparison to network evaluation, this architecture scales very graccfully as gets large in comparison to prior PINN approachos.
The practical challenges applying PINNs in dynamical systems lie in approximating and optimizing the loss function. Due to the nature of differential equations, the residual loss often not 'fixed' between time steps, it requires the model to be self-consistens with set of its own derivatives (Wang et al., 2021a). This objective means the regression target is not fixed, and the model is regressed against itsell during optimization until solf-consistency is achieved. This 'boosstrapping of a function approximator can lead to unstable optimization in practice2. As a result, sampling and optimization must be designed carefully and jointly to ensure successful optimization.
2 Bootstrapping and function approximation comprises of two parts of the 'deadly triad' in relnforcement learning (Sutton and Barto, 2018) where this model bootstrapping is also performed when computing value functions
12

---

Published in Transactions on Machine Learning Research (05/2015)
Sampling. Approximating an integral across a domain of interest requires extensive care computation in order to achieve good accuracy. Classical interpolation-baseed numerical quadrature methods (e.g., Gauss quadrature) typically prescribe weighted lattice-like evaluation points, which would make PINNs similar in implementation and memory complexity to mesh-based solvers. As an alternative, Monte Carlo approxi- mations are attractive as they provide the flexibility of evaluating points a non-lattice structure. The reduction in memory requirements due to minibatching can be used in conjunction with stochastic opti- mization methods such as stochastic gradient descent, widely utilized in deep learning. However, Monte Carlo's flexibility results in greater variance in the objective estimation, which impedes the convergence of the stochastic optimization. To mitigate this issue, quasi-Monte Carlo (Morokoff and Caflisch, 1995) methods introduce low-discrepancy pseudo-random sequences that significantly reduce variance. Examples include Sobol and Halton sequences, which have shown promise in PINN optimization (Pang et al., 2019).
Nabian et al. (2021) propose an optimization--rienaatee 'importance sampling' scheme. For a given set of points, they sample a minibatch according to the categorical distribution proportional to the loss at cach point. This prioritization is shown to accelerate convergence. However, this is not valid importance sampling scheme w.r.t. Monte Carlo methods but is rather 'active' sampling scheme that incorporates decision-making into the learning process, as done in active learning (Cohn et al., 1996).
Daw et al. (2023) proposes 'retain-resample-reeeee sampling, which carefully maintains a population of collocation points, where points with high residual loss are maintained, and the low loss points are resampled each iteration. They also present a biased sampling procedure that encourages points that match the predicted temporal evolution of the system. They also anneal the domain's horizon during optimization to mitigate the propagation of large model errors over time.
Subramanian et al. (2023) propose an adaptive self-supervision algorithm to enhance the performance and 'trainability' of PINNs. By reallocating collocation points to regions with higher errors and periodically resampling them, the proposed methods effectively reduce prediction errors and address optimization chal- lenges across various problems, without increasing computational costs.
Optimization. Wang et al. (2021b) investigate the training dynamics of PINNs and observed that the magnitude of the parameter gradient increased during training, in part due to the multi-objective loss function. To achieve stable convergence, the authors advocated for learning rate schedule.
Fonseca et al. (2023) studied the optimization landscape of PINNs, and found that their solutions lie in sharp minima, i.e. regions of the loss landscape where the loss varies significantly under small perturbations of the solution. This is in contrast to folklore in deep learning, where solutions in flat minima are preferred as they are interpreted as solutions that are not overfitting and, therefore, generalizing. Since PINNs models are trained in their domain, the issue of out-of-distribution generalization is less of a concern. This difference results in typical deep learning optimization techniques being less desirable. The authors found that quasi- Broyden Broyden-Fletcherr- Glolfabr-Gollffrbbb
Newton solvers such as the superior optimizers due to their strength in reaching sharp minima by incorporate loss curvature.
algorithm (LBFGS) were
Krishnapriyan et al. (2021) highlight the boolstrapping issue by isolating the residual loss term as the most difficult w.r.t. the combined PINN loss landscape and show that optimization can be improved by gradually annealing this term to be more significant during optimization. They also advocate modcling the solution autoregressively, i.e., U1td = fa, At), so the prediction can be bootstrapped from un, and the function approximate does not need to fit the solution for the entire time horizon.
Another approach to mitigating the bootstrapping problem is ensembling models. Haitsiukevich and Ilin (2023) trained an ensemble of PINN models and averaged their predictions in the residual loss term. By averaging over predictions, the errors from bootstrapping were reduced.
The temporal aspect of the bootstrapping issue has also been mitigated by introducing a base- weighting scheme to the residual loss (Wang et al., 2022a), which attenuates the residual loss at a point it demonstrates relatively high loss when integrated from the boundary condition, i.c.,
N
i-1 cr((x, 01) C Cy(l,, 0),
(0)=
exp
(22)
1=1
13

---

Published in Transactions on Machine Learning Research (05/2015)
where c > 0 controls the attenuation. Note that this loss is relevant to temporal PDE cases with known initial conditions.
Choosing the weights of each loss term is another practical challenge since different loss terms converge at different rates. Therefore, it is difficult to identify the optimal relative weights for each term, which has a al. (2023)).
significant impact on the training stability of the PINNs and predictions as well (Faroughi Bischof and Kraus (2021) proposed a self-adaptive loss balancing scheme named ReLoBRaLo (relative loss balancing with random lookback). This approach calculates the progress of each loss term by comparing it with the previous iteration values. Using 'random lookback,, the method can decide whether to consider the progress from the last iteration or the progress from the start of the training process. This approach enables multiple loss terms to be optimized simultaneously in an efficient way. Xiang et al. (2022) proposed a probabilistic interpretation to define the self-adaptive loss function through the adaptive weights for each loss term. The weights are updated in each epoch automatically based on maximum likelihood estimation.
PINNs have difficulty in learning search systems due spectral bias, which makes them learn low-frequency patterns in the data (Mustajab et al., 2024). Furthermore, even when trained several initial and boundary conditions, they still suffer when new extrapolated initial and boundary conditions are there (Nakamura et al., 2021).
If these implementation issues discussed above can be overcome, researchers can fully unlock the practical value of PINNs as an alternative to mesh-based solvers. They provide a less accurate but computationally cheaper solution that may be desirable in certain applications. The generality of the approach has led to extensive interest from the scientific community (Cuomo et al., 2022) due to the numerous application domains, such as fluid mechanics (Cai et al., 2021a), heat transfer (Cai et al., 2021b), power systems (Huang and Wang, 2022). There is an open quest.io regarding how and when PINNs are viable replacement to standard PDE solvers. Grossmann et al. (2023) benchmarked PINNs and FEM methods for several PDE test cases and found that, on the whole, the FEM method was faster (around x10) and more accurate (around 100x). The benefit of PINNs is that they are sometimes faster to evaluate and may potentially scale more gracefully to higher-dimennsional problems. However, given their poorer accuracy on lower-dimensional problems, they are not guaranteed to provide high accuracy in this setting.
3.4 Operator Learning
So far, this survey discussed learning the solution ua( to differential equation for specific configuration (possible) boundary conditions (equation (4)), and source terms f. However, this approach requires solving the system anew for each new configuration. A more efficient alternative is to learn an operator a model that maps an infmite-dimeesionaal function-space input to the corresponding solution. This enables generalization across different configurations without solving the differential equation from scratch. Specifically, we introduce learning a solution operator f(x) = ua(x) that maps from state-varying functions a(x)= Ω Rdd to the corresponding solution (Figure 4). Operating on functions introduces the same flexible discretization invariance discussed in section 3.1, but now extended beyond time to any continuous state space. Moreover, the training data for such models now extends across several instances of the physical system, so the model is extended in a infermee fashion to be conditioned on the system paramcters. We will revisit multi-instance learning in section 4 in the context of learning data-driven priors.
While the parameterized solution operator approximates a mapping between two function spaces So : A - U, its implementation is a point-to-point mapping evaluating the solution u for a given input function a point x. The training data. is commonly available from discretization of the domain D; {z, ?) for a specific inpul a; μ(a). Information about the functions and u are typically only available at point-wise evaluations. Thus, the loss is the relative L loss for K different inputs a; at the discretization points 2j E D.,,
[[Sa;(,,) (a;;
(0) =
(23)
[I,a;
14

---

Published in Transactions on Machine Learning Research (05/2015)
Operator Learning
DeepONet
Sya(z) go 4P (a(
NO
Spa(x)) Q Ly o... L4 Pa(x)
a: 12 R.
R &
R
9o
M1.M
(2)a)(x)
ua (2)
JF E Ω
C
PP
Uu (m)
P
V (o)
N E 2-ij-1 (Mai) (a:) (2,()
13
Figure 4: (Left) A schematic of operator networks that learn a functional mapping from the inputs a to the PDE solution at designated point æ. (Middle) DeepONets can approximate any nonlinear operator by learning a finite-dimensional mapping û = po(ā) in latent space which is spanned by lincar encodings à = (a) and linear decoding u = ge (t, æ)) (Right) Neural operators directly map a discretized representation of the input a1: to the solution at an arbitrary discretization U1:a-
Kovachki et al. (2023) report that training with the relative I,2 loss equation (23) is less prone to overfitting due to normalization compared to the alternative mean squared error (MSE) loss. The specific parameteri- zation of the operator networks has been the primary focus recent approaches to guarantee discretization invariance, which has a bounded generalization error while being computationally efficient. Following this, we will discuss the two prevailing operator learning architectures, DeepONets (Lu et al., 2019) and neural op- erators (Li et al., 2020a) highlighted in Figure 4. Although both architectures follow universal approximation theorems to learn arbitrary operators, their conceptual design is fundamentally different.
Operator learning with finite-dimensional mappings. In order to deal with difiniteddimessiona input and output spaces, deep operator networks (DecpONets) (Lu et al., 2019) are based on the idea of learning the solution operator in a finitedimenssooaal space 9 Rdd + Rds This requires a functional mapping of the input functions a into a latent representation e - Rd and a respective functional mapping g : Rds - ll that docodes. Thus, the operator learning architecture based on the finite-dimensional mappings can be formulated as Sœa 9e P eg. For this architecture, Chen and Chen (1995) first showed that for linear maps eg and go, and single-layer network yo; there exist fmite-dimennionall latent spaces dg and Iddo for which any nonlinear operator can be approximated. This universal approximation theorem for operator learning has recently been expanded to encompass arbitrary deep-lcarning architectures for mapping in finitedimennsional spaces (Lu et al., 2019). The different embodiments of the three modules lead to distinct approaches, all underlying the same concept of finite-dimensional mappings.
DeepONets (Lu et al., 2019) encode the input function by evaluating a on so-called sensory points {a! (9) de Jis as e a (a(x:!))... ((n!de))r The encoded input representation is processed by any deep neural architecture spg. Finally, the decoder is defined as the scalar product between the output of the finite- dimensional mapping and a latent encoding of the domain ge(x) = Ee 4e vi (z). Thus, the decoder approximales the solution operator Sa(r)) R (o(),, where ya and tho are parameterized by neural networks. In the literature, the fiite-dimensional mapping &e is commonly referred to as the branch net while the denotes the trunk net. The scalar product between the branch and trunk network yields scalar output, yet, the DeepONets architecture can be easily extended to rector-valued outputs by adding channels to the branch and trunk nets (Lu et al., 2022). Wang et al. (2022b) propose an improved DeepONet architecture with empirical advantages over the vanilla DeepONet architecture. In particular, the authors argue that the fusion of the encodings of the input parameters a and the domain representation æ only happens in the final scalar product step. They proposed intertwining the two input sources earlier. While empirically, the proposed architecture improved the approximation results, the architecture lacks the common universal approximation theorem that related models have shown.
Bhattacharya et al. (2021) propose PCA-based encoding and decoding schemes for the input and output spaces. The encoder projects the Hilbert space U onto the finite-dimensional sub-space (a, Ka, (,")),..,(, (a, ve odde)lJ spanned by de basis vectors (.) Given evaluations of at N observed
15

---

Published in Transactions on Machine Learning Research (05/2015)
data points æ, vs correspond to the de eigenvectors of the empirical covariance matrix C - a aa with the largest eigenvalues. Approximate reconstruction can be obtained through a ~ da (u, V y By also projecting the solution space U into a finite-dimensional space using PCA, the decoder can be formulated as g(x) = Σ Po(e(a))ng ( with ( being the cigenvectors of the covariance matrix of u.
Similar ideas within the framework of reduced-order methods have been presented by Peherstorfer and Willcox (2016). This method, known as operator inference, involves projecting the state-space onto a lower- dimensional representation using proper orthogonal dccomposition (POD) and subsequently learning the linear operators of systems of nonlinear ordinary differential equations (ODEs). However, these models often lack the generalization capabilities seen in more recent methodologies, such neural operators.
Neural operators. As noted by Kovachki et al. (2024), the previous methods rely on a finiteddimensional representation of and U, which might be restrictive as they describe mappings in the linear subset of A and u. The neural operator (NO) architectures presented in this section do not rely on finitediimensional mapping and, thus, are more principled. Neural operators are motivated by the theory of inhomogeneous linear differential equations. If La in equation (3) constitutes linear operator, the solution of u is characterized by Green's 's function
u(xx) u(xx== - G(x, () dy + Whomo (x).
(24)
Here, G(æ,, y) is the response impulse of the linear operator L.G( y) 8(x - y) and Whomo (a:) is the homogeneous solution which only takes values on the domain boundary An Generally, Green's function G( y) is not trivially available. Therefore, several works propose approximating Green's function with a neural network Ga(æ, y) (Boullé et al., 2022; Zhang et al., 2022a).
In numerous relevant applications, the PDE operator exhibits nonlinearity, rendering equation (24) inapplica- ble. Although approachess utilizing encoding and decoding schemes to transform the problem into sub-spaces amenable to linear operators have been suggested (Gin et al., 2021), they lack the theoretical assurances inherent in operator learning methods. Yet,, the idea of using integral kernel operators play pivotal role in designing neural operators with theoretical guarantees
Kv(x)) = f <(x,, ((())))(())
(25)
where k(æ;, y) D x D' + Rd x Rdd is a kernel function, and v(x): : D + Rd is a vector valued function. Each integral kernel operator maps v(x) to Ku(x): D' + Rd Here, D and D' refer a representation of the domain. The neural operator architecture is split into three sub-components: a pointwise lifting operator P, sequence of kernel integrations layers L; t= 1,....,, ,, and pointwise projection operator Q. The lifting operator maps the input function to a hidden projection vn(x) Pa(x). The series of kernel integration layers processes the lifted representation vx(x)) = L,0(-1(x), and is finally projected to the solution u(x)) = Qxr(x). Here, a pointwise operator means that the operator is applied on each action scparately, i..., Pa(x) = P(a(x)). The full architocture is a composition of the previously introducod operators,
(Sea) (w) = (Q L7 o Ll Pa) (,)=1( = (W,,, + Kvv,-1 + b.).
(26)
(27)
The kernel integration layers are composed of a linear operator, represented by the matrix W. xde.-
bias term b, E Rd=,, and the integral kernel operator (equation (25)). By parameterizing P, Q. W, by Ri:
the neural operator can be shown to approximale arbitrary nonlinear operators (Kovachki el al., 2023). Although NOs are not restricted to practice, NOs work on discretizod representations of the input domain a1.N- The main reason is that the integral kernel operator (equation (25)) remains computationally demanding operation. Therefore, we will examine different architectures to approximate K efficiently.
mappings, as noted by (Kovachki et al., 2024), in
16

---

Published in Transactions on Machine Learning Research (05/2015)
Parameterization of the Kernel Integrations. Numerical quadrature of the integral in equation (25) requires matrix-vector multiplications on the order of ((N2) operations, where N is the number of uni- formly sampled discretization points 32, Yj in the domains D;, Dr-1. Thus, recent approaches leverage the theoretical findings of kernel methods and spectral methods to construct efficient neural approximations of the integral kernel operator K.
Li et al. (2020a) propose graph neural operators (GNO), which are based on message-passinn passeng graph neural networks (Bronstein et al., 2021) to represent the neighborhood of data-point æ. Based the J nearest neighbors of the data point, the kernel integral can be efficiently computed using Nyström approximation and truncation. With the assumption that J < N, the computational efficiency can be significantly increased, even though it still requires matrix-vector multiplications in the order of (JJ2). Multipole GNOs (Li et al., 2020b) extend GNOs with the fast multipole method that enables a hierarchical graph structure that is capable of capturing global phenomena.
Instead of evaluating the kernel operator in the domain space, Fourier neural operators (FNOs) (Li et al., 2021a) leverage Fourier transformations to evaluate the integral kernel operator in its spectral representation. Assuming x(x, y) k(æ y) and applying the convolution theorem, the integral kernel operator K can be reprosented as
((F(x)))((((-))((a))) = / ne --y--yy--- y)m = K.,
(28)
where F and F-1 denote the Fourier transformation and inverse Fourier transformation, respectively Like GNOs, a discretized domain representation facilitates an efficient evaluation of equation (28) by using fast Fourier transformations (FFTs). Given an equidistantly spaced discretization of the do- main [= the Fourier coefficients of the discretized Fourier transform of the function u:-1 are -N-1 (1-(z//))-22ij///00 E R Likewise, we denote the Fourier coefficients of the ker- (k) = =5= nel by (k) E Rd xde Finally, the convolution can be expressed as
N
FF(((())) F(v)-1))(x)
((())))-1(k) exp(2mikn/NN).
(29)
Instead of projecting K₃ to the Fourier space, the kernel is directly parameterized in Fourier space. Specifically, the Fourier coefficients of the kernel are parameterized Rg. Empirically, it has been found that directly parameterizing the matrix Ro E IR" Nxde, xd. yields the best performance. In addition to their natural frequency domain processing, FNOs demonstrate computational efficiency with a sub-quadratic complexity of ONN N).
However, several studies (Fanaskov and Oseledets, 2023; Bartolucci al., 2023) emphasize that choosing the training data from a fixed grid representation leads a systematic bias. In addition, aliasing can occur for FNOs, the maximum frequency mode that be distinguished from lower frequency modes is bounded by the grid resolution. Fanaskov and Oseledets (2023) propose spectral neural operators that use Chebyshev or trigonometric polynomials. Additionally, multi-wavelet operators (Gupta et al., 2021) extend FNOs by utilizing multi-resolution wavelets to represent the kernel operator with a piecewise polynomial basis. This results in a sparse kernel representation, improving the operator learning performance.
In recent years, neural operators have gained considerable attention. Different architectures have been proposed based on the recent success in other research disciplines, such as attention-based (Cao, 2021), CNN- based (Raonié et al., 2023), or VAE-based (Seidman et al., 2023) architectures for operator learning. Finally, strides have been made to make neural operators universally applicable in various domains compliant to specific pre-conditions. As FNOs operate on uniform grids, domain-agnostic methods have been proposed that either map complex topologies to a latent uniform grid for the NO (Li al., 2023a;b) or apply the NO on an extended uniform grid and masking out points outside the domain (Liu et al., 2023a). Furthermore, as neural operators are fully data-driven approaches, there are no guarantees that the learned solution operator adheres to the boundary value conditions of the considered PDE. Therefore, Saad al. (2023) show that specific parameterizations of the kernel operator 25 can ensure compliance with Dirichlet, Neumann, and
17

---

Published in Transactions on Machine Learning Research (05/2015)
periodic boundary conditions while Négiar et al. (2023) propose to employ differentiable final layer which guarantees that. the boundary constraints are not violated. Further geometric biases have been employed with neural operator architectures to be equivariant to rotations, translations, and reflections (Helwig et al., 2023), or explicitly being SE(3)_qquivariant (Cheng and Peng, 2023). FNOs have also been adopted for general machine learning. FNOs have been incorporated into vision transformers to perform 'token mixing' in the Fourier domain rather than in pixel space (Guibas et al., 2022). These adaptive FNOs are developed to be more sparse, use fewer parameters for greater scalability, and have been deployed on weather forecasting tasks (Pathak et al., 2022).
The little regularization required for training and high flexibility of neural operators come at the cost of demanding plenty of training data to learn the desired operator. Therefore, one of the ongoing tasks is to incorporate physical priors to facilitate faster and more efficient learning. DeepONets and neural operators have been integrated with physics-informed losses, see section 3.3, thereby enhancing the data-driven operator learning loss with PDE constraints based on the PINN loss L = aLxo + (1 - CPINN, a E [o. 1] (Wang et al., 2021e; Li et al., 2021b). Setting a 0 recovers PINNs, but. instead of point-wise mapping, an operator is learned that maps an input function a to the solution the auxiliary PINN loss does not always benefit the training of neural operators and may hinder its final performance. Du et al. (2024) highlight the shortcomings of the PINN loss as it requires possibly multiple differentiations through the network architecture. Thus, the authors propose to replace the PINN loss term with a new loss based on spectral theory. The proposed loss evaluates the residual term of the PINN loss in its spectral representations, which can be shown to simplify gradient propagation through the network and thus improve the optimization of neural operators with added physics-inspired losses.
However, Saad et al. (2023) show that
The study of linear differential equations, particularly through Green's functions, has significantly shaped the development of operator learning architectures. Here, nonlinear operator approximation is achieved by compositions of linear operators with nonlinear activations. In the next section (section 3.5), we introduce a related but distinct approach, Koopman thoory, which also leverages lincar operators to approximate nonlinear dynamical systems. Unlike the compositional method, Koopman theory seeks to embed the domain into latent space where the dynamics are governed by a linear, though potentially operator.
Gin et al. (2021) demonstrated the close Lies between operator learning and Koopman theory by proposing to learn latent representations of input and solution spaces, facilitating the approximation of Green's 's function (equation (24)) for the solution of linear PDEs. For discretizations U and F of u(x) and f()) respectively, they propose learning embeddings us = (U) and fø = (F) such that Lus = fa. Solving the PDE in this embedding space is simply U = - (L-b)o)))) providing & and &b can be learned. The subsequent section will discuss the notion of learning effective representations for dynamical systems and the extension of linear dynamical systems theory to nonlinear operators through observable functions.
3.5 Latent Variable Models for Dynamical Systems
Latent variable models (LVM) (Bishop and Nasrabadi, 2006) refers to the assumption that an observed set of measurements are a consequence of an unobserved latent process. Much of statistics, signal processing and machine learning reduce to problems of this form. In the context of learning PDEs and dynamical systems, there is a desire to reduce complex systems into simpler ones using the idea of a latent coordinate system. Thus, LVMs motivate different view on previously discussed methods for operator learning on finite-dimensional embeddings, see section 3.4, and physically-struccure latent coordinate systems, see section 3.2. These methods construct latent coordinate representations enhancing the interpretability of system dynamics and enabling more efficient computational analysis. Building on this motivation, the following section surveys Koopman operator theory, which aims to find a latent coordinate system in which the dynamics become linear (Koopman, 1931; Koopman and Neumann, 1932). The linearization simplifies the forecasting and enables the use of well-established techniques from linear algebra for analyzing Another motivation is that when learning from measurement data, there is dogree of flexibility in the actual definition of the dynamical system and its internal state, and this ambiguity can be leveraged to learn dynamical system that may not reflect the physical laws of the process, but may be easier to learn, capture the observed
18

---

Published in Transactions on Machine Learning Research (05/2015)
Cd
K4
21+2
K
K
K
K
20
22
2T
21+1
g
g
S
S
S
S
uo
ut
u2
UT
Figure 5: Schematic overview of the Koopman operator theory. Left: Other than analyzing the evolution of dynamical system on X, the theory studies linear but potentially infnite-dimensional operator on F E {C, RJ. Right: The graphical model provides an overview of how recent methods often address Koopman theory. Given the observations {u.)/zo the objective twofold: (i) finding a feature representation transforming the observation into a latent coordinate system, and (ii) approximating the Koopman operator basod on latent variables { assuming linear latent dynamics.
data, or use as a prodictive model. In this section, we will discuss learning latent dynamics with both the linear and nonlinear assumption.
Linear latent dynamics. Koopman operator theory provides an alternative perspective dynamical systems (Kutz et al., 2016; Brunton et al., 2021). Instead of analyzing the evolution of the system itself, the theory takes an operator-theoreticc perspective. Consider u(ær, t) E U evolving on the Banach space depending on time and spatial coordinates E IR and a E Ω respectively. Let : 4 + U be the corresponding flow ( () = u( + At), propagating the current state u(x,t) t) by a time At and mapping it to the future state u(x, t + At). Koopman operator theory revolves around an infinite dimensional linear operator K acting Hilbert space H of all observable functionals glua] u - C. The continuous-time Koopman operator is defined by
k> gl = gl ++ At),
(30)
with K- glu being the observable at time + At initiated from gau](xr, t) at time t (Kutz et al., 2016). In the context of Hamiltonian systems, Koopman (1931); Koopman and Neumann (1932) introduced the Koopman operator. Later, Mezić (2005; 2013) extended the theoretical framework to address dissipative and non-smooth dynamics arising from the Navicr-Stokes equation. While the flow S comes from arbitrary and polentially highly nonlinear dynamics, the observable space represents an infinite-dimensional linear system (see Figure 5). Consequently, we can use spectral analysis to extract coherent spatio-temporal structures of the nonlinear system embedded in the observable space. Given that g(u] is a functional, spectral analysis on the Koopman operator for arbitrary PDEs lcads to cigenvalues X C and cigenfunctionals v(u): : C (Nakao and Mezić, 2020). In practice, however, the observables are oflen discretized in space and time resulting in discrete-time dynamical system (Kutz et al., 2018). The system states simplify to finitediimensional vectors u; u(z,, tx) E U and the observables to scalar functions spectral decomposition of the resulting discrete-time Koopman operator Ki+1 leads to to Ki+1 (w:) with {ıb >₂ _ representing the associated eigenfunctions and eigenvalues. Thus, a vector of observables g [g),"** [4,] I E ddz can be expressed by
Ω + C. A
g(u,) = Σ (uss)
Σ (u,)v,
(31)
(1b3, Man)
with the j-th Koopman mode ey E Cd projecting the observables g(4)) onto the j-th eigenfunction. Since the observables are linear combinations of the eigenfunctions, i.e. g E span(()) (2 the temporal evolution
19

---

Published in Transactions on Machine Learning Research (05/2015)
is given by the eigenvalues K** ((,,) = -y=v"y However, K acts on infinite-dimensional Hilbert space - with an infinite number of eigenvalues, eigenvectors, and modes. Therefore, applied Koopman theory aims to identify a finite-dimensional latent coordinate system that is invariant to actions of K (Brunton et al., 2016a). This latent space, spanned by d, cigenfunctions { is termed the Koopman invariant subspace (KIS). Consequently, we obtain a finiteddimensional linear operator K E (d. The approximation depends crucially on the correct choice of observables. For multiple dynamical systems, such as the Burger or Schrödinger equations, we can select specific observables and obtain accurale approximalions of K (Kutz et al., 2018; Nakao and Mezić, 2020). However, a general scheme for finding the observable functions is an ongoing challenge (Brunton et al., 2021).
Algorithms related to dynamic mode decomposition (DMD) provide entirely data-driven approaches to extract latent coordinate systems from time-series data (Schmid, 2022). They decompose complex systems into spatio-temporal coherent structures employing singular value decomposition (Kutz et al., 2016). Rowley et al. (2009) highlighted the connection between these approaches and Koopman operator theory. Given a temporal sequence (u:) they operate as supervised learning schemes, addressing least-squares problems of the form
T-1
J(K,0) = L 0) +
( Lite (K = Σ (ui+1) - K (o + aL C 0).
(32)
Here, J(K, 0) comprises a one-step prediction loss L [K 0) cnsuring lincarity and a regularization loss : Cd= and V Ω Cd represent linear basis models approximating the invariant subspace based on provided feature functions (Ø)(w) : S2 + C) The original DMD framework, introduced by Schmid (2010), considered a one-to-one mapping, i.c. l'(u)= = (u) = u, resulting in linear least-squares problem. Later research, such extended DMD (EDMD) (Williams et al., 2015) or SINDy (Brumton et al,, 2016b), explored more representative features employing nonlinear function approximations. However, choosing Vp' and & is challenging, commonll known as the bias-variance problem in machine learning (Bishop and Nasrabadi, 2006). It results in confficting goals between (i) extracting a valid KIS and (ii) not overfitting to the given data (Otto and Rowley, 2019). Therefore, Li et al. (2017) and Yeung et al. (2019) proposed dictionaries with a partly trainable set of features, i.e. (z)(u) : f2 + C) and k < dz. EDMD with dictionary learning (EDMD-DL) considers shallow networks as function approximations and utilizes an Le-reuularization for 0 (Li et al., 2017). In contrast, Yeung et al. (2019) proposed deep- DMD employing deep learning architectures and L z'-regularization. Batch methods itcrate between ridge regression to estimate K and a nonlinear gradient descent to optimize O. Furthermore, Terao el al. (2021) introduced a variant of EDMD-DL, replacing the MLPs with NODEs (section 3.1). These approaches have demonstrated promising results in capturing chaotic behavior, such as that cxhibited by the Duffing oscillator (Williams et al., 2015). Also, they found practical applications in fluid mechanics, for instance, in systems described by the Kuramoto-Sivashinsly equation (Li et al., 2017). However, they share the assumption of full state observable i.e. g(u)- u. This assumption is implemented in two ways. Firstly, through a linear reconstruction using the estimated cigenfunctions and Koopman modes. Alternatively, by incorporating constant feature d(u)= u into the dictionary. However, significant challenge remains in the robust forccasting of out-of-distribution dynamics. Mouli et al. (2024a) addressed this problem employing a meta-learning framework that learns relevant feature functions across tasks. Similarly to SINDy (Brunton et al., 2016b), their approach employs nonlinear function approximators together with task-specific coefficient matrices to learn the dynamics for each task. In addition, they introduce a iask-independent indicator matrix that learns the global causal structure across all tasks. Given the causal information, efficient adaptation to new tasks is achieved by learning only the task-specific coefficients. This method demonst the potential of meta-learning discussed in section 4, to address challenges such as out-of-distribution dynamics.
C 0) scaled by a E R+ The dictionaries
In recent years, researchers explored the application of autoencoder architectures to extract concise yet informative subspace from time series data (Takeishi et al., 2017; Lusch et al., 2018; Alford-Lago et al., 2022).
As a result, they relaxed the assumption of a full state observable. The linear reconstruction becomes obsolete and a nonlinear reconstruction takes place using the docoder network. The objective from equation (32) extends to
I (0c, O₂, K) a1 CReoon (0c, Os) + az Lpred (0,, 0a. K) + ag Lian (0c, 04, K) + a4 CReg (0,, 0g, K),
(33)
20

---

Published in Transactions on Machine Learning Research (05/2015)
with CRecou 02) and (Or, being a reconstruction loss and a prediction loss, respectively. The former penalizes inaccurate reconstructions of the system state, i.e. u, R (do, )u. The latter aims to correctly predict the next state after propagating the current one through the architecture, ie. u1+1 = (de, o K o co,) Several works, including Takeishi et al. (2017); Lusch et al. (2018) and Alford- Lago et. al. (2022) focused on forward prediction losses Preed (fe: 0d. K) and LLin(ee, f.: K). Azencot et al. (2020) studied the effect of an additional backward prediction loss, i.c. u;-1 (do, D o co,) Jo,).. They introduced an additional operator for the backward dynamics D satisfying K o D D o K = Il. However, these works solely looked at one-step predictions. Otto and Rowley (2019), on the other hand, introduced an autoencoder featuring linear recurrent latent embeddings. The proposed linearly-recurrent autoencoder network (LRAN) enhances the recurrent losses Cpred Od, K) and L0i 02, K) in equation (33) by considering a weighted average over r-step predictions. Similarly, Wang et. al. (2022c) and Liu et al. (2024a) both used recurrent structures to learn KIS and operators. However, they employ a composition of latent coordinate systems, cach designed to capture different aspects of the time series data. The Koopman neural forecaster (KNF) decomposes the embedding to capture local and global behavior patterns separately (Wang et al, 2022c). In contrast, Liu et al. (2024a) proposed hierarchy of Koopman forecaster blocks ("Koopa"). In each block, a Fourier filter separates the imput signal to capture time-varying and fime-invariant components. While these methods were effective for applications like weather forecasting (Liu et al., 2024a) and fluid mechanics, e.g., in modeling wake flow around cylinders (Otto and Rowley, 2019), they showed efficacy in quantifying the impacts of uncertainties in both model construction and forecasting. Therefore, several works, including Morton et al. (2019) and Pan and Duraisamy (2020), proposed using variational autoencoder (VAE) architectures (Kingma and Welling, 2014). These approaches enable the learning of a generative model that closely resembles the distribution over dynamics in the latent space.
Eigenvalues are crucial for understanding the temporal behavior of the dynamics in the latent embedding, Previous works, including Liu et al. (2024a) or Otto and Rowley (2019), have not explicitly addressed stabil- ity. As a consequence, the locally-lincar latent dynamics were prone to instability. Therefore, recent research proposes physics-informee architectures to promote properties like stability. Erichson et al. (2019) suggested an autoencoder framework largeting stable latent. Koopman dynamics, following Lyapunov stability prin- ciples. Pan and Duraisamy (2020), on the other hand, employed syem-symeetric representation of the Koopman operator. This approach inherently forms stable latent dynamics and deviates from Erichson et al. (2019), where stability is integrated as soft constraint within the objective function equation (33). For quasi-periodic systems, works like Lange et al. (2021) express the objective in the frequency domain leverag- ing the fast Fourier transformation (FFT). This dual formulation achieves globally linear latent embeddings by construction. To incorporate further physical principles such as conservation or measure preservation, Baddoo et al. (2023) or Colbrook (2023) reframed the regression problem from equation (32) as Procrustes problem. This formulation ensures that the solution lies on a manifold spanmed by the physically imposed constraints. While demonstrating convergence guarantees and promising results for systems such as the Volterra antegro-differente equation or linearized -tvier-Stokes equations, these approaches rely on the choice of feature functions and assume lincar reconstruction.
In our discussion so far, we have explored methods that use parameterized function approximations, e.g. polynomials or neural networks. However, these methods can be computationally intensive, especially when dealing with high-dimensional data. Autocncoders, as proposed by Takeishi et al. (2017); Lusch et al. (2018) and Alford-Lago et al. (2022), offer a solution by extracting latent coordinate system with linear dynamics. However, using autoencoders adds complexity and potentially increases the non-convexity of the optimization problem. Additionally, they usually perform dimensionality reduction, resulting in lower- dimensional latent embedding. Yet, in Koopman theory, the observable space may represent an infinite- dimensional linear system. An alternative solution to this problem is to utilize non-parametric models. Williams et al. (2014) introduced a version of DMD called kernel DMD by reframing the equation (32) in its dual form. The resulting dual operator K is obtained by inner products in feature space, facilitated by applying the kernel trick. Instead of explicitly defining the features for V, e.g., using parameterized function approximations, kernel function k Ω x Ω : R is employed to implicitly compute these inner products Seeger (2004). Several works, including Kawahara (2016); Klus et al. (2020) and Das and Giannakis (2020), studied Koopman operator theory in reproducing kernel Hilbert spaces (RKHS). Recently, Kostic et al. (2022) and Bevanda et al. (2024) highlighted the connection between these kernel-based methods for
21

---

Published in Transactions on Machine Learning Research (05/2015)
Koopman theory and statistical learning. This connection establishes a natural concept of risk into the estimation of the Koopman invariant subspace. However, comprehensive discussion on non-parametric methods is beyond the scope of our paper. We encourage interested readers to explore the relevant literature for further insights.
Despite its theoretical concepts, the practical implementation of Koopman theory ongoing (Brunton et al., 2021). While promising, autoencoder methods typically serve as dimensionality reduction techniques, con- flicting with the concept of potentially difinttedimmensionall latent space. Nevertheless, they have shown remarkable performance in finding low-dimensional embedding for tasks involving high-dimensional solution spaces (Brunton et al., 2021). Furthermore, several works, including Lusch et al. (2018), have rendered the linear latent dynamics as state-dependent resulting in locally-linear dynamics. Although these archilectures perform well on high-dimensional nonlinear fluid flow, they loosen the Koopman invariance assumption due to the operator's dependence on the latent state. These results motivate the discussion of architectures that employ dimensionality roduction while relaxing the linearity assumption for the dynamics in latent space. i.e., nonlinear latent. dynamics.
Nonlinear latent dynamics. Modeling dynamical system without assuming latent linear dynamics removes the inductive bias but yields more expressive parametric model. In the context of this survey, these models are less relevant as their only inductive bias is the Markovian structure of the data, but these models perform well on a range of complex tasks, such as modeling natural language (Sutskever et al., 2014), music (Roberts et al., 2018) and video data (Babaeizadeh et al., 2018). One relevant line of work investigated how to implement these models with physically--structured latent space, like the models discussed in section 3.2. These include position-velocity encoders (Jonschkowski et al., 2017), Hamiltonian generative networks (Toth et al., 2020) and Newtonian variational autoencoder (Jaques et al., 2021).
Another noteworthy line of research, discussed in section 3.4, concentrates on operator learning. DeepONets employ an autoencoder architecture to model nonlinear operators within a finite-dimensional latent space (Lu et al., 2019).
Encoder-decoder models can also be combined with neural ODEs (section 3.1). Dynamics-aware implicit neural representations (DINo, Yin et al. (2023)) uses an implicit neural representation (INR) as decoder. INRs describe parametric functions trained to describe domain, such those alrcady seen in PINNs (section 3.3) and also neural radiance fields used to render 3D scenes. DINo learns an additional latent variable a and corresponding neural ODE & = fa),,, such that the true system is decoded using go(c,, æ), where 9o is the INR. Encoding is performed using 'auto-decoding' where a is optimized with gradient-based mot to invert go. The benefit of this approach is that, like PINNs, we do not need to discretize the spatial domain. However, unlike PINNs, the LVM formulation is more expressive and a many different systems to be learned, for example different initial boundary conditions. A similar architecture is the continuous reduced-order modeling approach (CROM, Chen et al. (2023a)), which also combines neural ODEs and INRs, but unlike DINo uses learned discretization encoder during learning, rather than auto- decoding.
More recently, significant, progress has been made in machine learning for weather prediction using deep latent variable models. While these models are primarily data-driven and do not have physically-inteet interpretablle latent spaces, their performance has been achieved using several small but crucial ahysics-informer design decisions that we will discuss here.
Nonlinear latent dynamics for weather prediction. Medium-range numerical weather prediction (NWP) requires forecasting the global weather system over a period of several days using computation models of the Earth's climate system. There are several challenges in achieving accurate NWP. Weather forecasting is inherently multi-modal, multi-scale, and multivariate, predicting temperature, pressure, wind speeds, and precipitation from a diverse range of measurement sources. Forecasting must also accurately an- ticipate rare extreme weather events, whose effects may be the most consequential. Despite these challenges, NWP has advanced significantly due to progress in weather models, numerical methods, and availability of computational resources (Bauer et al., 2015).
Weather forecasting traditionally relies on computationally intensive plysically-derived models, often com- bined in ensembles, to mitigate uncertainties arising from chaotic atmospheric dynamics and measurement
22

---

Published in Transactions on Machine Learning Research (05/2015)
limitations. Despite advancements, the accuracy of these methods for medium-range, regional, and global climate modeling is dictated by the computational resources and model fidelity. Additionally, they cannot leverage extensive historical weather data beyond optimizing model parameters with system identification. Morcover, the complexity of weather dynamics requires combining models for each component (c.g. clouds, pollution, elc), and jointly luning these models to improve net prediction accuracy is challenging/ Machine learning approaches offer promise by directly predicting measurements from historical datascts, circum- venting some computational challenges. Recent years have witnessed a significant advancement in machine learning methods for weather prediction, enabled by large capacity models, hardware accelerators, and clean historical datasets (Voosen, 2023). We discuss the design of three prominent nonlinear latent variable models in this context. An empirical analysis of deep learning- learning-based weather prediction can be found in Karlbauer et al. (2024).
Cest (Kurth et al., 2023), a neural network for global-scale weather forocasting, combines adaptive Fourier neural operators (AFNO) with a vision transformer (ViT). AFNO serves as a resolution-invaree encoder/decodeer, adept at capturing physical phenomena. ViT tokenizes AFNO representation and employs self-attention to model complex spatial and temporal interactions in weather dynamics. Initially trained for single-time step prediction, it's fine-tuned for consecutive predictions, significantly speeding up forccasting compared to seysics-based NWP. However, its accuracy falls short of current NWP systems, prompting ongoing improvements.
Like FourCastNet, this model builds off vision transformers, but instead of using AFNOs, proposes a novel ciree-dimeniional (3D) Earth-specific transformer architecture (3DEST) to better model weather correlation with latitude, longitude, and altitude. The key design deci- sion is in the positional encoding used by the 3DEST, which encodes absolute position coordinates rather than relative ones and incorporates translation invariance. This encoding uses 527 times more embedding parameters, but the authors did not observe issues with prediction or optimization. The model also employs "iirarccicaa temporal aggregation' to address crror accumulation in long-horizon forccasts. To mitigate this source of error, the authors learn models that are trained on 1-, 3-, 6- and 24-hour intervals so that long- horizon forecasts can be computed using an aggregation of models, using 3-4 predictions at most. However, with each 3D ViT having 64M parameters, this aggregation strategy results in a. 256M parameter model, which presents engineering issues at deployment. Similarly, Schmude et al. (2024) proposed Prithvi-WxCh a ViT-based approach with multi-axis attention to facilitate long-range interaction with limited attention window sizes and, to mitigate block boundary artifacts, an attention window shifting method was introduced. The model uses 160 input variables in order to address a more diverse set of downstream tasks. To reason over this many inputs, it uses both local and global masking and attention mechanisms during training to learn to model local and global phenomena. However, the resulting model has 2.3B parameters, posing significant challenges to deployment.
Bi et al. (2023) proposed
Lam et al. (2023) introduced GraphCast, a graph neural network (GNN) approach for redium--aage fore- casting. GraphCast builds on the work of Pfaff et al. (2021) learning mesh-based simulations using graph neural networks. GraphCast utilizes a hierarchical graph structure over the globe, resembling 3DEST, with six levels of resolution. The model's encoder projects Earth-oriented grid measurements onto this multi- mesh representation while the decoder aggregates nearby mesh values back to the grid. Operating on a 40,962-node multi-mesh, the GNN conducts simulation via simultancous message passing at various spatial resolutions. Despite making G-hour predictions, during training, is evaluated autoregressively for up to 3 days. Remarkably, GraphCast can predict a 10-day forecast in under minute at 0,25 global resolution with approximately 40 million parameters. Comparative evaluations suggest GraphCast outperforms Pangu- Weather across most measurement modalities, possibly due to the GNN's inductive bias, which resembles finite-element PDE solvers and propagates computations coherently in space. This behavior contrasts with ViT architecture, which needs latent self-attention mechanisms to learn such computations. However, the graph architecture does complicate some aspects of forecasting, such as downscaling and regional prediction.
Further extensions of this work use diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) for decoding. GenCast (Price et al., 2025) extends GraphCast by using a graph transformer for the latent dynamics and conditional diffusion model for decoding. Compared to GraphCast, which has deterministic decoder, sampling GenCast produces diverse high-fidelity forecasts. GraphCast is less capable of such high-fidelity
23

---

Published in Transactions on Machine Learning Research (05/2015)
predictions since the deterministic decoder requires averaging over any present stochasticity. Precipitation diffusion (PreDiff, Gao et al. (2023)) also uses conditional diffusion model for decoding weather predictions. Using a VAE to encode several steps of observations, the conditional diffusion model is trained to forecast a short lookahead window. An interesting aspect of this model is incorporating physical plausibility into predictions. For image generation, classifer-based guidance (Dhariwal and Nichol, 2021) uses the gradient of a pretrained classifier to improve the quality of the image generation. Similarly, Gao et al. (2023) replaces for example, a function that scores samples that reflect energy con-
the classifier with servation, and uses this function to guide samples towards these physically plausible characteristies. The authors use energy conservation guidance in an n-body simulation experiment and anticipated precipitation intensity for a precipitation forecast experiment.
Despite progress, several challenges persist in current weather forecasting models. A significant hurdle is their reliance on 'reanalysis' data derived from complex Bayesian state estimation processes, which combine metcorological measurements with simulation models. This reliance on processed data prevents these MI models from forecasting in real time using raw data as desired. Additionally, challenges include scaling models to go beyond the 0.25% training data and match the high spatial resolution of 0.1º of top NWP systems, extending forccasting horizons, and increasing model capacity. There is also the question of the extent to which physics-based elements from NWP can be combined with ML models to enhance their performance, given that NWP forecasts are inherently physically plausible, while current MT.-based forecasts lack this feature. This question is an active topic of study, e.g., (Kochkov et al., 2024). The next section explores examples of incorporating physics-informae elements into model learning.
3.6 Bridging Domain Knowledge and Function Approximation for System Identification
System identification approaches lie on a spectrum depending on modeling assumptions. Tf a significant amount of domain knowledge is assumed, the dynamics model will be highly structured. If correct, it will generalize well but will most likely be infiexible to unanticipated observed phenomena. On the other hand, one could model with no domain knowledge and use a black-bor box function approximator", These models have gencralization guarantees and, therefore, require expansive datasets and regularization to ensure good modeling accuracy across the domain of interest. To achieve an effective combination of these two approaches, one would need to balance structural biases that facilitate generalization with adaptivity to the data that prevents the biases from leading to underfitting (Ljung, 2010; Lutter et al., 2021). This section reviews these approaches, which build on models discussed earlier.
Physics-informed neural networks can incorporate an additional dataset as an additional loss term in the objective. In addition residual and boundary loss terms, the combined objective (equation (19)) has an additional dataset loss similar to the boundary loss term (Raissi et al., 2017a; Moreover, since the PINNs objective is quite general to any differentiable function approximator, Wang et al. (2021d) extend DeepONets by adding the PINNs residual loss to the data-driver operator loss as a way of encoding physics knowledge.
Yang et al. (2020) consider a generative approach using PINNs. Generative adversarial networks (GANs) learn a generator that learns to predict samples that match a dataset and a discriminator that learns to differentiate real and generated samples. Learning both models jointly should converge to a rcalistic data generator. PI-GANs incorporate PINNs into the stochastic generator using a reference PDE, where samples take the form u and Lu and then passed into the discriminator d Rdd, + R with true measurements of u and f, so for a dataset of size N the discriminator is evaluated on all six terms, i.c.,
d< E.), uxx:), (x,, G.), ff (a:;,, us uo E,), E~ p(), x; EX 2, E 0x, i.j E (0. N).
The benefit of this approach is that the discriminator learns a data-driven metric between the model and the dataset, in rast to the squared error used by vanilla PINNs for all terms. However, the paper does not compare the performance between objectives, and PI-GANS are more expensive to train than PINNs
One could argue that using many popular function approximators, such Gaussian processes, come with smoothness assumption
24

---

Published in Transactions on Machine Learning Research (05/2015)
Chen al. (2021) propose physics-informet approach for using learning PDEs from data. They posit that the solution u is linear in broad set of pre-specified physics-informed features, e.g.
u( t) = W ((tt
(34)
( m) [1, u( t), u( t)² ...... t)...................... vx,
(35)
The training objective combines measurements of and its temporal and spatial derivatives similar to PINNs. Moreover, lo encourage sparsity in the learned solution with respect to the features, an L penalty on W is also incorporated into the training objective.
Haußmann et al. (2021) consider neural stochastic differential equations with partial knowledge of the dy- namics, captured by r(z, !), which caplures the mean function of the process with weights 7 E [0, 1]
dæ; fo( t) 1ot++9 t t) dt + g t) dwi, Y ~ p(- xx).
(36)
To optimize 0, PAC Bayesian approach is adopted to regularize the parameters against prior p(0),
Ee O E log(y: 2t, 0)] + DKLL90 (0) p(0) Log(4///8)//22
(37)
where N is the dataset size and E [0 1] is the probability of the PAC Bayes generalization bound. This objective is optimized using reparameterized gradients and the Euler-Maruyama integration of the stochastic process.
Long et al. (2022) combine differential equations and Gaussian processes by leveraging that the derivatives of Gaussian process are also Gaussian processes long the kernel is sufficiently differentiable. Their Bayeeian method that combines the PINNs residual objective with the typical Gaussian process prior and likelihood, minimizing the following expression w.r.t. pseudo-posterior g(u) over solutions u, where p(u| X) = GP(O, C(X)) and
method is
Eu-g(-) log pY
u, X)] + a- (Lu()) + DKI(l(u) I(()]]
(38)
As a result, the sgeudo-posteriog solution combines the lypical GP prior and data likelihood with the PDE residual speudo-likelihodd controlled by hyperparameter a >
quu1 DD = (0; f(X), P(Y u, X) p(u).
(39)
This posterior cannot be obtained in closed form, approximate inference is performed. This is done by parameterizing the nonparametric GP with parametric inducing points synthetic dataset and optimizing these points to minimize equation (38). These inducing points play the role of collocation points in PINNs The authors also demonstrate that an additional GP can be incorporated into the residual term to capture any unknown terms in the PDE, which means two GPs are jointly inferred. their experiments, this model was shown to perform better than PINNs with fewer collocation points. PINNs could outperform their method with more collocation points, and the authors did not demonstrate that their method scaled a high number of inducing points. This is because non-parametric GPs suffer from O(N3) complexity for N training points when arbitrary kernels are used. However, specialized kernels exist that enable more graceful scaling.
3.7 coysics-inspired Model Invariances
Physical models in the form of ODEs, PDEs, algebraic equations often invariant to specific types of operations or transformations. For instance, many dynamical processes such as molccular, rigid body, clasto or fluid dynamics, chemical reaction kinetics, etc. are modeled under the assumption of energy conservation and the laws of thermodynamics (Cueto and Chinesta, 2023). This also translates to continuum material models, which should be invariant to the orientation of the coordinate system and consider the orientation of the material's microstructure. Likewise, in computer vision, object detection should be invariant to the position, orientation, and scale of the object. Graph-based representations, e.g., in the context of molecular
25

---

Published in Transactions on Machine Learning Research (05/2015)
interactions particle systems (Sanalzz-Gonzalez al., 2020), should often also be invariant to rotations, translations, reflections, and permutations. To discover invariants and conservation quantities from datasets using machine learning, Wetzel et al. (2020) proposed a "Siamese twin" neural network architecture, which aims to classify data points as related to the same invariant value. Similarly, Ha and Jeong (2021) introduced an intuitive noise-variance loss function for training a NN to classify data points related and non-related to the same invariant value.
Mathematically defining and incorporating relevant invariance and equivariances into machine learning mod- els has led to several advancements in applying machine learning to complex physical domains, e.g., (Cohen and Welling, 2016; Fuchs et al., 2020; Satorras et al., 2021; Hoogeboom et al., 2022; Weiler et al., 2023; Thiemann et al., 2025). Denoting a set of transforms as T, the T-invariance of a function or operator can be expressed as:
((T=)))==(0 = TTET,
(40)
and the T-cquivariance as:
[(T())) = T(f(z)) ATET,
(41)
For instance, "rotation invariant" function f R33 + R is invariant under transformations in the 3D rotational group T SO(3), where T R + T(x) Ræ with R E R3x3 R I. According to Nocther's ss theorem, such rotational symmetries are also closely linked to conservation laws, which imply that some quantity / of a system, such as energy, momentum, mass, charge, etc., is conserved under any admissible trajectory T,, i.., invariant.
When applying machine learning in these contexts, embedding such known invariances and conservation properties into the models can greatly increase their accuracy, robustness, and generalization abilities and the same time reduce training data requirements. Prior knowledge about invariances can be considered at all stages of the design of thysics-informed ML pipcline, i.c., in the problem conception or feature engineering (inputs and outputs of the model), the training and test data collection and curation, the ML architecture design, and the loss function formulation. Thus, we propose the following taxonomy for incorporating a T- invariance, as defined above in equation (40), and likewise also a T-equivariance, into parametric a model fe:
(model-based)
fo(x) = fe(T(z)) TTET, (xx)))) = so()(T(z)) VTET,
(42)
blead))
(43)
(data-based)
D Daug (yyn, _0 n== M am (r(m (xr), 7(m) ~ p(())
(m)) N
(44)
(objective-based)
Linvar (0) = L()) + ET (D(Jo(mn), (o(((x))))
(45)
In the following, we now explore how these different implementations of invariances have been used in practice in various fields of application.
Thermodynamic consistency of material models. In mechanical, thermal, electromagnetic, and sim- ilar (including coupled, multi-physical) continuum theories, the governing PDEs, see equation (3) include constitutive models that describe the material behavior and must adhere to the laws of thermodynamies (Silhavý, 1997). Generally, energy conservation can be ensured by formulating material models in terms of potentials such the internal, Gibbs, or Helmholtz free energy densities. For instance, in hyperclastic material theory, the mechanical stress tensor P E R3x3 is derived as the gradient of the internal (strain) energy density (FF): 33x3 + R with respect to the deformation gradient tensor F E R3x3,
dv P(F) dF
dv - dt
dF dt
- =0
P:
(46)
Thus, the high degree of flexibility offered by ML methods can be exploited to formulate constitutive models that strictly ensure energy conscrvation in model-based fashion by parameterizing V. So far, such models have been developed for clastic, electro-clastic, and magneto-clastic materials in terms of internal energy potentials v using, e.g., neural networks (Te et al., 2015; Fernández et al., 2020; Linka et al., 2021; Klein
26

---

Published in Transactions on Machine Learning Research (05/2015)
et al., 2022a; Kalina et al., 2024), Gaussian process regression (Fuhg al., 2022; Ellmer et al., 2024; Pérez- Escolar et al., 2024), sparse regression (Flaschel et. al., 2021). In the training process of these models, since the main quantity of interest is in fact not the potential We itself, but its input-derivative P = / it is highly beneficial to include the stress MSE with a loss term, which is denoted as Sobolev training, see Czarnecki et al. (2017); Vlassis and Sun (2022):
Lsobolev (0) C
m
m Cp m
P. y
2
V (o(
(Fi) dF
(47)
m
Furthermore, it should be mentioned that the energy conservation and favorable properties of the material models can only be preserved throughout a simulation when dedicated spatial discretization and time inte- gration methods are used. For example, the mixed finite element methods and euergy-momentunn scheme developed for hyperelastic, paysics-angmentte NNs in Franke et al. (2023).
Strictly speaking, dissipative systems do not possess invariance property, but they must fulfill the second law of thermodynamics, which is mathematically formulated as an inequality. For dissipative material behaviors such as elastoplasticity, viscoelasticity, or damage in mechanics, magnetic hysteresis, viscous flows, etc., adherence to the second law of thermodynamics can be guaranteed by formulating the models according to the gencralized standard materials (GSM) framework (González et al., 2019). Here, additional internal (latent, non-observable) variables y are introduced, which the internal energy potential as well as a dissipation potential & depend on. The evolution of the internal variables y then defined in terms of a compatibility condition between both potentials:
dv dv d$ dy
0, dv
V 7),
P(F,7) = aF dy
+ 2 0.
(48)
dy
Flaschel al. (2024) demonstrated sparse regression concepts for obtaining GSM models for viscoelastic and clastoplastic material behaviors in the small strain regime, while Rosenkranz et al. (2023) compared various NN-based architectures, including RNNs with not guaranteed thermodynamies consistency and GSMs with FFNN potentials & and 4 and Holthusen et al. (2023) extended and applied the concepts in the finite deformation regime using constitutive ANNs. However, thermodynamic consistency and the fulfillment of the dissipation incquality can also be ensured by transferring classical rhcological constitutive models to the ML context, see e.g. (Masi et al., 2021; Fuhg et al., 2023; Meyer and Ekre, 2023; Abdolazizi et al., 2024), thus applying a very strong, potentially ovsrrrestrictive bias on the model architectures.
Rotational invariances of material models. In material modeling, also the consideration of rotational invariances is crucial. Hyperelastic material models, see equation (46), have to be objective, i.e., invariant to changes of the observer in terms of arbitrary rotations Q E SO(3), and material frame indifferent, i.e., invariant to changes of the reference configuration in terms of rolations R E G that are within their material symmetry group G C (3) (Silhavý, 1997). Mathematically, this can be formalized as:
(QQ F R*) (FF) E (3), Q E SO(3), REC. EG
(49)
which directly leads to the equivariances of the stress lensor as:
P R Q- R' GL+ SO(3), REG.
(50)
In the classical, analytical way of formulating material models, adherence to both invariance (and equivari- ance) conditions is usually achieved by not expressing v directly in terms of the second-order tensor F, but using scalar invariants. For the simplest case of isotropic materials, which exhibit completely direction- independent behavior as G = -3),, the three scalar invariants I1 I2, Ig E R of the tensor C FFFF
4 tCC) I = ((r((()))) (c²)) I3 det(C),
(51)
are both objective and material frame indifferent.
Thus, also machine learning- based hyperelastic material models can be thysics-augmentt and by construc- tion fulfill these rotational invariances in fealure-based way by representing the strain energy in terms of
27

---

Published in Transactions on Machine Learning Research (05/2015)
these invariants, i.c., (FF) = Vo([i. I2, Is); using neural networks, Gaussian processes, etc. (Klein et al., 2022b; Linka. et al., 2021; Fuhg et al., 2022). Similarly, for anisotropic materials with other symmetry groups 9 and also for electro- or magneto-elastic couplings, invariants can be derived and employed for machine learning of material models (Klein ct al., 2024; Kalina et al., 2024; Péroz-Escolar ct al., 2024). Furthermore, invariants can also be used to develop alternative representations of the material models, e.g., circumventing the learning of the internal energy density & and instead directly expressing stress coefficients through NNs (Fuhg et al., 2024a), which avoids the computational effort of differentiation of the NN through nack-propagation in order to compute the stresses.
Another model-based route for ensuring these rotational invariances can also be to express V in terms of the but augmenting the ML model architecture with algebraic group symmetrization approaches, see Itskov (2001) and its adaption for NNs in Fernández et al. (2020); Klein et al. (2022b):
tensors F or C' (where the latter already ensures objectivity as (QF) (QF)= F F²FF),
V ((F) =
Σ REG
Vo(F R).
(52)
|
Here, the training and evaluation of the model require multiple cvaluations of the NN Vo (or any other type of ML model), which is compulationally costly, especially for the stress evaluations. Furthermore, this approach can only exactly incorporate the material symmetry requirement if the symmetry group is finite, i.e., [G] < co. Nevertheless, in Klein et al. (2022b), it was also applied to transverse isotropy by sampling a uniformly distributed subset of 60 rotations from the corresponding infinite material symmetry group, yielding highly accurate results.
As the model formulation in equation (52) using F does not fulfill the objectivity condition, Klein et al. (2022b) also uses data-based augmentation. Here, the training dataset, which consisted of deformations F, with typically only a specific observer direction, was augmented by including additionally also rotated data points (Q- F;; V. } with Q E 50(3). Using 64 uniformly sampled rotations, the models could then learn to be objective up to the desired model accuracy. While this approach increases the training duration due to the extended dataset size and only weakly includes the physical requirement, it preserves the faster evaluation of the trained model compared to the algebraic model augmentation using group symmetrization. Similarly, instead of augmenting the dataset, the same effect could also be achieved using penalty approach, similar to the PINN objective section 3.3, by adding loss function term that penalizes violation of equation (49) for Q and R being sampled in (subsets of) the respective groups. Again, this would increase the training time but would not affect the evaluation of the trained models.
Functional requirements of material models. Machine learning models that are meant to substitute physical models or numorical simulations, or ingredients thereof, are typically subject to further mathemat- ical requirements such as continuity and continuous differentiability, monotonicity, convexity or concavity, invertibility, integrability, etc. Though these are not directly invariances, these functional requirements are often crucial for ensuring invariances, well-behavior, and robustness of ML models.
For instance, in continuum mechanics, the PDE boundary value problem only has a well-defined, unique solution if the included hyperelastic material model is elliptic (Šilhavý, 1997). This can be ensured by formulating the internal energy potential as polyconvex, i.e., ((F) = (F, det(F) det(F)) must be convex in terms of its three separate arguments. While analytically formulating multivariate functions as convex is a difficult task, the so-called input-convex neural networks (ICNN) proposed by Amos et al. (2017) have relatively simple to implement restrictions that ensure convexity: their weights must be non-negative (except for the first layer) and the activation functions monotonously increasing and convex. Thus, using ICNNs, the functional requirement of polyconvexity can be embedded into the architectures of NN-based models vo for hyperclastic matcrials, see Klein et al. (2022b); Linden et al. (2023); Fuhg et al. (2024a). When combined with the other aforementioned requirements, such as thermodynamic consistency, objectivity, NN material models can be accurately trained on very and material symmetry, these generalization behavior, and capture highly nonlincar (meta-) material sparse training data, show excellent behaviors (Klein et al., 2022b; Linden et al., 2023). Furthermore, the interpretability (and generalization can be increased by applying sparsification strategies such as I/-rregularization in the training process (Fuhg
28

---

Published in Transactions on Machine Learning Research (05/2015)
et al, 2024b). Similar convexity requirements also exist in electro- or magneto-clasticity and were successfully implemented into NN-based constitutive models, see Klein et al. (2022a); Kalina et al. (2024). However, it should be noted that ICNNs were also observed to be overly restrictive in some cases, as the conditions on the network architecture are sufficient but not necessary to ensure convexity, see Kalina et al. (2024); Klein et al. (2024).
Furthermore, when material models are be equipped with additional input parameters, such as material parameters microstructure descriptors, they should be polyconvex in terms of F, H,, and J, but may have no functional requirements regarding the additional features, be monotonously increasing decreasing, concave in those. Also, this prior knowledge can be incorporated into NN architectures, see also Amos et al. (2017) for general formulations of partially input-convex NNs and Klein et al. (2023) for the application to hyperelastic material models. In particular, thermoelastic material models of the form V = VF require polyconvexity in F but concavity in the temperature T, which has also been successfully realized using convex-concave NN architectures in Fuhg et al. (2024b). Furthermore, also GSM models, see equation (48), require polyconvexily of V 7 in F and convexity of $ 7) in 4, which can be realized by partially convex NN architectures, see Rosenkranz et al. (2023); Holthusen et al. (2023).
In summary, material modeling is subject to several types of invariance requirements such as thermodynamic consistency, rotational invariances, and functional requirements. It has already been demonstrated through various techniques that this prior knowledge can be successfully incorporated into ML frameworks for con- stitutive modeling. In particular, ML models can be fully physics-augmenteed i.e, the requirements can be embodded into the model architectures by careful choice of features, outputs, and NN structure. This additional structure generally greatly improves the model performance, generalization, robustness, and in- terpretability and reduces data requirements.
Invariances in dynamic systems and neural operators. Symmetries and energy conservation are also important aspects of the modeling of dynamic systems. As already discussed in section 3.2, the equations of motion of dynamical system can be formulated in an energy-conserving. i.c., thermodynamically con- sistent, fashion by deriving them through the Lagrangian or Hamiltonian formalisms (Lutter and Peters, 2023; Cranmer et al., 2020b; Greydanus et al., 2019). In these approaches, the invariance is model-based and incorporated through the model conception as the (Lagrangian L, Hamiltonian H, ...) potentials are learned/ paaned/parameterizee by a NN and not the dynamics themselves in the form of an ODE, e.g., through a neural ODE, see section 3.1. This concept has also been applied to Koopman operators (section 3.5), for the data-based discovery of conservation laws in Kaiser ct al. (2018) and could be further extended to neural operators. Likewise, enforcing conservation laws can also be achieved in the context of neural solution fields (Richter-Powell et al., 2022), c.f. section 3.3, neural operators (Liu et al., 2023b; 2024b), c.f. section 3.4, or neural processes (Hanson et al., 2023), c.f. section 4.3.
However, energy conservation is usually an idealization of real system behavior, and dissipative ellects are present in most real-world systems. Dissipation behavior can be considered in a similar fashion through application of formalisms such as the Port-Hamiltonian (Desai et al., 2021; Neary and Topcu, 2023; Roth et al., 2025), see equation (16), Seerssibbe--reeeesiiii Coupling) frameworks (González et al., 2019; Hernández et al., 2021), which is structurally very similar to the GSM framework mentioned above for material modeling.
(Eidnes et al., 2023), GENERIC (General Equation for
the Non- Non-Eqqilibrium
Further prior knowledge about the model structure can be incorporated in these frameworks, e.g., by ensur- ing convexity or a quadratic positive definite form of sub-potentials such as the kinetic energy component of the Lagrangian, see Lutter and Peters (2023), or the skew-symmetry of the cosymplectic matrix and the symmetry and positive semi-deftniteness of the friction matrix in the GENERIC framework, see Hernández et al. (2021; 2022). For dissipative systems, this further structure is crucial in ensuring the thermodynamic consistency of the models, and the GENERIC formalism even requires the fulfillment of an additional con- sistency condition, which is embedded by adding a further penalty-type loss term, c.f. equation (19), in Hernández et al. (2021). Furthermore, it should be mentioned that preserving the thermodynamic con- sistency of (classical or MI--basdd) models also requires suitable, tailored time integration schemes for the resulting ODEs, see section 2.1, which as already realized for the GENERIC framework in Hernández et al. (2021; 2022).
29

---

Published in Transactions on Machine Learning Research (05/2015)
Single-Tasie Learning
Multi-Task Learning
Meta- Meta-Learning
u
u
u
U04
0
i
u
uo
Figure 6: Visual illustration of the singlc-task learning, multi-task learning (MTL), and meta-learning paradigms for PDEs. In single-task learning, different models are being learned separately for each PDE. On the other hand, MTL learns a set of PDEs concurrently, hence finding set of parameters that compromises the single-task solutions and generalizes across them. Finally, meta-learning aims for a general solution that can be transferred later for novel PDEs (e.g., uo,) where few gradient steps can be taken arrive by the single-task models.
As already mentioned in section 3.4, neural operator architectures have also already been extended to be oquivariant to rotations, translations, and reflections in Hclwig et al. (2023) by applying group convolutions similar to equation (52), or explicitly being Squii-equivariant
Cheng and Peng (2023), as well as also being momentum conserving in (Liu et al., 2023b). Furthermore, PINNs can be enhanced to approximate the symmetries of PDEs using additional loss terms, see Akhound-Sadegh al. (2023), and neural processes, which will be discussed in detail in section 4.3, can be designed to be invariant to translations, rotations, and reflections, demonstrated by Holderrieth et al. (2021) also using group theoretic approach.
Lastly, should be mentioned that these concepts for employing invariances prior knowledge for ma- chine learning be applied to wide range of problems, e.g., in material science chemistry. For instance, Batzner al. (2022) developed potential-based and thus energy-conserving, invariant/equivariann graphiconvolution NN for learning interatomic potentials from ab-initio caleulations for molecular dynamics simulations. Furthermore, Döppel and Votsmeier (2023) enhanced the chemical re- action neural networks (CRNNs) for autonomous mechanism discovery, which a neural ODF-type model, with stoichiometric constraints that ensure mass conservation. Moreover, in these cases, informing the NNs with invariances drastically reduced training data requirements, accelerated training, and increased accuracy and robustness compared to previous state-of-the-art ML models.
well E(3)-
4
Priors from Experiments
While section and 3.7 focus on designing models with some form of prior knowledge from physical, mathe- matical, and numerical modeling, data and the learning process itself can also leveragod as form of prior knowledge. Meta- and multi-task learning are two learning paradignss that exploit knowledge gained from previous or concurrent experiments (Figure 6). Multi-task learning (Caruana, 1997) aims to use the knowl- edge gained from each model (or 'task') to improve the model through generalization, while meta-learning (Vilalta and Drissi, 2002) secks to improve the learning algorithm instead, either by optimizing the model initialization or how the model updates during learning. Neural processes (Garnelo et. al., 2018) combine ideas from meta- and multi-task learning by conditioning the model on a small 'contextual' dataset, which facilitates fast adaption different scttings. Since forecasting typically considers a range of domains practice, such as boundary conditions and variations in physical parameters, there ample opportunity leverage knowledge from multiple experiments across these different domains to improve performance.
30

---

Published in Transactions on Machine Learning Research (05/2015)
4.1 Multi-task Learning
System identification lypically considers learning model for single system under study. However, in practice, there may be fleet of systems of interest. In reality, these systems will differ in behavior and may have different operating regimes, which warrants training separate models. Multi-task Iearning (MTL) attempts to learn these models jointly to reduce the measurement burden. By balancing the need to model each system while acknowledging the similarity between systems, the aim that the chore of learning multiple models can, in fact, be leveraged to improve sample efficiency and performance by using the shared properties across the physical systems.
Following section 2, system identification can be extended to the MTL setting when the objective is to learn more dynamical systems at once. A dalaset of N input-oulput sequences D = {(w0 ( where Ti is the length of sequence and each sequence is defined by a distinct dynamical system with state variables ot E X, 1, ...., T, for each i I.... .V with ( being an initial state variable. Each dynamical system can be assumed as a different but related task.
A naive MTL approach for learning common dynamical system is to exploit task information (e.g., task ID, task description) in the learning process. There are two ways to integrate such information: initial state customization or bias customization. For initial state customization, task information T is used to customize the initial state Fo. On the other hand, the task information can be deeply integrated by being appended to the inputs w. + +TJT for each time-step t.
Spieckermann et al. (2015) use RNNs for multi-task system identification by structuring some weights per task, referred to tensor factorization. By considering similar system identification tasks with similar lask information, Spieckermann et al. (2015) use an indicator variable I = (io..... ). E [ = N for N tasks. These indicator variables are added to the dataset and used in the RNN architecture to index the parameter weights, such as linear weights or bias terms. This work explores different approaches to factorizing the parameters to trade off the number of parameters learned per task and across tasks.
On the other hand, Bird et al. (2022) proposes probabilistic approach, called multi-task dynamical sys- tem (MTDS), for extending MTL to time series models. MTDS learns set of hierarchical latent variables for modeling dynamical systems of different sequences. This model allows for adapting dynamical systems to the inter-sequence variations, hence enabling personalization or customization of the models. MTDS assumes different parameterization ( E O, which depends on the hicrarchical latent variable 2(0 E 2, for cach sequence i generated from a specific dynamical system. MTDS is defined mathematically by the following equations,
0 = ha a ~ p( y ply
p(z). (())
(53)
w
(54)
(55)
where h& 2 > R is vector-valued function that maps latent variable 200 to model parameter g Rd. MTDS explicitly provides visibility of the task specialization, which can be controlled directly. MTDS is evaluated on two applications: motion-capture data of people walking in various styles using a multi-task recurrent neural network and patient drug-response data using a multi-task pharmacodynamiic system.
PINNs suffer from issues related to varying initial conditions, boundary conditions, and domains due to the tradc-off between generalization vs. specificity. Pellegrin et al. (2022) proposed the use of transfer lcarning for increasing the generalization, where base neural network with different "heads" for various initial and boundary conditions is initially trained. After this initial training, they kept the base network unchanged and fine-tuned the heads for new conditions. This method allows the base network to retain general understanding of the physical systems, thus reducing training time for new scenarios because the base does not need retraining.
Flamant et al. (2020) tried to enhance PINNs across different scenarios by taking PINN models as solution bundles, which is a collection of solutions for the same PDE under various conditions. They adjust the model's loss function by adding more weight to errors at the initial state. This is beneficial because it
31

---

Published in Transactions on Machine Learning Research (05/2015)
Body
Heads
(La; ui (ar) f((a), (x) b
H1
(Lay (c))) fu(a), ux(x) bx(
S 2
H₂.
(Lay - (x))), ux(x) (x))
HN
Figure 7: A schematic of the multi-head physics-informet neural networks (MH-PINNs) from Zon and Karniadakis (2023), where body & is shared by N Tasks while having lask-specifie heads Hg, k = 1,..,, N.
locuses more on initial error reduction that prevents them from propagating and amplifying. Therefore, the network learns multiple solutions simultaneously for variable conditions and shows good convergence properties.
Thanasutives et al. (2021) propose an MTL approach of solving the original PDE along with auxiliary related PDE with different coefficients. In this work, they integrate MTL at the architecture and optimization levels. For the architecture, cross-stitch (Misra et al., 2016) is employed to share information between two parallel baseline networks for original and auxiliary tasks. At each layer, a cross-stitch unit is used to lincarly combine the input activations from the two networks to allow flexible sharing of representations between the two branches. The authors evaluate two MTL techniques for learning, an uncertainty- weighting loss (Kendall et al., 2017) and gradient surgery (Yu et al., 2020). The uncortainty- physics-inforai informed loss function (UNCERT) is defined combination of task-specific PINNs loss functions weighted by the uncertainty of each PDE solution. The loss function is defined
LUNCERT (0,
Σ
20 Frixx(0 + logo;)
(56)
where N refers to the number of tasks, 0; is a gradicnt-based trainable parameter that indicates the uncer- lainty of each PDE solution, and LPINN is a similar loss function as in equation (19). "Gradient surgery" is implemented using PCGrad (Yu et al., 2020) to modify the average of the unweighted PINNs loss functions for mitigating conflicting gradients between tasks. PCGrad projects the loss gradient task i onto the normal vector of task j's loss gradient and vice versa. The model parameters are updated using Spc shown below,
0 0
Eee {g/c) = PCGrad((Vooowwe (0.))),
where gée is the modified gradient of task after applying PCGrad. The proposed algorithm provides better generalization highly nonlinear domains in terms of lower error rates unseen data points in comparison to related baselines.
Zou and Karniadakis (2023) propose a new PINN architecture that suites MTL. In this work, a multi-head architecture is presented, named multi-head physics-informed neural network (MH-PINN), where a body t is shared by a collection of tasks (e.g., ODEs/ PDEs) while task-specific heads H&, = I,.... N are learned separately for each model. The body consists of nonlinear hidden layers, while linear layers are used for the different heads. Consequently, the body provides set of basis functions for each head to generate task-specific solutions. The body Φ _2 + Rd is function parameterized with parameter 0, and each head Hg is a linear function. The lask-specific solution is defined as (x) = Hy ((x), Vo Ω. Given an MTL dataset ( the MTL loss function is defined as follows:
1
Cam-Pin (1Da) (11)) (,
La (Di; Ha).
(57)
N
32

---

Published in Transactions on Machine Learning Research (05/2015)
where C3 is the PINN loss function as in equation (19). Moreover, Zou and Karniadakis (2023) makes use of the learned model to construct prior knowledge for downstream tasks. For instance, the heads are considered as samples used for estimating the PDF and a generator for I/ using normalizing flows (NEs) (Rezende and Mohamed, 2015). The acquired prior knowledge is useful for downstream tasks where limited data points are available.
Similarly, Wang et al. (2023) ease the training of PINNs by employing multi-task optimization paradigm where multiple auxiliary tasks are solved together with main task. Knowledge transfer from one task facilitates solving others. The problem of automatic traffic modeling is considered, which has multiple modalities such as traffic speed and density. Given several datasots measuring different traffic scenarios and MTL over datasets and modalities, a similar PINN architecture, as in Zou and Karniadakis (2023), is trained with a shared body network and only a final layer per task. This architecture design significantly increases the data available to train the initial PINN layers, which build an internal feature representation. The MTL ensures that this feature representation is useful across traffic conditions and modalitics. As a drawback, the model requires an adaplive training scheme in order for the multi-task aspect to be effective, which significantly increases the implementation complexity.
4.2 Meta Learning
Whereas multi-task learning aims to exploit common knowledge while learning given set of tasks in parallel, meta lcarning aims to optimize a learning procedure on the given tasks that can later be applied to novel lasks. Both paradigms can leverage a set of source lasks to learn better solutions on larget tasks and are therefore closely related to transfer learning (Zhuang et al., 2021b). However, whereas multi-task learning optimizes source and target tasks jointly, meta. learning considers the source tasks only during pre-training allowing for computationally efficient learning on the target tasks. In general, the result of meta-learning could be a complete algorithm. However, the methods discussed in the following are content with learning suitable initial parameters or other hyperparameters for given algorithms.
Meta learning methods for PDEs are often specific to a certain type of PDE but can adapt to different instances, such as different boundary conditions or different coefficients in the governing equations. We can distinguish between meta learning methods in which the network has an additional input to inform it about the particular instance and those in which adaptation happens solely via fine-tuning of the weights.
The latter class of methods is typically based on 'model agnostic' meta learning (MAMT.) (Finn et al., 2017), its first-order approximation FOMAML, or closely related meta learning techniques, such REP- TILE (Nichol et al., 2018). MAML optimizes initial model parameters to minimize the expected test loss on the given set of tasks, which is achieved by optimizing the tasks on a training set, e.g., using a small number of gradient updates. FOMAML and Reptile are computationally more efficient than MAML by discarding higher-order information, and thereby avoiding backpropagating through the inner optimization. Despite using less accurate meta-gradients, both methods have been shown to perform similarly well compared to MAML (Finn et al., 2017; Nichol et al., 2018). A comparison between MAML and REPTILE is shown in Algorithm 1.
Several works have applied these techniques for learning dynamics models (Lin et al., 2020; Qin et al., 2022). Meta-L (Lin et al., 2020) uses MAML for linear time-variant (LTV) systems. The model is optimized to achieve a small prediction error after updating the linear parameters of the dynamics with a single gradient step. During deployment, the fast adaptation makes possible to adapt to changes of the dynamics online. MctaPDE (Qin et al., 2022) employs a similar approach to nonlincar dynamics. They compare MAML with a slightly different meta learning technique, LEAP (Flennerhag et al., 2019), for meta learning a PINN that can be quickly adapted to different PDE parameters. While LEAP was faster during meta- leaanie, MAML performed faster in deployment. NRPINN (Liu et al., 2022) is a similar approach that uses REPTILE (Nichol et al., 2018) on a modified PINN loss, which contains an additional term that can make use of labeled data, if available.
Another MAML-based approach used by iMODE (Li et al., 2023c), which adapts a latent input to a more general PINN that is shared among different PDE instances. A bi-level optimization problem is solved to optimize both, the common PINN network and the initialization of its latent input. Furthermore, PCA is
33

---

Published in Transactions on Machine Learning Research (05/2015)
Algorithm 1 Meta learning with MAML and Reptile
Require: initial parameters O task distribution Peask (T), task loss function L(O, r), task batch size Nr: number of iterations Viter number of gradient steps K, task learning rate a; meta learning rate B.
1: for == 1.2....,,NV Niter do
2: sample batch of tasks Ti...., TN, from Prask (T)
3: for j 1.2.....A NN do
0 -- ((-1)
4:
5: for
1,,,.... "....KKK K do
6:
0 (%)) + 0 (k-1)
a 0 60
C(0)) :T²)
7:
end for
8: end for
9: ((0 f 0(-1) VVoc oci 1) Nr E L(O(k) .T)) // MAML
0 - 0(-1) { Nr Σド= [0 [00h] -0(i-1) // REPTILE
10:
11: end for
applied to learn compressed form of the latent input. When examples with labeled PDE parameters are available during training, iMode can learn diffcomorphism between the labeled parameters and the com- pressed latent imputs, which can be used for system identification. DyAd (Wang et al., 2022d) directly feeds the parameters of the PDE as input to the model. The architecture is geared towards 2D images/ images/velocity velocity fields. uses 3D convolutions and AdaPad, which predicts boundary conditions using a nonlinear transfor- mation of the latent used for padding the images. As the PDE paramcters are often unknown, Ay also trains an encoder network that predicts the parameters from a history of states. The encoder is trained using weak supervision, e.g., by exploiting when subset of the parameters are known.
Instead of predicting latent inputs, hypernetworks can be used to predict the weights of the common PINN. For example, HyperPINN (de Avila Belbute-Peres et al., 2021) trains a hypernetwork that takes the param- eters of given class of PDEs (e.g., Burger's equation) as input and outputs the parameters of a PINN. To reduce the number of the parameters of the PINN, Hyper-LR-PINNS (Cho et al., 2023b) restrict the weight matrices of the neural network to be low-rank, parameterized by the matrices of an SVD decompo- sition, that is, the weight matrix of layer i is given by W; U,E.V, with rectangular matrices U, and V,, and a diagonal matrix Σ. During offline pro-training, U and V are optimized directly along with the hopernetwork that predicts the elements of E. During deployment, only the diagonal elements are opti- mized, using the hypernetwork for predicting their initial values. An alternative way to reduce the number of environment-spocifc paramcters was proposed by Kirchmeyer et al. (2022). Their method, CoDA, Icarns shared nominal parameters of the dynamics model along with a shared linear projection matrix that maps environment-specifit low-dimensional parameters to an offset for the nominal parameters. By penalizing the L1 or & norms of the projection matrix and the enviromment--peeifie parameters, CoDA aims concentrate the resulting offset parameters around the nominal parameters for faster optimization during deployment.
While the aforementioned methods were targeted towards a specific type of PDFs, some approaches aim to learn even more general networks. LEADS (Yin et al., 2021) learns a dynamics model, where the dynamics may also change over time. The model predicts the state derivatives and decomposes them into shared and environment-speecfic components. It trained by minimizing the squared error between predicted derivatives and targets, with an additional convex regularizer penalizing the complexity of the cnvironment-specifn terms. Different regularizers are proposed for the linear and nonlinear cases, based on sample complexily analysis
Iwata et al. (2023) consider PDEs with polynomial governing equations. A single PINN is trained, which takes the query point and the coefficients of the PDE as input. Furthermore, Dirichlet boundary conditions are represented by a set of boundary points and their evaluations. The PINN optimized with respect to the PINN-crror, using self-generated data from randomly sampled PDEs. However, to keep the number of coefficients small, the experiments are limited to two-dimensional PDEs of order and use polynomials of degree? 2.
34

---

Published in Transactions on Machine Learning Research (05/2015)
Conditional set encoder
Conditional decoder
Latent set encoder
Latent decoder
(#1, 31
-+
{01, y1)
(#2,
y2)
12
Co
TC
dg
( ya)
P2
CA
2C
do 1 xx
y
TC
{ Jc)
TC
Figure 8: Neural processes compress context datasct C = faie, 1 in an invariant fashion into compact representation by an encoder. This context is then used during prediction to adapl the model appropriately given the context dataset. For conditional models, this representation is deterministic, but for latent neural processes, this context representation is a distribution that is sampled from at prodiction time.
The aforementioned methods can be used obtain initial model parameters that can serve as a starting point for fine-tuning. This fine-tuning is crucial for methods that learn single initialization for all tasks, such as the MAML-based approaches. However, we also discussed methods that learn hyper-networks to predict initialization based on a given task specification (de Avila Belbute-Peres et al., 2021; Cho et al., 2023b; Kirchmeyer et al., 2022) By using these methods without fine-tuning, we can construct zero-shot methods, that is, we can avoid training a new model when given a new PDE. In the next section, we consider neural processes, which are data-driven models that are particularly suitable for such fast adaptations.
4.3 Neural Processes
Neural processes (Garnelo et al., 2018) are probabilistic models over functions that are conditioned on a context represented by a set of measurements, cnabling the model to adapt to relevant data,
f~ g- C) C= C= y:). f
(58)
Constructing a model this way has many applications, and covers several domains discussed earlier. Deep operator networks also have a context as input, corresponding to the parameters the PDE being modeled. Multi-task learning can also be designed to identify tasks through an auxillary context input, and meta learning considers rapid adaptation to a new task, which is captured here by the context representing a small dataset. The idea of inference over functions was also covered as part of PINNs and neural operators, mesh-free modeling tochniques using function approximators provide flexible predictions and work will discuss how neural networks be used to
with more unstructured datasets. In this section, design these models and how they have been applied to modeling physical systems from data. For a more comprehensive review of these models, see Jha et al. (2022).
Architectures As a latent variable model, with latent variable z inferring from the contextual data, we discuss the architectural design decision of neural processes along three axes: the encoder, the decoder, and whether is modeled deterministically stochastically. The last point is required to differentiate conditional- and latent, NPs respectively (Garnelo el al., 2018). The trade-off here is that conditional NPs use simpler, deterministic context representations, whereas a latent NP uses a random variable for the context and, therefore, represents a richer probabilistic model but is burdened with requiring approximate inference during learning. The objective is now an amortized evidence lower bound (ELBO), which requires sampling from the encoded latent distribution and stochastic optimization using reparameterized gradients (Kingma and Welling, 2014).
max (ixx x]]] DK (C) pp)))
(59)
0
This approach benefits the model by allowing it to capture ambiguity in the approximate process induced by the finite context set of measurements. As a result, the decoder can focus on modeling statistical ('aleatoric') uncertainty while the encoder's 's latent variable captures model-based ('epistemic') uncertainty. This separa- tion important for uncortainty quantification when there is limited training data and over-parameterized models, as well as for statistical docision-making such as active learning, where the epistemic uncertainty is used actively to reduce model uncertainty.
35

---

Published in Transactions on Machine Learning Research (05/2015)
Beyond the latent variable assumption discussed above, a major component of the encoder is implementing set-based model that is invariant to the ordering of the context points. The implementation details of this aspect go beyond the remit of this survey, so we refer readers to sections 2 and 3 of Jha et al. (2022) for a comprehensive discussion, but on a high-level, many NP architectures learn a parametric feature space Φ to transform the measurement pair {me. into a combined representation Tex and then these representations are combined with a query point æ, in a set-invariant fashion with respect to C. Examples of set-invariant operations include averaging the representations into a single vector (Garnelo et al., 2018), or averaging over function evaluations (Murphy et al., 2018), such as convolution- (Gordon et al., 2019) and attention- based (Nguyen and Grover, 2022) operations. These representations are used directly by the conditional NP, while for the latent NP these representations are used for the amortized variational posterior shown in equation (59). These architectures are shown in Figure 8. The decoder model has more flexibility in incorporating useful inductive biases from deep learning, such as deconvolutions (Garnelo et al., 2018), recurrent architoctures (Willi et al., 2019), and transformers (Nguyen and Grover, 2022).
A Applications. While a relatively new architecture, NPs have been used for several prediction applications, especially for settings that have a spatial considcration that should be incorporated to improve performance.
Holderrieth et al. (2021) introduce 'steerable' conditional neural processes (CNP), which are motivated to model scalar- and vector-fields, such as temperature readings and wind maps. In this setting, relevant inductive bias is invariant to translation, rotation, and reflection, so that the model is invariant to any innocuous spatial perturbation of the data. NPs are well suited for this setting, and the authors use steerable convolutional layers in the encoder and decoder architecture to capture this invariance, as steerable feature maps have the desired invariances. The authors demonstrate that this model can effectively model mcasured wind direction vector fields from weather data, and the incorporated invariances improve generalization.
Vaughan et al. (2022) use a ConvCNP for downscaling climate data, Rather than downsample from a high to low resolution grid using an interpolation strategy, NPs are flexible enough to be evaluated at arbitrary locations, and can learn their interpolation behaviour in data-driven fashion. They show this approach outperforms an interpolation-based baseline and show the NPs contimuous nature is well suited for estimating maximum and minimum values.
One application of Bayesian prediction models is active learning (Cohn et al., 1996), where the model selects samples for its dataset to accelerate learning. This is relevant in settings where collecting data is expensive. One example of this is weather monitoring, where environmental sensors must be installed to monitor the weather, and it is desirable to place sensors optimally to improve the qualily of the dalasel and minimize hardware costs. Andersson et al. (2023) use a Gaussian ConvNP for active sensor placement. Gaussian ConvNPs are effective here because they are scalable models with tractable uncertainty quantification and can learn the non-stationary spatial and temporal phenomena seen in weather systems through the context set.
Scholz et al. (2023) consider the issue of training NP models on highly processod data, such rcanalysis data in weather forecasting, discussed in section 3.5. This data is close to ideal, e.g., spatially uniform, and smoothed with physical models, but there is a 'reality gap' between it and raw measurement data. The authors use ConvCNP to train the model first on reanalysis data, and then fine-tune on raw data. The fine-tuned model outperforms variants trained on only reanalysis or raw data
Hansen et al. (2023) consider the task of combining the data-driven NP with physical plausibility. They use an attentive NP as a solution prior, which provides an initial estimate of the solution through a multivariate Gaussian predictive distribution. To incorporate a soft baysics-based based conservation law constraint, they consider a linear constraint PDE of the form Lu(t, æ) = fx u(t, z)dr = b(t), which can be approximated using discretization, which essentially translates obcying the soft physics constraint penalty into solving regression problem subject to a weighting hyperparameter of the penalty. Note that this soft constraint becomes hard at the limit of the penalty weighting hyperparameter going to infinity. By adopting a multivariate Gaussian solution prior, this prior can be conditioned on the constraint points to yield an updated distribution that is more physically plausible in closed form. The authors primarily consider the porous medium equation, which is used to model underground fluid flow, nonlinear heat transfer, and water desalination. This PDE
36

---

Published in Transactions on Machine Learning Research (05/2015)
can incorporate challenging solution constraints and discontinuities, which is why the solution's physical plausibility is essential. The authors verify that the constraint correction step improves the solution towards the ground truth value.
5 Discussion on industrial applications
From the scale of this survey, it is clear that the combination of machine learning with physics knowledge is a rapidly expanding field, with many researchers across disciplines hoping to see the scale of progress seen in machine learning domains such as computer vision and language modeling transfer to the physical sciences. Various industrial applications of the methods described in this review paper have been reported in scientific literature. Besides more exotic applications such as utilizing physics-informee losses as accident scenario simulation in nuclear power plants (Antonello et al., 2023), utilizing meta learning to forecast industrial sales (Kück et al., 2016), predictive maintenance of batteries (Wen et al., 2023) or using physics-informer losses for non-intrusive power load monitoring (Huang al., 2022b). Additionally, we have identified several clusters of industrial applications, as detailed in Table 3, which lists relevant publications.
In sectors like mamufacturing, anomaly detection by utilizing meta-learning (Garouani et al., 2021a) physics-informed losses (Kim et al., 2022) forms a cluster of industrial applications. Another field of industrial application is the prediction and control of additive manufacturing processes with physics-informed losses (e.g- Kapusuzoglu and Mahadevan (2020), Zhu et al. (2021), Nguyen et al. (2022), Würth et al. (2023), Zhang et al. (2022b)). Guo et al. (2022) provide a thorough review of physics-informee ML applications in additive manufacturing from product design over process planning to fabrication quality control.
In the automobile industry, physics knowledge is commonly used for predictive maintenance (Wen et al. (2023)) and system identification (Hie et al., 2020; Nath et al., 2023). Using machine learning techniques, such as system identification using physics-informed neural networks (Grimm et al., 2021; Zhang et al., 2023) and operator learning for protein structure prediction (Jumper et al., 2021), has made significant advancements in the field of epidemiology and medical industries.
Industry sectors such as power (Zideh and Solanki, 2023; Hu et al., 2021; Antonello et al., 2023) and thermal (Yan et al., 2024; Cai et al., 2021c; Daw et al., 2019) have also widely incorporated the physies knowledge in machine learning for anomaly detection and parameterizing equations. Furthermore, physics-based machine learning has also found its application in robotics (Sanyal and Roy, 2023; Nicodemuss et al., 2022; Huang et al., 2022b; Kamranfar et al., 2021) where PLNNs have been used to control quadrotors and multi-lin manipulators. In optics, PINNs have also been used for system identification (Lu et al., 2021; Wang et al., 2020).
In the process industry (e.g., chemical engineering, oil and gas processing, water treatment) physics-informed lossess have been used to predict process signals (Asrav et al., 2023; Franklin et al., 2022; Ge and Chen, 2017). Failure detection was approached using meta-learning (Gao et al., 2024) and latent variable models (Kong and Ge, 2021)). Finally, plysics-informed losses have been used to create soft or virtual sensors in fluid dynamical settings in oil processing (Du et al., 2023; Franklin et al., 2022). The process control industry also utilizes physics-infor informed losses for modeling oil wells (Kittelsen et al., 2024) and water tanks (Antonelo et al., 2024).
Across industry segments, number of use cases gravitate around the prediction of wear in mechanical systems using meta-learning approaches (Chen et al., 2023b; Kamranfar et al., 2021; Li et al., 2021c; Hu et al., 2021) as well as thysics-informen losses (Hua et al., 2023).
Despite the promising applications, it is still uncertain whether these methods will replace traditional mu- merical approaches. In data-rich environments like weather forecasting, MI. has demonstrated significant progress (Lam et al., 2023; Kurth et al., 2023). However, the real challenges of applying machine learning in industrial settings, such as process industrics or manufacturing, lic their distinct requirements compared to those in the consumer market (Hoffmann et al., 2021a). Industrial settings often deal with sparse data, and operators may be skeptical of black-box systems due to their opacity.
37

---

Published in Transactions on Machine Learning Research (05/2015)
Industry
Method
Study
Aerospace
Operator learning
Percuequla (3024)
Mechanical
Parameterizing equations
Wüzth (3020)
System identification
(20226): (2023);
al. (2022)
Additive
Anomaly detection
Chen al. (2025b); Garouani al. (20221);; Kim al. (2022)
manufacturing
Industrial data science
Garouaal et al. (2021b; 2022)
Plysiic-informed informed losses
Hun et al. (2023): Kapusssoga und Mahudevan (2020)
Plysics-informed losss System identification
Automobile
Weu et al. (2023)
(2020);
(2028)
Epidemiology
System identification
(2021); Zhang (2023)
Operator learning
Engines
Anomaly detection (w/ PINNs)
Coben al. (2023); Zerasgen et al. (20255)
Solar energy
Anomaly detection (w/ PINNs) Zgtaggen et al. (2028)
Fluid
Operator Learning
Palae at al. (2020); Wen at al. (2022)
dynamics
System identification
Jin et al. (2021): Cal es al. (20214); Roi et al (2023)
Industrial
Operator Learning
Saayal Bay (2028); Nicodemaa (2022)
Physics-informed losses
robotics
Huung al. (20226)
Predictive maintenance
Kamraufar (2021)
Optics
System identification
(3021)
Physics-informe informedd losses
Wong al. (2020)
Anomaly detection (w/ PINNs) Zidah and Solanks (2023);; Ha es al. (2021)
Power industry
Physics-infeee informed losses
Antonelio es al. (2028)
Thermal
Parameterizing equations
Yan (2024); Cat es sl. (20216)
processes
Anomaly detection (w/ PINNs) Antonalo (2024): Kittolaen
al. (2019)
Process
Physics-informer losses
control
Process
Physics-informod losses
Du al. (2023); Pranieltn at al (20222); Asrave (2023)
monitoring
Anomaly detection
Gao et al. (2024); Kong and Ge (2021)
Latent variable model
Ge and Chen (2017)
Climate
System identification
Beueler al. (2019)
Cross-segment
Anomaly detection
at. (2021e)
System identification
(2016)
Table 3: Industrial application cases of machine learning with physics knowledge described in literature.
These challenges may be well addressed by combination of classical machine learning and understanding of the production process well as knowledge of the physics of the production assets. Such approaches would also improve the explainability of these models, which is a major challenge in industry rKotri et al., 2021). Furthermore, industrial solutions are usually embedded in cyber-physical systems. Hoffmann et al. (2021b) report that these pose additional challenges for various disciplines to work together. On the one hand, cyber-physical systems offer data of a physical asset as part of its "digital twin", which may ease the implementation of MT.-basod services. On the other hand, the entanglement of the physical and digital world also requires a better understanding and common terminology between developers from both worlds. inysics- machine learning stands poised to bridge this gap in the context of the industrial digital twin, tapping into substantial potential where both data and system knowledge are only partially available. In this survey, we focus on the prediction of the physical quantity of interest under the forward settings. However, many industrial applications, such as sensing, require solving a physical system under the inverse settings, namely estimating certain conditions from indirect information. The will also play an important role this regard bocause of advantages linking data and physics efficiently and robustness against data and system uncertainties. This dual capacity enhances both the applicability and reliability of machine learning in complex industrial environments.
framework
38

---

Published in Transactions on Machine Learning Research (05/2015)
6
Conclusion
This survey has covered the breadth of methods integrating machine learning with prior physics knowledge for predictive modeling, underscoring the potential to bridge the gap between traditional numerical methods and modern data-driven approaches. These methods aim to enhance reliability, robustness, and generalization out-of-distribution by leveraging scientific knowledge, particularly in environments with limited data. The various techniques surveyed, ranging from the physics-informed losses of PINNs to the purely data-driven neural processes, demonstrate the diverse ways in which prior knowledge can be encoded into machine learning models.
Significant progress has been made in domains such as weather forecasting (section 3.5), facilitated by large quantities of clean, open-access data. Such datasets have enabled significant advancements in predictive accuracy and reliability. In section A of the Appendix, we discuss the open-source ecosystem of software and datasets for physics-informet machine learning. However, in many industrial applications, similar data is unlikely to become open-source due to confidentiality concerns. Consequently, progress in these areas may be predominantly driven by private industry in a closed-soures manner, potentially limiting the collaborative and open development of new methods in academia.
Secondlly while many of the machine learning methods discussed in this survey can predict solutions rapidly, it seems unlikely they will fully replace classical solvers (Grossmann et al., 2023). The extensive time required to train ML models often diminishes their attractiveness compared to classical methods. A more effective approach could involve using ML. models to 'warm start' classical solvers, combining the speed of ML predictions with the robustness and reliability of traditional methods to achieve optimal performance.
Additionally, there is a need for more research on uncertainty quantification, especially for industrial appli- cations operating in data-scarce environments. Methods like neural processes (section 4.3), which can be trained extensive prior knowledge and fine-tuned with minimal contextual data, are particularly promis- ing. These models not only adapt quickly but also provide valuable uncertainty estimates, which are crucial for informed decision-making in practical settings. This need is further emphasized in recent. works such as Psaros et al. (2023), which provides a comprehensive overview of uncertainty quantification methods, metrics, and comparisons in scientific machine learning, and Mouli et al. (2024b), that explores the role of uncertainty quantification in improving out-of-domain generalization for PDE-based learning tasks. These contributions highlight the importance of probabilistic models like neural processes and Gaussian processes in addressing these challenges and represent key area for future exploration.
The potential for data scarcity suggests that real-world data might be effectively augmented with simulated data from physical models to enrich data-driven models. However, this decision introduces a reality gap, which suggests that the source of the data should be carcfully handled during model learning to carefully balance the sources of model bias (Scholz et al., 2023).
Finally, bhasid based machine learning will inevitably follow the trend of foundation models (Bommasani et al., 2021). This trend builds on the multi-task, meta, and contextual learning paradigms discussed in section 4 by using large capacity transformer models that can learn from large datasets, as seen for weather forecasting in section 3.5. The goal of foundation models for physics applications is to learn a model across many settings, which can then be fine-tuned on new instances, potentially enabling data-efficient models through data-intensive pretraining (Subramanian et al., 2024; McCabe et al., 2023; Herde et al., 2024).
This survey demonstrates that combining physics knowledge with machine learning has the power to reshape predictive modeling by building systems that are not only efficient and accurate but also generalizable and trustworthy. By leveraging centuries of scientific knowledge alongside the flexibility of modern machine learning, researchers can address problems that were previously deemed intractable.
As the field advances, the synergy between physics-based insights and data-driven hot will redefine scientific discovery and engineering design, unlocking solutions to challenges in domains as varied as climate modeling, materials seience, and industrial process optimization. The future of this interdisciplinary field lies in striking the perfect balance between computational rigor, data efficiency, and practical applicability, ensuring its impact across both academic and industrial landscapes.
39

---

Published in Transactions on Machine Learning Research (05/2015)
Acknowledgments
We wish to thank the anonymous reviewers for their feodback on this survey, and providing additional ref- erences related work. This work was partly funded by Hessian.ai through the project 'The Third Wave of Artificial Intelligence 3AI by the Ministry for Science and Arts of the state of Hessen, by the grant "Einrich- lung eines Labors des Deutschen Forschungseentrum für Künstliche Intelligenz (DFKI) an der Technischen Uni- versität Darmstadt", and by the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK). O.W. acknowledges the financial support provided by the Deutsche Forschungsgemensccaatt (DFG, German Research Foundation, project number 492770117).
References
Christopher M. Bishop and Nasser M Nasrabadi. Pallern recognilion and machine learning. Springer, 2006.
Alessandro Chiuso and Gianluigi Pillonetto. System identification: A machine learning perspective. Annual Review of Control, Robotics, and Autonomous Systems, 2019.
Alexandra L' L'heurenx, Katarina Grolinger, Hany Flyamany, and Miriam AM Capretz. Machine learning with big data: Challenges and approaches. IEEE Access, 2017.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.
George Em Karniadakis, Ioamnis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 2021.
Anuj Karpatne, Ramakrishnan Kannan, and Vipin Kumar. Knowledge guided machine learning: Accelerating discovery using scientific knowledge and data. CRC Press, 2022.
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 2000.
Martin G Schultz, Clara Betancourt, Bing Gong, Felix Kleinert, Michael Langguth, Lukas Hubert Leufen, Amirpasha Mozaffari, and Scarlet Stadtler. Can deep learning beat numerical weather prediction? Philo- sophical Transactions of the Royal Society A, 2021.
Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar. Fourcastnet: Accelerating global high- resolution weather forecasting using adaptive Fourier neural operators. In Proceedings of the Platform for Advanced Scientific Computing Conference, 2023.
Omar Ghattas and Karen Willcox. Learning physics-based models from data: Perspectives from inverse problems and model reduction. Acta Numerica, 2021.
Chris Aldrich and Lidia Auret. Unsupervised process monitoring and fault diagnosis with machine learning methods. Springer, 2013.
Joseph Pateras, Pratip Rana, and Preetam Ghosh. A taxonomic survey of physics- machine learning. Applied Sciences, 2023.
Shudong Huang, Wentao Feng, Chenwei Tang, and Jiancheng Lv. Partial differential equations meet deep neural networks: A survey. arXiv preprint arXit:2211.05500 2022a.
Steven L Brunton, Marko Budišié, Eurika Kaiser, and J Nathan Kutz. Modern Koopman theory for dynam- ical systems. SIAM Review, 2022.
Laura Von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, al. Informed machine learning- a taxonomy and survey of integrating prior knowledge into learning systems. IEEE Transoctions on Knowledge and Data Fngineering, 2021.
40

---

Published in Transactions on Machine Learning Research (05/2015)
Azra Seyyedi, Mahdi Bohlouli, and Seyedehsan Nedaace Oskoce. Machine learning and physics: A survey of integrated models. ACM Computing Surveys, 2023.
Rui Wang and Rose Yu. Physics-guided deep learning for dynamical systems: A survey. orXiv preprint 011.02200..01272, 2021.
Jack K Hale. Ordinary differential equations. Courier Corporation, 2009.
Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and differential- algebraic equations. Siam, 1998.
Lawrence C Evans. Partial differential equations. American Mathematical Society, 2022.
Randall J LeVeque. Finite difference methods for ordinary and partial differential equations: steady-state and lime-dependent problems. STAM, 2007.
RJ LeVeque. Finite volume methods for hyperbolic problems, 2002.
Thomas JR Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2003.
James William Thomas. Numerical partial differential equations: Finite difference methods. Springer Science & Business Media, 2013.
Robert Eymard, Thierry Gallouët, and Raphačle Herbin. Finite volume methods. Handbook of numerical analysis, 2000.
Barna Szabó and Ivo Babuška. Finile Element Analysis: Method, Verification and Validalion. John Wiley & Sons, 2021.
C. Canuto, M.Y. Hussaini, A. Quarteroni, and T.A. Zang. Spectral Methods: Evolution to Complex Geome- tries and Applications Fluid Dynamics. Scientific Computation. Springer Berlin Heidelberg, 2007.
Michael Griebel, Marc Alexander Schweitzer, et al. Meshfree methods for portial differential equations II. Springer, 2005.
David L Colton, Richard Ewing, William Rundell, et al. Inverse problems in partial differential equations. Siam, 1990.
Karl Johan Åström and Peter Fykhoff. System identification A survey. Automatica, 1971.
Arun K Tangirala. Principles of system identification: Theory and practice. CRC Press, 2018.
Simo Särkkä and Lennart Svensson. Bayesian filtering and smoothing. Cambridge University Press, 2023.
Carl Edward Rasmussen, Christopher KI Williams, al. Gaussian processes for machine learning. Springer, 2006.
arviv preprint Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412, 6980, 2014.
David E Rumelhart, Geoffrey E Hinton, Ronald Williams, al. Learning internal representations by error propagation, 1985.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukaasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. ar preprint arXio:2104, 04.1144..1344 2021.
41

---

Published in Transactions on Machine Learning Research (05/2015)
Jan E Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
Ee Weinan, A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 1 2017.
Eldad Haber, Lars Ruthotto, Elliot Holtham, and Scong-Hwan Jun. Learning across methods for convolution neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
Henning Voss, Jens Timmer, and Jürgen Kurths. Nonlinear dynamical system identification from uncertain and indirect measurements. International Journal of Bifureation and Chaos, 2004.
Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature Communications, 2021.
Marco Forgione and Dario Piga. Continuous-time system identification with neural networks: Model struc- tures and fitting criteria. European Journal of Control, 2021.
Alexander Luke lan Norcliffe and Mare Peter Deisenroth. Faster training of neural ODEs using gauB legendre quadrature. Transactions on Machine Learning Research, 2023.
Amir Gholaminejad, Kurt Keutzer, and George Biros. ANODE: unconditionally accurate memory-effcieet gradients for neural odes. In International Joint Conference on Artificial Intelligence, (IJCAI), 2019, 2019.
Juntang Zhuang, Nicha C. Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James S. Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ODE. In International Conference Machine Learning, 2020.
Derek Onken and Lars Ruthotto. Discretize-optimize vs. optimize-discretee discretize for time-series regression and continuous normalizing flows. CoRR, 2020.
Juntang Zhuang, Nicha C Dvornek, Sekhar Tatikonda, and James Duncan. Mali: memory efficient and reverse accurate integrator for neural odes. In ICLR, 2021a.
Katharina Out, Prateek Katiyar, Philipp Hemnig, and Michael Tiemann. Resnet after all? neural odes and their numerical solution. arxiv preprint oxXw::2007113866 2020.
Aditi S Krishnapriyan, Alejandro F Queiruga, N Benjamin Erichson, and Michael W Mahoney, Learning continuous models for continuous physics. Communications Physics, 6(1):319, 2023.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs. Advances in Neural
Information Processing Systems (NeurIPS), 2019.
Matthew J Zahr and P-O Persson. An adjoint method for high-order discretization of deforming domain conservation laws for optimization of flow problems. Journal of Computational Physics, 326:516 543, 2016.
Xiaoxiao Guo, Wei Li, and Francesco lorio. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 481-490 2016.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional modeling and uncertainty quantification. Journal of Computational Physics, 2018.
networks for surrogate
42

---

Published in Transactions on Machine Learning Research (05/2015)
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Compulational Mechanics, 2019.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 2019.
Daniel Greenfeld, Meirav Galun, Ronen Basri, Irad Yavnch, and Ron Kimmel. Learning to optimize multigrid PDE solvers. In International Conference on Machine Learning (ICML), 2019.
Jun-Ting Hsich, Shongjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning neu- ral PDE solvers with convergence guarantees. In International Conference on Learning Representations (ICLR), 2019.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations (ICLR), 2022.
Bing Yu et al. The deep Ritz method: deep learning-based mumerical algorithm for solving variational problems. Communications in Mathemalics and Statislãcs, 2018.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 2019.
Leah Bar and Nir Sochen. Unsupervised deep learning algorithm for PDE-based forward and inverse prob- lems. arXiv preprint arXi::1904.054 2019.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. DeepONet: Learning nonlinear operators for identi- fying differential equations based on the universal approximation theorem of operators. ar arXiv preprint arXiv:1910.0311 2019.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadnnsshcli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprinl XrXi::2003.088 03485, 2020a.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew M. Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations (ICLR), 2021a.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- operator learning for differential equa- tions. In Advances in Neurol Information Processing Systems (NeurIPS, 2021.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Trans. Neural Networks, 1995.
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. Advonces in Neural Information Processing Systems (NeurIPS), 2020a.
C Karen Liu, Aaron Hertzmann, and Zoran Popović. Learning physics- based motion style with nonlinear inverse optimization. ACM Transactions on Graphics (TOG), 2005.
Michael Lutter and Jan Peters. Combining physics and deep learning to learn contimuous-time dynamics models. The International Journal of Robolics Research, 2023.
Miles Cranmer, Sam Greydamus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXie preprint wxXiv:2003.04630 2020b.
Gene H Golub and Charles F Van Loan. Matrir compulations. JHU press, 2013.
43

---

Published in Transactions on Machine Learning Research (05/2015)
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advonces in Neural Information. Processing Systems (NewrIPS), 2019.
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina
Higgins. Hamiltonian generative networks. arXiv preprint 1Xix:1909.1170 2019.
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. arXiv preprint X 1909. 19334, 2019.
Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrinsic structure-preserving symplectic networks for identifying hamiltonian systems. Neural Networks, 2020.
Shaan A Desai, Marios Mattheakis, David Sondak, Pavlos Protopapas, and Stephen J Roberts. Port- Hamiltonian neural networks for learning explicit time-dependent dynamical systems. Physical Review E, 2021.
Cyrus Neary and Ufuk Topcu. Compositional learning of dynamical system models using Hamiltonln neural networks. In Learning for Dynamics and Control Conference, 2023.
Fabian J. Roth, Dominik K. Klein, Maximilian Kannapinn, Jan Peters, and Oliver Weeger. Stable Port- Hamiltonian Neural Networks. arXie preprint, wxXix:2502.04480 2025.
Mustafa Hajij, Ghada Zamzmi, Matthew Dawson, and Greg Muller. Algebraically-inoonnn deep networks (AIDN): deep learning approach to represent algebraic structures. arXiv preprint arXiu:2012.01141 2020.
Lóon Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade: Second Edition. Springer, 2012.
Christian Moya and Guang Lin. DAE-PINN: physics-informed neural network model for simulating dif- ferential algebraic equations with application to power networks. Neural Computing and Applications, 2023.
Arich Iserles. A first course in the numerical analysis of differential equations. Cambridge University Press, 2009.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (Part I): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXi::1711.10500 2017a.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (Part II):
Data-driven discovery of nonlinear partial differential equations. arXiv preprint arXiv:1711. 10566, 2017b.
MWMG Dissanayake and Nhan Phan-Thien. Neural-network-basss based approximations for solving partial dif-
forential equations. Communications in Numerical Methods in Engineering, 1994.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 1998.
Louis B Rall. Automatic differentiation: Techniques and Applications. Springer, 1981.
Ehsan Kharazmi, Zhongqiang Zhang, and George Em Karniadakis. Variational physics-informen neural networks for solving partial differential equations. ar preprint wrXXi:1912.00877: 2019.
Junwoo Cho, Seungtac Nam, Hyummo Yang, Scok-Bac Yun, Youngjoon Hong, and Eunbyung Park. Separable plysics-informed neural networks. Advances in Neural Information Processing Systems, 2023a.
Rui Wang, Danielle Maddix, Christos Faloutsos, Yuyang Wang, and Rose Yu. Bridging physics-based and data-driven modeling for learning dynamical systems. In Learning for dynamics and control, pages 385- 398. PMLR, 2021a.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
44

---

Published in Transactions on Machine Learning Research (05/2015)
William J Morokoff and Russel E Caflisch. Quasi-monte carlo integration. Journal of Computational Physics, 1995.
Guofei Pang, Lu Lu, and George Em Karniadakis. IPINNS: Fractional physics-informed neural networks.
SIAM Journal on Scientific Computing, 2019.
Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Meidami. Efficient training of physics-informet neural networks via importance sampling. Computer- Aided Civil and Infrastructure Engineering, 2021.
David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 1996.
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Mitigating propagation failures in informe- neural networks using resaapi-resanpl-reeeeess (13) sampling. In International Conference on Machine Learning (ICML), pages 7264 7302, 2023.
Shashank Subramanian, Robert M Kirby, Michael W Mahoney, and Amir Gholami. Adaptive self-supervision algorithms for physics-informed neural networks. In ECAI 2023, pages 2234-2241. IOS Press, 2023.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologics in physics-informed neural networks. SIAM Journal on Scientific Compuling, 2021b.
Nayara Fonseca, Veronica Guidetti, and Will Trojak. Probing optimisation in thysics-informet neural networks. In ICLR 2023 Workshop on Physics for Machine Learning, 2023.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in duysics-informer neural networks. In Advonces in Newral Information Processing Systems, 2021.
Katsiaryna Haitsiukevich and Alexander Ilin. Improved training of physics-informed neural networks with model ensembles. In International Joint Conference Neural Networks (IJCNN). IEEE, 2023.
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics- informed neural networks. arXiv preprint 2XX0:2203.07744, 2022a.
Salah A Faroughi, Nikhil Pawar, Celio Fernandes, Maziar Raissi, Subasish Das, Nima K. Kalantari, and Seyed Kourosh Mahjour. Physics-guided,, plysics-informed, and physics-encoded neural networks in sci- entific computing, 2023.
Rafael Bischof and Michael Kraus. Multi-objective loss balancing for physics- deep learning. arXiv, 2021.
Zixue Xiang, Wei Peng, Xu Liu, and Wen Yao. Self-adaptive loss balanced physics-informed neural networks. Newrocomputing, 2022.
Abdul Hannan Mustajab, Hao Lyu, Zarghaam Rizvi, and Frank Wultke. Physics-informed neural networks for figh-frequency and multi-scale problems using transfer learning, 2024.
Yo Nakamura, Suguru Shiratori, Hideaki Nagano, and Kenjiro Shimano. Physics-informed neural network with variable initial conditions. In Proceedings of the 7th World Congress on Mechanical, Chemical, and Material Engineering, Progue, Czech Republic, 2021.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics informed neural networks: Where we are and what's 's next. Journal of Scientific Computing, 2022.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica, 2021a.
Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Em Karniadakis. Physics- informed neural networks for heat transfer problems. Journal of Heat Transfer, 2021b.
45

---

Published in Transactions on Machine Learning Research (05/2015)
Bin Huang and Jianhui Wang. Applications of chysics-informed neural networks in power systems-a review. IEFE Transactions on Power Systems, 2022.
Tamara G Grossmann, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibiane Schönlieb. Can physics- informed neural networks beat the finite element method? arXiv preprinl. ar arXi::2302.04 0417, 2023.
Nikola B. Kovachki, Zongyi Li, Burigede Liu, Kamyar Arizzadenesheli, Kaushik Bhallacharya, Andrew M. Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applica- tions to PDEs. Journal of Machine Learning Research, 2023.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao; Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis, A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 2022.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Improved architectures and training algorithms for deep operator works Journal of Scientific Compulaling, 2022b.
Kaushik Bhattacharya, Bamdad Hosscini, Nikola B. Kovachki, and Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI Journal of computational mathematics, 2021.
Benjamin Peherstorfer and Karen Willcox. Data-driven operator inference for nonintrusive projection-based model reduction. Computer Methods in Applied Mechanics and Engineering. 2016.
Nikola B Kovachki, Samuel Lanthaler, and Andrew M Stuart. Operator learning: Algorithms and analysis. arXiv preprinl Xrx::22402. 15715, 2024.
Nicolas Boullé, Christopher J Earls, and Alex Townsend. Data-driven discovery of Green's functions with undaa--udeestanndalee deep learning. Scientific Reports, 2022.
LuluLuo Zhang, YaoyuE TaoZhang, WeinanJohn Xu, Zhi-Qin, and Zheng Ma. MOD-Net: A machine learning approach via model-operator-ddal data network for solving PDEs. Communications in Computational Physics, 2022a.
Craig R Gin, Daniel Shea, Steven L. Brunton, and Nathan Kutz. DeepGreen: Deep learning of Green's 's functions for nonlinear boundary value problems. Scientific Reports, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipolc graph neural operator for parametric partial differential equations. Advances in Newral Information Processing Systems (NeurIPS), 2020b.
Vladimir Sergeevich Fanaskov and Ivan Oseledets. Spectral neural operators. In Doklady Mathematics. Springer, 2023.
Francesca Bartolucci, Emmanuel de Bezenac, Bogdan Raonic, Roberto Molinaro, Siddhartha Mishra, and Rima Alaifari. Representation equivalent neural operators: a framework for alias-frce operator learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Shuhao Cao. Choose a transformer: Fourier or Galerkin. In Advances in Neural Information Processing
Systems (NeurIPS), 2021.
Bogdan Raonić, Roberto Molinaro, Tim de Ryck, Tobias Rohner, Francesca Bartolucci, Rima Alaifari, Siddhartha Mishra, and Emmanuel de Bezenac. Convolutional neural operators for robust and accurate learning of PDEs. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Jacob H. Seidman, Georgios Kissas, George J. Pappas, and Paris Perdikaris. Variational autoencoding neural operators. In International Conference on Machine Learning (ICML), 2023.
Zongyi Li, Nikola B. Kovachki, Christopher B. Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Mo- hammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, and Animashroe Anandkumar. Geometry-informed neural operator for large-scale 3d pdes. In Conference on Neural In- formation Processing Systems, (NeurIPS), 2023a.
46

---

Published in Transactions on Machine Learning Research (05/2015)
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes general geometries. J. Mach. Learn. Res., 24:38811-388:66, 2023b.
Ning Liu, Siavash Jafarzadeh, and Yue Yu. Domain agnostic Fourier neural operators. In Advonces in Neural Information Processing Systems (NeurIPS), 2023a.
Nadim Saad, Gaurav Gupta, Shima Alizadch, and Danielle C. Maddix. Guiding continuous operator learning through physics-based boundary constraints. In Conference on Learning Regresentations (ICLR), 2023.
Geoffrey Négiar, Michael W. Mahoney, and Aditi S. Krishnapriyan. Learning difforentiable solvers for systems with hard constraints. In Conference on Leorning Representations (ICLR), 2023.
Jacob Helwig, Xuan Zhang, Cong Fu, Jerry Kurtin, Stephan Wojtowylsch; and Shuiwang Ji. Group equiv- ariant Fourier neural operators for partial differential equations. In International Conference Machine Learning (ICML), 2023.
Chaoran Cheng and Jian Peng. Equivariant neural operator learning with graphon convolution. In Advances in Neurul Information Processing Systems (NeurIPS), 2023.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adap- tive Fourier neural operators: Efficient token mixers for transformers. In International Conference Learning Representations (ICLR), 2022.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, al. FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv preprinl. arXiv:2202.112 11214 2022.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science advances, 2021c
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzade- nesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equa- tions. ACM/JMS Journal of Data Science, 2021b.
Yiheng Du, Nithin Chalapathi, and Aditi S. Krishnapriyan. Neural spectral methods: Self-supervised learn- ing in the spectral domain. In International Conference on Learning Representations (ICLR), ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Bernard O Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences, 1931.
Bernard O Koopman and J v Neumann. Dynamical systems of continuous spectra. Proceedings of the
National Academy of Sciences, 1932.
J Nathan Kutz, Steven L Brunton, Bingni W Brunton, and Joshua L Proctor. Dynamic mode decomposition: Data-driven modeling of compler systems. SIAM, 2016.
Steven L Brunton, Marko Budišió, Eurika Kaiser, and J Nathan Kutz. Modern Koopman theory for dynam- ical systems. arXie preprint orXiv:2102.19066, 2021.
Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 2005.
Igor Mezić. Analysis of fluid flows via spectral properties of the Koopman operator. Annual Review of Fluid Mechanics, 2013.
Hiroya Nakao and Igor Mezić. Spectral analysis of the Koopman operator for partial differential equations. Chaos: An Inderdisciplinary Journal of Nonlinear Science, 2020.
47

---

Published in Transactions on Machine Learning Research (05/2015)
Nathan J Kutz, Joshua L Proctor, and Steven L Brunton. Applied Koopman theory for partial differential equations and data-driven modeling of spatio-temporal systems. Complexily, 2018.
Steven I, Brunton, Bingni W Brunton, Joshua I. Proctor, and J Nathan Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PloS one, 2016a.
Peter J Schmid. Dynamic mode decomposition and its variants. Annual Review of Fluid Mechanics, 2022.
Clarence W Rowley, Igor Mezić, Shervin Bagheri, Philipp Schlatter, and Dan S Henningson. Spectral analysis of nonlinear flows. Journal of Fluid Mechanics, 2009.
Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 2010.
Matthew O Williams, Toannis G Kevrekidis, and Clarence W Rowley. A data driven approximation of the Koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 2015.
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 2016b.
Samuel E. Otto and Clarence W Rowley. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 2019.
Qianxiao Li, Felix Dietrich, Erik M Bollt, and loannis G Kevrekidis. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2017.
Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for Koop- man operators of nonlinear dynamical systems. In American Control Conference (ACC). IEEE, 2019.
Hiroaki Terao, Sho Shirasaka, and Hideyuki Suzuki. Extended dynamic mode dccomposition with dictionary learning using neural ordinary differential equations. Nonlinear Theory and Ils Applications, IEICE, 2021.
S Chandra Mouli, Muhammad Alam, and Bruno Ribeiro. Metaphysica: Improving ood robustness in physics- informed machine learning. In The Twelfth International Conference on Learning Representations, 2024a.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning Koopman invariant subspaces for dynamic mode decomposition. Advances in Neural Information Processing Systems (NeurIPS). 30, 2017.
Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal lincar embeddings of nonlinear dynamics. Nalure communications,, 2018.
Daniel J Llgord-Lago, Christopher W Curtis, Alexander T Thler, and Opal Issan. Deep learning enhanced dynamic mode decomposition. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2022.
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning (ICML), 2020.
Rui Wang, Yihe Dong, Sercan Ô Arik, and Rose Yu. Koopman neural forccaster for time series with temporal distribution shifts. Xixio preprint arXio:2210. 0.06675, 2022c.
Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series
dynamics with Koopman predictors. Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Jeremy Morton, Freddic Witherden, and Mykel Kochenderfer. Deep variational Koopman models: Inferring koopman observations for uncertainty-aware dynamics modeling and control. arXiv preprint arXiv 09744,, 2019.
48

---

Published in Transactions on Machine Learning Research (05/2015)
Shaowu Pan and Karthik Duraisamy. Physics-informee probabilistic learning of linear embeddings of non- linear dynamics with guaranteed stability. STAM Journal on Applied Dynammical Systems, 2020.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR)) 2014.
N Benjamin Erichson, Michael Muehlebach, and Michael W Mahoney. Physics-informed autoencoders for Lyapunov-stable fluid flow prediction. arXiv preprint arx 10866, 2019.
Henning Lange, Steven L Brunton, and J Nathan Kutz. From Fourier to Koopman: Spectral methods for long-term time series prediction. Journal of Machine Learning Research, 2021.
Peter J Baddoo, Benjamin Herrmann, Beverley J McKeon, J Nathan Kutz, and Steven Brunton. Physics- informed dynamic mode decomposition. Proceedings of the Royal Society A, 2023.
Matthew J Colbrook. The mpEDMD algorithm for data-driven computations of meassre--peeserving dynam- ical systems. SIAM Journal on Numerical Analysis, 2023.
Matthew Williams, Clarence W Rowley, and Ioannis Kevrekidis. kernel-based approach to data-driven Koopman spectral analysis. arXiv preprint 1rXix:141..20 2014.
Matthias Seeger. Gaussian processes for machine learning. International Journal of Neural Systems, 2004.
Yoshinobu Kawahara. Dynamic mode decomposition with reproducing kernels for Koopman spectral analysis. Advances in Newral Information Processing Systems (NeurIPS), 29, 2016.
Stefan Klus, Ingmar Schuster, and Krikamol Muandet. Eigendecomposiiioos of transfer operators in repro- ducing kernel Hilbert spaces. Journal of Nonlinear Science, 2020.
Suddhasattwa Das and Dimitrios Giannakis. Koopman spectra in reproducing kernel Hilbert spaces. Applied and Compulational Harmonic. Analysis, 2020.
Vladimir Kostie, Pietro Novelli, Andreas Maurer, Carlo Ciliberto, Lorenzo Rosasco, and Massimíliano Pon- til. Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces. Advances in Neural Information Processing Systems (NewrIPS), 2022.
Petar Bevanda, Max Beier, Armin Lederer, Stefan Sosnowski, Eyke Hüllermeier, and Sandra Hirche. Koop- man kernel regression. Advances in Neural Information Processing Systems (NewrIPS), 2024.
llya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2014.
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hicrarchical latent vector model for learning long-term structure in music. In International Conference on Machine Learning (ICML), 2018.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representations, 2018.
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller. PVEs: Position-vclocity en- coders for unsupervised learning of structured state representations. arXiv preprinl arXiv:1705.0000 09800, 2017.
Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020.
Miguel Jaques, Michael Burke, and Timothy M Hospedales. NewtonianVAE: Proportional control and goal identification from pixels via physical latent spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
49

---

Published in Transactions on Machine Learning Research (05/2015)
Yuan Yin, Matthieu Kirchmeyer, Jean- Franceschi, Alain Rakotomamonjy, and Patrick Gallinari. Con- timuous PDE dynamics forecasting with implicit neural representations. In International Conference Learning Representations (ICLR), 2023.
Peter Yichen Chen, Jinxu Xiang, Dong Heon Cho, Yue Chang, G A Pershing, Henrique Teles Maia, Mau- rizio M Chiaramonte, Kevin Thomas Carlberg, and Eitan Grinspun. CROM: Continuous reduced-order modeling of PDEs using implicit neural representations. In International Conference on Learning Repre- sentations (ICLRR), 2023a.
Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 2015.
Paul Voosen, Al churns out fggtning-fast forecasts as good as the weather agencies'. Science, 2023.
Matthias Karlbauer, Danielle C Maddix, Abdul Fatir Ansari, Boran Han, Gaurav Gupta, Yuyang Wang, Andrew Stuart, and Michael W Mahoney. Comparing and contrasting deep learning weather prediction backbones on navier-stokes and atmospheric dynamics. arXiv preprint 141:12407.14299 2024.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3D neural networks. Nature, 2023.
Johannes Schmude, Sujit Roy, Will Trojak, Johannes Jakubik, Daniel Salles Civitarese, Shraddha Singh, Ju- lian Kuchnert, Kumar Ankur, Aman Gupta, Christopher E Phillips, Romeo Kienzler, Daniela Szwarcman, Vishal Gaur, Rajat Shinde, Rohit Lal, Arlindo Da Silva, Jorge Luis Guevara Diaz, Anne Jones, Simon Pfreundschuh, Amy Lin, Aditi Sheshadri, Udaysankar Nair, Valentine Anantharaj, Hendrik Hamann, Campbell Watson, Manil Maskey, Tsengdar Lee, Juan Bernabe Moreno, and Rahul Ramachandran. Prithvi wxe: Foundation model for weather and climate, 2024.
Remi Lam, Alvaro Ganzaez- Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful mediun-range global weather forecasting. Science, 2023.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez- Gonzalz,, and Peter Battaglia. Learning mesh-based simula- tion with graph networks. In International Conference on Learning Representations (ICLR), 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using noncquilibrium thermodynamics. In International Conference on Machine Learning (ICML), 2015.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in Neural
Information Processing Systems (NeurIPS), 2020.
llan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott,, Shakir Mohamed, Peter Battaglia, al. Probabilistic weather forecasting with machine learning. Nature, 2025.
Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Yuyang Bernie Wang. PreDiff: Precipitation nowcasting with latent diffusion models. Advances Neural Information Processing Systems (NeurIPS), 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Newral Information Processing Systems (NeurIPS), 2021.
Dmitrii Kochkov, Janni Yuval, Ian Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Klöwer, James Lottes, Stephan Rasp, Peter Düben, Sam Hatfield, Peter Battaglia, Alvaro Banchez-Gonzalee Matthew Willson, Michael P. Brenner, and Stephan Hoyer. Neural general circulation models for weather and climate. arXiv preprint arXiv:2311.0792) 2024.
Lennart Ljung. Approaches identification of nonlinear systems. In Proceedings of the 29th Chinese Control Conference. IEFF, 2010.
50

---

Published in Transactions on Machine Learning Research (05/2015)
Michael Lutter, Johannes Silberbauer, Joe Watson, and Jan Peters. A differentiable Newton-Eulee algorithm for real-world robotics. arXiv preprint arXiv:2110. 1222222,, 2021.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial diffcrential equations with physics-informed DeepONets. Science Advances, 2021d.
Liu Yang, Dongkun Zhang, and George Em Karmiadakis. Physics-informed generative adversarial networks for stochastic differential equations. STAM Journal on Scienlific Compuling. 2020.
Manuel Haußmann, Sebastian Gerwinn, Andreas Look, Barbara Rakitsch, and Melih Kandemir. Learning partially known stochastic dynamics with empirical PAC Bayes. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics, 2021.
Da Long, Zheng Wang, Aditi Krishnapriyan, Robert Kirby, Shandian Zhe, and Michael Mahoney. AutolP: A united framework to integrate physics into Gaussian processes. In International Conference on Machine Learning (ICML), 2022.
Elias Cucto and Francisco Chinesta. Thermodynamics of Learning Physical Phenomena. Archives of Com- pulational Methods in Engineering, 2023.
Alvaro Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to Simulate Complex Physics with Graph Networks. In International Conference Machine Learning (ICML), 2020.
Sebastian J. Wetzel, Roger G. Melko, Joseph Scott, Maysum Panju, and Vijay Ganesh. Discovering symmelry invariants and conserved quantities by interpreting siamese neural networks. Physical Review Research, 2020.
Scungwoong Ha and Hawoong Jeong. Discovering invariants via machine learning. Physical Review Research, 2021.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning (ICML), 2016.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Sran)-transffommrs:: 3D roto-translation equivariant attention networks. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In International Conference on Machine Learning (ICML), 2021.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant Diffusion for Molecule Generation 3D. In International Conference on Machine Learning (ICML), 2022.
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Equivariant and Coordinate Independent Convolutional Networks, 2023.
Fabian L. Thiemann, Thiago Reschützegger, Massimiliano Esposito, Tseden Taddese, Juan D. Olarte-Plata and Fausto Martelli. Force-free molecular dynamics through autoregressive equivariant networks, 2025.
Miroslav Silhavý. The Mechanics and Thermodynamics of Continuous Media. Springer Berlin Heidelberg, 1997.
B. A. Le, J. Yvonnet, and Q.-C. He. Computational homogenization of nonlinear elastic ic materials using neural networks. International Journal for Numerical Methods in Engineering, 2015.
Mauricio Fernández, Mostafa Jamshidian, Thomas Böhlke, Kristian Kersting, and Oliver Weeger. Anisotropic hyperclastic constitutive models for finite deformations combining material theory and data- driven approaches with application to cubic lattice metamaterials. Compulational Mechanics, 2020.
51

---

Published in Transactions on Machine Learning Research (05/2015)
Kevin Linka, Markus Hillgärtner, Kian P. Abdolazizi, Roland C. Aydin, Mikhail Itskov, and Christian J. Cyron. Constitutive artificial neural networks: A fast and general approach to predictive data-driven constitutive modeling by deep learning, Journal of Computational Physics, 2021.
Dominik K. Klein, Rogelio Ortigosa, Jesús Martincz-Frutos, and Oliver Weeger. Finite clectro-clasticity with plysics-auggmented neural nelworks. Compuler Methods in Ayplied Mechanics and Engineering, 2022a.
Karl A. Kalina, Philipp Gebhart, Jörg Brummund, Lennart Linden, WaiChing Sun, and Markus Käst- ner. Neural network-based multiscale modeling of finite strain magneto-elasticity with relaxed convexity criteria. Computer Methods in Applied Mechanics and Engineering, 2024.
Jan Niklas Fubg, Michele Marino, and Nikolaos Bouklas. Local approximate Gaussian process regression for data-driven constitutive laws: Development and comparison with neural networks. Compuler Methods in Applied Mechanics and Engineering, 2022.
Nathan Ellmer, Rogelio Ortigosa, Jesús Martínez-Frutor and Antonio J. Gil. Gradient enhanced gaussian process regression for constitutive modelling in finite strain hyperclasticity. Computer Methods in Applied Mechanics and Engineering, 2024.
A. Pérez-Escolar, J. Martinez-Frutee R. Ortigosa, N. Elimer, and A. J. Gil. Learning nonlinear constitutive models in finite strain electromechanies with Gaussian process predictors. Computational Mechamics, 2024.
Moritz Flaschel, Siddhant Kumar, and Laura De Lorenzis. Unsupervised discovery of interpretable hypere- lastic constitutive laws. Computer Methods in Applied Mechanics and Engineering, 2021.
Wojeiech Marian Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirzzez, and Razvan Pascanu. Sobolev Training for Neural Networks. arXiv preprint arXiv:1706. SSt::1706.048555 04889, 2017.
Nikolaos N. Vlassis and WaiChing Sun. Sobolev training of thermodynamicc-nformee neural networks for interpretable elasto-plasticity models with level set hardening. Computer Methods in Applied Mechanics and Engineering, 2022.
Marlon Franke, Dominik K. Klein, Oliver Weeger, and Peter Betsch. Advanced discretization techniques for hyperelastic physics-augmented neural networks. Computer Methods in Applied Mechanics and Engineer- ing, 2023.
David González, Francisco Chinesta, and Elías Cueto. Thermodynamically consistent data-driven computa- tional mechanics. Continuum Mechanics and Thermodynamics, 2019.
Moritz Flaschel, Siddhant Kumar, and Laura De Lorenzis. Automated discovery of generalized standard material models with EUCLID. Computer Methods in Applied Mechanics and Engincering, 2024.
Max Rosenkranz, Karl A. Kalina, Jörg Brummund, and Markus Kästner. A comparative study on differ- ent neural network architectures to model inelasticity. International Journal for Numerical Methods in Engineering, 2023.
Hagen Holthusen, Lukas Lamm, Tim Brepols, Stefanie Reese, and Ellen Kuhl. Theory and implementation of inclastic Constitutive Artificial Neural Networks. arXiv preprint arXio:2311.0638) 2023.
Filippo Masi, Ioannis Stefanou, Paolo Vanmucci, and Victor Matfi-Berthier. Thermodynamics- based Artificial Neural Networks for constitutive modeling. Journal of the Mechanics and Physics of Solids, 2021.
Jan Niklas Fubg, Craig M. Hamel, Kyle Johnson, Reese Jones, and Nikolaos Bouklas. Modular machine learning-based clastoplasticiy: Generalization in the context of limited data. Computer Methods in Applied Mechanics and Engineering, 2023.
Knut Andreas Meyer and Fredrik Ekre. Thermodynamically consistent neural network plasticity modeling and discovery of evolution laws. Journal of the Mechanics and Physics of Solids, 2023.
52

---

Published in Transactions on Machine Learning Research (05/2015)
Kian P. Abdolazizi, Kevin Linka, and Christian J. Cyron. Viscoelastic constitutive artificial neural net- works (vCANNs) - A framework for data-driven ropso nonlinear finite viscoelasticity. Journal of Computational Physics, 2024.
Dominik K. Klein, Mauricio Fernández, Robert Martin, Patrizio Neff, and Oliver Weeger. Polyconvex anisotropic hyperelasticity with neural networks. Journal of the Mechanics and Physics of Solids, 2022b.
Dominik K. Klein, Rogelio Ortigosa, Jesús Martincz-Frutos, and Oliver Weeger. Nonlinear clectro-clastic finite element analysis with neural network constilutive models. Compuler Methods in Ayplied Mechanics and Engineering, 2024.
Jan Fuhg, Nikolaos Bouklas, and Reese Jones. Stress representations for tensor basis neural networks: Alternative formulations to Finger-Rivlin-Eriic Journal of Computing and Information Science in Engineering, 2024a.
Mikhail Itskov.
A generalized orthotropic hyperelastic material model with application to incompressible shells. International Journal for Numerical Methods in Engineering, 2001.
Brandon Amos, Lei Xu, and J. Zico Kolter. Input Convex Neural Networks. In International Conference Machine Learning (ICML), 2017.
Lennart Linden, Dominík K. Klein, Karl A. Kalina, Jörg Brummund, Oliver Weeger, and Markus Kästner. Neural networks meet hyperelasticity: A guide to enforcing physics. Journal of the Mechanics and Physics of Solids, 2023.
Jan Niklas Fubg, Reese Edward Jones, and Nikolaos Bouklas. Extreme sparsification of physics-angmented neural networks for interpretable model discovery in mechanics. Computer Methods in Applied Mechanics and Engineering, 2024b.
Dominik K. Klein, Fabian J. Roth, Iman Valizadch, and Oliver Weeger. Parametrized polyconvex hyperc- lasticity with plysics-augmented neural networks. Dala-Centric Engineering, 2023.
Eurika Kaiser, J. Nathan Kutz, and Steven L. Brunton. Discovering conservation laws from data for control. arXiv preprint ar Xim: 1atc:181.00000 811000961, 2018.
Jack fichter-Poweld Yaron Lipman, and Ricky T. Q. Chen. Neural conservation laws: A divergence-free per- spective. In Proceedings of the 36th International Conference on Newral Information Processing Systems. 2022.
Ning Liu, Yue Yu, Huaiqian You, and Necraj Tatikola. INO: Invariant Neural Operators for Learning Complex Physical Systems with Momentum Conservation. arXiv preprint ar Xiv:22212. 143655,, 2023b.
Ning Liu, Yiming Fan, Xianyi Zeng, Milan Klöwer, Lu Zhang, and Yue Yu. Harnessing the power of neural operators with automatically encoded conservation laws. In Proceedings of the 41st International Conference on Machine Learning, 2024b.
Derek Hansen, Danielle C Maddix, Shima Alizadeh, Gaurav Gupta, and Michael W Mahoney. Learning physical models that can respect conservation laws. In International Conference Machine Learning (ICML), 2023.
Solve Eidnes, Alexander J. Stasik, Camilla Sterud, Eivind Bohn, and Signe Riemer-Sorensen. Pseudo- Hamiltonian neural networks with state-dependent external forces. Physica D: Nonlinear Phenomena, 2023.
Quercus Hernández, Alberto Badias, David Gonzalez, Francisco Chinesta, and Elias Cucto. Structure- preserving neural networks. Journal of Compulational Physics, 2021.
Quercus Hernández, Alberto Badías, Francisco Chinesta, and Elias Cueto. Thermodynamics- informed graph neural networks. IEFF Transactions on Artificial Intelligence, 2022.
53

---

Published in Transactions on Machine Learning Research (05/2015)
Tara Akhound-Sadegh, Laurence Perreault- Johannes Brandstetter, Max Welling, and Siamak Ravanbakhsh. Lie point symmetry and physios-informed networks. In Advances in Newral Information Processing Systems, 2023.
Peter Holderrieth, Michael Hutchinson, and Yee Whye Teh. Equivariant learning of stochastic fields; Gaussian processes and stecrable conditional neural processes. In International Conference on Machine Learning (ICML), 2021.
Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P.
Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data- efficient and accurate interatomic potentials. Nature Communications, 2022.
Felix Döppel and Martin Votsmeier. Robust Mechanism Discovery with Atom Conserving Chemical Reaction Neural Networks. ChemRzie pre-print 2023- Ir389, 2023. doi: /1/433/chemxxiv 2023-1r389.
Rich Caruana, Multitask learning. Machine learning, 1997.
Ricardo Vilalta and Youssef Drissi. perspective view and survey of meta-learning. Artificial intelligence revieu, 2002.
Marta Garnelo, Dan Rosenbaun, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning (ICML), 2018
Sigurd Spieckermann, Siegmund Düll, Steffen Udluft, Alexander Hentschel, and Thomas Runkler. Exploiting similarity in system identification tasks with recurrent noural networks. Neurocomputing, 169, 2015.
Alex Bird, Christopher KI Williams, and Christopher Hawthorne. Multi-task dynamical systems. The Journal of Machine Learning Research, 2022.
Zongren Zou and George Em Karniadakis. L-HYDRA: multi-head physics-informed neural networks. arXiv preprinl xrX::22301002155 2023.
Raphaël Pellegrin, Blake Bullwinkel, Marios Mattheakis, and Pavlos Protopapas. Transfer learning with physics-informed neural networks for efficient simulation of branched flows, 2022.
Cedric Flamant, Pavlos Protopapas, and David Sondak. Solving differential equations using neural network solution bundles, 2020.
Pongpisit Thanasutives, Masayuki Numao, and Ken-ichi Fukui, Adversarial multi-task learning enhancod plysics-informed neural networks for solving partial differential equations. In Indernational Joint Confer- ence on Neural Networks (LJCNN). IEEE, 2021.
Ishan Misra, Abhinav Shrivastava, Abhinay Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. arXie preprint arXiv: 705.07115 2017.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Con-
ference on Machine Learning (ICML), 2015.
Bo Wang, AK Qin, Sajjad Shafiei, Hussein Dia, Adriana-Siman Mihaita, and Hanna Grzybowska. Training inysics-informed neural networks via multi-task optimization for traffic density prediction. In International Joint Conference on Neural Networks (IJCNN). IEEE, 2023.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEFF, 2021b.
54

---

Published in Transactions on Machine Learning Research (05/2015)
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (TCMT.), 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithmss arxiv preprinl arXiv:1803. 02999, 2018.
Sen Lin, Hang Wang, and Junshan Zhang. System identification via meta-learning in linear time-varying environments. arXiv preprint orXie:2010.14604 2020.
Tian Qin, Alex Beatson, Deniz Oktay, Nick McGreivy, and Ryan P Adams. Meta-PDE: Learning to solve PDEs quickly without a mesh. arXiv preprint arXiv:2211.01604, 2022.
Sebastian Flennerhag, Pablo Garcia Moreno, Neil Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. In International Conference on Learning Representations (ICLR), 2019.
Xu Liu, Xiaoya Zhang, Wei Peng, Weien Zhou, and Wen Yao. A novel meta-learning initialization method for physics-informed neural networks. Newral Computing and Applications, 2022.
Qiaofeng Li, Tianyi Wang, Vwani Roychowdhury, and M. K. Jawed. Metalearning generalizable dynamics from trajectories. Physical Review Letters, 2023c.
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. In Advances in Newral Information Processing Systems (NeurIPS), 2022d.
Filipe de Avila Belbute-Pores, Yi-fan Chen, and Fei Sha. HyperPINN: Learning parameterized differen- tial equations with physics-informed hypernetworks. The Symbiosis of Deep Learning and Differential Equations, 2021.
Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork- based meta-learning for low-rank thysics-informed neural networks. arXiv preprint arXiv:2310. 09528, 2023b.
Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamoniy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In International Conference on Machine Learning (ICML), 2022.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. LEADS: Learn-
ing dynamical systems that generalize across environments. Advances in Newral Information Processing Systems (NeurIPS), 2021.
Tomoharu Iwata, Yusuke Tanaka, and Naonori Ueda. Meta-lcarning of plysics-informed neural networks for efficiently solving newly given PDEs. ai preprind Xri::22310. 132700, 2023.
Saurav Jha, Dong Gong, Xuesong Wang, Richard E Turner, and Lina Yao. The neural process family:
Survey, applications and perspectives. arXiv preprint "Xxx:2209.00511 2022.
Ryan L Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy pooling: Learning deep permutation-invariant functions for variable-size inputs. In International Conference on Learning Representations (ICLR), 2018.
Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In International Conference on Learning Representa- lions (ICLR), 2019.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via se- quence modeling. In International Conference on Machine Learning (ICML), 2022.
Timon Willi, Jonathan Masci, Jürgen Schmidhuber, and Christian Osendorfer. Recurrent neural processes. Xiviv preprinl 1566:990000991 2019.
Anna Vaughan, Will Tebbutt, J Scott Hosking, and Richard Turner. Convolutional conditional neural processes for local climate downscaling. Geoscientific Model Development, 2022.
55

---

Published in Transactions on Machine Learning Research (05/2015)
Tom R Andersson, Wessel P Bruinsma, Stratis Markou, James Requeima, Alejandro Coca-Castro Anna Vaughan, Anna-Louise Ellis, Matthew A Lazzara, Dani Jones, Scott Hosking, et al. Environmental sensor placeeent with convolutional Gaussian neural processes. Environmental Data Science, 2023.
Jonas Scholz, Tom R Andersson, Anna Vanghan, James Roqueima, and Richard E Turner. Sim2real for environmental neural processes. arXiv preprint or Xix:2300.. 19932, 2023.
Federico Antonello, Jacopo Buongiorno, and Enrico Zio. Physics informed neural networks for surrogate modeling of accidental scenarios in nuclear power plants. Nuclear Engineering and Technology, 2023.
Mirko Kück, Sven Crone, and Michael Freitag. Mota-learning with neural networks and landmarking for forecasting model selection an empirical evaluation of different feature sets applied industry data. In International Joint Conference on Neural Networks (IJCNN). TEEE, 2016.
Pengfei Wen, Zhi-Sheng Ye, Yong Li, Shaowci Chen, Pu Xie, and Shuai Zhao. Physics-informed neural networks for prognostics and health management of lithium-ion batteries, 2023.
Gang Huang, Zhou Zhou, Fei Wu, and Wei Hua. Physics-inford informed time-aware neural networks for industrial nonintrusive load monitoring. IEEE Transactions on Industrial Informatics, 2022b.
Moncef Garouani, Adeel Ahmad, Mourad Bouneffa, Mohamed Hamlich, Gregory Bourguin, and Arnaud Lewandowski. Towards meta-learning based data analytics to better assist the domain experts in industry 4.0. In International Conference Artificial Intelligence and Big Data in Digital Era. Springer, 2021a.
Seokgoo Kim, Joo-Ho Choi, and Nam Ho Kim. Data-driven prognosties with low-fidelity physical information for digital twin: physics-infrmme neural network. Structural and Multidisciplinary Optimization, 2022.
Berkcan Kapusuzoglu and Sankaran Mahadevan. Physics-informed and bybrid machine learning in addi- tive manufacturing: Application to fused filament fabrication. The Journal of The Minerals, Metals & Materials Society, 2020.
Qiming Zhu, Zeliang Liu, and Jinhui Yan. Machine learning for metal additive manufacturing: Predict- ing temperature and melt pool fluid dynamics using physics-infe informed neural networks. Computational Mechanics, 2021.
Thi Nguyen Khoa Nguyen, Thibault Dairay, Raphaël Meunier, and Mathilde Mougeot. Physics-informed neural networks for non-newtonian fluid thermo-men mechanical problems: An application rubber calender- ing process. Engineering Applications of Artificial Intelligence, 2022.
Tobias Würth, Constantin Krauß, Clemens Zimmerling, and Luise Kärger. Physics-informed neural networks for data-free surrogate modelling and enginecring optimization An example from composite manufac- turing. Materials & Design, 2023.
Enrui Zhang, Ming Dao, George Em Karniadakis, and Subra Suresh. Analyses of internal structures and defects in materials using ihysics-informed neural networks. Science Advances, 2022b.
Shenghan Guo, Mohit Agarwal, Clayton Cooper, Qi Tian, Robert X Gao, Weihong Guo Grace, and YB Guo. Machine learning for metal additive manufacturing: Towards physics-informed data-driven paradigm. Journal of Manufacturing Systems, 2022.
QiZhi Hc, David Barajas-Solano, Guzel Tartakovsky, and Alexandre M. Tartakovsky. Plysics-informed neural networks for multiphysics data assimilation with application to subsurface transport. Advances in Water Resources, 2020.
Kamaljyoti Nath, Xuhui Mong, Daniel Smith, and George Em Karniadakis. Physics-informed neural networks for predicting gas flow dynamics and unknown parameters diesel engines. Scientific Reports, 2023.
56

---

Published in Transactions on Machine Learning Research (05/2015)
Viktor Grimm, Alexander Heinlein, Axel Klawonn, Martin Lanser, and Janine Weber. Estimating the time- dependent contact rate of sir and seir models in mathematical epidemiology using physics-informed neural networks. Electronic Transactions on Numerical Analysis, 2021.
Zhen Zhang, Zongren Zou, Ellen Kuhl, and George Em Karniadakis. Discovering a reaction-diffusion model for alzheimer's disease by combining pinns with symbolic regression, 2023.
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Iunyasuvunakol,, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Semior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 2021.
Mehdi Jabbari Zideh and Sarika Khushalani Solanki. Physics-informed convolutional auloencoder for cyber anomaly detection in power distribution grids, 2023.
Yidan Hu, Ruonan Liu, Xianling Li, Dongyue Chen, and Qinghua Hu, Task-sequencing meta learning for intelligent few-shot fault diagnosis with limited data. IEEE Transactions on Industrial Informatics, 2021.
Bicheng Yan, Manojkumar Gudala, Hussein Hoteit, Shuyu Sun, Wendong Wang, and Liangliang Jiang, porsice-informed machine learning for noniterative optimization in geothermal energy recovery. Applied Energy, 2024.
Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Em Karniadakis. Neural Networks for Heat Transfer Problems. Journal of Heat Transfer, 2021c
Arka Daw, R. Quinn Thomas, Cayclan C. Carcy, Jordan S. Read, Alison P. Appling, and Anuj Karpatne. Physics-guided architecture (PGA) of neural networks for quantifying uncertainty in lake temperature modeling, 2019.
Sourav Sanyal and Kaushik Roy. RAMP-Net: A robust adaptive MPC for quadrotors via physics-informed neural network. In IEEE International Conference Robotics and Automation (ICRA), 2023.
Jonas Nicodemus, Jonas Kncifl, Jörg Fehr, and Benjamin Unger. Physics-informed neural networks-based model predictive control for multi-link manipulators. IFAC- MTAC-PuperSOnliin, 2022.
Parastoo Kamranfar, Jelf Bynum, David Lattanzi, and Amarda Shehu. Meta-learning for industrial system monitoring via multi-objective optimization. In Advances in Data Science and Information Engineering: Proceedings from ICDATA 2020 and IKE 2020. Springer, 2021.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G. Johnson. Physics- informed neural networks with hard constraints for inverse design, 2021.
Chulin Wang, Eloisa Bentivegna, Wang Zhou, Levente Klein, and Bruce Elmegreen. Physics-informed neural network super resolution for advection-diffusie models, 2020.
Tuse Asrav, Ece Serenat Koksal, Elif Ecem Esenboga, Ahmet Cosgun, Gizem Kusoglu, Duygu Aydin, and Erdal Aydin. Physics-informed neural network based modeling of an industrial wastewater treatment unit. In Computer Aided Chemical Engineering. Elsevier, 2023.
Taniel S Franklin, Leonardo S Souza, Raony M Fontes, and Márcio AF Martins. A networks (pinn) oriented approach to flow metering in oil wells: an esp lifted oil well system as a case study. Digital Chemical Engineering, 2022.
neural
Zhiqiang Ge and Xinru Chen. Dynamic probabilistic latent variable model for process data modeling and regression application. IEFE Transactions on Control Systems Technology, 2017.
57

---

Published in Transactions on Machine Learning Research (05/2015)
Yu Gao, Jin Qi, Ying Sun, Xiaoxuan Hu, Zhenjiang Dong, and Yanfei Sun. Industrial process fault diag- nosis based on feature enhanced meta-learning toward domain generalization scenarios. Knowledge--aase Systems, 2024.
Xiangyin Kong and Zhiqiang Ge. Deep learning of latent variable models for industrial process monitoring. IEEE Transuctions on Industrial Informalics, 2021.
Jian Du, Jianqin Zheng, Yonglu Liang, Ning Xu, Jiří Jaromír Klemeč, Bohong Wang, Qi Liao, Petar Sabev Varbanov, Khurram Shahzad, and Arshid Mahmood Ali. Deeppipe: A two-stage physics-informed informed neural network for predicting mixed oil concentration distribution. Energy, 2023.
Jonas Ekeland Kittelsen, Eric Aislan Antonelo, Eduardo Camponogara, and Lars Struen Imsland. Physics- informed neural networks with skip connections for modeling and control of gas-lifted oil wells. Applied Soft Compuling, 2024.
Eric Aislan Antonelo, Eduardo Camponogara, Laio Oriel Seman, Jean Panaioti Jordanou, Eduardo Re- hbein de Souza, and Jomi Fred Hübner. Physics-informed neural nets for control of dynamical systems. Neurocompuling, 579:127419, 2024.
Jiao Chen, Jianhua Tang, and Weihua Li. Industrial edge intelligence: Feclerated-meta learning framework for few-shot fault diagnosis. IEEE Transactions on Network Science and Engineering, 2023b.
Chuanjiang Li, Shaobo Li, Ansi Zhang, Qiang He, Zihao Liao, and Jianjun Hu. Meta-learning for few-shot bearing fault diagnosis under complex working conditions. Newrocompating, 2021c.
Jiaqi Hua, Yingguang Li, Changqing Liu, Peng Wan, and Xu Liu. Physics-informed neural networks with weighted losses by uncertainly evaluation for accurate and stable prediction of manufacturing systems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Adolfo Perrusquía, Weisi Guo, Benjamin Fraser, and Zhuangkun Wei. Uncovering drone intentions using control physics informed machine learning. Communications Engineering, 2024.
Moncef Garouani, Adeel Ahmad, Mourad Bouneffa, Arnaud Lewandowski, Grégory Bourguin, and Mo- hamed Hamlich. Towards the automation of industrial data science: A meta-learning based approach. In International Conference on Enterprise Information Systems, 2021b.
Moncef Garouani, Adeel Ahmad, Mourad Bouneffa, Mohamed Hamlich, Gregory Bourguin, and Arnaud Lewandowski. Using meta-learning for automated algorithms selection and configuration: an experimental framework for industrial big data. Journal of Big Dala, 2022.
Joseph Cohen, Xun Huan, and Jun Ni. Fault prognosis of turbofan engines: Eventual failure prediction and remaining useful life estimation. International Journal of Prognostics and Health Management, August 2023.
Jannik Zgraggen, Yuyan Guo, Antonio Notaristefano, and Lilach Goren Huber. Fully unsupervised fault de- tection in solar power plants using physics-informed deep learning. In 33rd European Safely and Reliability Conference (ESREL). Research Publishing, 2023.
Solon Falas, Charalambos Konstantinou, and Maria K. Michael. Physics-informed neural networks for sccuring water distribution systems, 2020.
Gege Wen, Zongyi Li, Kamyar Azizwaeenshelii, Anima Anandkunar, and Sally M. Benson. U-FNO An enhanced Fourier neural operator- based deep-learning model for multiphase flow. Advances in Water Resowrces, 2022.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. NSFnets (Navicr-Stokes flow nets): Physics- informed neural networks for the incompressible Navier-Stokes equations. Journal of Computational Physics, 2021.
58

---

Published in Transactions on Machine Learning Research (05/2015)
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics-informet neural networks (pinns) for fluid mechanics: A review, 2021d.
En-Ze Rui, Zheng- Wei Chen, Yi-Qing Ni, Lei Yuan, and Guang-Zhi Zeng. Reconstruction of 3D flow field around a building model in wind tunnel: A novel physics-informed neural network framework adopting dynamic prioritization self-adaptive loss balance strategy. Engineering Applications of Computational Fluid Mechonics, 2023.
Tom Beucler, Stephan Rasp, Michael Pritchard, and Pierre Gentine. Achieving conservation of energy in neural network emulators for climate modeling, 2019.
Martin W. Hoffmann, Rainer Drath, and Christopher Ganz. Proposal for requirements on industrial AT solutions. In Machine Learning for Cyber Physical Systems, 2021a.
Arzam Kotriwala, Benjamin Klöpper, Marcel Dix, Gayathri Gopalakrishnan, Dawid Ziobro, and Andreas Potschka. Xai for operations in the process industry-appliication, theses, and research directions. In AAAI spring symposium: Combining machine learning with knowledge engineering, 2021.
Martin W Hoffmann, Somaych Malakuti, Sten Grüner, Socren Finster, Jörg Gebhardt, Ruomu Tan, Thorsten Schindler, and Thomas Gamer. Developing industrial CPS: A multi-disciplinary challenge. Sensors, 2021b.
Apostolos F Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty quan- tification in scientific machine learning: Methods, metrics, and comparisons. Journal of Computational Physics, 477:111902, 2023.
S Chandra Mouli, Danielle C Maddix, Shima Alizadeh, Gaurav Gupta, Andrew Stuart, Michael W Mahoney, and Yuyang Wang. Using uncertainty quantification to characterize and improve out-of-domain learning for pdes. Xixi preprint arXiv:2403. 10642, 2024b.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, al. On the opportunities and risks of foundation models. Xrxiv preprint Xrx:::1108.0 072558, 2021.
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Michael McCabe, Bruno Régaldo-Saing Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Al- berto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawczik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. In NewrIPS 2023 Al for Science Workshop, 2023.
Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient. foundation models for pdes. arx preprinl arXiv:2405. 19101, 2024.
Zongyi Li, Miguel Liu Ath-Schiffini, Nikola Kovachki, Kamyar Azizzadeneshelli Burigede Liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Learning chaotic dynamics in dissipative systems. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for PDEs on general geometries. arXie preprint arXi::2207.0524 055209, 2022b.
Feiyu Chen, David Sondak, Pavlos Protopapas, Marios Mattheakis, Shuheng Liu, Devansh Agarwal, and Marco Di Giovanni. Neurodiffeq: A python package for solving differential equations with neural networks. Journal of Open Source Software, 2020.
Makoto Takamoto, Timothy Praditia, Raphael Leitcritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger,
and Mathias Niepert. PDEBench: An extensive benchmark for scientific machine learning. In Advances in Neural Information Processing Systems Track Datasets and Benchmarks, 2022.
59

---

Published in Transactions on Machine Learning Research (05/2015)
Jayesh K Gupta and Johannes Brandstetter. Towards multispattioeemppoaa-ssael generalized pde modeling.
arviv preprint arXi::200. 15616, 2022.
Felix Koehler, Simon Niedermayr, Rüdiger Westermann, and Nils Thuerey. APEBench: A benchmark for autoregressive neural emulators of PDEs. Advances in Newral Information Processing Systems (NeurIPS). 38, 2024.
Karl Otness, Arvi Gjoka, Joan Bruna, Daniele Panozzo, Benjamin Peherstorfer, Teseo Schneider, and De- nis Zorin. An extensible benchmark suite for learning simulate physical systems. arXiv preprint urXiv:208.007999 2021.
Ruben Ohana, Michael McCabe, Lucas Thibaut Meyer, Rudy Morel, Fruzsina Julia Agoes, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B. Dalziel, Drummond Buschman Fielding, Daniel For- tunato, Jared A. Goldberg, Keiya Hirashima, Yan-Fei Jiang, Rich Kerswell, Suryanarayana Maddu, Jonah M. Miller, Payel Mukhopadhyay, Stefan S. Nixon, Jeff Shen, Romain Walleaux, Bruno Régaldo- Saint Blancard, François Rozet, Liam Holden Parker, Miles Cranmer, and Shirley Ho. The well: large-scale collection of diverse physics simulations for machine learning, In The Thihty- Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: //openreiiew wwweeeiviiieeeeeeeeeeeeeeeeeeeee www./eettouuueee wtt.::eeeettttttttttddt////////a 0d00055777BT.
60

---

Published in Transactions on Machine Learning Research (05/2015)
A Open-source Libraries and Datasets
This section reviews the open-source projects that are related to physics-informed prediction. denotes the number of GitHub stars at the time of writing,
Iululxvi/DeepXDE 21 is a library for PINNs and Deep Operator models that is implements for PyTorch, Jax and TensorFlow backends. The library accommodates intricate domain geometrics without mandat- ing mesh generation, utilizing diverse primitive shapes like intervals, triangles, rectangles, polygons, disks, ellipses, cuboids, spheres, hypercubes, and hyperspheres, with the ability to construct other shapes using con- structive solid geometry operations. It also handles geometries represented by point clouds. DeepXDE offers five boundary condition types, including Dirichlet, Neumann, Robin, periodie, and general BC adaptable to arbitrary domains or point sets, alongside approximale distance functions for hard ronsts The library incorporates diverse sampling methods such as uniform, pseudorandom, Latin hypercube sampling, Halton sequence, Hammersley sequence, and Sobol sequence. DeepXDE's loosely coupled components contribute to its well-structured nature, enabling high configurability and ease of customization to cater to evolving requirements.
paml/neuromancer [601 (Neural Modules with Adaptive Nonlinear Constraints and Efficient Regular- izations) is library for solving parametric constrained optimization problems, physies-informed system identification, and parametric model-based optimal control using automatic differentiation. It contains im- plementation of PINNs, neural ODEs, Koopman operator-based timeseries models. It is written in PyTorch and integrates optimization libraries such as evx.
NVIDIA/modulus [456 is a library for physics-based ML written in PyTorch. While focusing on Neural Operators and PINNs, also supports projects involving diffusion models and graph neural networks. It also has front-end for processing symbolic equations, and back-end for multi-node, multi-GPU large-scale training. It also supports complex domains, contraints and boundary conditions.
SciML is a collection of libraries in julia for physics- ML. To date, it contains 156 repositories, includ- ing SciML/Diff www.ifffeeeiaaaaaaaeeae ialeaatiioinn j1 [ for differential equation solvers and SciML/Diifffqllux j1
ji [200*} for neural operators. The SciMI suite in Julia leverages the language's 's design, emphasizing performance and ease of use, allowing users to seamlessly combine symbolic mathematics, automatic differentiation, and efficient numerical solvers. This ecosystem offers a variety of solvers for differential equations and flexible interface for building models, enabling streamlined process for scientific simulations and model develop- ment. Compared to Python, Julia's SciML packages often demonstrate superior performance due to Julia's 's focus on high-performance computing and its just-in-time (JIT) compilation, resulting in faster execution times for numerical simulations and complex scientific computations. However, Python libraries such as JAX also provide JIT optimization as well.
1806 for neural ODEs, SciML/NeuuaaPDD..... j1, [852 + for PINNs and SciML//euuaalperaattt
[1600 in PyTorrc,, is the accompanying imple- mentation from Li et al. (2021a). The github profile also has reference implementations of thysics-informed neural operators (Li et al., 202Tb), graph kernel network (Li et al., 2020b), (Li et al., 2022a) and geometry- aware Fourier neural operator (Li et al., 2022b).
For neural operators, muuraloperatornnurraaa
Many more librarics exist. For neural differential equations, there is MeuroDifffGmm//eurood by (Chen et al., 2020) in Tyrorch. However, somewhat confusingly, the 'neural differential equations' implemented this library are essentially PINNs, but the authors cite the older litcrature. For PINNs, there is analysiscenterrpydee which implements a few extensions to PINN models. idrl-lab/idrlnet, also tackling PINNs in PyTorch, but can handle complex domain geometrics. tensordiiffeq/TensoDDffft [102*) also implements PINNs, but using Tensorflow 2. In general, while these libraries appear impactful, they are typically older and less maintained as those discussed above.
[554+]
[266*] and Beatom--A---ssaarcc/lllaaaaloo
[118*] in PyTorch,
[545*) by Takamoto et al. (2022) a benchmark suite for PDE solvers to evaluate MLsbased methods across a range of PDEs, parameters, and boundary conditions. Additionally, PDEArena [241 + by Gupta and Brandstetter (2022) provides an extensible platform for benchmarking PDE
For evaluation, wwwpppcc/peeeeeeeeeeeepppp
61

---

Published in Transactions on Machine Learning Research (05/2015)
solvers, while APEBench [59 x] focuses on adaptive physics-based based machine learning techniques (Kochler et al., 2024). nn mn-benchmark [25 is designed for evaluating neural networks on scientific computing tasks (Otness et al., 2021), and The Well [746*] serves by Ohana et al. (2024) as a benchmark for geophysical modeling and machine learning in subsurface environments.
62


Reviews:
[{"review_id": "eAzrignVAm", "reviewer": "Reviewer_Zh69", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a really interesting paper that has done a lot of useful work, but has weaknesses in the execution. I suspect that this mostly terminology difficulties and easy to fix, but I am not 100% certain.\n\nStrengths:\n\n* Comprehensive overview of many methods \n* Lineages of research are well connected\n* surfaced some excellent lesser-known papers.\n\nWeaknesses\n\n* Non-standard terminology use — I do not think that the distinctions that they draw with  _inductive bias_  and _function approximation_  are used in the standard way\n* taxonomy seems muddy to me; I think an attempt made to draw out some different axes of variation amongst the ML models, but the axes are either confounded or not clearly communicated\n* many small typos and unclear statements (For the sake of time I will skip these in my initial review and add a comment with them later)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors itemize and summarize various lineages of methods that incorporate both physics and machine learning. The taxonomy explores a few axes of variation, in terms how physics is imposed, the structure of the interaction that may be learned, and how general the model learned is.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It seems to me that a few concepts are confounded in the authors' taxonomy. This might be a matter of definitions being unclear, or of deeper confusion. I recommend that it resolved other by crisper and more consistent definition of important terms, or by adjusting the actual taxonomy. I will mention what seem to me to be the biggest examples of the difficulties.\n\n## Figure 1\n\nI applaud the authors for diagramming the taxonomy they are understanding. great idea. Figure 1 seems plausible, but there are some questions that I feel I must request answers for. Let us examine the categories\n\n* Physics-informed models\n* Physics-informed losses\n* Data-driven generalization\n* Physics-informed data augmentation \n* Learning with contexts\n\nSince all these topics relate to the methods taxonomised in this paper, would it not make sense to link to which sections of the paper deal with these, e.g. should \"physics informed losses\" link to section 3.2\n\nWhy is there a color bar on this graph from \"informed\" to \"uninformed\"? There are no shade of gray on the graph that I can see.\n\n## Table 1\n\nThe central distinctions made between \"solve/infer/generalize\" seem very important but are only briefly explained:\n\n> learning can refer to the solution (or simulation) of a physical system, inferring the\n> physical system from data, and learning a model that generalizes across several instantiations of the physical system, e.g., different boundary condition\"\n\nI think this need to be mathematically formalised, since we use it a lot. \"Solve\" sounds like \"produce values on the solution surface $u_\\theta$, infer means something like learn $\\theta$ which is a parameter generating  $u_\\theta$, and generalize means something like learning $u_\\theta| x$ for some covariate $x$? If so, may I request the authors clarify? \n\nThe entries in the column headed _inductive bias_ suggest a non-standard use of the term, or at least I don't understand the use here fully because it seems to mean something more specific than I am used to. What does \"Structure objective\" mean, for example? It sounds like the authors wish to describe when and how the known laws of physics are applied to the learning algorithm. If so, this is an excellent idea, but the table should be clarified so that we know what they mean. Is the \"Structure objective\" the part of the inductive bias which includes known laws of physics? Or is the the structure of the generating process which produces the data, such as a hypothesised structural equation model underlying the data? Or is it both?\n\nConcretely, when I say Structural Equation Model, I mean this kind of thing:\n```text\n     +--------+       +--------+       +--------+  \n     | θ      |--->   | x_t-1  |--->   |  x_t   |---> ...\n     +--------+       +--------+       +--------+  \n                        |               |\n                        v               v\n                +--------------+   +--------------+\n                |  y_t-1       |   |   y_t        |\n                +--------------+   +--------------+\n```\nThe above is the graphical model encoding of a system identification problem. See, e.g. Koller and Friedman 2009. I suspect that conceptually separating the graphical structure, and the physics structure of the problems might make the paper much clearer.\n\n\n## 3.6 Bridging System Identification and Function Approximation\n\nSection 3.6 is a good example of a non-standard terminology use and confusing taxonomy used together. _Function approximation_ (introduced in section 2.4) is simply learning to approximate a function. System identification (section 2.3) is about learning systems with serial dependence in time conditional upon known inputs $w_t$ and latent parameter $\\theta$.\n\n>System identification involves defining the dynamics of a system through a sequence of inputs $\\boldsymbol{w}_t$, measurements $\\boldsymbol{y}_t$, and an initial state $\\boldsymbol{x}_1$ over a finite horizon $\\mathcal{T}$ as a trajectory, i.e. $\\boldsymbol{Y}=\\left[\\boldsymbol{y}_1, \\ldots, \\boldsymbol{y}_T\\right]$ . [where ] the state $\\boldsymbol{x}_{t+1}=\\boldsymbol{f}_\\theta\\left(\\boldsymbol{x}_t, \\boldsymbol{w}_t\\right)$ and the observation $\\boldsymbol{y}_t=\\boldsymbol{g}_{\\boldsymbol{\\theta}}\\left(\\boldsymbol{x}_t\\right)$ are both parameterized by $\\boldsymbol{\\theta}$, […] A probabilistic approach considers the joint distribution $p(\\boldsymbol{Y}, \\boldsymbol{X} \\mid \\boldsymbol{W}, \\boldsymbol{\\theta})$ which, continuing the Markov assumption, factorizes into $p\\left(\\boldsymbol{x}_1\\right) \\prod_{t=1}^T p\\left(\\boldsymbol{y}_t \\mid \\boldsymbol{x}_t, \\boldsymbol{\\theta}\\right) p\\left(\\boldsymbol{x}_{t+1} \\mid \\boldsymbol{x}_t, \\boldsymbol{w}_t, \\boldsymbol{\\theta}\\right)$. \n\nSo far I think this is not to controversial. I have seen the problem of identifying $\\theta$ as system identification in the literature, but also sometimes the identification of the unobserved $X$ is included. Then the uasge of this term gets confusing:\n\n>System identification approaches lie on a spectrum depending on modeling assumptions. If a lot of domain knowledge is assumed, the dynamics model will be highly structured. If correct, it will generalize well but will most likely be inflexible to un-modelled phenomena. On the other hand, one could model with no modeling assumptions and use a black-box function approximator\n\nI am not sure what is being said here. As far as I understand it, almost _everything_ in this paper is function approximation according to the definitions introduced by the authors. So there is nothing to \"bridge\". If we read further into section 3.6 I cannot work out what the common theme is: there are GANs, there are PINNs, there are GPs with a physics-informed kernel.  What is the common theme in 3.6? Possibly it is simply _system identification_. If so great! This could be clarified in the first paragraph of the section.\n\n## What are the dimensions of the paper's taxonomy?\n\n_System identification_ seems to me to be an assumption about the underlying graphical model generating the data, and that does not need to involve physics, necessarily. But we can imagine that it *might*, if for example the system to be identified was generated by the laws of physics. If I can propose my idea about what the authors are trying to get at here I think there might be two separate axes of variation, the probabilistic graphical model underlying the system of interest, and the means by which physics is integrated. For example, my way of writing that would be as follows:\n\n| Section / Technique              | Generating Structural Equation Model      | Means of Physics Integration                     |\n|-----------------------------------|----------------------------------------------------|---------------------------------------------------|\n| **3.1 Neural ODEs/PDEs**         | Continuous-time state-space models                 | Architectural physics (ODE/PDE structure)        |\n| **3.2 Physics-Informed Losses**   | Deterministic regression with residual constraints | Strong physics (PDE residuals as objectives)      |\n| **3.3 Hamiltonian/Lagrangian NNs**| Structured dynamical systems                       | Full physics (energy conservation built-in)       |\n| **3.4 Neural Operators**         | Operator-valued regression                        | Physics-informed architectures (Green's functions)|\n| **3.5 Koopman Operators**         | Linear latent space models                         | Physics-inspired (spectral decomposition)       |\n| **3.6 Hybrid System ID**          | Hierarchical Bayesian models                       | Partial physics (blackbox + known terms)         |\n| **3.7 Invariance Methods**        | Factor graphs with symmetry constraints            | Strong physics (Noether's theorem applications)   |\n| **4.1 Multi-task Learning**     | Hierarchical Bayesian models                    | Data-driven               |\n| **4.2 Meta Learning**           | Hierarchical Bayesian models           | Data-driven                  |\n| **4.3 Neural Processes**       | Hierarchical Bayesian models              | Data-driven                  |\n\nHere is an alternative table that categories the methods differently, generated by an LLM:\n\n| Section & Method              | Integration Mechanism          | Learning Objective              | Generating Structural Equation Model     |\n|-------------------------------|----------------------------------|----------------------------------|---------------------------------------|\n| **3.1 Neural ODEs/PDEs**       | Architectural constraints       | Solve PDEs/Infer parameters      | Continuous-time dynamics              |\n| **3.2 PINNs**                  | Loss-based physics              | Solve PDEs                       | Structured objective (PDE residuals) |\n| **3.3 Hamiltonian/Lagrangian** | Architectural constraints       | Infer system parameters          | Symplectic structure                  |\n| **3.4 Neural Operators**        | Architectural constraints       | Generalize across conditions     | Multi-instance data                   |\n| **3.5 Koopman Autoencoders**    | Architectural constraints       | Infer system parameters          | Structured latent models             |\n| **3.6 Hybrid System ID**        | Mixed architectural/loss-based   | Solve & Infer parameters         | Structured models/objectives          |\n| **3.7 Invariance Priors**       | Architectural constraints       | Solve & Generalize              | Structured models/features/data/obj.  |\n| **4.1 Multi-task Learning**     | Data-driven                     | Generalize across tasks          | Multi-instance data                   |\n| **4.2 Meta Learning**           | Data-driven                     | Generalize across conditions      | Multi-instance data                   |\n| **4.3 Neural Processes**       | Data-driven                     | Solve & Generalize              | Contextual data                       |\n\nBoth these seem like reasonable taxonomies to me, but I am open to other ones. They key point is that the paper should consistently use terminology so that we can infer such a taxonomy.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ODYtGbj6W4", "reviewer": "Reviewer_MFpK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: comprehensive survey.\n\nWeakness: I don't foresee any. This is a much needed paper in this evolving domain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper covers the recent development of ML + Physics, with a focus on PDE approximation. The survey is comprehensive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A schematic graph at the beginning to describe the relationship between different concepts/approaches would be great.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QgfgKAskNn", "reviewer": "Reviewer_4egC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Nice comparison of methods that add physics into the architecture or loss function vs. just having physics satisfied in the data.\n- Nice overview of an underexplored and important topic of using machine learning in scientific fields through PDEs and so the survey is very relevant and needed in the field.\n- Nice overview in the introduction and the challenges for adopting ML for real-world scientific applications especially with limited data.\n- Good survey across different state-of-the-art methods for SciML, e.g., PINNs and Neural Operators rather than just focusing on one of them.\n- Nice perspective from the practical engineering community\n- Good that boundary conditions are introduced and the three common ones are addressed. See Saad et. al \"Guiding continuous operator learning through Physics-based boundary constraints\", ICLR, 2023 for a general representation of linear boundary conditions using G operator,\n- Nice connection at the beginning of section 3 that there are other areas of machine learning where domain knowledge is incorporated through the architecture, i.e., CNNs in computer vision.\n- Good motivation on solving PDEs on very fine spatio-temporal resolutions and the challenges, need for ML methods, ways to improve them.\n- Nice discussion and overview of the limitation of PINNs in terms of the optimization.\n- Good discussion about the comparison to PINNs and classical methods, e.g., FEM and the current limitations of PINNs on practical problems.\n- Nice physical application discussion in Section 3.7 and the importance of invariances and conservation.\n- Table 3 provides a useful and practical summary\n- Good motivation in identifying the limitations in applying these methods in industry and with real sparse data\n- Good survey to bridge the gap between classical numerical methods and ML models and emphasizing that the accuracy of the ML models still does not match that.\n\n## Weaknesses\n-  Missing citation of another overview of ML for physics methods: Wang, R., Yu, R., \"Physics-Guided Deep Learning for Dynamical\nSystems: A Survey\", https://arxiv.org/pdf/2107.01272, 2023.\n- It is true that physics-informed methods are useful for weather forecasting and FourCastNet is cited but GraphCast which is state-of-the-art is purely data-driven GNN method and Pangu-Weather is data-driven and Transformer based. Both works should be cited.\n- The Solve column in Table 1 is a bit unclear with the description of the methods.\n- While PINNs and Neural Operators are discussed, references to another state-of-the-art GNN-based MeshGraphNet (Pfaff et. al, \"Learning Mesh-Based Simulation with Graph Networks\", ICLR 2021) model is not directly discussed - only through GraphCast which is based on this architecture.\n- Missing reference Saad et. al, \"Guiding continuous operator learning through Physics-based boundary constraints\", ICLR, 2023 on hard-constraining Neural Operators to satisfy boundary conditions and Negiar et al. \"Learning differentiable solvers for systems with hard constraints\", ICLR, 2023 on enforcing PDEs as a hard-constraint rather than a soft-constraint in PINNs, which can also be added to the Operator Learning section.\n- Additional references include Chalapathi, N. et. al, \"Scaling physics-informed hard constraints with mixture-of-experts\", ICLR 2024 and Du et. al, \"Neural Spectral Methods: Self-supervised learning in the spectral domain\", ICLR 2024 especially the latter would be relevant to cite since spectral methods are also discussed.\n- For the physics-informed loss on the medical and ODEs, I think Wang et. al, \"Bridging physics-based and data-driven modeling for learning dynamical systems\", L4DC, 2021 would be good to cite.\n- Missing reference to LeVeque, \"Finite Difference Methods for Ordinary and Partial Differential Equations\", SIAM, 2007 and LeVeque, \"Finite-Volume Methods for Hyperbolic Problems\", Cambridge University Press, 2002 and Hughes, \"The Finite Element Method: Linear Static and Dynamic Finite Element Analysis\" for the numerical methods\n- A smoother transition between sections 2.1-2.2 and 2.3 would be nice. It currently jumps from ODEs and PDEs to probabilities and loss functions. It is also unclear where $\\mathcal{L}$ is defined in Eqn. 5.\n- I think the NN background in Section 2.4 should be familiar to the reader and can be moved to an appendix.\n- On page 5, add reference for a RNN\n- While the background on Neural ODEs is thorough, I'm not sure if it is needed. Additional references here include Ott et. al, \"ResNet After All? Neural ODEs and their Numerical Solution\", 2020 and Krishnapriyan et. al, \"Learning continuous models for continuous physics\", Communications Physics 6 (1), 319, 2023.\n- There is also a nice potential connection to highlight between the adjoint method in Neural ODEs and the adjoint method using by Per-Olof Persson and Zhar in constrained optimization numerical methods for PDEs.\n- On page 8 it would be good to add reference to the Multi-wavelet Neural Operator: Gupta et. al, \"Multiwavelet-based Operator Learning for Differential Equations\", NeurIPS, 2021.\n- For clarity in Eqn. 13 should refer back to Eqn. 3 for the PDE definition and variables\n- Typo in Poisson's Eqn at the end of page 8.\n- Another method to address the limitations with PINNs to be cited is Subramanian, S., et. al, \"Adaptive self-supervision algorithms\nfor physics-informed neural networks,\" arXiv:2207.04084, 2022, which proposes an adaptive update of collocation points.\n- I've also seen in practice that BFGS works better in PINNs optimization. It would be nice if the authors explained a bit more into why.\n- The authors may consider reordering the subsections 3.2 and 3.3. There is quite a jump from learning the solution with PINNs in 3.2 to learning the equations in 3.3 and going back to ODEs.\n- The Moore-Penrose inverse is briefly discussed at the end of page 11 and I think the numerical challenges of computing it should also be addressed.\n- Since POD and DMD is mentioned on pg. 14, I think adding references to works on ROM for background would also be useful.\n- There are recent works on general geometries Li, Z. \"Geometry-Informed Neural Operator for Large-Scale 3D PDEs\", NeurIPS 2023 and Li, Z. \"Fourier Neural Operator with Learned Deformations for PDEs on General Geometries\", Journal of Machine Learning Research 24 1-26, 2023, which should be added to address the limitations of a fixed uniform mesh\n- Multi-wavelet Neural Operator should be included with the spectral methods on page 16\n- Chebyshev is spelled incorrectly\n- Du et. al, \"Neural Spectral Methods: Self-supervised learning in the spectral domain\", ICLR 2024 is another recent spectral-based method to add.\n- There is a discussion about adding PINNs to Neural Operators with PINO. PINO paper and Saad et. al \"Guiding continuous operator learning through Physics-based boundary constraints\", ICLR, 2023 show that sometimes adding this physics-informed loss does not improve performance.\n- There is a jump from Neural Operators in 3.4 to Koopman Operator Theory in 3.5 and the flow is not that clear.\n- Mouli, S.C., et.al, \"Metaphysica: Ood robustness in physics-informed machine learning\", ICLR 2024 improves upon SINDy for OOD learning and should be cited.\n- For details on weather benchmarking of DL models, it would be good to cite Karlbauer, M. \"Comparing and Contrasting Deep Learning Weather Prediction Backbones on Navier-Stokes and Atmospheric Dynamics\", https://arxiv.org/abs/2407.14129, 2024.\n- I think the weather forecasting part should be it's own section. It doesn't really fit in Section 3.5\n- For diffusion based models for weather forecasting see PreDiff (Zao et. al, NeurIPS 2023) and GenCast (Price et. al, 2024)\n- The PINNs discussion in 3.6 could be moved up to the PINNs section where it is first discussed. In general, PINNs seem to be randomly discussed throughout the paper (also see Section 4.1) and the organization/outline could be improved.\n- Minor: Capital S typo in PINNs at the top of page 23\n- The motivation of Hansen et. al, \"Learning Physical Models that Can Respect Conservation Laws\" is satisfying conservation and can be discussed in Section 3.7. In addition see Richter-Powell, J., \"Neural Conservation Laws: A Divergence-Free Perspective\", NeurIPS, 2022, Liu et. al, \"Harnessing the Power of Neural Operators with Automatically Encoded Conservation Laws\", ICML 2024, Liu et. al, \"INO: Invariant Neural Operators for Learning Complex Physical Systems with Momentum Conservation\", AISTATS, 2023 and Beucler, T., et. al, \"Enforcing Analytic Constraints in Neural-Networks Emulating Physical Systems\". Physical Review Letters, 126(9):098302, 2021 and adding the corresponding Neural Operator papers to page 27.\n- More recent methods using diffusion methods for PDEs, e.g., DiffusionPDE (Huang et. al, \"DiffusionPDE: Generative PDE-Solving Under Partial Observation\", https://arxiv.org/abs/2406.17763, 2024) could also be discussed.\n- The authors may also want to consider discussing uncertainty quantification (see Psaros et. al, \"Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons. Journal of Computational Physics, 477:111902, 2023) and out-of-domain generalization  (see Mouli et. al, \"Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEs\", ICML 2024) especially since probabilistic models, e.g., Neural Processes) and Gaussian Processes are discussed. I see this is discussed as future work.\n- I'm not sure why Neural Processes explicitly at put in Section 4 with meta-learning and multi-task learning subsections. I think it would fit better in the sections with the discussions of the other models.\n- Additional recent works to add include DINo (Yiu et. al, Continuous PDE Dynamics Forecasting with Implicit Neural Representations, ICLR 2023) and CROM (Chen et. al, \"CROM: Continuous Reduced-Order Modeling of PDEs Using Implicit Neural Representations\", ICLR 2023).\n- Typo on page 34, \"conversation\" should be \"conservation\"\n- The constraint method in Hansen et. al does not need to be a soft constraint and with the $sigma_G$ parameter set to 0 it can be a hard-constraint projection method and satisfy the constraint exactly unlike soft-constrained PINNs type methods.\n- For additional references of ML in the automotive domain, see Ashton, N. et. al,  \"WindsorML--High-Fidelity Computational Fluid Dynamics Dataset For Automotive Aerodynamics\", NeurIPS Datasets and Benchmarking Track, 2024, Ananthan, V et. al, Machine Learning for Road Vehicle Aerodynamics, SAE Technical Paper and Li et. al, Geometry-Informed Neural Operator for Large-Scale 3D PDEs\", NeurIPS 2023.\n- A stronger concluding sentence would be good.\n- Subramanian et al., 2024 is cited twice on page 36.\n- For more PDE foundation models to add see Herde et. al, Poseidon: Efficient Foundation Models for PDEs, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work covers an important survey about the use of machine learning in the physical sciences and computational fluid dynamics community. In particular, it provides a comprehensive review of works that incorporate physics into the learning process via various mechanisms including through the architecture, loss function and in the data itself, i.e., meta-learning and data augmentation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend acceptance of this paper upon revision and review. In particular, the main critical component is to add the below references and discussions as well as the references mentioned in the weakness section so that the survey is thorough to cover all the state-of-the-art SciML methods and their advantages and limitations.\n\n- Need to add clear contribution of this work in comparison to the similar survey by Wang et. al. and add reference with appropriate discussion.\n- Also I think MeshGraphNet GNN-based models need to be added and thoroughly discussed especially since GraphCast which is based on these methods is discussed.\n- I think breaking up Section 3 into different sections and reorganizing it would also be helpful for the flow (see the notes in weaknesses) and possibly have separate sections for the ODEs and dynamical systems works from the PDEs works.", "broader_impact_concerns": "I do not have ethical or broader impact concerns with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is a long review of physics-connected machine learning. Such works are inherently useful and interesting for the community. The reviewers agree on acceptance on both criteria. The reviewers gave exceptionally thorough commentary and suggestions on the paper, which the authors addressed sufficiently. This is a well-written review paper that helps categorise a fast-moving domain. I recommend the survey certification.
Decision: Accept (poster)