Metamizer: A Versatile Neural Optimizer for Fast and Accurate Physics Simulations
Authors: Nils Wandel
Stefan Schulz
Reinhard Klein
Venue: ICLR 2025 Poster (2025)
Keywords: Physics-based Deep Learning
Physics Simulations
Meta-Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Metamizer is a fast and accurate neural optimizer that iteratively solves a wide range of physical systems, including Poisson, advection-diffusion, wave, Navier-Stokes, and Burgers equations.

Abstract: Efficient physics simulations are essential for numerous applications, ranging from realistic cloth animations in video games, to analyzing pollutant dispersion in environmental sciences, to calculating vehicle drag coefficients in engineering applications. Unfortunately, analytical solutions to the underlying physical equations are rarely available, and numerical solutions are computationally demanding. 
Latest developments in the field of physics-based Deep Learning have led to promising efficiency gains but still suffer from limited generalization capabilities across multiple different PDEs. 

Thus, in this work, we introduce **Metamizer**, a novel neural optimizer that iteratively solves a wide range of physical systems without retraining by minimizing a physics-based loss function. To this end, our approach leverages a scale-invariant architecture that enhances gradient descent updates to accelerate convergence. Since the neural network itself acts as an optimizer, training this neural optimizer falls into the category of meta-optimization approaches. 
We demonstrate that Metamizer achieves high accuracy across multiple PDEs after training on the Laplace, advection-diffusion and incompressible Navier-Stokes equation as well as on cloth simulations. Remarkably, the model also generalizes to PDEs that were not covered during training such as the Poisson, wave and Burgers equation.

Published as a conference paper at ICLR 2025
METAMIZER: A VERSATILE NEURAL OPTIMIZER FOR FAST AND ACCURATE PHYSICS SIMULATIONS
Nils Wandel, Stefan Schulz & Reinhard Klein Department of Computer Science University of Bonn 53115 Bonn, Germany wandel 1n@cs unu-uoonn
ABSTRACT
Efficient physics simulations are essential for numerous applications, ranging from realistic cloth animations in video games, to analyzing pollutant dispersion in environmental sciences, calculating vehicle drag coefficients in engineering applications. Unfortunately, analytical solutions to the underlying physical equa- tions are rarely available, and numerical solutions are computationally demand- ing. Latest developments in the field of physics-based Deep Learning have led promising efficiency gains but still suffer from limited generalization capabilities across multiple different PDEs.
Thus, in this work, we introduce Metamizer, a novel neural optimizer that iter- atively solves wide range of physical systems without retraining by minimiz- ing a physics-based loss function. To this end, our approach leverages scale- invariant architecture that enhances gradient descent updates accelerate conver- gence. Since the neural network itself acts as an optimizer, training this neural optimizer falls into the category of meta-optimization approaches. We demon- strate that Metamizer achieves high accuracy across multiple PDEs after training the Laplace, ddlvectionddiffuiinn and incompressible Navier-Stoke, equation well as on cloth simulations. Remarkably, the model also generalizes to PDEs that were not covered during training such the Poisson, wave and Burgers equation.
Linear PDEs
Non-Linear PDEs
Lackaoe Equation
AdvectionOMltsion Equation
Wave Equation
Nevier-Stakes Equalion
Burgers Equation
Cleth Smulation
Metamizer
Figure 1: Metamizer able to simulate various linear and non-linear physical systems. All of the depicted results were produced by the same neural model (same architecture and same weights). PDEs marked in red were not considered during training.
INTRODUCTION
Countless physical systems can be described by partial differential equations (PDEs). For example, electrostatic or gravitational fields can described by the Poisson equation, heat diffusion or pol- lutant dispersion can be described by the advection-diffusion equation, pressure waves by the wave equation, fluids by the incompressible Navier-Stoker Stokes equation, and so on. Unfortunately, most of these equations do not have analytical solutions, numerical solvers must be used. To reduce the computational burden of numerical methods, numerous neural surrogate models have been devel- oped in recent years that achieve fast and highly efficient physics simulations (Thuerey al. 2021

---

Published as a conference paper at ICLR 2025
Moreover, recent developments (Kaneda et al. 2023 Chen al. 2024 show that neural networks do not fall behind numerical solvers in terms of accuracy anymore. However, neural surrogate models are usually tailored to specific PDEs and do not generalize well across mul- tiple different PDEs without retraining. Zero-shot PDE solvers, analogous to current endeavours in foundational models for natural language processing (Brown, (020) or computer vision (Rad- ford et al.) 2021, Kirillov et al., 2023). could be of great benefit for instant realisite animations in computer games or computer aided engineering.
Cuomo et al., 2022).
Thus, in this work, we propose Metamizer, a neural optimizer to accelerate gradient descent for fast and accurate physics simulations. To this end, we introduce novel ccale-invariant architecture that suggests improved gradient descent update steps that are independent of arbitrary scalings of the loss function or loss domain. We train Metamizer without any training data directly on the physics-based loss that it is supposed to optimize. After training, unlike traditional gradient descent methods, Metamizer requires no tuning of parameters such as learning rate or momentum, and allows tradeoffs between runtime and accuracy for fast and highly accurate results. We evaluate the very same network (same architecture and same weights) across a wide range of linear and non- linear PDEs and showcase its generalization capabilites even to PDEs that were not covered during training.
Code well as pretrained model are available on Github
2 RELATED WORK
typically rely on discretization scheme (e.g. Finite Differences, Finite Volumes, Finite Elements etc. (Peiró & Sherwin (0055) in order to transform PDEs into large systems of equations. Depending on properties such as symmetry or linearity of this system, various numerical solvers such as (nonlinear) CG, GMRES, MINRES etc. can be employed. To alleviate their computational burden (Farmaga et al. 2011), already in the last century, several specialized methods have been developed to accelerate for example fluid simulations (Chen & Doolen| 1998] Stam 1999) or cloth simulations (Baraff & Witkin [998).. More recently, neural preconditioners (Kaneda et al. 2023; Lan et al.] 2023) significatly accelerated conjugate gradient methods for the Poisson equation. However, these specialized methods exploit domain-specifie assumptions and, thus, do not generalize well across different linear and nonlinear PDEs.
Numerical Methods
Gradient Descent based methods like AdaGrad (Duchi al.) 2011), Adam (Kingma & Ba 2017
Reddi et al. 2019). AdamW (Loshchilov) 2017) and many more (Norouzi & Ebrahimi] 2019) build the backbone of most Deep Learning approaches. These optimization methods can be generally ap- plied to a wide range of differentiable minimization problems including solving PDEs (Nurbekyan) al. (023). However, Gradient Descent methods require tuning of hyperparameters such as learn- ing rate momentum and many iterations until convergence. As result, they are not yet efficient enough for real-time physics simulations.
Meta-Learning, and in particular "Learning to Optimize" (L20), is the field of research that deals with the automatic improvement of optimization algorithms,, typically by means of machine learning techniques. Pytorch libraries such "higher" (Grefenstette al. 2019) "Jearn2learn" (Arnold et al. 2020) allow for example to automatically optimize hyperparameters of optimizers like Adam through gradient descent. (Anryychoicc et al. (2016) train LSTMs to optimize neural networks for classification tasks and style transfer and [Cen
and/Chen et al. (020)) present several training techniques that suggest there still room for potential improvement in the field L20. However, to the best of our knowledge, L20 has not yet been applied in the context of physics simulations.
Data-driven Deep Learning approaches have been widely established to generate neural surro- gate models for efficient physics simulations for example in context of fluid dynamics (Tompson) cct al. 2017 Gonzzllz- et al. 2020), cloth simulations (Pfaff et al. 2020), weather forecast- ing (Bonev et al. 2023 Lam et al.) 2023) or aerodynamics (Li et al. 2024). While these methods allow efficient simulations at coarse spatial and temporal resolutions or on physical systems without knowledge of the underlying mathematical equations, they require large amounts of high-quality training data, which can be expensive to generate. Furthermore, generalization and accuracy beyond the training data is often fairly limited.
2

---

Published as a conference paper at ICLR 2025
Physics-driven Deep Learning approaches in contrast, require only little no ground truth data and rely on physics-base based loss. Here, 2 major strands of research can be discerned:
First, implicit neural representations (often referred to PINNs) popularized by Raissi et al. (2019) have become a new and quickly growing field of research (Cuomo et al. 2022 ranging from climate simulations (Chen et al., 2022: Lai al. 2024) highly accurate simulations of Burgers equation (Chen et al. 2024) and fluid dynamics Cai et al. (021 Ghosh et al.) 2023). 3 all the way to
robotics (Sanyal & Roy| 2023; Liu al. 2024). Unfortunately, the learned implicit representations
need to be retrained for every physical simulation and thus are not real-time capable.
Second, neural surrogate models that evolve an explicit state representation in time presented by /Zhu et al. (2019) have been successfully applied to the coupled 2D Burgers equation (Geneva Zabaaass| 2020), incompressible fluid dynamics in 2D and 3D (Wandel et al., 2 (b) and cloth simulations (Santesteban et al., [2022] SStoko et al., 2024). While these methods do not require any training data and show good generalization performance within their PDE domain, so far, they do not generalize across multiple PDEs.
FOUNDATIONS
In this chapter, we introduce some basic concepts and definitions for PDEs, explain, how finite dif- ference schemes can be used to formulate physics-based loss, and give some intuition to motivate our gradient-based optimization approach.
Partial Differential Equations (PDEs) Partial differential equations are equations that constrain the partial derivatives of a multivariate function. In its most general form, PDE can be written as:
F F(11.... .... Ln, U, Oz, u, Oz. G2 ...) = O'(1z. Fn) E Ω
(1)
where u(I1,.... .... In)) is the unknown function that we want to solve. Zi. En are the independent variables inside domain 52 and Üz, denotes the partial derivative of u with respect to I3. If one of the independent variables corresponds to time t, F is called a dependennndent PDE. If u does not depend on i, is called stationary. If F is linear in u and its derivatives, F' is called a linear PDE. Otherwise, F' is called a non-linear PDE.
Boundary and Initial Conditions Usually, additional constrains at the domain boundaries are given such as boundary or initial conditions. Although various types of different boundary con- ditions could be considered, in this work, for simplicity, we focus solely Dirichlet boundary conditions:
u(I,,...,,,, d(z)...., Fn) V ((z)..... E an
Here, d(x1. .... In) directly specifies the values of u at the domain boundary M.. Typically, time- dependent PDEs also require an initial state (or initial condition) from which the solution of PDE evolves over time.
Finite Differences To compute u and its partial derivatives numerical manner, u needs to be discretized. Here, we consider a regular grid in space (and time). This allows to compute derivatives like gradients (V), divergence (V.), Laplace (A) or curl (Vx) numerically by performing convolu- tions with finite difference kernels. By plugging the results of these finite difference convolutions for every grid point into the general formulation of PDEs (Equation []. we obtaín a large system of equations. If the PDE is linear, this set of equations becomes (typically sparse) linear system of equations.
Time-Integration Schemes case of time-dependent PDEs, a strategy must be chosen to inte- grate the simulation in time for given initial conditions. There exist several strategies to calculate and integrate the time derivative with finite differences. The most popular strategies are:
Forward Euler: Here, the computation of the time derivative depends only on the current time step. This explicit strategy makes computations easy and efficient, but tends to cause unstable simulation behavior for many physical systems the time step is chosen too large. Therefore, explicit schemes not considered in this work.
3

---

Published as a conference paper at ICLR 2025
Backward Euler: Here, the computation of the time derivative depends the following time step. This implicit strategy results in stable simulations, but can suffer from numerical dissipation and requires solving large systems of equations.
Trank-Nicolson Here, the centered finite difference of the current and the following time step is used to calculate the time derivative. This implicit strategy results in more accurate computations compared to forward or backward Euler, but requires, like backward Euler, solving large systems of equations.
Physics- Based Loss Solving large (potentially non-linear) systems of equations obtained by finite differences challenging task that is typically computationally expensive. Thus, we formulate physics-constrained loss, that penalizes the mean squared residuals (left-hand side of Equation T as follows:
L(u)
Σ (eE,),
[F Uny U, Ozy U, m.. Oz u, 0z u, u....
This way, we can search a solution u² of the PDE by minimizing L:
u arg min L
Dirichlet Boundary conditions (Equation can be typically applied directly by setting the corre- sponding boundary grid points of equal to d. However, sometimes, this is not possible and we have consider an additional loss term for the Dirichlet boundary conditions:
(u(u)=
1
Σ
|u(r1,..., Fn) d(.,,.... F.]
(4)
[asa]
(#1
In this case, have to minimize L + LD to solve Equation I and Equation 2
Exemplary visualization 2D loss function with steep valley. Naive gradient descent fails since it ei- ther converges very slowly (blue curve 1) or diverges (red curve 2). Ideally, we would like adapt the step size and direction for accelerated convergence (green curve 3).
Figure 2:
L
Gradient Descent Intuition In order to minimize the physics-basee loss L (Equation and we follow gradient descent approach. Figure 2 provides some intuition on common problems with naive gradient descent in the vicinity of ill conditioned minima and how convergence could be improved: If we take too small steps along the gradients (blue curve 1), convergence becomes very slow. If we take too large steps (red curve 2), the gradient descent scheme diverges. Ideally (green curve 3), we would first take some small initial steps to check the local gradients and avoid initial divergence. Then, the step size can be carefully increased. Instead of directly following the gradients, a better step direction can found by taking the gradients of the previous steps into account well. Once come close to the minimum, the step size should be decreased again to achieve higher accuracy. As discussed by Holl et al.] (2022), scaling the loss function and thus its gradients by constant factor e does affect its minima (arg min,, L(u) arg min,, c- L())). Thus, the gradient descent procedure should be invariant with respect to constant gradient scalings. Similarly, scaling the coordinate system of L should result in equally scaled update steps (arg min,, L(u)
arg min, L(c u), see solid vs dashed lines in Figure 2). This motivates our scale invariant Metamizer architecture that can deal with arbitrarily scaled gradients and can automatically adapt its step size (see Figure
4 METHOD
Our approach aims to accelerate convergence of a gradient descent scheme by proposing improved update steps with neural network.
4

---

Published as a conference paper at ICLR 2025
b)
high precsiin
reguler
O-iriliaizze
V.L.
L--elt
V 0oom
Aug
Traning Pool
V V
A
state
L
V
state
A
optimize
Foom
Metamizer
8;
Figure 3: a) Metamizer architecture: a scale invariant neural optimizer. b) Training Cycle.
Metamizer Architecture The neural network architecture as depicted in Figure 3 a) works as follows: First, the gradient VuL; of the current iteration i as well as the gradient V ,,, and update step Au of the previous iteration - 1 are taken input. Then, the gradients of the current and previous iteration are normalized by the standard deviation of the current gradient and the step size normalized by its standard deviation as well. These values are then concatenated and fed into a U-Net (Ronneherger et al., 2015) in order to predict an update scale factor for 8; and an update direction that gets multiplied with s; to obtain the update step Au; for the next iteration. The U- Net architecture was chosen since it allows to model long-range dependencies at coarse resolutions while preserving small details fine resolutions, reflecting the various domains of dependence of different PDEs. Finally, the computed update step is applied to u to obtain U4++1 + Au; For the initial state, we set ug Vo Auo and set so 0.05 avoid initial divergence by making only small steps at the beginning.
These normalization and scaling measures make the network invariant with respect arbitrary scal- ings of the loss or domain and allow the network to make reasonable progress at all stages of the optimization: At the beginning, when large update steps must be made the presence of large gra- dients, as well as in later optimization stages, when fine adjustments must be made in the presence of small gradients. To allow for very high accuracies, the gradient computations and update steps need to be calculated in double precision. Nevertheless, due the input normalizations, the U-Net can be evaluated using a more efficient regular float precision. Since the outputs of the network get scaled by 8,, the training gradients get scaled as well. This would result in huge gradients for the first optimization iterations compared to basically no training signal at later optimization iter- ations. Thus, we employ gradient normalization (Chen al. 2018) to normalize gradients during backpropagation and provide the network with an evenly distributed training signal during all opti- mization stages and across all different PDEs that we train on in parallel. Since we want to reuse the same network for different PDEs that might also contain different numbers of channels, Metamizer optimizes channels separately (e.g. the Laplace equation requires only channel while incom- pressible fluids require 2 channels for the velocity and pressure field and cloth dynamics require channels to describe motions in z/ z/y/z directions).
Training The neural optimizer trained in cyclic Meta-learning manner inspired by Wandell et al. (2021a): As visualized in 3 we first 0-initialize a training pool of randomized PDES (e.g. randomized boundary conditions for the Laplace / dddvection-diffusion Navier-Stokes equa- tions). Note that in contrast to data-driven methods, no pre-generated ground truth data is needed. Then, we draw random mini-batch that contains the current gradient V ,,L, of the physics-bars baseed loss as well as optimizer state information (i.e. V,Li,1, (.Li-1,A;-11. -;-1; and feed it into the Metamizer architecture to compute an update step Au, This update step, together with updated state informa- tion (i.e. L, Aw; 8;) is fed back into the training pool to update the training environments with more realistic data and to recompute the physics-based loss based on the updated values of u. This recomputed loss is then used to optimize the Metamizer with Adam (1r=0.001). By iterating this training cycle, the training pool becomes filled with more and more realistic training data and syysics-based losses. Furthermore,
the neural optimizer becomes better and better at minimizing since this way the training pool constantly generates new training data on the fly, its memory foot- print is small enough such that the entire training pool can be kept in GPU memory. After training
5

---

Published as a conference paper at ICLR 2025
for about 6 hours Nvidia Geforce RTX 4090, we obtained single model that was able solve various PDEs at high precision and produced all of the results shown in the following section.
RESULTS
In this section, we show that the very same model (same architecture and same weights) can deal with wide variety of PDEs linear well non-linear, stationary well time-dependent.
5.1 LINEAR SECOND ORDER PDES
Linear second order PDEs can be grouped into 3 categories: Elliptic PDEs (c.g., Poisson or Laplace equation), Parabolic PDEs (c.g.. advection -iffiionniifuiion equation) and Hyperbolic PDEs (c.g., wave equation). Here, show that model excels in all 3 categories:
5.1.1 POISSON AND LAPLACE EQUATION
The Poisson equation is stationary, elliptic PDE and is important find e.g. minimal surfaces, steady state equilibria the Diffusion Diffusion-Equation (see Section 5.1.2) or to calculate energy potentials of electrostatic gravitational fields (see Figure 12 in Appendix):
F Au f
(5)
The Laplace Equation is the homogeneous (f 0) special case of the Poisson equation.
iteration: 0
iteration:
iteration: 2
iteration: 3
iteration: 20
iteration: 50
1o-4
1e-4
16-4
16-11
1.0
0.0
1e 1s 1e
2.5
0
0.0
0
-25
Figure 4: Iterative refinements by Metamizer to solve the Laplace equation. Top row: Intermediate solutions of Middle row: corresponding gradients of the physics-based loss. Bottom update steps computed by Metamizer. An animation this process presented in the supplemental video.
ddepicts intermediate solutions u, of the Laplace equation, the gradients of the physics- based loss V.,i, and update steps Au; computed by Metamizer at iterations 0, 1, 3, 20 and 50. At the beginning, O-initialiized. Thus, high gradients of L only appear the domain boundaries where the boundary conditions result in discontinuities of u. Nevertheless, Metamizer is able to anticipate better update steps that also affect the surroundings of the boundaries to achieve faster convergence. After only 3 update steps, the shape of u is visually close to the final solution. After 20 iterations, the gradients of are already in the range of 10-7 and after 50 steps, Metamizer hits the limits of machine precision (see noise in V. Lso in Figure 4 rendering further improvements impossible. Figure 5 shows how Metamizer automatically adjusts the scaling 8;: At the very beginning, so starts relatively small value (0.05) to avoid initial divergence. Then, for the next 5 iterations. Metamizer increases 8; to lower more efficiently. After that, 8; is steadily decreased for finer and finer adjustments. Remarkably, this learned behavior corresponds very well with "natural" gradient descent intuition provided Section 3 After around 50 iterations, when machine level precision is reached, further progress is possible and 8; remains constant very low level.

---

Published as a conference paper at ICLR 2025
We compared our approach to several gradient descent based methods of the pytorch optimizer package (SGD, Adam, AdamW, RMSprop, Adagrad, Adadelta) and iterative sparse linear system solvers from CuPy (conjugate gradients, minres, gmres, lsmr). Figure|Jshowe that traditional gradi- ent based approaches are not competitive with current sparse linear system solvers regarding runtime accuracy. Furthermore, these gradient based optimizers require tuning of hyperparameters such as learning rate or momentum (more information is provided in Appendix (B). While Metamizer does yet reach the performance of specialized and highly optimized algebraic multigrid solvers such as AMGX (see Appendix C. it reaches competitive performance to sparse linear system solvers like CG, MINRES or GMRES and reaches a MSR of 6.7 x 10 -33 after only 0.08s while maintaining the generality of gradient based approaches. Thus, it can be directly applied for example to nonlinear and nonsymmetric PDEs as well (see Section 5.2) On top of that, Metamizer shows good scaling behavior on larger grid sizes (see Appendix B
a]
Laglace Equation
b) Advection Diffusion Equataon (10
Iteratione
Time Stop
16-3
10:1
10-1
10
10-12
10:2
0
10
20
30
40
50
60
20
30
50
60
Iteration
iteration
Figure 5: The scale parameter s; (see Figure 3 a) is automatically adjusted by Metamizer to make appropriate update steps. a) stationary Laplace Equation b) time-dependent advection-diffusion equation with 10 iterations per time step.
Metamizer and GPU based Solvers 100x100
(1=0,00
+0.011
10-9
Adem 3r-d.811
10-2
Adem 01-0.811
Ademmi dr-0.033
Adamii [t-18 03)
10-41
RMSSprop (1-0 0.011
10-14
RMSSorop
109
10-1
Adagriad (1w0.11
10-1
Adaggadd fr-0..1
Adadett dr-
Adadeba (ire
cata,, ginnes
10
ouds, gimtee
cuda, ism
oudb, Smn
cutaa mirres
10-21
ouda, minves
49
10-20
10
10
10*
107
10
10P
Iterations
Figure 6: Comparison of mean squared residual errors for the Laplace equation as a function of time (left) and number of iterations (right) between Metamizer and various iterative sparse linear system solvers of the CuPy package (Okuta et al.]|201 (017) and gradient based optimizers from pytorch (various learning rates are investigated in Appendix The goal is to lower the loss in as little time as possible (bottom left corner of the left plot).
100 x 100 grid
5.1.2 ADVECTION-DIFTUSI EQUATION
The Advection- -iiffusionifuuio equation is an important parabolic PDE that describes for example heat transfer or the dispersion of pollutants. For an unknown function u, it can written as:
= 0,u DAu (Ju) - R 0
(6)
Here, D denotes the diffusivity, 5 denotes the velocity field for the advection term and R corresponds to the source sink term. In contrast to the stationary Poisson equation, the cddvection-diffuseee equa- tion is time-dependent. Thus, the implicit backward Fuler scheme for stable unfolding of

---

Published as a conference paper at ICLR 2025
the simulation in time. For every simulation time step, perform a certain number of optimization steps to reduce L. By choosing the right number of iterations per time step we can make a trade-off between speed and accuracy. Figure ()) shows how Metamizer automatically adjusts the scaling 8; when simulating the advection-diffusiii equation at 10 iterations per time step. At the beginning of each time step, the neural optimizer increases its stepsize quickly adjust Then, is decreased again for fine adjustments.
Figure 1 shows results for constant localized source field I and rotating velocity field o (for example a laser that heats a spinning disk). More examples with different diffusivity parameters are shown in Appendix A Table shows how accuracy increases with the number of iterations per time step.
5.1.3
WAVE EQUATION
The wave equation an important hyperbolic PDE that describes wave like phenomena such as water waves, pressure waves, clectro-magnetic waves and many more. It can be written as:
F F= ou 0f
Here, corresponds to the wave propagation speed. Figure 7 shows results for different values of e that were obtained by Metamizer using an implicit Crank-Nicolson scheme. Note, that the wave equation not included during training. Table shows accuracy levels for 1, 5, 20 and 100 iterations per timestep. Using only iteration results in divergent simulations, 20 iterations yield fairly accurate results and 100 iterations, the simulation runs close to machine precision.
a) c
b) c= 2
c) c = 10
Metamizer is capable of simulating waves different propagation speeds without having seen the wave equation during training.
Figure 7:
5.2 NON-LINEAR PDES
Non-linear PDEs are particularly hard to solve as the underlying systems equations become non- linear well. In this section, demonstrate Metamizers capabilities to deal with non-linear PDEs.
5.2.1 INCOMPRESSIBLE NAVIER- -TROESS EQUATION
The incompressible Stoier-Stokes equation describes the dynamics of an incompressible fluid by means of velocity field T and pressure field p:
F F= (B.V) Vp PAU Fex
(8)
Here, p denotes the fluids density, y its viscosity and Fexs external forces. The incompressibility equation V ō = can be automatically fulfilled by utilizing a vector potential a and setting the velocity field ū V However, using vector potential prohibits from directly manipulating the velocity field. Thus, we need an additional boundary loss term (Equation to deal with the Dirichlet boundary conditions for the velocity field at the domain boundaries. To compute the partial derivatives of a, 5 and p efficiently with centered finite differences, we rely on a staggered marker and cell (MAC) grid (Harlow et 1963 Holl al. 2020, Wandel et al., 2021a).
Figure 8 shows results for various Reynolds numbers. quantity defined as:
The Reynolds number Re is unit less
p|| D
Re
8

---

Published as a conference paper at ICLR 2025
PDE
lierations
I
20
100
leraatons
Time Step
Second
Advection-Diffusee Diffusson
7.9 x 10-5
2.0 x 10
5.6 X 10
2.2 X 10 33
420
Wave Equation
4.4 x 1021
4.6 x 10
3.2 x 10
3.5 x 10
420
Navier-Stokes Lp
7.4 10-6
2.0 10 10-5
2.4 10
2.4 10 10-16
230
La
1.7 X 10-7
3.7 x 10
9.5 x 10-9
1.9 X 10
2D coupled Burgers
4.1 x 10-3
8.5 x 10
2.5 x 10-10
5.0 x 10-32
330
Table 1: Mean Squared Residuals for different dime-dependent PDEs at different iterations per time step
where D corresponds to the diameter of the obstacle and ||o|| to the flow velocity. The Reynolds number has a big effect on the qualitative behavior of the flow field. For example, in case of very small Reynolds numbers, the flow-field becomes stationary and the Navier-Stokes equation can be approximated by the Stokes equation. In case very high Reynolds numbers, the flow-field becomes turbulent and can be approximated by the Euler equation for inviscid fluids.
a) p= 5,p 1, Re
b) p 1, 10, Re 30
0.1, 4, Re 200
Figure 8: Fluids of various viscosities μ and densities at wide range of Reynolds numbers can be simulated by Metamizer. The obstacle diameter D 10 and the fluid velocity | 0.5
Table shows results for Lp (mean squared residuals of Equation 8 and La (mean squared residuals of V that capture if the Dirichlet boundary conditions are not properly met). Lp and La are lower for only iteration compared to 5 iterations per time step, because 1 iteration per time step results in an incorrect steady-state solution for the wake dynamics. Compared to Wandel et al. (202 la), Metamizer reaches similar performance at iterations per time step. At 20 or even 100 iterations, our approach is orders of magnitudes more accurate while still being reasonably fast.
5.2.2 BURGERS EQUATION
The 2D coupled Burgers equation is an important non-linear PDE to study the formation of shock patterns.
F = 0- V pAT = 0
(10)
Here, is a viscosity parameter. Figure I shows an exemplary simulation result by Metamizer that exhibits clear shock discontinuities at μ 0.3. Additional qualitative results are shown in Figure TO of Appendix A Note that the Burgers equation was not considered during training. Mean squared residuals for different iterations per time step are shown Table [
5.2.3 CLOTH DYNAMICS
To describe cloth dynamics, we closely follow the approach of SSntestteban et al. (2022) and Stotko al. (2024). Instead of using classical PDE formulation, their approach relies spring-mas mass system consisting of a regular grid vertex positions F, velocities and accelerations ă that get integrated by backward Euler scheme:
J == カーサー + Al al and मe = + Al T
(11)

---

Published as a conference paper at ICLR 2025
To obtain the accelerations a", the following physics- based loss L must be minimized by Metamizer:
L (in( At a²) + (a Mā²)
(12)
Here, Fext corresponds external forces (such as gravity wind), M the matrix and Eint are the internal energies of the cloth specified as follows:
Bine
Cstiff
Σ
(TET - 1)2 Cshear
Σ
- -9-) + Cbend
Σ
(0 - 180
2
BEodges
Erigghh angles
@Estright angles
where Catiff, Cshear; Gbend are parameters that describe stiffness, shearing and bending properties of the cloth.
a)
b)
Figure 9: Cloth with various material parameters (Cain/ CShear/ (bend) can be simulated. a) (1000/10/0.01) b) 10000/100/1100 c) (1000/10/1 1000000)
Figure shows results by Metamizer for various cloth parameters. The simulations were performed at 20 iterations per time step and around 185 iterations per second. Note that L can take negative values and cannot be interpreted as easily as the mean squared residuals of the previous PDEs. Thus ommited L in Table [ our cloth simulations. Instead, we provide visualizations for the reduction of the gradient norm |0aL||2 for different iterations per timestep in Appendix
CONCLUSION
In this work, we introduced Metamizer, a novel neural optimizer for fast and accurate physics sim- ulations. To this end, propose a scale-invariant architecture that suggests improved gradient descent update steps to speed up convergence on a physics- loss at arbitrary scales. Metamizer was trained without any training data but directly the mean square residuals of PDEs such the Laplace, advection-diffusion or Navier-Stokes equations and cloth simulations in a meta-learning manner. Remarkably,
method also generalizes PDEs that ing such as the Poisson, wave and Burgers equation. By choosing proper number of iterations per timestep. trade-off between speed and accuracy can be made. Our results demonstrate that Metamizer produces fast as well as highly accurate results across wide range of linear and nonlin- ear physical systems.
covered during train-
In the future, further physical constrains such as Neumann or Robin boundary conditions and self- collisions for dlthh-dynamics could be included. At the moment, Metamizer is trained for a single grid size. Thus, increased flexibility could be achieved by training Metamizer multiple grid sizes simultaneously or by extending the network architecture to Mesh- or Graph-Neural Networks. For simulations Mesh structures, physics-infome loss based on Galerkin neural networks (Gao et al. 2022) could practical. Furthermore, multiple PDEs such as the Navier-Stokes and equation could be coupled.
We believe that our approach may have applications in future numerical solvers and speed up ac- curate simulations in computer games computer-aided engineering. Since gradient descent ap- proaches are ubiquitous, our approach may also serve as an inspiration for other gradient-based applications outside of physics, such style transfer, scene reconstruction, optimal control.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Federal Ministry of Education and Research of Germany and the state of North Rhine-We Westphalia as part of the Lamarr Institute for Machine Learning and Artificial Intelligence and by the Federal Ministry Education and Research under Grant No. PB22-063A INVIRTUO and Grant No. 011S22094A WEST-AL. Furthermore, it is affiliated to VRVis GmbH in scope of COMET Competence Centers for Excellent Technologies (879730, 1654) which managed by FFG.
REFERENCES
Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances neural information processing systems, 29, 2016.
Sébastien M R Arnold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learn: A library for Meta-Learning research. August 2020. URL http:// Doxivorr/abs//200 12284
David Baraff and Andrew Witkin. Large steps in cloth simulation. In Proceedings of the 25th Annual Conference Computer Graphics and Interactive Techniques, SIGGRAPH '98, pp. 43-54, New York, NY. USA, 1998. Association for Computing Machinery. ISBN 0897919998. doi: 10.1145/ 2808 14.20821.
10. 1145 280814 220.4///288844288811 280821
Nathan Bell, Luke N. Olson, Jacob Schroder, and Ben Southworth. PyAMG: Algebraic multigrid solvers in python. Journal of Open Source Software, 8(87):5495, 2023. doi: 10.21 10.21105/joss.05495,
URL //ttps://doi... org 11../.1...../1.1. foss 054 95
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dy- namics the sphere. In International conference on machine learning, pp. 2806-2823. PMLR, 2023.
Tom B Brown. Language models few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Kariiadakis. Physics- informed neural networks (pinns) for fluid mechanies: review, 2021. URI L https://arxxii
org/ abs/ 2105. 09506
Linwei Chen, Bowen Fang, Lei Zhao, Yu Zang. Weiquan Liu, Yiping Chen, Cheng Wang, and Jonathan Li. Deepurbandownscale: A physics informed deep learning framework for high- resolution urban surface temperature estimation via 3d point clouds. International Journal of Applied Earth Observation and Geoinformation, 106: 106:002650, 2022. ISSN 1569-8432. doi:
URL https: www. sciencedirect. com/
Shiyi Chen and Gary D Doolen. Lattice boltzmann method for fluid flows. Annual review of fluid mechanics, 30(1):329-364, 1998.
Tianlong Chen, Weiyi Zhang, Zhou Jingyang, Shiyu Chang, Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. Advances in Neural Information Processing Systems, 33:7332-7343. 2020.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. International conference machine learning. pp. 794 803. PMLR, 2018
Zhuo Chen, Jacob McCarran, Esteban Vizeaino, Marin Soljacic, and Di Luo. TENG: Time-evolving natural gradient for solving PDEs with deep neural nets toward machine precision. In Forty-first International Conference on Machine Learning, 2024. URL https: /ttpps:://pennee wpwprevien.eweeeeoemeeeee net/
forum? ddvi T ZRA Mb.

---

Published as a conference paper at ICLR 2025
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics- informed neural networks: Where we and what's 's' next. Journal of Scientific Computing, 92(3):88, 2022.
2022.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning reseurch, 12(7), 2011.
Ihor Farmaga, Petro Shmigelskyi, Piotr Spiewak, and Lukasz Ciupinski. Evaluation of computa- tional complexity of finite element analysis. In 2011 11th International Conference The Experi- ence of Designing and Application of CAD Systems in Microelectronics (CADSM), pp. 213-214, 2011.
Han Gao, Matthew J Zahr, and Jian-Xun Wang. Physics-informed graph neural galerkin networks: A unified framework for solving pde-governed forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 390: 14502, 2022.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks. Journal of Computational Physics, 403:109056, 2020.
Shinjan Ghosh, Amit Chakraborty, Georgia Olympia Brikis, and Biswadip Dey. Rans-pinn based simulation surrogates for predicting turbulent flows, 2023. htt //axivv org/ abs/
2306. 06034
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho. and Soumith Chintala. Generalized inner loop meta- learning. arXiv preprint 1rXiv:1910.01727, 2019.
Francis H Harlow, J Eddie Welch, et al. Numerical calculation of time-dependent viscous incom- pressible flow of fluid with free surface. Physics of fluids, 8(12):2182, 1965.
Philipp Holl, Vladlen Koltun, Kiwon Um, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In NeurlPS workshop, volume 2, 2020.
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Scalc-invariant learning by physics inversion.
Advances in Neural Information Processing Systems, 35:5390-5403, 2022.
Ayano Kaneda, Osman Akar, Jingyu Chen, Victoria Alicia Trevino Kala, David Hyde, and Joseph Teran. A deep conjugate direction method for iteratively solving linear systems. In International Conference on Machine Learning, pp. 15720-15736. PMLR, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https::/aaxxz: www.orrg//b.... 1412.611660 6980
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Ching- Yao Lai, Pedram Hassanzadeh, Aditi Sheshadri, Maike Sonnewald, Raffaele Ferrari, and Venkatramani Balaji. Machine learning for climate physics and simulations, 2024. URL https:
2204. 13227
Remi Lam, Alvaro Sanchez-Gonzaler Matthew Willson, Peter Wirnsberger, Meire Fortunato, Fer- ran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, al. Learning skillful medium-range global weather forecasting. Science, 382(6677)1111--1421 2023.
Kai Weixian Lan, Elias Gueidon, Ayano Kaneda, Julian Panetta, and Joseph Teran. A neural- preconditioned poisson solver for mixed dirichlet and neumann boundary conditions. arXiv preprint arXiv:2310.0017. 2023.
Zongyi Li, Nikola Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Otta, Mohammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, et al. Geometry- informed neural operator for large-scale 3d pdes. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Jingyue Liu, Pablo Borja, and Cosimo Della Santina. Physics-informnt neural networks to model and control robots: A theoretical and expermmnntll investigation. Advanced Intelligent
Systems, 6(5):230038,, 2024. doi: /atoo//////lee 1002 202300338
URL https:
//on1 ine ineeibrar Sibrarrrr... wi co..ooooooo///a/a/oooo
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Sajad Norouzi and M Ebrahimi. A survey proposed methods to address adam optimizer deficien- cies. Department of Electrical and Computer Engineering, University of Toronto, 2019.
Levon Nurbekyan, Wanzhou Lei, and Yunan Yang. Efficient natural gradient descent methods for large-scale pde-based optimization problems. SIAM Journal on Scientific Computing, 45(4): A1621-A1655, 2023.
Ryosuke Okuta, Yuya Unno, Daisuke Nishino, Shohei Hido, and Crissman Loomis. Cupy: A numpy-compatible library for nvidia gpu calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-frrst Annual Conference Neural Information Processing Systems (NIPS), 2017. htL/httpp//eerniiee http://gggyy..oo//sp//////////e/
papers/paper_16.22
Joaquim Peiró and Spencer Sherwin. Finite difference, finite element and finite volume methods for partial differential equations. In Handbook of Materials Modeling: Methods, pp. 2415 2446. Springer, 2005.
Tobias Pfaff, Meire Fortunato, Alvaro Sanzaez, and Peter W Battaglia. Learning mesh- based simulation with graph networks, arXiv preprint arXiv:2010.03409, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763, PMLR, 2021.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-707, 2019, ISSN 0021-9991. doi: https://doi.
100/100.0116/1.jpp2011... 10.045. URL https: //t:///wwww www. scwencedi com/ wtomm/ccience/////oocc/co axticie//11/5002119111833
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond, 2019.
URL http::httttstttts org. abs/ 1904. 09237
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-20 234-241. Springer, 2015.
Alvaro Sanchez-Gonzaler Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference PMLR, 2020.
machine learning, pp.
Igor Santesteban, Miguel A Otaduy, and Dan Casas. Snug: Self-supervised neural dynamic gar- ments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8140-815555 5150, 2022.
Souray Sanyal and Kaushik Roy. Ramp-net: A robust adaptive mpc for quadrotors via physics- informed neural network. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 1019-1025, 2023. doi: 1.1109//CRRA4899122222000000
Jos Stam. Stable fluids. 1999.
David Stotko, Nils Wandel, and Reinhard Klein. Physics-guide caop--fromteemppe monocular video perception through neural surrogate models. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 1 11895- 1904, 2024.
13

---

Published as a conference paper at ICLR 2025
Nils Thuerey, Philipp Holl, Maximilian Mueller, Patrick Schnell, Felix Trost, and Ki- WWW, 2021.
won Um. Physics-based Deep Learning.
URL https://
alysicchaeedeeeplern..........
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. In International conference on machine learning. pp.
PMLR, 2017.
Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dynamics from scratch-towards fast, differentiable fluid models that generalize. ICLR, 2021a.
Nils Wandel, Michael Weinmann, and Reinhard Klein. Teaching the incompressible navier-ster stokes equations to fast neural surrogate models three dimensions. Physics of Fluids, 33(4), 2021b.
Yinhao Zhu, Nicholas Zabaras, Phaedon Stelioo Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high- high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56-81, 2019.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL QUALITATIVE RESULTS
In this Section we provide additional qualitative results for Burgers equation with μ= 0.3/1 (Figure equation with D = 0.1 /0.5///1110 (Figure 11 and the Poisson equation (Figure 12) on 100 x 100 grid. Furthermore, we trained Metamizer on a larger 400 400 grid to enable fluid simulations at higher Reynolds numbers. Figure 13 demonstrates that Metamizer is able to simulate complex turbulent fluid dynamics at Re 2000.
10 the advection-diffuse
M- = 0.3
Figure 10: Burgers equation with μr 0.3/1
a) D 0.1
b) 0.5
c)
D 10
Figure 11: Advection-Difftssin equation with D 0.1/0.5/2/10. Dirichlet =C
(Domain Boundaries have
Figure 12: Solution of the Poisson equation for the electrostatic potential of two oppositely charged particles within a box with O-boundary conditions. Note, that Metamizer was only trained on the Laplace equation (f 0) while, here, f * 0 at the particle positions.
15

---

Published as a conference paper at ICLR 2025
-2
-00
-8
110
1000
1.5
1.0
1.5
1.0
0.5
1.6
0.5
0.0
0.0
0.0
-0.5
-05
-10
-1.0
Figure 13: Simulation of turbulent fluid dynamics at Re 2000 (μu 0.1, = 4, 100, | 0.5) performed by Metamizer on 400 400 grid.
B DOMAIN-SIZE DEPENDENT PERFORMANCE
In order investigate the domain size dependent performance scaling of Metamizer, trained another network on a 400 x 400 grid to solve the Laplace equation. In the following, we compare Metamizers performance scaling GPU and CPU based solvers.
B. COMPARISON TO GPU BASED SOLVERS
We run Metamizer against various GPU based solvers of the CuPy package on the same Nvidia Geforce RTX 4090. As shown in Figure Metamizer shows similar convergence speed for the Laplace equation a 100 x 100 grid. On a larger 400 x 400 grid (see Figure [4), Metamizer demonstrates improved scaling behavior compared sparse linear system solvers resulting in about 2x faster convergence than conjugate gradients or MINRES.
This is remarkable because Metamizer a more general optimizer that relies only on local gradients and can be used to solve for example nonlinear PDEs or cloth simulations as well. Conjugate gradient methods on the other hand require full knowledge the underlying (symmetric) matrix A to orthogonalize update steps and compute appropriate step sizes a.
Metamizer CuPy (GPU) Sparse Linear System Solvers 400x400
10:2
10
10
10
10-15
10-11
10
10
cuda_gmres
cuda, gmrees
l0es
cuda
1652
cuda Reda
cuda_ minres cuda_cg
10-22
10:22
cuda_rg
10-21
10-21
10-30
10-31
10
10
10a
10a
10a
10:
102
100
10*
time [s]
Iterations
Figure 14: Performance comparison of Metamizer with various GPU based solvers of the CuPy package on 400 x 400 grid. Metamizer is about 2x faster than CG on the same GPU,
16

---

Published as a conference paper at ICLR 2025
B.2 COMPARISON TO CPU BASED SOLVERS
We compared Metamizer to CPU based solvers as well. On 100 x 100 grid. CPU based solvers are actually slightly faster than GPU based solvers and Metamizer (see [3. This is because CPUs exhibit a smaller computational overhead to GPUs and run at higher clockrate. However, on larger 400 x 400 grid, GPU parallelization pays off significantly resulting in 10x faster convergence speed of Metamizer compared to CPU based solvers (see Figure T6].
Metamizer and Scipy (CPU) Sparse Linear System Solvers 100x100
10
10
10-4
10-4
Metamizer
Metamizer
10-15
10-1
cg
bicgstab
bicgstab
10-14
10-44
minres
minres
l05s
14
qmr
10
qmr
tfomr
tfgmrr
10-22
gcrotmik
10-2
gcrotmk
gmree
10-26
Igmres
10-21
lamres
10
10-
10
10
10-1
10"
10º
101
10'
10a
10
time [s]
Iterations
Figure 15: Performance comparison of Metamizer with various CPU based solvers of the SciPy package on 100 x 100 grid.
Metamizer and Stipy (CPU) Sparse Linear System Solvers 400x400
10
10-2
10-4
10.4
Metamicer
10-1#
Metamizer
cg
cg
10-14
bicgstab
10-1
bicgstab
minres
minres
10-14
qmr
1655
10-11
qmr
tfomr
tfamr
10: 21
gcrotmk
10 21
gcrotmk
gmres
gmres
10-2
Igmres
10-2
Igmres
10-1c
10-1
10
10-1
10a
10a
10a
10:
10a
10a
10a
[s]
Figure 16: Performance comparison of Metamizer with various CPU based solvers of the SciPy package on a 400 x 400 grid. Metamizer is about 10x faster than CG on a CPU.
C PRECONDITIONERS
C.I INCOMPLETE LU (ILU) AND ALGEBRAIC MULTI GRID (AMG) PRECONDITIONERS
We compared Metamizer with preconditioned conjugated gradient methods using incomplete LU factorization and algebraic multigrids. To this end, we relied on pyAMG (Bell et al.) (023) for CPU based implementation and pyAMG: based on Nvidia AMGX. AMG preconditioning resulted in significant speed-up over oon-preconditioned solvers on the CPU as well as the GPU (see e.g. Figure 17 and Figure 18 While Metamizer is still faster than CPU based AMG methods, the highly optimized AMGX package by Nvidia is up to 10 times faster than Metamizer domains of this size, this performance difference is in accordance with previous results for neural preconditioners (see Figure 4 La et al. (2033). However, Metamizer is more general approach
the GPU. For
17

---

Published as a conference paper at ICLR 2025
compared to AMG approaches and neural preconditioners since it only requires local gradients and can be applied for example to non-symmetric Matrices, nonlinear PDEs or cloth simulations as well.
Metamizer and preconditioned SciPy (CPU) Sparse Linear System Solvers 400x400
10
10
10
10
10 13
10
Metamizer
Metamizer
10
AMG
le5
10
AMG
10-21
10-2
25
10 25
19
10 24
10
10
10"
30'
10a
time [s]
Iterations
Figure 17: Performance comparison of Metamizer with conjugated gradients preconditioned on the incomplete LU factorization and the algebraic multigrid method (pyAMG (Bell et al. 2023) ) 400x400 grid.
Metamizer and GPU based AMGX 400x400
Necamizer
Motamizer
10.
AMG CLASSICAL TRUNE
10
AND CLASSIOAL
CLASSICAL L1LAGGGESSNE_EHHE HIIS
AMG, CLASSICAL L1, AGGRESSIEE
10
FOVRES AGGRESSIVE
10'
FOWRSS AGURESSIVE
AEGEREGATION JACDRI
10-3
AMG SEEEEESSEEEEEEEELEEEEEEEEE CCHE LRLTUI
10-3
AMIECCLASBICAL CUAASIICALLGGGGGERVEEEE CHEB_LLEEBLL__EEEEEEEEL____L CHEUUL TRUNC
PRlCOSTAS
ralcos
10
10
10-11
10-21
10-25
10-2
10-21
10-2
10--
10
30
10
30
10
zime (4)
Ior allons
Figure 18: Performance comparison of Metamizer with algebraic multigrid methods based on Nvidia AMGX (pyamgx) on a 400x400 grid. Note: the plotted timings of AMGX include initialization overheads. The pure setup and solve timings until convergence were 0.019s for AMG CLASSICAL TRUNC, 0.1s for AMG.CLLSSICALLLI AGGRESSIVVE HMIS, 0.1s for FGMRES 0.028s for AMG.CLASE CLLASSICALAAGGESSSTII CCEESSIIEEEEEEBBEEEEEEEEE LI TRUNC and 0.035s for PBICGSTAB. Metamizer reaches a loss of 6.22 x 10-32 32 after 0.22s.
HMIS, HMIS, 0.069s for PCG AGGREGATIIO JACOBI,
C.2 JACOBI PRECONDITIONER
We tested the diagonal Jacobi preconditioner for CG. GMRES and MINRES. Since the diagonal of the Laplace operator regular grid corresponds closely to the identity matrix, this preconditioner did not help to significantly reduce the number of iterations but resulted in a slight computational overhead and thus slowdown of convergence (see Figure 19
D COMPARISON TO NEWTON-CG, NONLINEAR CG AND L-BFGS-B
We added further comparisons for optimizers that. like Metamizer, rely only on local gradients of the loss. As can be seen in Figure optimizers like Newton-CG, Nonlinear CG or L-BFGS exhibit much slower convergence.
18

---

Published as a conference paper at ICLR 2025
Metamizer and Cupy (GPU) Sparse Linear System Solvers (jacobi Precond/tioner) 400x400
10
10
10.
10-14
10-
10"41
10
cuda, pmres cuda minres
165
10:14
cuda, gmrres cuda, cuda_minres minres
cuda_co
cudo
10-22
10-22
10 28
10
10
10
10
10-2
10
100
101
10P
10
10a
102
10*
iterations
Figure 19: Performance comparison of Metamizer with various GPU based solvers of the CuPy package that make use of Jacobi Preconditioner on a 400x400 grid. comparison to non- preconditioned solvers (Figure 14) this resulted in slight slowdown of convergence.
Metamizer and Scipy (CPU) Optimizers 100x100
10
10
10
10
10
10
Newton CG
loss
Newto n DG
10-18
10-14
Nonlinear
L-FGS a
L-FFGS-B
10-22
10-23
10
10
10-10
10-
10-1
10a
101
10'
10a
time [s]
iterations
Figure 20: Performance comparison of Metamizer with various optimizers that, like Metamizer, rely only on local gradients of the loss. Here, we make use of the SciPy package on 100 100 grid on CPU.
COMPARISON OF DIFFERENT LEARNING RATES
We compared various different learning rates for all gradient descent solvers and only reported results for learning rates that resulted in good trade-offs between convergence speed and accuracy in Figure 6 Figure [1] shows a performance comparison of Metamizer with Adagrad, Adam and AdamW different learning rates.
F COMPARISON TO GROUND TRUTH
We compared the mean squared errors of Metamizer and various GPU based linear system solvers to analytical ground truth values of the Laplace equation. Figure 22 shows fast convergence to highly accurate results already indicated by the mean residual errors Figure 14
G RESIDUAL LOSS GRADIENTS FOR CLOTH SIMULATIONS
Figure 23 shows how Metamizer reduces the gradient norm | of the physics-based Loss
defined in Equation 12 with respect to the accelerations of the cloth for 10, 20 and 50 iterations per timestep. With 10 iterations, Metamizer reaches gradient of around 3 x 10-2, with 20
19

---

Published as a conference paper at ICLR 2025
Metamizer and Adagrad (GPU) 100x100
10-
10
10
10-4
10 18
Metamizer
10 11
Metamizer
Adegrad 0rm0.1)
Adagrad (lr00.1)
10-14
Adagrad 0r-0.03)
10:14
Adagrad (Ir-0.03)
Adagrad 0rm0.01)
Adagrad (lr=0.01)
Adagrad 0r=0.003)
10
Adagrad (l==0,003)
Acagrad 0r=0.001)
Adagrad (I==0.001)
10-22
Anagrad Or-0.0003)
10-22
Adagrad (Ir-O 00003)
Adagrad 0r=0.0001
Adagrad (lr00.0001
10-21
10-2
10-11
10-2
10
10
10
100
10P
10
10a
10P
time [s]
iterations
Metamizer and Adam (GPU) 100x100
10-1
10-4
10
4
10
Metamizer
10-
Metamizer
Adam 0r-0.1)
Adam (Ir=0.1)
10-11
Adam 0r=0.03)
10-11
Adam (lr=0,03)
1055
Adam (r=0.01)
Adam (I=00001)
Adam 0r-0.003)
10-11
Adam (I=000003)
0r=0.001)
(I=00.001)
10-22
Adam 0r=0 00033)
10-2
Adam (Ir=0. 0.00)
Adam 0r=0.0001)
Adam (.0000
10
10
10
10
10-1
10
10-1
10a
10'
10
10a
10
time [s]
iterations
Metamizer and AdamW (GPU) 100x100
10-2
10-2_
10-18
Metamizeer
10-1
Melamizer
AdamW 0r=0.1)
AdamW (r=0011)
10-14
AdamW 0r-0.08)
10-14
AdamiW (l~00003)
100s
AdamW 0r-0.01)
AdamiW (l=00.01)
10-1
Adami 0rm0.003)
10-1*
AdamiW (lrm0.003)
AdamW (r=0.001)
(I=00.001)
10-21
AdamW 1r-0.0003)
10-22
Adamw (r=0.0003)
AdamW 0r-0.0061)
Adami (Ir-0.0001)
10-24
10-21
10-41
10-11
10-1
10-2
10
10a
10P
10a
10a
10a
time [s]
iterations
Figure 21: Performance comparison of Metamizer with various GPU based Gradient Descent Meth- ods at different learning rates on 100 x 100 grid.
iterations, it reaches gradient norm of around x 10-2 and with 50 iterations, it reaches gradient norm of around 4 x 10
H SCALING PARAMETERS
we show additional results how Metamizer scales 8; for 3, 5, 10, 20, 50, 100 iterations per timestep on the of iterations is "wasted"" to scale up 8; for the next timestep. This could be improved in the future by
In Figure 24
equation. For 50 and 100 iterations, a significant amount
20

---

Published as a conference paper at ICLR 2025
CuPy (GPU)
System
10
10">
Iooe with a'alptral sctution
10a
MSE loce with saallticlll sotution
10a1
10"4
10-12
10"14
Netamizer
10:14
Metambzer cuda gmres
cude gmres
10-40
10-44
cuda,
10
cudee
20-12
ouda_59
10-44
10
10a
10
Bmc Ec]
Iterations
Figure 22: Mean squared errors of Metamizer and various GPU based solvers of the CuPy package on a 400 400 grid compared to analytical ground truth values Laplace equation.
Gradient Norm reduction Metamizer. iteretions per timestep. 100 100
109
greden
10-1
100
130
200
250
Earation
Gradient Norm reduction Meternieer, iterations timestep.. 100
100
30'
10-2
200
500
Gradiert reduction Metamizer, iterations per
100
1009
16
200
400
600
800
10000
1200
Iteration
Figure 23: Metamizer iteratively decreases the gradient norm of a physics-basee loss nonlinear cloth simulations 100 x 100 grid. The more iterations per timestep are taken the more accurate the simulation.
automatically scaling s; to a certain percentage (c.g. 20 %) of the maximum scale of the previous time step when starting with time step.
21

---

Published as a conference paper at ICLR 2025
X Saaee Step
11.4
13.1
1341
1077
13:4
1101 Tirae Step
Invise
Figure 24: Scaling parameters for Diffftionn for different numbers of iterations per time step
I LAPLACE OPERATOR DETAILS
Figure 25: Implementation detail for Laplace operator example in ID
During our experiments, we found that naively applying the finite-differenn Laplace operator results in very small gradients even if the residuals are still quite large. Intuition on this problem is provided in Figure 255 If we want to lower the Laplacian for the center vertex (marked in black), we could increase the value of that vertex (marked in green) but we could also decrease the values of its neighboring vertices (marked in red). The same is true for all vertices, such that in the end the gradients pointing upwards get canceled out fairly exactly be the neighboring gradients pointing downwards. To avoid this issue, we detach (and thereby remove) all the neighbor- gradients srighbor-gradiets (marked in red) so that we are only left with the green gradients and can make much better progress in solving the Laplace operator.
22

---

Published as a conference paper at ICLR 2025
J TRAINING POOL DETAILS
The training pool strategy as described in Section 4 does not require any precomputed training data. Instead, Metamizer learns to solve the PDEs purely based on the physics-constrained loss and ran- domized boundary conditions that can be generated on the fly similar to Wandel et al. (2021a) Stotko et al. (022) Geneva Zabaras (2020). For example, in case of fluid simulations we gen- erate random boundaries by placing randomly moving boxes within the fluid domain, randomizing j and p or changing the flow velocity. In case of cloth simulations, we randomize cloth parameters such as Caiff; Cabear and Chend well as the points at which the cloth is fixed. For the advection- diffusion equation, we randomized the diffusivity parameter as well as the velocity field if and the domain boundary. Full details about the different randomization strategies will be provided with our code. After training, Metamizer shows good performance across a wide range of PDEs and generalizes to PDEs that were not covered during training such the wave, Poisson or Burgers equation.
K VIDEO
The supplementary video demonstrates how Metamizer solves the Laplace, diffusion diffusion-advectioon advection, wave, Navier-Stokes and Burgers equations well cloth simulations. The time-dependent PDEs are visualized at the speed of the simulation.
23


Reviews:
[{"review_id": "LuwxKfZu0i", "reviewer": "Reviewer_EsdF", "summary": "The paper proposes a meta-optimization technique for solving physics simulation problems, including linear and nonlinear PDE problems.  The proposed Metamizer framework is an iterative optimizer that uses a scale-invariant architecture to improve gradient descent updates to accelerate convergence.", "questions": "1. The authors criticize gradient descent-like methods like AdaGrad and Adam, yet their method is based on improving gradient descent.  Can the authors speculate on how their ideas could work if based on other techniques like Adam?\n\n2. L251-254 The authors describe how they want to use one network for all PDEs but then describe how different PDEs require different numbers of channels.  This sounds like different networks need to be trained depending on how many variables are in the problem (e.g., you'd have a 1-variable network, a 3-variable network, etc.).  Can the authors clarify this?\n\n3. Figure 6 and the discussion around it doesn't say what equation and what boundary conditions are being solved here.  If this is Laplace/Poisson (at least without all-Neumann boundary conditions) - I don't really believe the results, because this should be a symmetric positive-definite sparse linear system that conjugate gradients should perform quite well on (and certainly better than GMRES and other slower/more general Krylov solvers).\n\n4. With regards to the Figure 6 results, it's a little unfair to compare against constant-learning-rate variants of the other methods shown.  Of course things like SGD can't converge to 1e-33 precision with a learning rate of 0.01 - it is probably doing a very good job but the learning rate is so large that it literally cannot land on the minimum.  And it's not a mystery how to choose step sizes that enable convergence - e.g., for gradient descent, the Wolfe conditions (for instance) can inform what step size you should take at each step.\n\n5. These results are further unfair because Metamizer - along with any other L2O methods used for these types of problems, see for instance the discussion in Kaneda et al. 2023 - are effectively acting as preconditioned solvers.  In Metamizer's case you're preconditioning gradient descent by using data about PDE losses.  So it would be fairer to compare against preconditioned CG, MINRES, GMRES, etc.  Unfortunately, the timings and iteration counts are so close that I bet if you used even just a Jacobi/diagonal preconditioner on those methods, they would all be outperforming Metamizer.\n\n6. Figure 4 indicates that Metamizer-produced solutions may not enforce much smoothness (e.g., there is some rough noise around 20 iterations, where solutions are still not in the regime of double precision limits).  As mentioned earlier, I might venture a guess that this is due to the U-Net architecture chosen.  However, I am curious for the authors' thoughts?  This may be a useful thing to discuss in the paper, too.\n\nAlthough my score leans somewhat negative (I would prefer to mark this as a 4, but it's not an option), it is mostly due to the evaluation and presentation, and I believe this is a worthwhile idea that could be published in an appropriate venue and with suitable manuscript improvements.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. A nice advantage of Metamizer is that it's able to be applied to PDEs that weren't seen during training.  I don't know if there are other state-of-the-art methods like that, but I appreciated that feature.\n\n2. Metamizer appears to be scale-invariant and to not need retraining for different PDEs, which again, is quite useful from a practical perspective.\n\n3. Metamizer seems like a simple idea with a relatively straightforward architecture, which means that future research in this direction should be possible to build upon the present work.", "weaknesses": "1. \"However, to the best of our knowledge, L2O has not yet been applied in the context of physics simulations\"\nUnfortunately, I must inform the authors that L2O has absolutely been applied in the context of physics simulations.  Please see for example the works \"A deep conjugate direction method for iteratively solving linear systems\" (ICML 2023) and \"A Neural-Preconditioned Poisson Solver for Mixed Dirichlet and Neumann Boundary Conditions\" (ICML 2024) and the references / related work discussions in there.\n\n2. \"Data-driven Deep Learning approaches\"\nThe authors may be interested recent works like \"Toward Improving Boussinesq Flow Simulations by Learning with Compressible Flow\" (PASC 2024), which show that the addition of a neural surrogate can produce results that are more accurate than what is possible with a classical numerical solver - because a solver may be using a less accurate physical model.  This is an important point to be aware of or mention, since many deep learning approaches overlook the fact that one way to improve efficiency and accuracy is to simulate things with a cheaper model and then use learning models trained on more accurate models.\n\n2. \"Unfortunately, the learned implicit representations need to be retrained for every physical simulation and thus are not real-time capable.\"\nThis criticism of PINNs does not make sense to me.  When you train a PINN, you can evaluate is very quickly.  The fact that PINNs are best when trained on a specific physics problem, does not seem correlated to the performance of evaluating (performing inference on) a neural network like a PINN.  I suggest the authors remove this criticism or clarify what they mean.\n\n3. (Minor) In the section \"Time-Integration Schemes,\" it is likely worth tossing in a reference to a textbook or some similar resource that discusses additional common time integration schemes like RK4 - just to avoid giving the impression that the three schemes mentioned are the only ones.\n\n4. It is interesting the authors' technique seems to essentially just be a U-net with carefully-chosen features (Fig. 3).  With physics problems, solutions are often smooth/continuous, so practitioners often use CNNs or perhaps mix convolutional layers with U-nets.  There is no discussion of why a plain U-net architecture was chosen, and adding something like that would be helpful for readers to gain intuition into the method.\n\n5. There's not much that can be gleaned from Figs. 7 or 8 without quantitative metrics or labeled error/scale bars.\n\n6. L757 \"This is because Metamizer can be effectively parallelized on a GPU.\" Sparse iterative linear solvers can be effectively parallelized on a GPU - in fact, this is one of the things GPUs are best at.  This statement makes me believe the authors are using CPU or perhaps even serial implementations of things like CG or MINRES, which is a severely (10x+) unfair comparison.\n\nMinor:\n\n7. L245 you call it a CNN but it's specifically a U-Net?\n\n8. L264 You say you use gradient descent but in parentheses you say Adam - do the authors understand these are distinct numerical methods?\n\n9. L268 \"ca 6 hours\" should just be \"about 6 hours,\" or even better, just specify the time more precisely, e.g., \"6 hours and 15 minutes\"\n\n10. The abstract is a little long and split into a few paragraphs; I would recommend condensing things a bit.\n\n11. Formatting mistakes like not doing LaTeX quotes properly - the authors should know better than this in a venue like ICLR\n\n12. L235 citation not formatted right (\\citet not \\citep)\n\n13. L277-281 (for instance - there are other places for this) Put commas after e.g., this is a grammar thing\n\n14. L434 probably a typo in the table here", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzNFyaTvYx", "reviewer": "Reviewer_gC5k", "summary": "The paper presents a neural-network-based meta-optimizer, Metamizer, designed to minimize the residuals of physical systems, especially partial differential equations (PDEs), in pixel space. The authors conduct experiments comparing the performance of Metamizer to both gradient-base optimzers and linear system solvers. Metamizer is evaluated on different PDEs and the results presented.", "questions": "## Missing details/questions\n\n**Q1**: Insufficient details on residual computation (148-154, 175), specifically handling spatial resolution and Navier-Stokes pressure. More information should be provided in the appendix (related to **MW3**).\n\n**Q2**: In (198-211) you say that the main problem with directly optimizing the gradient is the scaling, therefore you introduce a scale-invariant optimizer. What about local minima? The solution of the physical problem is the global minimum of the problem, so getting stuck in a local minimum is a potential issue that is not necessarily solved by scale invariance.\n\n**Q3**: Is there no cold start issue? In the **Training** paragraph (256-269) it is stated that the training pool is initialized with randomized solutions. How exactly?. Fig. 6 shows that the gradient-based optimizers (Adam, SGD, etc) fail to converge. If the gradient signal is so poor how can the UNet learn from it? How does this relate to the scale vs local minima issues from Q1? Can Metamizer, once trained, minimize for randomized solutions drawn from a different distribution than the one it was trained on?\n\n**Q4**: I appreciate the wall time and GPU usage training information (269). How many update steps were performed on the model? How many data samples did it see (presumably the number of update steps * batch size)?\n\n**Q5** I find the description of the time integration and how it is used in the context of Metamizer unclear. Is the integration scheme used to define the discretization used of the time derivative? E.g. for forward Euler it would be approximated as $u_t' = \\frac{u_{t+1} - u_{t}}{\\Delta t}$ \n\n**Q6** Why is time integration needed at all? Can the residual computed on the full trajectory not be minimized for all the states at once?\n\n\n## Things to improve the paper that did **not** impact the score:\n\nAbstracts usually have a single paragraph.\n\nSpace before comma on line 93.\n\n\n## Refrences\n\n1) Brenner, S. C., & Scott, L. R. (2008). The Mathematical Theory of Finite Element Methods. In Texts in Applied Mathematics. Springer New York. https://doi.org/10.1007/978-0-387-75934-0 \n2) Chassaing, P. (2022). Fundamentals of Fluid Mechanics. Springer International Publishing. https://doi.org/10.1007/978-3-031-10086-4\n\n\n## Edit history\n1. Confidence reduced from 4 to 3 in light of the discussion with the authors.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The problem the paper tries to address is of interest and has wide applicability.", "weaknesses": "The paper has 3 main weaknesses:\n- **MW1**: Lack of quantitative evaluation. Only qualitative images and residuals are presented, making it difficult to assess the method's performance objectively. This issue affects all the experiments.\n- **MW2**:  No baseline comparisons with other Learning to Optimize (L2O) methods, classical numerical solvers, or neural PDE methods.\n- **MW3**:  Limited experimental scope. Experiments use only fixed-size grids, 100x100 (331) and 400x400 (775), which restricts generalizability and does not test varying resolutions, where new dynamics may arise, especially in the Navier-Stokes equation.\n\n\n## Main weaknesses\n\nThe proposed method can be viewed as a meta-learning method, a PDE solver, or an emulator but it lacks comparisons with other methods from these fields. Ideally, the method would be compared against methods from each of these fields. I acknowledge that such an exhaustive comparison involves a considerable effort, but the paper omits even a basic evaluation against standard methods in these areas.\n\nThe authors assert that L2O has not been applied to physics (99) but fail to demonstrate how Metamizer performs in the physics context against other L2O methods. Furthermore, the paper’s accuracy claims lack quantitative evidence, as it provides only qualitative images and residuals. This is inadequate for judging solution quality, especially when standard neural PDE literature benchmarks against ground truth obtained from numerical methods. Moreover, the lack of comparison to classical numerical methods is striking, since a comparison is performed against linear solvers for a linear system built in pixel space. Effectively, the paper is comparing against different linear solvers for the finite element method.\n\nIn short, the experimental evidence does not convincingly support the method's efficacy or comparative performance and extensive existing literature has not been engaged with.\n\n## Additional Weaknesses\n\n**W1**: The claim that no hyperparameters are used (68-69) is inaccurate, as $s_0$ is a hyperparameter (238).\n\n**W2**: The claim on (332-333) is misleading. In the deep learning literature, accuracy typically reflects error metrics like MSE between the ground truth and the prediction. In this paper, accuracy has a different meaning (value of the residual) so it is not directly comparable. This is related to **MW1-2**.\n\n**W2**: The accuracy claim on (332-333) is misleading; in deep learning, accuracy typically reflects error metrics like MSE. It is not possible to directly compare it to the claimed MSR.\n\n**W3**: Misleading claim about linear solvers and their usage. In Fig. 6 Metamizer is compared to linear solvers, and in (327) it is claimed that linear solvers are limited to linear PDEs. This is only true in pixel space. Most PDE-solving techniques, such as the FEM, rely on solving a linear system derived from the equation through linearization [1]. Related to **MW2**.\n\n**W4**: The caption of Fig. 8 claims that Metamizer can simulate a wide range of Reynolds numbers and shows snapshots for $Re=1$, $Re=50$ and $Re=200$. This range is insufficient compared to the typical Reynolds number range (1 to millions) in fluid dynamics, missing the complex behaviors at higher Reynolds numbers [2].\n\n**W5**: No limitations whatsoever are discussed or acknowledged.\n\n\n-------\nGiven all of the above, I recommend the paper be rejected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myUxdAaKnq", "reviewer": "Reviewer_shqZ", "summary": "This paper targets an optimizer for training deep neural networks for improving the solving of tasks related to PDE solving. The goal if scale invariance is identified as a central challenge for physics-related learning tasks. The paper then proposes a U-net that is employed to infer the scaling of an update in order to improve conditioning of the neural network update.\n\nThe importance of scale invariance has been studied before for physics problems, e.g. in Holl et al., Scale-invariant learning by physics inversion, NeurIPS 2022. I just double checked , and e..g fig 1 there and figure 2 here are actually have a lot in common. This previous paper in the end takes a different approach, so I think it's no direction competition, but the unclear parts of the current submission definitely lead to a few questions.\n\nThe paper evaluates this idea for several canonical PDEs (Poisson & Laplace, advection-diffusion), as well as more complicated ones (waves, Navier-Stokes). Interestingly, cloth dynamics are included as well. While the rang of experiments is very nice, the motivation in terms of theory behind the approach remains unclear. Nonetheless, I think it’s an interesting and novel direction. So I liked the paper and its direction, but several unclear topics remain, which I hope the authors can address during the rebuttal phase.", "questions": "Closely related to the topics above, can the authors shed light on the following topics?\n\n- How does the approach differ from the scale-invariant learning proposed in the NeurIPS'22 paper above?\n\n- Does the theory there affect or relate to the current submission? \n\n- How are the scaling factors s_i computed, and how accurately are they inferred?\n\n- Have the authors considered Newton methods, that approximate the inverse Hessian for \"scale invariance\"? I think this would also be worth a discussion, at least.\n\n- The paper focuses on solving optimization problems in terms of PDE states. How would this extend to a second neural network, i.e. backpropagating further than just the u?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The central goal of addressing \"scale invariance\" is not new, but the proposed learning of scale invariance is new if I understood the approach correctly.\n\nThe breadth of results is also great to see.", "weaknesses": "One central part that was unclear to me was how the scale factors for learning are actually precomputed. Is this simply a scaling of the reference solutions from the solver? As this is the central component that is supposed to make the method work, I think it would be important to be clear where this comes from. Correspondingly, I was missing an evaluation in terms of how well the network actually learned to predict these scaling factors. The results seem to only show the loss in terms of the state.\n\nThe Holl'22 paper above also provides some theory on how updates from scale invariant solvers improve learning. Unfortunately the submission here does not take up on these topics, and focuses on experiments.\n\nThe appendix is also very brief on how the different learning tasks were actually set up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u4YKVGF8PX", "reviewer": "Reviewer_8DMR", "summary": "The paper studies solving partial differential equations (PDEs) numerically. The proposed framework builds upon a finite-difference spatial discretization of the unknown fields on a regular grid. Next, it minimizes a residual loss similar to that in the physics-informed neural network (PINN) via gradient-based optimization. The new idea comes in a neural network that predicts the update rule (scaling factor and direction) used in each step of the gradient-based optimization. The paper demonstrates this idea on several linear and nonlinear PDE problems.", "questions": "- Does CG fail to converge in Fig. 6? Is the matrix not symmetric positive definite?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of applying meta-optimizers to learning-based PDE solvers looks new to me. I think this is an interesting thought worth further discussion and exploration.\n- The technical method is simple, straightforward, and easy to implement.\n- The paper writing is good; it articulates its technical details very well.\n- The paper considers a diverse set of PDE problems, including N-S equations and 3D cloth simulation (albeit with limited features).", "weaknesses": "**Introduction/Related Work**\n- The paper claims numerical methods for solving PDEs “are computationally\nexpensive”/“are still relatively expensive to compute”/“require high computational resources.” This argument is generally valid for large-scale problems. However, for the small-sized problems demonstrated in this work (400 x 400), Chances are that numerical methods are not as expensive as the paper indicates and may be faster than Metamizer (See Poisson/Laplacian problem below). I suggest the paper evaluate its method on problems of much larger sizes, e.g., on >=1024^2 (2D) or >=64^3 (3D) grids.\n\n**Foundations**\n- Equations 3 and 4 are not new and should cite the original PINN paper. It is probably also worth mentioning that minimizing Equation 3 may get stuck in a local minimum (grad L = 0, but L is nonzero), which is not a solution to the original PDE. For several problems (linear second-order PDEs) demonstrated in the experiments, a variational loss from the PDE’s weak form is a standard and a naturally better candidate for the loss design. The incremental potential in Eqn. 12 is one such example that highlights this point.\n- The issue in Fig. 2 and the main intuition for taking gradients from previous steps into account have been extensively studied in numerical optimization methods. The former is typically handled with (exact) line searches, and the latter is discussed in (nonlinear) CG, L-BFGS, etc. (“Numerical Optimization” by Nocedal & Wright.) It is still very interesting to contemplate the possibility of training a neural network to exploit gradients from previous steps, but the paper should discuss its relation to these classic numerical optimization methods and compare them in an experiment.\n\n**Experiments**\n- The Poisson/Laplacian example has the most complete analysis, but the main results in Fig. 6 look dubious for several reasons:\n1) It looks like Metamizer has used an Nvidia RTX 4090 GPU (line 330 and Appendix B), so it should be compared with linear solvers implemented on the same GPU. However, I don’t recall scipy having a GPU implementation of them. Please correct me if I am wrong.\n2) The state-of-the-art GPU implementation of a Krylov subspace solver (CG, GMRES, etc.) is likely to be better than both the scipy baselines and Metamizer reported in Fig. 6 and Appendix B. Back in 2005, deploying a standard CG algorithm on a GPU could solve wave/Poisson equations on a 1024^2 grid at a rate of 10-30 fps (Chapter 44 in “GPU Gems 2” edited by Pharr & Fernando and the referenced papers), already arguably faster than the speed of Metamizer reported in the much smaller 400^2 problem. I feel this paper didn’t identify the right baseline for comparison.\n\n- The wave equation example does not compare Metamizer’s performance with a baseline. I suggest comparing it with the CG-GPU baseline mentioned above.\n\n- Eqn. 12 in the cloth example is not new and should cite its original source, e.g., Stotko 2024. This equation can be traced back to at least “Optimization Integrator for Large Time Steps” by Gast et al. in 2015, if not earlier.\n\n- “Note that L can take negative values and cannot be interpreted as easily as the mean squared residuals of the previous PDEs. Thus we omitted L in Table 1 for our cloth simulations.” In this case, grad L = 0 is the right indicator for successfully solving the PDE. There should be a table reporting grad L after certain iterations.\n\n- On the algorithmic side, I feel some classic approaches for exploiting gradients from past iterations should be considered for comparison to establish the advantage of the network module in Fig. 3a. Two candidates are nonlinear CG and L-BFGS (with history size = 2). These methods do not require training and can be directly used to minimize Eqns. 3 and 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)