Scale-invariant Learning by Physics Inversion
Authors: Philipp Holl
Vladlen Koltun
Nils Thuerey
Venue: NeurIPS 2022 Accept (2022)
Keywords: Higher-order Optimization
Neural Networks
Inverse Problems
Physical Simulations

Abstract: Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science. Many experiments repeatedly collect data and rely on machine learning algorithms to quickly infer solutions to the associated inverse problems. We find that state-of-the-art training techniques are not well-suited to many problems that involve physical processes. The highly nonlinear behavior, common in physical processes, results in strongly varying gradients that lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions.
We propose a novel hybrid training approach that combines higher-order optimization methods with machine learning techniques. We take updates from a scale-invariant inverse problem solver and embed them into the gradient-descent-based learning pipeline, replacing the regular gradient of the physical process.
We demonstrate the capabilities of our method on a variety of canonical physical systems, showing that it yields significant improvements on a wide range of optimization and learning problems.

Scale-invariant Learning by Physics Inversion
Philipp Holl
Vladlen Koltun
Nils Thuerey
Technical University Munich
Apple
Technical University of Munich
Abstract
Solving inverse problems, such as parameter estimation and optimal control, vital part of science. Many experiments repeatedly collect data and rely machine learning algorithms to quickly infer solutions to the associated inverse problems. We find that state-of-the- training techniques are not well-suited many problems that involve physical processes. The highly nonlinear behavior, common in physical processes, results in strongly varying gradients that lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions. We propose novel hybrid training approach that combines higher- order optimization methods with machine learning techniques. We take updates from a scale-invariant inverse problem solver and embed them into the gradient- baacent-based learning pipeline, replacing the regular gradient of the physical process. We demonstrate the capabilities of our method on variety of canonical physical systems, showing that it yields significant improvements on a wide range of optimization and learning problems.
Introduction
Inverse problems that involve physical systems play central role in computational science. This
class of problems includes parameter estimation (54) and optimal control [57]. Among others, solving inverse problems is integral detecting gravitational waves (22), controlling plasma flows [38]
searching for neutrinoless double-beta decay
and testing general relativity 19 35
Decades of research in optimization have produced wide range of iterative methods for solving inverse problems 47 Higher-order methods such as limited- memoory BFGS [36] have been espe- cially successful. Such methods compute approximate the Hessian of the optimization function addition the gradient, allowing them to locally invert the function and find stable optimization directions. Gradient descent, in contrast, only requires the first derivative but converges more slowly, especially in ill-conditioned settings [49].
Despite the success of iterative solvers, many of today's experiments rely on machine learning methods, and especially deep neural networks, find unknown parameters given the observations [I] 25]3 3) While learning methods typically cannot recover a solution up to machine precision, they have a number of advantages over iterative solvers. First, their computational cost for inferring solution is usually much lower than with iterative methods. This is especially important in time- critical applications, such the search for rare events in data sets comprising of billions of individual recordings for collider physics. Second, learning-basee based methods do not require an initial guess to solve a problem. With iterative solvers, a poor initial guess can prevent convergence to global optimum lead to divergence (see Appendix C. Third, learning-basee prone to finding local optima than iterative methods because the parameters are shared across large collection of problems [30]. Combined with the stochastic nature of the training process, this allows gradients from other problems to push a prediction out of the basin of attraction of a local optimum.
solutions can be less
"Corresponding author, philipp. hww.puuml...m.m..
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Practically all state-of-the art neural networks are trained using only first order information, mostly due the computational cost of evaluating the Hessian w.r.t the network parameters. Despite the many breakthroughs in the field of deep learning, the fundamental shortcomings of gradient descent persist and are especially pronounced when optimizing non-linear functions such as physical systems. In such situations, the gradient magnitudes often vary strongly from example to example and parameter to parameter.
In this paper, we show that inverse physics solvers can be embedded into the traditional deep learning pipeline, resulting in a hybrid training scheme that aims to combine the fast convergence higher- order solvers with the low computational cost of backpropagation for network training. Instead of using the adjoint method to backpropagate through the physical process, we replace that gradient by the update computed from higher-order solver which can encode the local nonlinear behavior of the physics. These physies updates are then passed on to traditional neural network optimizer which computes the updates to the network weights using backpropagation. Thereby our approach maintains compatibility with acceleration schemes [8] 33 and stabilization techniques [2 31
developed for training deep learning models. The replacement of the physics gradient yields simple mathematical formulation that lends itself to straightforward integration into existing machine learning frameworks.
In addition to a theoretical discussion, we perform an extensive empirical evaluation on wide variety of inverse problems including the highly challenging Navier-Stoker equations. We find that using higher-order or Nomain-speciine solvers can drastically improve convergence speed and solution quality compared to traditional training without requiring the evaluation of the Hessian w.r.t. the model parameters.
Scale-invariance in Optimization
We consider unconstrained inverse problems that involve differentiable physical process P. X C Rd + Y C dy which can be simulated. Here X denotes the physical parameter space and Y the space of possible observed outcomes. Given an observed or desired output y" E Y, the inverse problem consists of finding optimal parameters
1 x² = arg ((r with L(x) (1P()) - 11.
(I)
Such problems are classically solved by starting with an initial guess zi and iteratively applying updates I = -a Ark. Newton's method [5] and many related methods 110 36 23
41
46
g
12 6 approximate L around zę as parabola L(x) = L(xx)+ aL(sz)
82 (x Ix) + 1 H4 xx)²
where H₃ denotes the Hessian an approximation thereof. Inverting L and walking towards its minimum with step size 1) yields
(aT T
Ark = - H-1
Oxy
The inversion of H results in scale-invariant updates, i.e. when rescaling a or any component of r, the optimization will behave the same way, leaving L(xx) unchanged. An important consequence of scale-invariance that minima will be approached equally quickly in terms of no matter how wide or deep they are.
Newton-type methods have one major downside, however. The inversion depends the Hessian H which is expensive to compute exactly, and hard to approximate in typical machine learning settings with high- nigh-dimensional parameter spaces (25 and
(5T).
Instead, practically all sfate-of-the-art deep learning relies on first-order information only. Setting
H the identity in Eq. 2 yields gradient descent updates Ar -1 (OL)² which not scale-
(oz))
invariant. Rescaling by À also scales Ar by X, inducing factor of X2 in the first-order-accurate loss change [(x)- L(x + Ar) - ( [0L )2 + (()2)). Gradient descent prescribes small updates to parameters that require a large change to decrease L and versa, typically resulting in slower convergence than Newton updates [56]. This behavior is the root cause of exploding vanishing aL gradients in deep neural networks. The step size 7) alone cannot remedy this behavior whenever du

---

varies along z. Figure| | shows the optimization trajectories for the simple problem P(z) = (I1, x2)) illustrate this problem.
When training a neural network, the effect of scaling-variance can be reduced through nor- malization in intermediate layers [32 7 and
3
8
N
regularization [37 53) but this level of control is not present in most other optimization tasks, such as inverse problems (Eq. I More ad- vanced first-order optimizers try to solve the scaling issue by approximating higher-order
2
2
0
information 33 29
45 550. de- creasing the loss scaling factor from X to
18.
39
40
55
such as Adam where
diag
8m
Figure 1: Minimization (Eq. 1 with y P(z) (x1, zğ). contours in gray. Trajectories of gradient descent (GD), Newton's method (N), and perfect physics inversion (PI) shown as lines (infinitesimal 7)) and circles (10 iterations with constant 77).
However, these methods lack the exact higher- order information which limits the accuracy of the resulting update steps when optimizing nonlinear functions.
3 Scale-invariant Physics and Deep Learning
We are interested in finding solutions Eq. Dusing a neural network, x" NN(y* 10), parameterized by 0. Let y {v: 1...., N) denote set of N inverse problems involving P. Then training the network means finding
8. = arg ming Σ IP 0)) - 1
(3)
Assuming large parameter space & and the use of mini-batches, higher-order methods are difficult to apply to this problem, described above. Additionally, the scale-variance issue of gradient descent especially pronounced this setting because only the network part of the joint problem NN P can be properly normalized while the physical process P is fixed. Therefore, the traditional approach
of computing the gradient OL using the adjoint method (backpropagation) can lead to undesired 09
behavior.
Consider the problem P(x) = et with observed data '² E (0, 1] (Appendix C.2). Due to the
exponential form of P, the curvature around the solutions x² = log(y" strongly depends y² This causes first-order network optimizers such as SGD or Adam to fail in approximating the correct solutions for small y" because their gradients overshadowed by larger y" (see Fig, 2 Scaling the gradients to unit length in drastically improves the prediction accuracy, which hints at a possible solution: we could employ ieale-invrriant physics solver, we would be able to optimize all examples, independent of the curvature around their respective minima.
3.1 Derivation
P has unique inverse and is sufficiently well-behaved, we can split the joint optimization problem (Eq. 3) into two stages. First, solve all inverse problems individually, constructing a set of unique solutions 2 {x+" P- (y?)) where p-1 denotes the inverse problem solver. Second, use sv
as labels for supervised training
0_ arg ming
NN (0) -=,"'1
This enables scale-invariant physics (SIP) inversion while fast first-order method can be used to train the network which can be constructed to be normalized using state-of sate-of-thhear procedures [32 7 53]
Unfortunately, this two-stage approach is not applicable in multimodal settings, where #v depends on the initial guess 2i used in the first stage. This would cause the network interpolate between possible solutions, leading to subpar convergence and generalization performance. To avoid these

---

problems, we alter the training procedure from Eq in two ways. First, we reintroduce p-1 into the training loop. yielding
8, = arg ming
NN (0) - (yt)
(5)
Next, we condition p-1 on the neural network prediction by using it as an initial guess, (1)) PP! (v" INN(v" ()). This makes training in multimodal settings possible because the embedded solver P searches for minima close to the prediction of the NN. Therefore 0 can exit the basin of attraction of other minima and does not need to interpolate between possible solutions. Also, since all inverse problems from y are optimized jointly, this reduces the likelihood of any individual solution getting stuck in local minimum, discussed earlier.
The obvious downside to this approach is that P-1 must be run for each training step. When P-1 is iter- ative solver, we can write it as P~1 (y² NN [0) = NN 0) + Ano + update Azy = (v" (0).
Ar,. We denote the first
Predicted x
Instead of computing all updates Az, we approximate p-1 with its first update U. Inserting this into Eq.
with (a) = (u) 0) yields
SGD
Adam
-10
Rescaled
8. arg ming
Σ
NN (N)-(N((o) + U()))
-10
(6)
This can be further simplified Σ (v 101 but this form is hard to optimize directly as is requires backpropagation through U. In addition, its minimiza- tion is not sufficient, because all fixed points of U, such as maxima or saddle points of L, can act as attractors.
Figure 2: Networks trained according to Eq. 3 with P(xx) e" Stochastic gradient descent (SGD) and Adam fail approxi- mate solutions for small values due to scale variance. Normalizing the gradient in space (Rescaled) improves solution accu- racy by decreasing scale variance.
Instead, we make the assumption 0P-1 the (o) dependencies in Eq. 6, treating them as constant.
By
to remove
This results in simple 2,2 Toss for the network output.
As we will show, this avoids both issues. It also allows us to factor the optimization into network and physics graph (see Fig. 3). so that all necessary derivative computations only require data from one of them.
3.2 Update Rule
The factorization described above results in the following update rule for training with SIP updates, shown in Fig.
1. Pass the network prediction zo the physics graph and compute Azo Uv( 20) for all examples in the mini-batch.
2. Send ã zo Amo back and compute the network loss i. 1 |200 5 :
3. Update using any neural network optimizer, such as SGD or Adam, treating a constant.
Network Graph
Physics Graph
y*
NN
X
U
Ax
Backprop
Figure 3: Neural network (NN) training procedure with embedded inverse physics solver (U).

---

To see what updates A result from this algorithm, we compute the gradient w.r.t. 6,
ai
N
àNN
UUS
20)
00
30
Comparing this to the gradient resulting from optimizing the joint problem NN P with a single optimizer (Eq. 3
ÔL Ayj 00
OP ONN
dr 00
8P the adjoint vector that where Ay (NN 10) - - we see that U takes the place of Ay ag would otherwise be computed by backpropagation. Unlike the adjoint vector, however, 5 25 + U encodes an actual physical configuration. Since E can stem from a higher-order solver, the resulting updates A can also encode non-linear information about the physics without computing the Hessian w.r.t. 6.
3.3 Convergence
It is priori not clear whether SIP training will converge, given that OP-1 is not computed. We By start by proving convergence for two special choices of the inverse solver p-1 before considering the general case. We assume that NN is expressive enough to fit our problem and that it is able to converge to every point x for all examples using gradient descent:
En > 0
Vi Vz Vc > 0 3n E N: INNN,, - =x112 S
where 0n is the sequence of gradient descent steps
OONN 00
Ox+1 On
(NNo, xx)
(10)
with 7 > 0. Large enough networks fulfill this property under certain assumptions 17/ and the universal approximation theorem guarantees that such configuration exists [13]
In the first special case, U2x) U(y |x) points directly towards solution x²" This models the
case of a known ground truth solution in unimodal setting. For brevity, we will drop the example
indices and the dependencies on y²
Theorem 1. V2 EY E (0, 0,+1 = On + ( (ONK)² ) (z) converges to x².
U(x) X(x - x),, then the gradient descent sequence NNo,, with
Proof. Rewriting Uzr) = a (21lz - - x²
yields the update On+1 0, = =nn ( aL ONN
T
de de
where i. || 11 This describes gradient descent towards x" with the gradients scaled by Since E (Ó, 1]. the convergence proof of gradient descent applies.
The second special case has (z) pointing the direction of steepest gradient descent in æ space.
Theorem 2. Va YE E (0,1] U(x) -X (
(#))² T then the gradient descent sequence NN with
On+1 On v
ONN 59
Uxx) converges to minima of L.
Proof. This is equivalent to gradient descent in L( (NN ()). Rewriting the update yields
0n+1 - 02 - oe 09
AL
ANN T which is the gradient descent update scaled by X.
Next, we consider the general case of an arbitrary P-1 and U. We require that U decreases L by minimum relative amount specified by T.
Er 0
Vr L(x)= L(r + U(x)) 2 T (L(x) - I(x"))
(11)
To guarantee convergence to a region, we also require
W > 0 Yz ||U(z)] K(L(z) L(x")).
(12)
5

---

Theorem 3. There exists an update strategy 0x+1 S(O,) based on single evaluation of U for which oN o. (v)) converges to a minimum x² or minimum region of L for all examples.
Proof. We denote H, In + (x,)) and AL" L(x,) - L(x"). Let 1g denote the open set of all x for which L(x) - [(nn) > JAL. Eq. 11 provides that En E Iz. Since Iy is open, Be 0: Vo E B. (5.) (xnn) [(x) JAL" where (x) denotes the open set containing all
r' Rd for which || x|I2 i.c there exists a small ball around ā, which is fully contained in
I2 (see sketch in Fig.
can find finite N for which NNo. E (.,, and therefore L(x,.) L(NNo, SAL". We can thus use the following strategy S for minimizing L: First, compute In. In + + (T.)) Then perform gradient descent steps in 9 with the effective objective function | ||NN, - 2n. until [(x,,, o(NNg) Z TAL'. Each application of S reduces the loss to A
Using Eq.
so any value of L L(x") can be reached within finite number of steps. Eq. 12 ensures that [x)( + as the optimization progresses which guarantees that the optimization converges to minimum region.
While this theorem guarantees convergence, it requires potentially many gradient descent steps in for each physics update U' This can be ad- vantageous in special circumstances, e.g. when U is more expensive to compute than an up- date in 0, when 0 is far away from a solution. However, in many cases, we want to re-evaluate U after each update 0. Without additional assumptions about U and NNo. there is no guar- antee that I. will decrease every iteration, even for infinitesimally small step sizes. Despite this, there good reason assume that the optimiza- tion will decrease L over time. This can be seen visually in Fig. where the next prediction In+1 is guaranteed to lie within the blue circle. The region of increasing loss is shaded orange and always fills less than half of the volume of the circle, assuming we choose a sufficiently small step size. We formalize this argument in appendix A.2.
Xn
4
(L
0x
Figure Convergence visualization for the proof of theorem [ All shown objects are visualized in x space for one example
Remarks and lemmas the case that U Azo + Arm. for any E N. When we let the solver convergence, i.e. m large enough, theorem guarantees convergence if p-1 consistently converges to the same o/- Also note that any minimum 0" found with SIP training fulfills U for all examples, i.e. we implicitly minimizing EN [[(( 0))2
We considered the case that = Axg. This can be trivially extended to
3.4 Experimental Characterization
We first investigate the convergence behavior of SIP training depending on characteristics of P. We construct the synthetic two-dimensional inverse process
((x) (sin()1)/E, di with f -= Rs x;
where Rs E SO(2) denotes rotation matrix and O. The parameters & and allow to continuously change the characteristics of the system. The value of & determines the conditioning of P with large representing ill-conditioned problems while o describes the coupling of xı and T2. When & = 0, the off-diagonal elements of the Hessian vanish and the problem factors into two independent problems. Fig. 5h shows one example loss landscape.
We train a fully-connected neural network to invert this problem (Eq. 3). comparing SIP training using saddle-free Newton solver [14] to various state-of-the-art network optimizers. We select the best learning rate for each optimizer independently. For 0, when the problem is perfectly conditioned, all network optimizers converge, with Adam converging the quickest (Fig. 5p). Note

---

0.5
10
10"
Roletww X-Dadincs
Moan X Diitaance
10"
SP (Ouns
Adm
0.0
16
10
10"
Adakita
SGD
Adogna
-0.5
250
500
25
40
(a)
(b)
Time (8)
(c)
Time
(d)
Conditininn
(8)
Rotation 9
Figure 5: (a) Example loss landscape with y" = (0.3, 0.5),
15 (b,c) Learning curves
with o f. averaged over 256 min-batches. For (b) ξ 1. and (c) & 32. (d) Dependence
problem conditioning & (with & = 0). (e) Dependence on parameter coupling & (with { = 32).
that the relatively slow convergence of SIP mostly stems from it taking significantly more time per iteration than the other methods, on average 3 times as long as Adam. As we have spent little time in eliminating computational overheads, SIP performance could likely be significantly improved to mear-Adam performance.
When increasing & with o 0 fixed (Fig. 5t1, the accuracy of all traditional network optimizers decreases because the gradients scale with (T/E, æ, elongating in I2, the direction that requires more precise values. SIP training uses the Hessian to invert the scaling behavior, producing updates that align with the flat direction in r to avoid this issue. This allows SIP training to retain its relative accuracy over wide range of ç. At 5 = 32, only SIP and Adam succeed in optimizing the network to significant degree (Fig. 5t).
Varying & with 32 fixed (Fig. (5) sheds light on how Adam manages to learn in ill-conditioned settings. Its diagonal approximation of the Hessian reduces the scaling effect when 21 and z2 lie on different scales, but when the parameters are coupled, the lack of off dffgonaa terms prevents this. SIP training has no problem with coupled parameters since its updates are based on the full-rank Hessian LLL Og
3.5 Application to High-diieasioonl Problems
Explicitly evaluating the Hessian is not feasible for high-dimensional problems. However, scale- invariant updates can still be computed, e.g. by inverting the gradient or via domain knowledge. We test SIP training on three high-dimensional physical systems described by partial differential equations: Poisson's ss equation, the heat equation, and the Navier-Stokes equations. This selection covers ubiquitous physical processes with diffusion, transport, and strongly non-local effects, featuring both explicit and implicit solvers. All code required to reproduce our results is available at httpps //github. com/ tum- /SPP A detailed description of all experiments along with additional visualizations and performance measurements can be found in appendix B.
Poisson's equation Poisson's ' equation, P(z) = V 22, plays an important role in electrostatics, Newtonian gravity, and fluid dynamics [4] It has the property that local changes in z can affect P Pax) globally. Here we consider a two-dimensional system and train a U-net [48] to solve inverse problems (Eq. T pseudo-randomly generated y We compare SIP training SGD with momentum, Adam, AdaHassian [55]. Fourier neural operators (FNO) 33 and Hessian-free optimization (H-free) [39]. Fig. 66 shows the learning curves. The training with SGD, Adam and AdaHessian drastically slows within the first 300 iterations. FNO and H-free both improve upon this behavior, reaching twice the accuracy before slowing. For SIP, we construct scale-invariant Ax based on the analytic inverse of Poisson's 's equation and use Adam to compute A6. The curve closely resembles an exponential curve, which indicates linear convergence, the ideal case for first-order methods optimizing an L2 objective. During all of training, the SIP variant converges exponentially faster than the traditional optimizers, its relative performance difference compared to Adam continually increasing from a factor of at iteration 60 to full order of magnitude after 5k iterations. This difference can be seen in the inferred solutions (Fig. (b)) which are noticeably more detailed.
Heat equation Next, we consider system with fundamentally non-invertible dynamics. The heat equation, du = V models heat flow in solids but also plays part in many diffusive B4
systems [16.. It gradually destroys information as the temperature equilibrium is approached (26). causing VP to become near-singular. Inspired by heat conduction in microprocessors, generate
7

---

(a)
y
Xasam
XSIP
x
(c)
y
XsGo
(b)
(d)
SGD
AdaHessian
H-free
Inferred MAE
Inferred borredx.o MAE
500
Adam
FNO
SIP (Ours)
10"
100
0
1000
2000
3000
1000
2000
3000
4000
Time (s)
Time (s)
Figure 6: Poisson's equation (left) and the heat equation (right). (a,c) Example from the data set: observed distribution (y"), inferred solutions, and ground truth solution (x*). (b,d) Learning curves, running average over 64 mini-batches (except for H-free).
examples IGT by randomly scattering four to ten heat generating rectangular regions on plane and simulating the heat profile y Px as observed from outside heat-conducting casing. The learning curves for the corresponding inverse problem are shown in Fig. od.
When training with SGD, Adam AdaHessian, observe that the distance to the solution starts rapidly decreasing before decelerating between iterations 30 and 40 to a slow but mostly stable convergence. The sudden deceleration is rooted in the adjoint problem, which is also diffusion problem. Backpropagation through PP removes detail from the gradients, which makes it hard for first-order methods recover the solution. H-free initially finds better solutions but then stagnates with the solution quality slowly deteriorating. FNO performs poorly on this task, likely due to the sharp edges x".
The information loss in P prevents direct numerical inversion of the gradient or Hessian. Instead, add a dampening term derive numerically stable ocale-invariant solver which use for SIP training. Unlike SGD and Adam, the convergence of SIP training does not decelerate as early as the other methods, resulting in exponentially faster convergence. At iteration 100, the predictions are around 34% more accurate compared to Adam, and the difference increases to 130% after 10k iterations, making the reconstructions noticeably sharper than with traditional training methods (Fig. (6)). To test the dependence of SIP on hyperparameters like batch size or learning rate, we perform this experiment with range of values (see appendix B.3). Our results indicate that SIP training and Adam are impacted the same way by non-optimal hyperparameter configurations.
Navier-Stokes equations Fluids and turbulence are among the most challenging and least un- derstood areas of physics due their highly nonlinear behavior and chaotic nature [2] We consider a two-dimensional system governed by the incompressible Navier-Stokes equations: 8u 2 - V Vu Vp. V O, where p denotes pressure and y the viscosity. At = 0, at
region of the fluid is randomly marked with massless colorant my that passively moves with the fluid, dm
Vn. After time I, the marker is observed again to obtain initialized as a superposition linear motion, large vortex and small-scale perturbations. example observation pair y² {mo, mz} is shown in Fig. 77. The task is to find an initial fluid velocity æ up such that the fluid simulation P matches my at time Since is deterministic, encodes the complete fluid flow from 0 to We define the objective in frequency space with lower frequencies being weighted more strongly. This definition considers the match the marker distribution on all scales, from the coarse global match to fine details, and is compatible with the definition in Eq. We train a U-net 48 to solve these inverse problems; the learning curves are shown in Fig.
The fluid motion
When training with Adam, the error decreases for the first 100 iterations while network learns to infer velocities that lead to an approximate match. The error then proceeds to decline a much lower rate, nearly coming standstill. This is caused by an overshoot in terms vorticity, as visible in Fig. a right. While the resulting dynamics roughly approximate the shape of the observed m

---

(a) mo
YAdam
YsIP
XAdam
X5IP
Inferred MAE
10
5
Adam
SIP (Ours)
www
2
20000
40000
600D0
80000
Time (s)
Figure 7: Incompressible fluid flow reconstruction, (a) Example from the data set: initial marker distribution (mo); simulated marker distribution after time t using ground-truth velocity (y") and network predictions (ye); predicted initial velocities (zo); ground truth velocity (x*).. (b) Learning curves, running average over 64 mini-batches.
they fail match its detailed structure. Moving from this local optimum the global optimum is very hard for the network as the distance in z space is large and the gradients become very noisy due the highly non-linear physics. similar behavior can also be seen when optimizing single inverse problems with gradient descent where it takes more than 20k iterations for GD to converge on single problems.
For SIP training, we run the simulation reverse, starting with my, to estimate the translation and vortex of in acale-invariant manner. When used for network training, observe vastly improved convergence behavior compared to first-order training. The error rapidly decreases during the first 200 iterations, which point the inferred solutions are more accurate than pure Adam training by factor of 2.3. The error then continues to improve at slower but still exponentially faster rate than first-order training. reaching a relative accuracy advantage of 5x after 20k iterations. To match the network trained with pure Adam for 20k iterations, the SIP variant only requires 55 iterations. This improvement is possible because the inverse-physics solver and associated SIP updates do not suffer from the grongly-varying gradient magnitudes and directions, which drown the first-order signal in noise. Instead, the SIP updates behave much more smoothly, both in magnitude and direction.
Comparing the inferred solutions from the net- work to iterative approach shows large dif- ference inference time (table [ To reach the solution quality the neural network prediction, Prt needs iterations on average, which takes more than 10,000 times as long, and gradient descent (GD) does not reach the same quality even after 20k iterations. This difference caused by Pxá having to run the full forward and backward simulation for each iteration. This cost is also required for each training itcration of the network but once converged, its inference is extremely fast, solving around 9000 problems per second batch mode. For both iterative solver and network, we used a batch size of and divide the total time by the batch size.
Table 1:
Time reach equal solution quality in the fluid experiment, measured as MAE in space. The inference time is given per example in batch mode, followed by the number of iterations in parentheses.
Method Training time
Inference time
NN
17.6 h k15.6
0.11 ms
P
n/a n/a
2.2 (7)
4h (20k)
3.6 Limitations and Discussion
While SIP training manages find vastly more accurate solutions for the examples above, there are some caveats to consider. First, approximately scale-invariant physics solver is required. While in low-dimensional x spaces Newton's 's method is a good candidate, high-dimensional spaces require another form of inversion. Some equations can locally be inverted analytically but for complex

---

problems, domain-speent specific knowledge may be required. However, this is a widely studied field and many specialized solvers have been developed 9
Second, SIP uses traditional first-order optimizers to determine A0. As discussed, these solvers behave poorly in ill-conditioned settings which can also affect SIP performance when the network outputs lie on different scales. Some recent works address this issue and have proposed network
optimization based on inversion [39 40 20)
Third, while SIP training generally leads to more accurate solutions, measured in x space, the same not always true for the loss L Σ, Lg- SIP training weighs all examples equally, independent of the a near a chosen solution. This can cause small errors in examples with large curvatures curvature 0x² to dominate L. In these cases, when the accuracy in space is not important, like in some control tasks, traditional training methods may perform better than SIP training.
4 Conclusions
We have introduced scale-invariant physics (SIP) training, a novel neural network training scheme for learning solutions to nonlinear inverse problems. SIP training leverages physics inversion to compute scale-invariant updates in the solution space. It provably converges assuming enough network updates AO are performed per solver evaluation and we have shown that it converges with a single A update for wide range of physics experiments. The scale-invariance allows it to find solutions exponentially faster than traditional learning methods for many physics problems while keeping the computational cost relatively low. While this work targets physical processes, SIP training could also be applied to other coupled nonlinear optimization problems, such as differentiable rendering or training invertible neural networks.
Scale-invariant optimizers, such as Newton's method, avoid many of the problems that plague deep learning at the moment. While their application high- dighedimessionll parameter spaces is currently limited, we hope that our method will help establish them as commonplace tools for training neural networks in the future.
References
[1] Craig E Aalseth, N Abgrall, Estanislao Aguayo, SI Alvis, M Amman, Isaac J Arnquist, FT Avi- gnone III, Henning O Back, Alexander Š Barabash, PS Barbeau, et al. Search for neutri- noless double-3 decay in ge 76 with the majorana demonstrator. Physical review letters,
122512)) 2018.
[2] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large-scale machine learning. In 12th [USENIX] symposium on operating systems design and implementation ((ODDI) 16), pages 265-283, 2016.
[3] M Agostini, M Allardt, E Andreotti, AM Bakalyarov, Balata, Barabanov, M Barnabé Heider, N Barros, L Baudis, C Bauer, et al. Pulse shape discrimination for gerda phase i data. The European Physical Journal C, 73(1))25883, 2013.
[4] William Ames. Numerical methods for partial differential equations. Academic press, 2014.
[5] Kendall E Atkinson. An introduction to numerical analysis. John wiley & sons, 2008
[6] Mordecai Avriel. Nonlinear programming: analysis and methods. Courier Corporation, 2003.
[7] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In International Conference on Learning Representations (ICLR), 2016.
[8] Ernst Berndt, Bronwyn Hall, Robert E Hall, and Jerry Hausman. Estimation and inference in nonlinear structural models. In Annals of Economic and Social Measurement, Volume number pages 653.665. NBER, 1974.
[9] A. Borzå, G. Ciaramella, and M. Sprengel. Formalation and Numerical Solution of Quantum Control Problems. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2017.
10

---

[10] Charles George Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1):76-90, 1970.
[11] Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Nafall Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Reviews of Modern Physics, 91(4):04500, 2019.
[12] Andrew R Conn, Nicholas IM Gould, and Ph L Toint. Convergence of quasi-newton matrices generated by the symmetric rank one update. Mathematical programming, 50(1-3):177-195, 1991.
[13] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314 1989.
[14] Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv preprint arXiv: 140. 2572, 2014.
[15] S Delaquis, MJ Jewell, I Ostrovskiy, M Weber, T Ziegler, J Dalmasson, LJ Kaufman, T Richards, Deep neural networks for energy and position reconstruction in
JB Albert, G Anton, exo-200. Journal of Instrumentation, 13(08):P08023, 2018.
[16] Jerome Droniou. Finite volume schemes for diffusion equations: introduction to and review of modern methods. Mathematical Models and Methods in Applied Sciences, 24708):1555-1619 2014.
[17] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02055 2018.
[18] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[19] W. Dyson, A. S. Eddington, and C. Davidson. A Determination the Deflection Light by the Sun's Gravitational Field, from Observations Made at the Total Eclipse of May 29, 1919, January 1920.
[20] Israel Elias, José de Jesús Rubio, David Ricardo Cruz, Genaro Ochoa, Juan Francisco Novoa, Dany Ivan Martinez, Samantha Muñiz, Ricardo Balcazar, Enrique Garcia, and Cesar Fe- lipe Juarez. Hessian with mini-batches for electrical demand prediction. Applied Sciences, 10(6):2036, 2020.
[21] Giovanni Galdi, An introduction to the mathematical theory of the Navier- Stokes equations: Steady-start state problems. Springer Science Business Media, 2011.
[22] Daniel George and EA Huerta. Deep learning for real-time gravitational wave detection and parameter estimation: Results with advanced ligo data. Physics Letters B, 778:64-70, 2018.
[23] Philip E Gill and Walter Murray. Algorithms for the solution of the nonlinear least-squares problem. SIAM Journal Numerical Analysis, 15(5):977-95 1978.
[24] Alexander Gluhovsky and Christopher Tong. The structure of energy conserving low-order models. Physics of Fluids, 11(2):334-443, 1999.
[25] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT press Cambridge, 2016.
[26] Matthew Grayson. The heat equation shrinks embedded plane curves to round points, Journal of Differential geometry, 26(2):285-314, 1987.
[27] FH Harlow. The marker-and-cell method. Fluid Dyn. Numerical Methods, 38, 1972.
[28] Francis Harlow and Eddie Welch. Numerical calculation of time-dependent viscous incompressible flow of fluid with free surface. The physics of fluids, 8(12):2182-2188) 1965.
11

---

[29] Magnus R Hestenes, Eduard Stiefel, et al. Methods of conjugate gradients for solving linear systems. Journal of research of the National Bureau of Standards, 49(6):409-436, 1952.
[30] Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning to control pdes with differentiable physics. In International Conference on Leurning Representations (ICLR), 2020.
[31] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pages 646-661, Springer, 2016.
[32] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In arXiv: 1502.03177 February 2015.
[33] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[34] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108. 08481. 2021.
[35] GV Kraniotis and SB Whitehouse. Compact calculation of the perihelion precession of mercury in general relativity, the cosmological constant and jacobi's inversion problem. Classical and Quantum Gravity, 20(22))4817, 2003.
[36] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503-525 1989.
[37] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[38] Rajesh Maingi, Arnold Lumsdaine, Jean Paul Allain, Luis Chacon, SA Gourlay, CM Greenfield, JW Hughes, Humphreys, V Izzo, H McLean, et al. Summary of the fesac transformative enabling capabilities panel report. Fusion Science and Technology, 75(3):167-177, 2019.
[39] James Martens al. Deep learning via dessian-free optimization. In ICML, volume 27, pages 735-742, 2010.
[40] James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approx- imate curvature. In International conference on machine learning, pages 2408-2417. PMLR, 2015.
[41] Jorge J Moré. The levenberg levenberg-marquarde algorithm: implementation and theory. In Numerical analysis, pages 105-116. Springer, 1978.
[42] Patrick Mullen, Keenan Crane, Dmitry Pavlov, Yiying Tong, and Mathieu Desbrun. Energy- preserving integrators for fluid animation. ACM Transactions Graphics (TOG), 28-3):1-8, 2009.
[43] Mykola Novik. auch-oppimizzer-ooleee of optimization algorithms for PyTorch., 1 2020.
[44] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems, pages 1-8, 2017.
[45] J Gregory Pauloski, Qi Huang, Lei Huang, Shivaram Venkataraman, Kyle Chard, Ian Foster, and Zhao Zhang. Kaisa: an adaptive second-order optimizer framework for deep neural networks. In Proceedings of the International Conference for High Performance Computing, Networking. Storage and Analysis, pages 1-14, 2021.
[46] Michael Powell. A hybrid method for nonlinear equations. In Numerical methods for nonlinear algebraic equations. Gordon and Breach, 1970.
[47] William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian Flannery. Numerical Recipes. Cambridge University Press. 3 edition, 2007.
12

---

[48] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-ussisted intervention, pages 234 241. Springer, Sprrinrrr 2015.
[49] Sirpa Saarinen, Randall Bramley, and George Cybenko. Illconditioning in neural network training problems. SIAM Journal on Scientific Computing. 14(3):693-714, 1993.
[50] Patrick Schnell, Philipp Holl, and Nils Thuerey. Half-inverse gradients for physical deep learning. arXiv preprint arXiv:2203. 10131, 2022.
[51] Nicol N Schraudolph, Jin Yu, and Simon Günter. A stochastic quasi-newton method for online convex optimization. In Artificial intelligence and statistics, pages 436 443, 2007.
[52] Andrew Selle, Ronald Fedkiw, ByungMoon Kim, Yingjie Liu, and Jarek Rossignac, An unconditionally stable maccormack method. Journal of Scientific Computing, 35(2-3):350-371 June 2008.
[53] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1229-1988 2014.
[54] Albert Tarantola. Inverse problem theory and methods for model parameter estimation. SIAM, 2005.
[55] Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael W Mahoney. Adahessian: An adaptive second order optimizer for machine learning, arXiv preprint arXiv:2006.00719, 2020.
[56] Nan Ye, Farbod Koosta-Khorasann and Tiangang Cui. Optimization methods for inverse problems. In 2017 MATRIX Annals, pages 121-140. Springer, 2019.
[57] Kemin Zhou, John Comstock Doyle, Keith Glover, et al. Robust and optimal control, volume 40. Prentice hall New Jersey, 1996.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not expect any negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We declare all assumptions the main text and provide an additional compact list in the appendix.
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the full source code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We list learning rates and our training procedure the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We show learning curves with different seeds in the appendix.
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Our hardware listed in the appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All used software libraries given in the appendix.
(b) Did you mention the license of the assets? [N/A] No proprictary licenses involved.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include our source code.
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zrmsPKmf5x3", "reviewer": "Reviewer_ukwX", "summary": "This work proposes scale-invariant learning for solving inverse problems. The state of the art machine learning techniques used to infer solutions to inverse problems rely on first order optimizers such as SGD or Adam. The highly non-linear behavior in physical processes result in suboptimal optimization direction. This work tackles this issue by proposing a 2 stage process which combines higher order optimization methods with Deep Learning. Specifically, their method uses the solutions from a scale invariant physics solver as targets for the neural network during training. Finally, they demonstrate that their method is much more effective than the existing methods on numerous problems such as - poisson's equation, Heat equation, Navier-Stokes equation.", "questions": "The authors have only compared their methods only against optimizers such as SGD, Adam, AdaHessian which seems a bit 1-dimensional. Currently the story is that SIP is better than the very traditional training pipeline and to me, the contributions seem limited to some extent. I believe the authors can paint a stronger picture if they compared their method against other popular approaches such as - neural operators [1] and Physics Informed Neural Networks [2]. I'm happy to increase my scores if the authors could compare their methods against these baselines.\n\n\n[1] Kovachki N, Li Z, Liu B, Azizzadenesheli K, Bhattacharya K, Stuart A, Anandkumar A. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481. 2021 Aug 19.\n\n[2] Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nature Reviews Physics. 2021 Jun;3(6):422-40. ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The 2-stage SIP pipeline Clearly outperforms the traditional training pipeline that uses gradient based optimizers to optimize the joint problem NN \\odot P. \n* This work has provided convergence proof for their algorithm.\n* Empirically demonstrate the effectiveness of their work on problems such as - poisson's equation, Heat equation, Navier-Stokes equation.\n\nWeakness:\nThe authors have only compared their methods only against optimizers such as SGD, Adam, AdaHessian which seems a bit 1-dimensional. Currently the story is that SIP is better than the very traditional training pipeline and to me, the contributions seem limited to some extent. I believe the authors can paint a stronger picture if they compared their method against other popular approaches such as - neural operators [1] and Physics Informed Neural Networks [2]. I'm happy to increase my scores if the authors could compare their methods against these baselines.\n\n\n[1] Kovachki N, Li Z, Liu B, Azizzadenesheli K, Bhattacharya K, Stuart A, Anandkumar A. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481. 2021 Aug 19.\n\n[2] Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nature Reviews Physics. 2021 Jun;3(6):422-40. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "tOw0eHgUcD", "reviewer": "Reviewer_3Np3", "summary": "Inverse problems that involve physical systems usually have strongly varying gradients. This may lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions. To handling this issue, the paper embeds a scale-invariant inverse problem solver into the gradient-descent-based learning pipeline. More specifically, the paper reformulates the objective function in the observed outcome space to that in the physical parameter space. It needs to compute the inverse of the physical process for each training step. Some theoretical and experimental results are provided.", "questions": "After reading the response and the comments from other reviewers, most of my concerns are addressed. So I increase my score to 6. My remaining concern is the approximate derivation in subsection 3.1. If there are no approximations, the performance is worse or better?\n==================================================\nThere are some comments and suggestions.\n1.\tHighly related methods are missing for comparisons. There are many optimization methods to avoid compute the inverse of Hessian matrix. For example, the Hessian-free method. James Martens. Deep learning via Hessian-free optimization. ICML, 2010. It is better to include these methods for comparisons.\n2.\tThe paper assumes that the physical process P has a unique inverse and is sufficiently well-behaved. It is better to give plentiful examples. \n3.\tAt lines 134-147, the paper approximates P^{-1} with its first update. It needs more elaborations on this. What about iteratively solve P^{-1} precisely? \n4.\tAt line 249, x has a square.\n", "limitations": "The paper did not have the limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper presents a new approach to obtain scale-invariant updates of physical parameters, i.e., when rescaling the physical parameters, the optimization will behave the same way. Namely, it uses the objective function in the physical parameter space instead of the observed outcome space. Some acceleration tricks and convergence analyses are also presented. It is a useful complement to scale-invariant optimizers, e.g., Newton’s method, for high-dimensional parameter spaces. \n\nWeakness:\n\nSee the comments for the \"Questions” below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iR87-UnC_-", "reviewer": "Reviewer_8hA3", "summary": "In this paper, the authors look at solving inverse problems with a neural network training scheme. They use a scale-invariant physics solver to compute the gradient update, and then train the neural network using this information. They show that this SIP training approach can converge faster to inverse problem solutions over optimizers such as Adam and SGD for various physical problems.", "questions": "- What is the advantage of the SIP solver vs. using the adjoint method as a way to back propagate through the weights?\n\n- Can you clarify the problem setting better for the problems (such as Poisson and heat equations)? That is, exactly what the parameters being learned correspond to (i.e., by solving the inverse problem).\n\n- How does the method compare to traditional numerical inverse solvers?\n\n- Could the authors clarify what you mean by the network weights are not being altered?", "limitations": "Limitations are discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Solving inverse problems is a long-standing problem in science and engineering, and the authors propose an interesting technique to integrate the physics solver with a neural network to address this challenge\n- The method seems to converge more quickly compared to other optimization approaches, and can reconstruct solutions more accurately\n- The paper is well-written for the most part\n\nWeaknesses:\n- The authors consider an infinite data setting, so it isn’t entirely clear what the generalization benefits of this method are\n- The authors compare to mainly first-order optimizers and one second-order optimizer. It would be interesting to see how well the Newton’s method-based second-order optimizers do here, as are often used in solving numerical inverse problems (L-BFGS, etc.). There are also some promising results using other second-order optimizers, as well as exploring the use of starting with a first-order optimizer and then switching to a second-order optimizer. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept