Stabilizing Backpropagation Through Time to Learn Complex Physics
Authors: Patrick Schnell
Nils Thuerey
Venue: ICLR 2024 poster (2024)
Keywords: backpropagation through time
unrolling
differentiable physics
differentiable simulators
optimization
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We show how modifying backpropagation and counteracting rotation in the resulting vector fields can improve learning in physical simulations.

Abstract: Of all the vector fields surrounding the minima of recurrent learning setups, the gradient field with its exploding and vanishing updates appears a poor choice for optimization, offering little beyond efficient computability. We seek to improve this suboptimal practice in the context of physics simulations, where backpropagating feedback through many unrolled time steps is considered crucial to acquiring temporally coherent behavior. The alternative vector field we propose follows from two principles: physics simulators, unlike neural networks, have a balanced gradient flow and certain modifications to the backpropagation pass leave the positions of the original minima unchanged. As any modification of backpropagation decouples forward and backward pass, the rotation-free character of the gradient field is lost. Therefore, we discuss the negative implications of using such a rotational vector field for optimization and how to counteract them. Our final procedure is easily implementable via a sequence of gradient stopping and component-wise comparison operations, which do not negatively affect scalability. Our experiments on three control problems show that especially as we increase the complexity of each task, the unbalanced updates from the gradient can no longer provide the precise control signals necessary while our method still solves the tasks. Our code can be found at https://github.com/tum-pbs/StableBPTT.

Published as a conference paper at ICLR 2024
STABILIZING BACKPROPAGATION THROUGH TIME TO LEARN COMPLEX PHYSICS
Patrick Schnell & Nils Thuerey School of Computation, Information and Technology Technical University of Munich
Bolzzmannstr. 3, 85748 Garching, Germany
Serrick.schhelle nils. thuerey @t de
ABSTRACT
Of all the vector fields surrounding the minima of recurrent learning setups, the gradient field with its exploding and vanishing updates appears poor choice for optimization, offering little beyond efficient computability. We seek to improve this suboptimal practice in the context of physies simulations, where backpropa- gating feedback through many unrolled time steps is considered crucial to acquir- ing temporally coherent behavior. The alternative vector field we propose follows from two principles: physics simulators, unlike neural networks, have balanced gradient flow, and certain modifications to the backpropagation pass leave the po- sitions of the original minima unchanged. As any modification of backpropa- gation decouples forward and backward pass, the rotation-free character of the gradient field is lost. Therefore, discuss the negative implications of using such a rotational vector field for optimization and how to counteract them. Our final procedure is easily implementable via sequence of gradient stopping and component-wisst comparison operations, which do not negatively affect scalabil- ity. Our experiments on three control problems show that especially as we increase the complexity of each task, the unbalanced updates from the gradient can no longer provide the precise control signals necessary while our method still solves the tasks. Our code can be found at
INTRODUCTION
Simulators are gaining popularity as an alternative way to train neural networks in an otherwise data-dominated deep learning field. These computational replicas of real-world phenomena can be directly built into the training loop and, if diffcrentiable, combined with gradient- based optimizers. This was demonstrated in numerous areas, from fluid dynamics to robotics (Amos & Kolter 2017; Toussaint et al. 2018 Schenck Fox 2018 Ingraham et al. 2018). The close interaction between neural network and simulator provides faster training than sample-inefficient trial-and-error search, does not suffer from the presence of multimodal solution modes, and integrates outlier cases into training they occur.
In modern training setups, these simulator-network constructs are typically unrolled over many time steps, enabling the network to overlook long time spans and acquire sense of long-term plausible behavior- that is, long-range planning instead of greedy solutions in control tasks (Belbute-Pene
Holl al. 2020), temporal coherence instead of abrupt jumps in surrogate models (Kim 2019 Stachenfeld et al., 2021), or long- term stability instead of diverging trajectories in error et al.
al. 2018
correction tasks (Um al. 2020) Kochkov al. 202T). As the time step typically short to ensure numerically accurate results, the number of these time steps has then to be large to cover a sufficient time span, leading to sequence of many recurrent network and simulator calls in each training step.
From a mathematical point of view, optimizing such long unrollments difficult problem: re- currently evaluating the same operator quickly creates an unbalanced optimization landscape. This similar to how repeatedly multiplying a variable with itself leads to a high-order polynomial f()) = xx with increasingly impractical gradient: the larger n, the more f and its derivatives vary in size. This is a simple form of what in the context of neural networks is often called the "ex- ploding and vanishing gradient problem", a situation in which gradient-bart based optimizers perform

---

Published as a conference paper at ICLR 2024
poorly (Goodfellow al. 2016). The gradient flow in such unrollments has been widely studied under the name "Backpropagation Through Time" (Werbos, 1988) and addressing its mathematical weaknesses will facilitate the effective learning of temporal relationships.
Complementary to the existing approaches that focus on model architectures or loss formulations, target the backpropagation step. As outlined above, the gradient field is only among many vector fields that can guide a neural network toward an optimal configuration. We employ gradient stopping, one of the simplest ways use that freedom, to propagate feedback only through the physics simulator but still over the full time trajectory. This has a positive effect on the magnitude of the update because physical systems, the rate of change over time is typically limited, for instance by conservation laws. As any gradient stop not only stops feedback propagation but in general also literally stops the resulting vector field from being a rotation-free gradient field, the flow lines of this new vector field do not have to be perpendicular to minima surfaces anymore. This causes new challenges for existing optimizers, for instance, momentum terms can clash with spiraling flow lines around minima and prevent convergence. Since the regular gradient field gives the correct direction near minima, our final algorithm performs only updates in those components that coincide in sign with the original gradient to counteract rotation. Both aspects above central contributions of our work: the specific form of employing gradient stopping and the compensation of rotations are, to the best of our knowledge, novel. We evaluate the improvements of this algorithm three control model, a cart pole scenario, and a quantum system. Interestingly, the proposed algorithm fares especially well once the complexity of these scenarios is increased.
tasks:
PROBLEM FORMULATION
We consider training setups in which a neural network N parametrized by 0 and a simulator S interact with each other over several time steps n. Such setups are typically of the form:
C Na () for i 0, "..,....
Ti+1 = SiZ; (i) for i = 0, 1, I2..
2 (#n y)²
A physical context for this setup would a control task, where the goal is to transform an initial state zo into target state y via controllable external force To induce that transition, neural network N estimates the control value e for each time step based on the current physical state ai-- The effect of this action is then computed via the simulator S, which outputs the next physical state J1+1- After repeating this process for a specified number times ni, a time trajectory is constructed with final state nn. whose distance L to the target state y is measured with quadratic loss function. Besides control tasks, training setup [ has also been applied to improve temporal coherence in error correction, reduced order modeling, and surrogate tasks (Belbute- Peres al. 2018 Bar-Sinai al.) 2019 Wang et al. 2020a). We also introduce notations for the time stepping operator T. mapping 33 1++1, and its n-times application F, mapping the initial state To to the final state T,,
T( 0) S(x,, N (2,,, 0)) Fs+1
F(zo. 0) = T(. (T No: 0), 0) )). Fr
(2)
n-times
Finding network parameters 0 that minimize the loss T. by using the gradient is inherently difficult: on the level of linear approximations, recurrently applying the same operator corresponds to a repeated multiplication by the same Jacobian matrix. As result, the output of the backpropagation pass, the gradient, is dominated by the self-enforcing and self-damping mechanics of eigenvectors belonging large and small eigenvalues. This means the gradient is no longer representative for the deviation between In and and therefore a questionable choice for optimization.
Illustration with toy example: We illustrate these difficulties of the gradient for simple case of that will serve as the guiding example throughout our paper. We choose:
Nrr, 0) = 0,27 + 0224
(3)
S(r,, (4) x4 +
While the choice of the simulator S. an identity map in the physical state z; plus control c, may appear overly simple at first, it allows us to avoid unrelated problems, such as discretizing differen- tial operations or processing different physical time scales, and instead focus the mathematical
2

---

Published as a conference paper at ICLR 2024
a) Loss Landscape
b) Gradient Field
c) Moditied Field
>10
104
10
108
10c
8
10-2
10-1
-2
10-1
-2
4
-2
0
Figure 1: Toy example: a) loss landscape, b) regular gradient field (flow lines in red), c) modified vector field (yellow). For b) and c) the background color shows the L2 norm of the vectors. To improve optimization we trade an unbalanced gradient field for a balanced but rotating vector field.
essence Backpropagation Through Time. In addition, choosing a polynomial controller provides non-trivial solutions yet keeps the mathematical expressions simple: composition of polynomials is intuitive while the composition of other analytical functions, c.g. tanh, quickly gives convoluted expressions. Most importantly, we restrict ourselves two optimization variables f and 02. This enables to visualize the optimization landscape in its entirety and convey sense of what the dy- namics of an optimization algorithm look like. In Figure we depicted this toy example for initial state 20 0.3, target y 2.0 and
time steps.
Figure D and show landscape and gradient field. The unbalanced landscape consists of an exploding outer (white) with large gradients, flat area mainly in the middle (light blue) and minima (dark blue). The imbalance worsens with increasing n, visualizations can be found |n as the exponential effects at play increase the order of the saddle point, hence the vanishing gradient area extends and the minima become sharper. Gradient- gradient size is measure of optimality, i.e. large gradients far away from minima, and small gradients nearby. Here, such a relationship is not only disrupted but inverted: with increasing n, ever more sharper minima with larger gradients around them, and farther away a growing of near-constant loss with vanishing gradients. In the worst case, an optimizer would fight tediously through the area of vanishing gradients just shoot off once a minimum comes within reach. Under these cireumstances, we cannot expect gradient optimizers to do well. In the appendix C we further study these landscapes for more complex systems such as linear quadratic regulator the trained networks in our later experiments.
optimization works best when the
3
METHOD
To improve this situation, our method creates more suitable vector field for optimization:
3.1 MODIFYING BACKPROPAGATION
The backpropagation pass is where the gradient imbalance originates; modifying this step is there- fore the natural starting point. our equations, use d for the total derivative, dg for the gradient of the loss, and dg for the corresponding result of the modified backpropagation. To begin with, restrict ourselves to modifications that only change FgF:
L (F 0) y)²
dgL = (F(xo, 0) - y)) d
Lo = (F(#o. 0) v) do
This restriction is motivated by considering critical points of the non-modee non-modified, regular gradient, i.e. points with dg = O. We distinguish these points depending on whether not F(zo, 0) y. If yes, such a critical point is a global minimum and the modified update Lo will also vanish, ie. criticality of global minima is conserved. If not, such critical point is not global minimum, for instance, a non- coongllbal local minimum or a saddle point. For these, Fg is 0 while do needs not to be and neither needs do T.. Therefore, these points removed through such modification.
3

---

Published as a conference paper at ICLR 2024
a) Divergence
b)+ Rotation
c) Combination
d) Loss curve
Divergence
Robation
100
40
Coméimation
2-
10
0
10
20
10-
4
Adam Trajectory
-2
0
0:
Iteration
Figure 2:
Minimization of a loss function with Adam using a) the gradient field b) a rotating vector field c) our combined vector field constructed from the vector fields in a) and b). Background color indicates vector length. d) shows the loss curves. The rotational contribution in b) prevents Adam from converging while our combined vector field in e) allows Adam to approach the minimum.
From the many vector fields that could be constructed in this way, have to choose one that addresses the issues of vastly varying gradients across the optimization landscape. We decide to block any feedback flowing over the network N to its input at any time step. To implement this, we use gradient stop operations that set N, O, while all other partial derivatives are the same as in the unmodified case. We refer to plg computed with these modifications during backpropagation as modified update in the following. Such a partial stop of the feedback backpropagation does not violate original goal to carry feedback back over the full temporal trajectory since the physics path Sz, e) remains untouched. Backpropagating only along this path improves the situation of the gradient sizes as most physies simulators come with well-behaved gradient flow; the key lies in separately considering the partial derivatives for and While it is true that physical optimization task find a control can arbitrarily ill-conditioned, we do not repeatedly propagate through Sz c) but only rerun through (zz, c). A physical state and its successor hold close relationship to each other reflecting the physical characteristics of the system. Norms computed on them can correspond to various quantities that are either conserved, such as probability sum in quantum systems, or at least controlled, such as energy in a mechanical system when a state is brought from energy level to another. Such restrictions themselves also in limitations, upper and lower, on the size its gradient (,,,
the changes of a state reflect
In our toy example, the simulator S is identity map in the physical state æ. therefore the gradient flow over the physics path is trivially optimal. We construct our modified vector field by neglecting any feedback from the polynomial controller N the state æ and show this alternative vector field in [.. We observe improved optimization landscape: the saddle point and its extended area of vanishing gradients vanish and the new gradient field also visually resembles the loss landscape more closely, indicating improved conditions for optimization with gradient-bard based optimizers.
3.2 EMERGENCE OF ROTATIONAL. VECTOR FIELDS
The observed improvements come at cost: any modifications of the backpropagation pass decouple from the corresponding forward pass and can destroy the rotation-free character of the gradient field, an unfamiliar situation in optimization. In general, rotation of vector field can be determined from the antisymmetrical part of its Jacobian, requiring us to further differentiate equations 4
L2 (F(zo, 0) y) deF + (dg F)2 doda T. (F(so, 0) u) dado F dg F do
(5)
For unmodified backpropagation, d3 L is a Hessian matrix, therefore symmetric and free of rotation. If modifications are applied, such an argument cannot be made. Even near global minima, where the second summands dominate, the inequality of dg and dg makes dødg I. non-symmetric, creating rotation in our modified vector field I.
This can be directly seen in our toy example. The gradient of a function points in the direction of steepest ascent, making its flow lines orthogonal level sets and curves of minima, clearly fulfilled for the regular gradient dg Figure However, it further means that curves minima close to each other surrounded by abruptly changing gradients, another aspect worsening with more time
4

---

Published as a conference paper at ICLR 2024
steps and visible in figure In contrast in Figure Ic. the rotation in our modified field de allows flow lines to intersect minima curves von-orthog.n.lly. This additional freedom balances the vector field, removing abrupt variations near close minima but also around the original saddle point.
So far we have explained why Le is rotating, but it is not clear yet why this could be a problem. In Figure [P. rotation adds tangential movement along minima curves that should at worst lead to a slower convergence towards a minimum. We consider now the case when rotation adds a circular movement around minima as the second way of how a rotational vector field can be organized around a minima manifold. For a simple illustration, we consider a quadratic loss L around a minimum at (0, 0), a rotation-free vector field D, a purely rotating vector field R. and try to minimize the loss with Adam (Kingma& & Ba 2014) and different vector fields:
L(01, 02) 03 + 02
D 02) 3,L = 1-011-00
(6)
R(01,02) = (-02,0) 11)
Figure 2 shows the behavior of Adam moving in D, the simple case of a gradient field where Adam succeeds easily. Next, P shows Adam moving the vector field D + 2R, a vector field with rotational contribution, corresponding to what we encounter when modifying backpropagation. The circular contribution makes the flow lines spiral around the minimum and instead of following those, Adam constantly overshoots, ending up in a circle-like trajectory around the center without converging. This is caused by the momentum terms in the update procedure of Adam but also partly by the discrete nature of the algorithm. For completeness, Figure 2 contains the corresponding loss curves. Together with the other, tangential contribution, this completes our picture of the positive and negative effects that rotation can cause in an optimization landscape.
3.3 COMBINATION TO COUNTERACT ROTATION
To address the problem Adam had in the rotating vector field, it is helpful to understand why opti- mization in a rotational field is to some degree uncharted territory. Discrete time-stepping methods used in numerical simulation just like optimizers perform iterative updates to follow the flow lines of a vector field. After all, gradient descent steps are nothing different than Euler steps in the negative gradient field. In numerical simulation, there exists abundance of methods designed to closely follow the flow of a vector field, no matter if rotational or not. However, these are not applicable here since in optimization, it is important to accelerate along flow lines in flat valley environments, or even leave the flow lines to avoid suboptimal local minima. Since these are contradicting goals, possible solution cannot be obtained by changing the way the modified update do L, is processed. An example is momentum, which is widely accepted to give the explorative behavior required in optimization but is exactly what prevented convergence the minima in Figure 22b.
Therefore, we instead focus on jointly processing dg and de L. Although we criticized many prop- erties of the regular gradient L in Figure Ta. its direction near global minima one thing that is undoubtedly good. This motivates that for our final update vector we introduce a component-wise condition that sets the j-th component to zero if do, and da, I. have opposite signs:
do T if (ign I.) = sign (do T.)
Uj 0 otherwise
We call this quantity u the combined update. Geometrically, u is at most orthogonal to L and to Lg In optimization context, it has several desirable properties: it has the same well-balanced behavior as dg L but improved convergence near rotational minima and compatibility with momen- tum, can be seen in Figure 2
3.4 COMPUTATIONAL COMPLEXITY
Computing the modified update do L requires fewer operations as we no longer have to backprop- agate through any path including a network output to network input connection. Computing the combined update u requires two backpropagation passes and a component-wist sign comparison that is negligible mentation of the two backpropagation passes leads to doubled runtime and memory requirements. However, since they are independent, a parallel implementation is possible to avoid the runtime increase. On top of that, the forward pass and its recorded intermediate results required for back- propagation are the same. Hence, most of the memory increase can be avoided as well, In practice, the additional computational requirements are manageable, as we show in appendix B.T
comparison to backpropation. Therefore, a straight-forward sequential imple-
5

---

Published as a conference paper at ICLR 2024
a) Visualization
b) Regulbriztioon:
Regularization:
Bvader 1
Driver1 Driver2
Evader
Evader
Target
2-
2-
Evader
0.7
0.7-
0.5
0.5
200
4000
600
8000
1000
200
400
600
800
1000
Regularization on: 0.01
Rc larizatzzon 0.001
5-
LosS
2
0.7
0.5
0.5
-2
200
400
600
800
1000
0
200
400
600
800
1000
Epochs
Epochs
Figure 3: Guidance-by-rrpuusiio model: a) visualization, smaller circles indicate the configuration at earlier points in time, b) e) learning curves for different regularization coefficients. (S) is not able to learn. With less regularization, (C) and (M) increasingly outperform (R). Same-color curves differ by clipping mode. Curves were smoothed over 20 epochs for clarity.
EXPERIMENTS
In our experiments, we compare our proposed combined update vector (C) to the established regular gradient (R). Additionally, we include the modified update (M) that only backpropagates through the physics simulator as an ablation study. As a further baseline, we consider a stopped update vector (S) resulting from backpropagating only through the last unrollment step. This used in some works (Prantl et al.] 2022 Brandstetter et al. 2022) to trade temporal look-ahead for stability, but can be expected to perform less well due to the truncated temporal feedback. These four techniques are comparable since the forward pass, and hence the loss formulation is the same for all. Based on our derivation, we investigate the following hypotheses our experiments:
(i) (M) and (C) will outperform (R). (Motivated by their more balanced updates.)
(ii) (C) will outperform (M). (Thanks its ability counteract rotation.)
(iii) These differences become more apparent the task complexity increases. (In easy tasks the negative effects of unbalanced updates can be more easily corrected.)
To investigate these hypotheses, we study three control tasks: guidance-by-repuine pole swing-up, and quantum control problem. We also present cart pole variation with walls as a simple contact task the appendix D The diffcrential equations describing these systems and the simulation details can be found in appendix B Across the tasks, vary model architecture and choose either a final loss, where the physical state is compared to the target state only at the last time step, or accumulated loss, where the physical state is compared the target state after every time step. We choose Adam as optimizer to process the four different backpropagation vectors with learning rate of 0.001 and batch size of 8. This of hyperparameters performed well across our tests, but we include an extensive search over 792 training runs with variations of the optimizer, learning rate and batch size in the appendix.
model, a cart
All following learning curves show end-of-epoch losses test data against the number of epochs; we provide runtime data appendix [.I] The plots contain curves of four different colors for the four different backpropagation methods, the regular update (R) in red, modified (M) in yellow, combined (C) in green, and stopped (S) in black, and for each method, three same-color curves corresponding to three clipping modes (value, norm, no clipping). Including different clipping techniques serves a double purpose: showing problems from exploding gradients are not resolved by simple cut-offss and highlighting how consistently the methods behave over different runs.
4.1 CUIDANCE-BY- BY-REPLSSIO
This coupled system of differential equations from the class of collective behavior models (Ko & Zuazua 2020) has two types of agents, controllable drivers and non-controllable evaders. Several

---

Published as a conference paper at ICLR 2024
al Visualization
h) Number poles:
c) Number of poles: 2
1.0
105
10 -
10-
10
200
400
600
800
1000
200
400
600
800
1000
Number poles:
e) Number of poles: 4
10
10
10-1
10-1
200
400
600
R00
1000
0
200
400
600
Abco
1000
Epochs
Epachs
Figure 4: Cart pole: a) visualization of the two-poles task, upper plots and more transparent style indicate earlier points in time, for easier visualization the two poles plotted attached to two carts instead of one, b) e) learning curves for 1-4 poles. With more poles, (C) outperforms (R), (M) and (S). Same-color curves differ by clipping mode. Curves were smoothed over 20 epochs for elarity.
types of interactions are present such flocking term for evaders, a spreading term for drivers, and a repelling term causing evaders to run away from nearby drivers. As the last term grows quickly when drivers come near evaders, this resembles the interactions found in contact systems In con- junction, they resemble the dynamics of group of shepherds, see Figure 3a for an illustration. The learning goal to find control strategy for two drivers that will move four evaders position. We increase complexity and allow for more sophisticated behavior by decreasing a regu- larization coefficient, which allows drivers to go faster and change direction more frequently. We unroll over 60 time steps and use accumulated loss. The controller is implemented as fully connected network with 13004 parameters. We train with 256 initial states, created by placing a group of four evaders randomly at minimum distance around the target state and the two drivers farther away in the outer parts of the system.
specified
Figure D-e shows the results of this control task for different regularization values. In all cases, (S) does not learn a meaningful control strategy with a constant loss of around With the strongest regularization, the other three techniques are behaving similarly with loss ranging from 2 to 3. Only (R) shows slightly more unstable behavior with slightly higher loss values. As we decrease regularization, (R) falls back, and (M) shows a similar performance as (C). This supports hypothesis (i) and (iii) but not (ii).
4.2 CART POLE SWING-UP UP
The widely-used cart pole system consists of pendulum attached to a controllable cart. The goal is to move the cart in such a way that the pendulum exits its lower position and swings up to reach its highest possible point. In our variant, we have several poles to increase the difficulty of the task, see Figure 4h for an illustration. When the poles' starting positions differ, the network is required to apply stronger and faster varying control to swing up all of the poles. In setup, we unroll the cart pole simulator over 100 time steps and apply a loss on the final physics state. We use a fully connected network architecture with around 1000 trainable parameters. We train with 256 different initial configurations, where the poles are randomly swung out by an angle up to 30".
Figure A shows the result of these training runs, where the number of poles is increased from to 4 to vary complexity. In the easiest 1-pole task, all methods are able to solve the task with final loss values of around 0.01 and lower. This picture changes in favor of our combined update (C) for the harder tasks: while there are some outlier cases in the 2-poles case, (C) fares consistently better than the (R), (M), and (S) with 3 poles and more, and is the only method able to solve the task with loss below 0.01. This is in line with hypothesis (iii), and (ii) for to 4 poles; (i) is partially fulfilled, (R) is worse then (C) but better than (N) is these training runs.
7

---

Published as a conference paper at ICLR 2024
a) Visualization
Taraget
Target stake:
1.0
0014
60
60
0.10
20
30-
0.6
20
0.08
d) Target state:
c) Update size
0.06
1087
90:
-0.44
12 norn
1047
0.2
40
6.02
20
0.0
0.5
1.0
1.5
2.0
200
400
600
800
1000
0 200
400
600
800
1000
Epochs
Epochs
Figure 5: Quantum control: a) visualization of a state transition over time and space x, background color indicates the probability density b) learning curves for different target states, e) update size. For higher target states, (C) outperforms (R). Same-color curves differ by clipping mode. Curves were smoothed over 20 epochs for clarity.
4.3 QUANTUM CONTROI
In our last experiment, we target the quantum dipole task (Von Neumann 2018). control problem formulated on the Schroedinger equation. The goal to find a control signal to induce transitions from lower to higher energy states, see 5h for visualization of such transition. The higher the energy difference, the more difficult the task. Concretely, we build the data set for the initial state from superpositions of the ground and first excited state and consider three tasks with the target state being either the second, third or fourth excited state. We consider the case of accumulated loss, unroll for 128 time steps and use a convolutional neural network with 11701 parameters.
Figure 5p-d shows the learning curves. Reaching the second state is the simplest task, for which the different methods perform the most similarly. Nonetheless, is already apparent that (M) and (C) outperform (R) and (S). For the third state, (S) still achieves some improvement but (R) is no longer able to solve the task. Best performing are one run of (M) and two runs of (C) minimizing the loss to 30 and below, For the forth state, the hardest case, the picture becomes even clearer: while (R) and (S) do not make any progress, (M) is able to achieve a decent result, and (C) performs even better with a run achieving a loss of 20. This again is in line with (i) and (iii), and weakly supports (ii) with individual instances of good performance for (M), but (C) performing better in general.
This example encapsulates several key arguments our derivation: in mathematical formulations of quantum systems, the squared physical state is interpreted as probability density, and its sum has to add up to one. Neither natural time evolution nor external forces can violate this property, making the L2 norm of the state a perfectly conserved quantity. This forces the eigenvalues of the simulator to be one and therefore feedback can be backpropagated without creating an imbalance. To visualize this, we tracked the L2 norms of the backpropagation results for the last, most difficult task in figure 5e. The graph shows that (R) varies by more than 35 orders of magnitude while the magnitude of the (M) and (C) preserved over the entire training. This illustrates our reasoning and gives an intuition for the improved convergence of networks trained with the modified and combined updates.
4.4 HYPERPARAMETER STUDY
We conclude with a short summary of our hyperparameter study (details can found in the ap- pendix). For the cart pole task, we observed that 24 (M)-runs versus only 7 (R)-runs minimized the loss successfully to 0.01, supporting hypothesis (i). The calraancce----pplliooo- system behaves sim- ilarly. Noteworthy for quantum control, not single training run with (R) solved the task. Regarding (M) versus (C), both worked equally well on two tasks while on the cart pole, (C) performs better, experiments provide strong evidence in
adding up support of hypothesis (ii). In summary, favor of hypothesis (i) and (ii), and evidence for hypothesis (ii). This constitutes, to the best of our knowledge, fair and comprehensive analysis of our method.
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Machine learning techniques have a rich history of integrating physical models (Crutchfield & Mc-
Namara 1987 Kevrekidis et al.) 2003: Brumton al.] 2016). For instance, researchers have em-
ployed machine learning to discover formulations of partial differential equations (Long et al. 2017
Raissi & Karniadakis 2018, Sirignano & Spiliopoulos 2018), explore connections to dynamical
systems [Weinan 2017), and leverage conservation laws (Greydanus et al., 2019 Cranmer et al.) 2020). Previous investigations have also uncovered discontinuities in finit-diffferece solutions through deep learning (Ray & Hesthaven, 2018) and have concentrated on enhancing the iterative
performance of linear solvers (Hsieh et al. 2019). Another interesting avenue for the application of
deep learning algorithms lies in Koopman operators (Morton et al.) 2018 Li et al.] (211)) Differ-
entiable simulations (Thuerey et al.] 2021) have been combined with machine learning to describe a wide range of physical phenomena including molecular dynamics (Wang et al.] (020b), weather (Seo et al. 2020), higid-bodies (Battaglia et al., 2013 Watters et al., 2017 Bapst et al. 22019), fluid flows (List et al. 2022), or robotics (Toussaint et al. 2018)) Gradient-based optimization can be used in conjunction to solve inverse problems (Liang et al. 2219)] Schenck & Fox 2018). By now, sev-
eral frameworks exist for differentiable programming (Schoenholz & Cubuk 2019 Hu et al. 2020,
Holl et 2020). Unrolling can be regarded as an advanced technique in the long history of fusing machine learning with scientific computing improve learning of temporal relationships.
Backpropagation Through Time (BPTT) leads to the exploding and vanishing gradient problem (Bengio et al. 1994) and few techniques are known to address this problem in unroliments including a simulator. Specialized architectures, such as long short-term memory (Hochreiter & Schmidhuber, 1997.. avoid exploding gradients but are targeted to work on priori known and therefore normaliz- able data sets, whereas in control tasks the scales of the network outputs are beforehand unknown. One applicable strategy with simulators is initialize networks to output zero and start exploring the physical system from its natural, control-free time evolution outside an exploding gradient re- gion (Degrave et al., 2019). In terms of loss formulation, reducing the unrollment steps limits the extent of the gradient imbalance at the cost of encouraging greedier actions on a shorter time horizon 1998). Truncating BPTT has long tradition to stabilize long recurrent rollouts [Sutskever) 2013). Recently, it has been proposed for controlling and predicting different physical 2021 Brandstetter et al. 2022 Prantl et al. 2022). These approaches for trunca- tion primarily reduce unrollment steps, alleviating the underlying issues without directly addressing them. Apart from backpropagation, suggested improvements include clipping (Sutskever) 2013) or the use of higher-order optimization (Martens & Sutskever (01T)) which reverts any observed gradi- ent imbalance in a computationally expensive process, limiting practical applicability. We presented a scalable method with a similar effect by analyzing the structure of the computation graph. In par- ticular in Reinforcement Learning, many tasks include contact (Freeman et al. (02T). destabilizing gradient methods (Metz et al. [002]]Sun [Suh et al.
Grzzszzzzk et al.
systems (Xu et al.
2022
Antonova et al. and making the use of
specialized techniques necessary (Parmas et al.
2018 Qiao et 2021).
6 CONCLUSION
We focused on improving the training of recurrent iimulator-netorkk setups, an important topic that facilitates the learning of more complex temporal relationships. For this, we targeted the backprop- agation pass and constructed an alternative vector field for optimization without the widely known weaknesses of regular gradients. The modifications in the form of gradient stopping and compar- ison operations were motivated by the well-behaving gradient flows of physics simulators and the emerging rotation in this new vector field, respectively. We demonstrated our method successfully on three control problems with a diverse physical nature.
A current limitation of our work is that we have not yet evaluated contact-rich scenarios, a potentially very interesting avenue for our method. Their discontinuous dynamics have the potential to conflict with our assumption that physical simulators have a well-behaved gradient flow. An interesting direction for future work will also be to re-evaluate various limitations associated with temporal unrollments: an improved optimization method could reveal that the issue was not the formulation of the problem, but rather the way it was solved. Since the actual modifications we proposed are simple, our work could provide the starting point for research into more complex modifications of the backpropagation pass. We believe this to be promising area of research because, despite our investigations of two of them above, all the other vector fields surrounding the minima of recurrent learning setups are still unexplored.
9

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our work we will publish the source code for our experiments upon acceptance. Details on our experiments are also provided in the appendix.
ACKOWLEDGEMENTS
This work supported by the ERC Consolidator Grant 2019-863850 SpaTe, and by the DFG SFB-Transrege Transsregggg 109 DGD. We would also like to express our gratitude to the reviewers and the area chair for their helpful feedback.
REFERENCES
Brandon Amos and Zico Kolter. OptNet: Differentiable optimization layer in neural networks. In International Conference on Machine Learning, 2017.
Rika Antonova, Jingyun Yang, Krishna Murthy Jatavallabhula, and Jeannette Bohg. Rethinking optimization with differentiable simulation from a global perspective. In Conference on Robot Learning. pp. 276-286. PMLR, 2023.
Victor Bapst, Alvaro Sanchez-Gonzaled Carl Doersch, Kimberly Stachenfeld, Pushmeet Kohli, Pe- Battaglia, and Jessica Hamrick. Structured agents for physical construction. In International Conference on Machine Learning, pp. 464 474, 2019.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven dis- cretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):5344-1554 2019.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45), 2013.
Filipe Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. In Advances in neural information processing systems, 2018
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In International Conference on Learning Representations, 2022.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification nonlinear dynamical systems. Proceedings of the National Academy of Sciences, (13(15)333323393) 2016.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv:2003.046300, 2020.
James P Crutchfield and Bruce S McNamara. Equations of motion from data series. Complex systems, 14417-452):121, 1987.
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis wyffels. A differentiable physics gine for deep learning in robotics. Frontiers in Neurorobotics, 13, 2019. ISSN 1662-5218. doi: 10.33899/mbot.2119.0000 URL https: //wp:://www... http://.wooo.iiooc www.tiersiinnnnnmo wwtp://.ggggarrrrrreeeeeeee
10. (3389/fmbbc 00006
Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of pol- icy gradient methods for the linear quadratic regulator. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1467-1476. PMLR, 10-15 Jul 2018. URL https: http:s//pprcceedigg... http::/g........eeeeeee////aaaea
Razvan V. Florian. Correct equations for the dynamics of the cart-pole system. 2005. URL https:
7/api.. semant i cachol cr..oorrorooorpps111111444444110
10

---

Published as a conference paper at ICLR 2024
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint 12Xx::2106.13281, 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pp. 15353-15363, 2019.
Radek Grzeszezuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural net- work emulation and control of physics-based models. In Proceedings of the 25th annual confer- ence on Computer graphics and interactive techniques. pp. 9-20, 1998.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-17800 1997. doi: (l.1162/neco.1999 1997.9.8.1735
Philipp Holl, Vladlen Koltun, and Nils Thuerey. Learning control pdes with differentiable physics. International Conference on Learning Representations (ICLR), 2020.
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning neural pde solvers with convergence guarantees. arXiv:1906.01200. 2019.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. International Conference Learning Representations (ICLR), 2020.
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with a differentiable simulator. In International conference on learning representations, 2018.
Ioannis G Kevrekidis, C William Gear, James Hyman, Panagiotis Kevrekidid, Olof Runborg, Constantinos Theodoropoulos, et al. Equation-free, coarse-grained multiscale computation: En- abling mocroscopic simulators to perform system-level analysis. Communications in Mathemati- cal Sciences, 1(4):715-762, 2003.
Byungsoo Kim. Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In Computer Graphics Forum, volume 38(2), pp. 59-70. Wiley Online Library, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 11X1v01412.6980 [cs], December 2014.
Dongnam Ko and Enrique Zuazua. Asymptotic behavior and control of a "guidance by repulsion" model. Mathematical Models and Methods in Applied Sciences, 30(04):7655 804, 2020. doi: 10. 010790021822522440000
URL http::///ooo.o111../ wtc/ /tt.11///22222220000000000
D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, and S. Hoyer. Machine learning- accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118 (21))e2101784118; 2021.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koop- operators for model-based control, 19X.v:/900.0800 2019.
Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. In Advances in Neural Information Processing Systems, pp. 771-780, 2019.
Bjöm List, Li-Wei Chen, and Nils Thuerey. Learned turbulence modelling with differentiable fluid
solvers: physics- based loss functions and optimisation horizons. Journal of Fluid Mechanics, 949:A25, 2022. doi: 10.1017/ffm.2022.7338
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-Net: Learning PDEs from data. 1rXiv:1710.0966 2017.
James Martens and Ilya Sutskever. Learning recurrent neural networks with hessian- free optimiza- tion. In Proceedings of the 28th international conference on machine learning (ICML-1I), pp- 1033-1040, 2011.
11

---

Published as a conference paper at ICLR 2024
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients not all you need. arXiv preprint arXiv:2111.05803, 2021.
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In Advances in Neural Information Processing Systems, 2018.
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, PP- 4065-4074, PMLR, 2018.
Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guaranteed conserva- tion of momentum for learning particle-based fluid dynamics. Advances in Neural Information Processing Systems, 35, 2022.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning, pp. 8661-8671. PMLR, 2021.
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125-14]] 2018.
Deep Ray and Jan Hesthaven. An artificial neural network troubled-cell indicator. Journal of Computational Physics, 367:166-191, Aug 2018. ISSN 0021-9991. doi: 10. 10.166//..cp.20188 029. URL http: dx. dei org/10. wttp:/o..//.../.................
(.p.1///........................ 2018..001...mm 029
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. In Conference Robot Learning, pp. 317-335, 2018.
Samuel Schoenholz and Ekin Cubuk. Jax, md: End-to-end differentiable, hardware accelerated, molecular dynamics in pure python. arXiv:1912.04232. 2019.
Sungyong Seo, Chuizheng Meng, and Yan Liu. Physics-aware difference graph networks for sparsely-observed dynamics. International Conference on Learning Representations (ICLR), 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339- 1364, 2018.
K. Stachenfeld, D. B. Fielding, Kochkov, M. Cranmer, Pfaff, Godwin, C. Cui, S. Ho, P. Battaglia, and A. Sanchez- Learned coarse models for efficient turbulence simu- lation. 12.1222 2021.
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pp. 20668- 20696. PMLR, 2022.
Nya Sutskever. Training recurrent neural networks. University of Toronto Toronto, ON, Canada, 2013.
Nils Thuerey, Philipp Holl, Maximilian Mueller, Patrick Schnell, Felix Trost, and Ki- won Um. Physics-based Deep Learning. WWW, 2021. URL https:// org
physi www.sssadeaeeeeeeae.......
Mare Toussaint, Kelsey Allen, Kevin Smith, and Joshua B Tenenbaum. Differentiable physies and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
K. Um, R. Brand, Y. Fei, P. Holl, and N. Thuerey. Solver-in-the-loop: Learning from differentiable physics to interact with iterative pde-solvers. In Advances in Neural Information Processing Systems, pp. 6111-6122. Curran Associates, Inc., 2020.
John Von Neumann. Mathematical foundations 2018.
foundaaatiiio of quantum mechanics. Princeton university press,
12

---

Published as a conference paper at ICLR 2024
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1457-1466, 2020a.
Wujie Wang, Simon Axelrod, and Rafael Hommez-Bombarelle Differentiable molecular simulations for control and learning. arXiv:2003. 00000,,,0,,0088,, 2020b.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 2017.
E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1): 2017.
Paul Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural networks, 1(4):339-356, 1988.
Greg von Winckel, Alfio Borzi, and Stefan Volkwein. globalized newton method for the accurate solution of dipole quantum control problem. SIAM Journal Scientific Compuling, (6)) 4176-4203, 2009. doi: 10.1137/997749611.
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. In Interna- tional Conference on Learning Representations, 2021.
13

---

Published as a conference paper at ICLR 2024
Appendix
A TOY EXAMPLE: VARYING THE NUMBER OF TIME STEPS
In Figure [ we illustrate how the landscape and gradients in the toy example change depending on the number of time steps n.. We show n = 2, 4, 6, 8 and 10 steps. The optimization difficulties with increasing n are sharper minima (left column) that are harder approach, and more minima curves closer to each other (middle column) with highly varying gradients in between. A saddle point increasing in order with an extending area vanishing gradients is visible dark area the middle column, and exploding gradients in the outer areas (also middle column). As one can see, these quantities with highly varying magnitudes are not only problem for optimization but also for visualization; the flow lines of the gradient can barely be plotted (middle column, bottom row). In contrast, our modified update has more balanced update sizes (right column).
Lozz Landscape
Graclennt Field
Modified field
-- teteeme steps
10
101
13
10
10
0p time steps
101
10a
10a
10a1
1012
- time steps
10F
10a2
10t1
10
>30
2307
0, trme stegs
19
101
10F
10a4
10
5104
2110
10 time stops
10-3
10-1
19r1
10-2
9
Figure 6: Further visualization of the toy example for 2, 4, 6, 8 and 10 time steps from top to bottom. Loss landscape on the left, regular gradient field in the middle, modified vector field the right. With time steps increasing, optimization becomes for difficult due to sharper minima and higher- order saddle point.
14

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL DETAILS
Here we provide details on our three control tasks: suidance-by-repuliie cart pole swing-up and quantum control. Moreover, present run time data and hyperparameter study for the four meth- ods considered in the experiments, the regular gradient (R), the modified update (M), the combined update (C) and the stopped update (S). Below, these methods are visualized with the same colors as before: red for (R), dark green for (S), and the modified version (M) in yellow, while the combined update (C) is colored green.
B. RUNTIME COMPARISON
To illustrate the computational effort of each method. Table lists the epoch durations for the four considered backpropagation methods. As expected, (S) is the fastest, For guidance-by-repulsii
and
cart pole swing-up. (M) is just as fast as (R), and (C) needs almost double the time. In contrast for quantum control, (M) is even faster than (R) and (C) is only slightly slower than (R). Especially the
quantum control case demonstrates that (M) can be computed faster and (C) almost as fast as (R),
which is in line with our theoretical argumentation in 3.4
Table 1: Epoch duration in seconds for our three control tasks
Method
(R)
(M)
(C)
(S)
Seedance-by-Repuisee Cart Pole Swing-Up
3,40
3.42
6.50
1.54
1.57
1.58
3.03
1.18
Quantum Control
11.97
8.42
12.34
7.10
B.2 GUIDANCE-DY-REPOLSS
Differential equation and numerical simulation This coupled system (Ko Zuazua/ 2020) con- sists of drivers (d) and evaders (e) and follows Newtonian dynamic, i denoting the i-th driver, the j-th driver, x their positions in a two-dimensional plane, dots time derivatives, and F the sum of all forces acting upon them:
(t Fud (1 175
(8)
The total force on drivers a sum of friction force, a spreading force, and control force. Fach driver can be steered with two controllers c(2) and (2).
Fid F. pfric-d + pspread + Fcon
pfric-d
-
xx x
Espread
5 Nofi
|| 12
(9)
Feon = c. (1 B(x²)) + + (2 )((x))
B(xa) = = xx
P(za) = ((x B(r*)))
|| denotes the L2 norm. The total force on evaders is a sum of a friction force, a repulsive force, and flocking force:
15

---

Published as a conference paper at ICLR 2024
Fj = pfric-e + Free + Fflock
poficce -4ř;
(1x - 29)
Fjep 15 Σ
| xÿy ||2
(10)
Fllock
Σ
g(1lzf 2f (a5 - xx)
1 1 g(p) 2p²
p
We use 60 Euler steps to simulate time span of T = 4. For the training and test data set, we generate 256 initial states with 2 drivers randomly at distance from 3 to 4 around the origin, and evaders randomly in square of side length which is randomly placed at a distance from to 2 around the origin. The goal to guide the evaders towards the origin. The network controller is fully connected neural network: input layer (input shape (24)), fully connected layer (100 features, sanb-activation), fully connected layer (100 features, tanh-activation), output layer (output shape (4), no activation), all layers using weights and biases, totaling to 13004 parameters. We use an accumulated loss function, computing the distance to the origin of each evader after every time step plus the L2 norm of the control values multiplied by regularization coefficient. This coefficient changeable to vary the complexity of the task. In the learning curves shown in the main part, we used Adam with learning rate of 0.001 and batch size of 8 with three different clipping modes (value clipping to 1.0, norm clipping to 1.0, and no clipping) and trained for 1000 epochs.
Hyperparameter study Figure shows training runs for different learning rates and optimizers with batch size 8. Figure [8] shows training runs for different learning rates and batch sizes with Adam. (S) is not able to solve this optimization task. Both (M) and (C) minimize the loss success- fully and often behave similarly. In comparison, (R) sometimes achieves just good result but is more unstable with several instances of a much worse loss value, usually for higher learning rates. The quantitative summary in Table 2 confirms that (M) and (C) both equally suited to solve this control task and clearly better than (R).
Table 2: Quidance-by-repulsiie quantitative summary of our hyperparameter study
Method
(R)
(M)
(C)
(S)
Loss of the best run
0.590
0.465
0.495
5.24
Average loss of the best 5 runs
0.639
0.492
0.503
5.24
Average loss of the best 25 runs
0.839
0.547
0.560
5.24
Number of runs below loss 0.5 0
0
Number of runs below loss 0.8
11
37
38
0
Total runs
60
60
60
60
16

---

Published as a conference paper at ICLR 2024
SGD
SGD Momentum
Adagrad
Rinal Loss
109
109
10a
C
10-1
10-2
10-1
10
10-1
10
Adadelta
RMS prop
Adam
Final
10a
10"
10"
10
10-2
10-3
10-2
10-3
10
Learning rate
Learning rate
Learning rate
Figure Guiddace--y-repulsionn, hyperparameter study with different learning rates and optimizers.
Batch Size:
Batch Size:
Batch Size: 16
Final Loss
10a
10a
R
M
10
10-1
10-1
10-3
10-2
10-3
10
10-3
10-2
Batch Size: 32
Batch Size: 64
Final Loss
109
10a
10
10-1
10-3
10-2
10
10-2
Learning rate
Learning rate
Figure 8:
hyperparameter study with different learning rates and batch sizes.
17

---

Published as a conference paper at ICLR 2024
B.3 CART POLE SWING-UP
Differential equation and numerical simulation We consider poles attached to a cart and ne- glect friction. Our equations are a generalization the classic cart pole equations with one pole (Florian 2005). We use I for the position of the cart, 8, for the angle of pole i, and F for the control variable. Furthermore, A,B, and are used keep the notation compact. The remaining parameters are g.m, M, and I are the gravity constant, masses of the components, and the length of the poles.
F + mlC
M
C [de sin(0,) - 0, cos(O;))
0, 9 sin(O,), (0:) A; cos(O,)
(1I)
B.
F + mllf sOn(
A;
M
B, 1
m cos(O,) {1
4
M
For simulation, set 9.8, 0.1, M = 1.1, 0.5 and vary the number of poles from i == I to 4. We use 100 semi-implicit Euler steps with a time step of 0.01. For the training and test data set, generate 256 different initial configurations, where the poles an angle up to 30 and the goal is to swing all the poles up to the highest possible point. The network controller fully connected neural network: input layer (input shape (2 2- i)). fully connected layer (100 features, tanh-activation) fully connected layer (100 features, tanh-activation), output layer (output shape (1), no activation), all layers using weights and biases, totaling 10501 + 1-000- parameters. We use final loss function, computing the distance of the poles to the highest point only after the last time step. In the learning curves shown the main part, we used Adam with a learning rate of 0.001 and batch size of 8 with three different clipping modes (value clipping to 1.0, norm clipping to 1.0, and no clipping) and trained for 1000 epochs.
randomly swung by
Hyperparameter study Figure shows training runs for different learning rates and optimizers with batch size 8. Figure TO shows training runs for different learning rates and batch sizes with Adam. (S) not able to solve this optimization task. (R) better but rarely able to minimize the loss below 0.01. (M) is second best, often better then (R) but worse than (C). This can also be seen the quantitative summary Table 3 which shows (M) is an improvement over (R) and (C) the best method in all metrics.
Table 3: Cart pole swing-up: quantitative summary of our hyperparameter study
Method
(R)
(M)
(C)
(S)
Loss of the best
0.00194
0.00165
0.00145
0.644
Average loss of the best 5 runs
0.00273
0.00217
0.00161
0.803
Average loss of the best 25 runs
0.0694
0,00451
0.00386
0.915
Number of runs below loss 0.002 1
2
6
Number of runs below loss 0.01
7
37
Total runs
78
78
78
78
18

---

Published as a conference paper at ICLR 2024
SGD
SGD Momentum
Adagrad
10a
10a
Final Loss
10
10-1
10
10
10
10
10-3
10
10
10
10-1
10-7
10
10-1
10-1
10-7
10-1
Adadelta
RMSprop
Adam
10a
10a
10a
Final Loss
10
10
10
10
10
10
10
10
10-1
10-3
10
10
10
10
10-1
Learning rate
Learning rate
Learning rate
Figure 9: Cart pole swing-up, hyperparameter study with different learning rates and optimizers.
Batch size:
Batch size:
Batch size:
101
101
101
Final Loss
10a
10a.
10a
10-1
10-1
10-1
10-2
10-2
10-2
10
10
10
10
10 -2
10-3
10-2
10-2
10-2
Batch size:
Batch size: 16
Batch size: 32
10a
10a
104
Final L Loss
10a
10a
10-1
10-1
10-2
10-3
10-3
10-1
10-3
10
10-3
10-2
10-1
10-7
Learning rate
Learning rate
Learning rate
Figure 10: Cart pole swing-up. hyperparameter study with different learning rates and batch sizes.
19

---

Published as a conference paper at ICLR 2024
B.4 QUANTUM CONTROL
Differential equation and numerical simulation: problem on the Schroedinger equation (Von Neumann
The quantum dipole task is temporal control 2018):
id, V (- (- - u(t) 2)
(12)
The notation is for the imaginary unit, 3 for the time derivative, v for the quantum state wave function, A for Laplacian, for an uffinit-wwlll potential, u(t) the control function we are solving for, and for position operator. Similarity of quantum states is measured with an inner product loss:
L(vi, 1222 = 1 1(th, w/2)1
(13)
We discretize the equation on a spatial domain [0. 2] with 32 spatial discretization points. We simu- late from physical time 0 to 1 in 128 time steps using a modified Crank-Nicolssn scheme (Winckel) cct al., 2009). For training and test data set, we generate 256 random superpositions ground and first excited state, both equally weighted, which serve the initial states the control problem. The target state either the second, third or fourth excited state, depending on task difficulty. The net- work controller is convolutional neural network: input layer (input shape (32, 2)), two-dimensional convolution layer (60 features, kernel (3. 2). stride 2, tani-activation), one-dimensional convolution layer(60 features, kernel 3, stride 2, Lanh-activation), output layer (output shape (1), activation), all layers using weights and biases, totaling to 11701 parameters. We use an accumulated loss func- tion, computing the similarity to the target state after each time step. In the learning curves shown in the main part, we used Adam with a learning rate of 0.001 and batch size of 8 with three different clipping modes (value clipping to 1.0, norm clipping to 1.0, and no clipping) and trained for 1000 epochs.
Hyperparameter study Figure II shows training runs for different learning rates and optimizers with batch size Figure 12 shows training runs for different learning rates and batch sizes with Adam. (R) is not able to solve this optimization task. (S) generally fares similarly; only in individual cases it comes close to (M) and C, for instance when RMSprop was used as optimizer. Altogether it is clearly visible that (M) and (C) perform much better here, and generally exhibit matching performance. This can also be seen Table 4 the best performing runs achieved the best loss and counting the number of runs which solved the task well with loss of 30 or below, we count 14 for both methods. With the inability of (R) to solve this problem and the very good performance of (M) and (C) across a wide range of hyperparameters, we consider this quantum control problem true paragon of our advice on the beneficial gradient flow of physies simulators. The normalized states of the quantum system provide rell-behaved gradient flow that clearly benefits from the modified backpropagation pass.
Table 4: Quantum control: quantitative summary of our hyperparameter study
Method
(R)
(M)
(C)
(S)
Loss of the best run
85.6
19.0
19.0
38.4
Average loss of the best 5 runs
97.7
19.7
20.0
40.5
Average loss of the best 25 runs
99.4
28.9
30.4
68.9
Number of runs below loss 30 0
14
14
0
Number of runs below loss 50
0
27
27
Total runs
60
60
60
60
20

---

Published as a conference paper at ICLR 2024
SGD
SGD Momentum
Adagrad
10a
10a
10a
Rinal Loss
101
101
10-1
10-2
10-1
10
10
10-7
Adadelta
RMSprop
Adam
102
Loss
101
10a
10a
10
10-2
10-3
10-2
10-3
10
Learning rate
Learning rate
Learning rate
Figure 11: Quantum control, hyperparameter study with different learning rates and optimizers.
Batch Size: 4
Batch Size:
Batch Size: 16
102
10a
10a
Final Loss
10
10
10-2
10-3
10
10-1
10-2
Batch Size: 32
Batch Size: 64
10a
10a
Final Loss
M
10a
10a
10
3
10
10-3
10-2
Learning rate
Learning rate
Figure 12: Quantum control, hyperparameter study with different learning rates and batch sizes.
21

---

Published as a conference paper at ICLR 2024
C FURTHER VISUALIZATIONS
Here we present further visualizations of optimizations landscapes, the gradient, and our modified update for control tasks different from our toy problem the main part. These visualizations high- light that many of our observations regarding the modified updates carry over to more complicated settings.
C.I LINEAR QUADRATIC REGULATOR
A common control problem is the linear quadratic regular (LQR), which following our Notation in and using constant square matrices A, B can be defined as follows:
N (Z,, 0) Oz; S(r,, ci) = Ax, + Bc,
(14)
Gradient Descent is known to converge to a global minimum this optimization task, for appropri- ate A and B (Fazel et al., 2018). We do not offer similar mathematical proof here this typically requires a unique minimum or a loss function, which is then shown to iteratively decrease. As LQR non-convex problem and distinctive feature of method is the absence of loss function, established proof techniques or their ideas do not carry over. Instead, we present a visualization of the flow fields of the gradient field and our modified field similar those the toy example, which allows us to assess whether points in the optimization space flow to global minimum. As for visu- alization, we are restricted two-dimensional problems, consider the dynamics in hyperplane of the full optimization space and we have to choose one with global minimum. Technically for restricted case LQR, the convergence gradient descent is not guaranteed according Fazel et al. (2018), so it is also open if the gradient lines end on a global minimum. We choose n 5 time steps, and A and B to be:
0.8
0.5
(-0.5 (-...)
A
B =
(15)
1.2
1.0
13 Four global minima are located around the points (-2, 1). parts b) and c) of the figure, we observe that every flow line ends global minimum. For the minima at (-2, 1) and (0.5.-6), typically called an ill-conditioned landscape: it locally resembles a valley with quickly changing loss in one direction and slowly changing in the other. Also noteworthy is that the global minimum at (-0.5, -2) is one of those with rotating flow field of our modified method in part c). As all of the flow lines of the modified field, just for the gradient field move toward global minima, we have no obvious evidence for globally suboptimal behavior, indicating the modified field can be used solve that optimization task-
We visualize the results in Figure (-0.5,-2) -2), (4, 0.5) and (0.5,-6)
what is
a) Loss Landscape
b) Gradient Field
<) Modified Field
>00
10a
10a
-2
10
6
10-1
-4
-6
10-1
10
10-t
-5.0
-2.5
0.0
2.5
5.0
-2
2
-500
-2.5
0.0
2.5
5.0
Figure 13: Linear quadratic regulator: a) loss landscape, b) regular gradient field (flow lines in red), modified vector field (yellow). For and the background color shows the length of updates.
22

---

Published as a conference paper at ICLR 2024
C.2 NEURAL NETWORKS
We present a similar visualization for a network trained on the quantum control task. We again show a two-dimensional subspace of the optimization space and zoom in on key structures that arise:
Figure 14a) shows sawtooth-like loss landscape with several local minima in the form of multiple dark blue lines. The white areas are local maxima. The gradient field in b) behaves as expected, orthogonal on the lines of equal loss, pointing from local maxima to local minima, and takes on extremely large values. In c), several properties we explained about our modified field can be seen. The flow lines are not orthogonal anymore and have more balanced scale, Additionally, the updates move straight over the local minima, reducing the chance of getting stuck.
Figure T5p) shows the loss different region. closed manifold local minima located the upper part of the plot, which draws in the large majority of the flow lines in the plot, shown in b). Morcover, there are saddle points in the lower left part, leading to gradients with large variations in their directions small scales. In contrast, our modified vector field in c) traverses these local minima and unaffected by the saddle points.
8.100 In-3
AlLoss Landscage
Giadient Field
Moditied Field
3.0
8.075
8.050
8.025
107
2.0
8.000
7.975
1.5
10a
0.035
12
7,900
3.60
5.65
5.70
5.75
5.80
5.60
5.65
5.70
5.75
5.80
5.60
5.65
5.70
5.75
5,80
10:5
10-4
20-15
Quantum control task: a) loss landscape, b) regular gradient field (flow lines in red), c) modified vector field (ycllow). For b) and c) the background color shows the length of updates.
Figure 14:
Loss Landscape
b) Gradient Field
c) Moditied Field
-2.4
0.02
-2.6
10a
-2.8
-3.0
10
-3.2
0.01
-3.4
1
-366
10 2
-5
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
Se-4
36-4
36-4
Figure 15: Quantum control task: a) loss landscape, b) regular gradient field (flow lines in red), c) modified vector field (yellow). For b) and e) the background color shows the length of updates.
23

---

Published as a conference paper at ICLR 2024
D A SIMPLE CONTACT TASK
Here we present short discussion of the influence of a more problematic physics solver as those encountered in contact-rich tasks. This addresses the question if our method, which focuses on the gradient flow through the physics simulator, still works.
D.1
TOY EXAM WITH CONTACT
We start with the changes in the analysis of our toy example. We modify equations 3 by using an absolute value function instead of the identity as abstraction of non-differentiable interaction certain points.
(1; 0) = -81 + -22 ((r,, [2:
(16)
We consider n = 3 time steps and visualize the landscape in Figure 17 In the gradient field, we can see that the optimization space divided by discontinuities originating from the absolute function. One example would be the line from (-5, 0) to (5, Outside these discontinuities, the loss still a smooth function. So we conclude the differences between gradients and our modified update still hold inside the smooth regions, such global minima are conserved and local minima or saddle points can vanish. One example would the saddle point around (3, (3.4). At the lines of discontinuity, what happens depends on the vector fields both sides. If they point in the same direction, an optimizer can move through this line without difficulties. If on both sides the points toward the discontinuous connection, this would create structures that optimizer could get stuck in. For the gradient, one would call this a local minimum and this connects to a widely-accepted observation that contact tasks can give rise new local minima. We think this argumentation holds equally for the gradient and the modified field and consider it hint that contact affects both methods in the same way negatively.
a) Loss Landscape
b) Gradient Field
c) Modified Field
10a
10a
10a
10
6
10
10
10
-2
-2
Figure 16: Toy example with contact: a) loss landscape, b) regular gradient field (flow lines in red), modified vector field (yellow). For and the background color shows the length of updates.
24

---

Published as a conference paper at ICLR 2024
D.2 CART POLE SWING- UP WITH WALLS
We modity our cart pole task to investigate simple contact problem. With a parameter w > introduce walls, left at position -w and one right that reflect the cart's movement once it hits them. Concretely for the right wall, if the position x of the cart at the end of time step ws + d with the position, this follows the dynamics of the absolute value function considered above, and for the velocity, this even a discontinuous map.
we instead update the position to d and negate the cart's velocity. For
In our experiments, we used 25 time steps with duration dt = 0.04, continual loss formulation, 4 poles, and wall parameters w from 0.2 1.0. In this setting, contacts typically happen 4-times per episode for the narrow case 0.2 and around once for the widest case = 1.0.
The results are shown in We observe in all cases diverging of the stopped update (S). The regular gradient (R) is able solve the task but outperformed by our modified update (M) and our combined update (C). Both of these are able to solve the task equally well.
Wall
Wall 0.3
Wall 0.5
Wall 0.75
Wall 1.0
Mmma
0.6
Maaa
500
1000
500
1000
500
1000
500
1000
500
1000
Epochs
Epochs
Epochs
Epochs
Epochs
Figure 17: Cart pole with contact for different wall parameters 0.2, 0.3, 0.5, 0.75 and 1.0. (C) and (M) outperform (R). (S) diverges. Curves were smoothed over 20 epochs for clarity.
25


Reviews:
[{"review_id": "Ob0JBNseDB", "reviewer": "Reviewer_kKUm", "summary": "An learning algorithm is proposed for neural network systems that interact with a differentiable simulator. The idea is to train the neural network policy network with gradient descent, except to stop the gradient from backpropagating through the network multiple times. It is argued that backpropagating through time leads to optimization difficulties for gradient methods (e.g. the well-known gradient explosion in recurrent neural networks). Stopping the gradient in this way doesn't change the objective function, but can improve the dynamics of learning. The authors present nice simulated examples and a series of three experiments", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Very well written. Excellent presentation with nice examples.  \n- The toy examples and figures really helped with illustrating the points.\n- Experiments are clear and support the claims.\n- Differentiable control of simulators is a topic of interest. This could have high impact.", "weaknesses": "- I think the paper could have benefited from having more discussion of the chosen experiment applications. Knowing how these applications compare to potential real-world applications in terms of complexity would have been valuable context for the section on computational cost. It also would have given some context on how well this method might scale to complicated simulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9mQNZTl3C", "reviewer": "Reviewer_Z3pv", "summary": "The paper studies the gradient exploding and vanishing problem due to recurrent operations in the context of control optimization with differentiable physics in the loop. It constructed a toy example to illustrate the problem and proposed a simple method to modify the gradient such that a better optimization landscape can be obtained. \n\nThe method is validated on three tasks and compared to alternative approaches (regular gradients, no combination, no long-range back-propagation).", "questions": "1. The analysis assumes a simplified simulator (identity mapping + control). However, the transition between steps could be complex (e.g., non-linear contact) but the controller is simple (PD controller). How does this change the analysis? One concrete example is robot control.\n2. In the cart-pole experiment, Fig 4 suggests optimization is less stable with fewer poles. Is there an explanation for why this happens?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written. The toy example and visuals are very helpful in terms of understanding the problem and solution. The choice of the example is well explained.\n- The proposed method is conceptually simple (gradient stopping and sign check) but sheds light on using gradient modification to stabilize optimization.", "weaknesses": "**Experiments**\n- The visualization of results could be improved. For example in Fig 3, multiple curves have the same color and overlap each other, making it difficult to draw a clear conclusion. It might be better to draw a mean-std plot for each method where the std (computed over trials) is shaded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mnDrzjw52u", "reviewer": "Reviewer_BicD", "summary": "This paper tries to tackle the gradient explosion / vanishing problem in backpropagation through time. The authors first propose a method to stop the gradient of the feedback policy at each step with respect to the state while keeping the rest of the states active. Then, the authors point out that this update has problems due to rotation, and proposes to combine the original gradient and the modified gradient to tackle how rotation can slow down convergence of gradient-based methods. The authors compare their method against regular gradient descent and stopped gradient descent, and show that the method performs better.", "questions": "(1) I think the biggest question is: why is stopping the gradient $\\partial_x N$ more beneficial, and how do we ensure it's taking more globally beneficial steps compared to the actual gradient? How do we know it's not completely wrong by ignoring these terms, or if there is some pathological system where throwing away these terms will completely make the optimization fail?\n\nA convincing case that the authors could consider is the Linear Quadratic Regulator problem where we have linear dynamics $S(x,c) = ax + bc$ and a linear policy $N(x,\\theta)=\\theta x$ over some horizon. It is known that gradient descent will converge to the optimal parameters $\\theta$ for this problem [5]. But if we assume that $\\partial_x N = 0$, can we actually converge to the minima of the original problem at all?\n\n[5] Fazel et al., \"Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator\", ICML 2018", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The visualizations of the problems are well done and the problems that the authors are trying to convey are clearly communicated to the reader.\n2. The method shows convincing improvement over competitors on simple experiments.", "weaknesses": "(1) I have spent some time trying to understand the authors' argument on why it would be beneficial to stop the gradient of the policy with respect to the current state (i.e. $\\partial_x N$ in author's notation). But I still find that the motivations and the justifications for this is quite weak.\n \nFundamental theorem of calculus (generalized Stokes) tells us a nice connection about the loss and its gradient, so for smooth systems that the authors are considering, if we integrate back the gradient, we should get the loss. So when the authors set some of the terms to zero, there must be some surrogate loss that the modified gradient is considering. I'm willing to agree with the authors that this loss might have a better landscape compared to the original one; but how do we know that whether or not this landscape is completely unrepresentative or the original one?\n\nI think this is the most important section and contribution of the paper that is relatively not very well motivated in the paper, and am willing to give the paper a much better score if the authors can be more convincing about these points with some theory to back it up.\n\n(2) Related to above points, in the experiment results, the performance of combined (C) is much better than the Modifed (M). But it's not clear if this is because rotation is fixed, or if modified is simply not good because of the above issue.  \n\n(3) The authors are missing a large branch of relevant work on studying the efficacy of gradient-based methods in the setting of optimization through differentiable simulation [1,2,3,4], a lot of the issues that the authors have mentioned for Back Propagation Through Time (BPTT) could have benefited from citing these works. These works also have existing methods for improving the performance of BPTT (e.g. total propagation from [1], alpha-order gradients from [3]), which would have made stronger baselines. \n\n(4) It seems to me like the authors are mainly considering the difficulty of BPTT as considering complicating feedback from control actions, but previous works have mainly motivated the shortcomings of BPTT through the lens of characteristics in the dynamics such as chaos [1,2,3] or discontinuities through contact [3,4]. It is unclear if the author's method will improve performance for these difficult systems.\n\n(5) This is minor, but I wished the authors used a more standard notation from nonlinear control (where state-actions are (x,u), dynamics are f, and policy is k) or Reinforcement Learning (states-actions are (s,a) dynamics can still be f, policy is $\\pi$). \n\n\n[1] Parmas et al., \"PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos\", ICML 2018\n\n[2] Metz et al., \"Gradients are not all you need\"\n\n[3] Suh et al., \"Do Differentiable Simulators Give Better Policy Gradients?\" ICML 2022\n\n[4] Antonova et al., \"Rethinking Optimization with Differentiable Simulation from a Global Perspective\", CoRL 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JkDQ30wWSC", "reviewer": "Reviewer_Djj8", "summary": "This paper proposes two simple gradient manipulation strategies to stabilize control learning via backpropagation through time in physics simulation. The approaches are motivated by the fact that naively executing back propagation through time along a long horizon in physics simulation with neural network feedback policy results in gradient exploding and vanishing problems. The first proposed approach cuts the gradients of the neural network inference step and the computed pseudo-gradients have the potential problem of non-rotational-free. The second proposed approach alleviates the issue of the first approach by zeroing out the gradient components with wrong signs. The proposed approaches are evaluated in three dynamics control problems and are shown to outperform the optimization using regular gradients.", "questions": "1. What does the color in Figure 1(b) and 1(c) mean? Does it mean the norm of the gradient?\n\n2. $\\theta$ in Eq. (1) is not defined.\n\n3. For the gradient stopping approach (section 3.1), does it guarantee the modified gradient is zero at global minimal?\n\n4. The underlining assumption of the approaches is “most physics simulators come with a well-behaved gradient flow”, which is usually not true in most robotics problems especially when contacts happen [1, 2]. In those cases, the gradients from the simulation usually have much larger gradients than the gradients from the neural network. It would make the proposed approach much stronger if it could be evaluated on more complex robotic tasks such as the ones in [1, 2].\n\n5. Is there any theoretical guarantee of the correctness of the combination approach (section 3.3). It would be helpful to have two proofs: (1) the modified gradients keep the global optimality of the problem, (2) the modified gradient field is rotation-free.\n\n6. The complex version of the cart pole swing-up task is a bit strange in that the network is trained to control multiple poles with the same mechanism. Does it mean the network outputs 4x the number of actions to control 4-pole concurrently? The problem complexity does not indeed increase since basically, four copies of the single-pole policy should work well in this 4-pole task. A more meaningful task would be controlling a 4-linkage pole (i.e. multi-pendulum) which has much more complex dynamics than a single-linkage cart pole.\n\n\n[1] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation.\n\n[2] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, Miles Macklin. Accelerated Policy Learning with Parallel Differentiable Simulation", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed gradient modification approaches are simple and concise. The approach should be easy to implement.\n\n2. The proposed approach shows great and consistent performances on the designed problems.", "weaknesses": "1. Lack of theoretical guarantee for the proposed approach.\n\n2. The approach is only validated on the problems with simple dynamics. Its generalizability to more complex tasks is unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)