Accelerated Policy Learning with Parallel Differentiable Simulation
Authors: Jie Xu
Viktor Makoviychuk
Yashraj Narang
Fabio Ramos
Wojciech Matusik
Animesh Garg
Miles Macklin
Venue: ICLR 2022 Poster (2022)
Keywords: Robot Control
Policy Learning
Differentiable Simulation
Reinforcement Learning

Abstract: Deep reinforcement learning can generate complex control policies, but requires large amounts of training data to work effectively. Recent work has attempted to address this issue by leveraging differentiable simulators. However, inherent problems such as local minima and exploding/vanishing numerical gradients prevent these methods from being generally applied to control tasks with complex contact-rich dynamics, such as humanoid locomotion in classical RL benchmarks. In this work we present a high-performance differentiable simulator and a new policy learning algorithm (SHAC) that can effectively leverage simulation gradients, even in the presence of non-smoothness. Our learning algorithm alleviates problems with local minima through a smooth critic function, avoids vanishing/exploding gradients through a truncated learning window, and allows many physical environments to be run in parallel. We evaluate our method on classical RL control tasks, and show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms. In addition, we demonstrate the scalability of our method by applying it to the challenging high-dimensional problem of muscle-actuated locomotion with a large action space, achieving a greater than $17\times$ reduction in training time over the best-performing established RL algorithm. More visual results are provided at: https://short-horizon-actor-critic.github.io/.

Published as a conference paper at ICLR 2022
ACCELERATED POLICY LEARNING WITH PARALLEL DIFFERENTIABLE SIMULATION
Jie Xu! Viktor Makoviychuk', Yashraj Narang' Fabio Ramos
Wojciech Matusik?, Animesh Garg'., Miles Macklin'
NVVIDIA "Massacuusetts Institute of Technology UUniversity of Sydney 'University of Toronto
ABSTRACT
Deep reinforcement learning can generate complex control policies, but requires large amounts of training data to work effectively. Recent work has attempted to address this issue by leveraging differentiable simulators. However, inherent problems such as local minima and exploding/vanishing numerical gradients pre- vent these methods from being generally applicd to control tasks with complex contact-rich dynamies, such as humanoid locomotion in classical RL benchmarks. In this work we present a high-performance differentiable simulator and a new policy learning algorithm (SHAC) that effectively leverage simulation gra- dients, even in the presence of non-smoothness. Our learning algorithm allevi- problems with local minima through smooth critic function, avoids vanish- ing/exploding gradients through a truncated learning window, and allows many physical environments be run parallel. We evaluate our method on classi- cal RL control tasks, and show substantial improvements in sample efficiency and algo-
wall-clock time over state-of-the-an RI and differentiable simulation-banl rithms. In addition, we demonstrate the scalability of our method by applying it to the challenging high-dimensional problem of muscle-actuated locomotion with large action space, achieving greater than 17x reduction in training time over the best-performing established RL algorithm. More visual results are provided at: https::/shortthhorieee actor- wwwooorcctitiiiiiii github.io.
INTRODUCTION
Learning control policies is an important task robotics and computer animation. Among various policy learning techniques, reinforcement learning (RL) has been a particularly successful tool to learn policies for systems ranging from robots (e.g., Cheetah, Shadow Hand) (Hwangbo et al. 2019 Andrychowicz et al. 2020) complex animation characters (e.g., muscle-actuated humanoids) (Lee et al. 2019) using only high-level reward definitions. Despite this success, RL requires large amounts of training data approximate the policy gradient, making learning expensive and time- consuming, especially for high-dimensional problems (Figure Right). The recent development of differentiable simulators opens up new possibilities for accelerating the learning and optimization of control policies. A differentiable simulator may provide accurate first-order gradients of the task per- formance reward with respect to the control inputs. Such additional information potentially allows the use of efficient gradient-based based methods to optimize policies. As recently Freeman et al. (202T) show, however, despite the availability of differentiable simulators, it has not yet been convincingly demonstrated that they can effectively accelerate policy learning in complex digensimensi and contact-rich tasks, such as some traditional RI. henchmarks. There are several reasons for this:
1. Local minima may cause gradient-based optimization methods to stall.
2. Numerical gradients may vanish/explode along the backward path for long trajectories.
3. Discontinuous optimization landscapes can occur during policy failures/early termination.
Because of these challenges, previous work has been limited to the optimization of open-loop control policies with short task horizons (Hu et al. 2019 Huang et al. (021), or the optimization of policies for relatively simple tasks (e.g.. contact- free environments) (Mora et al.) 2021| Du et al. 2021). l). In
this work, we explore the question: Can differentiable simulation accelerate policy learning in tasks with continuous closed-looe control and complex contact-rich dynamics?

---

Published as a conference paper at ICLR 2022
of our environments for evaluation. Three classical physical control RL benchmarks inereasing difficulty, from left: Cartpole Swing Up Balance, Ant, and Humamoid. In addition, we train the policy for the high- high-dimensional muscle-tendon driven Humanoid MTU model from Lec cet al. (2019). Whereas model-free reinforcement learning (PPO, SAC) needs many samples for such high- dimensional control problems, SHAC scales efficiently through the use of analytic gradients from differentiable simulation with a parallelized implementation, both in sample complexity and wall-clock time.
Figure 1: Environments: Here
Inspired by actor-critic RL algorithms (Konda & T'itsiklis 2000 we propose an approach to ef- fectively leverage differentiable simulators for policy learning. We alleviate the problem of local minima by using critic network that acts smooth surrogate to approximate the underlying noisy reward landscape resulted by complex dynamics and occurrences of policy failures (Figure 2). In addition, problems with vanishing/exploding gradients, reduce memory requirements, and improve learning efficiency.
truncated learning window shorten the backpropagation path to address
A further challenge with differentiable simulators that the backward pass typically introduces some computational overhead compared to optimized forward-dynamies physics engines. To en- sure meaningful comparisons, we must ensure that our learning method not only improves sample- efficiency, but also wall-clock time. GPU-based physics simulation has shown remarkable effective- ness for accelerating model-free RL algorithms (Liang et al. 2018 Allshire et al.) 202T), given this, we develop a GPU-based differentiable simulator that can compute gradients of standard robotics models over many environments in parallel. Our Py Torch-bbssed simulator allows us to connect high-quality simulation with existing algorithms and tools.
To the best of our knowledge, this work is the first to provide fair and comprehensive compar- ison between gradient- based and RL-based policy learning methods, where fairness is defined as (a) benchmarking on both RL-favored tasks and iifffrentiaall-smuuaatooon complex tasks (i.e., contact-rich tasks with long task horizons), (c) comparing to the state-of-the-a implementation of RL algorithms, and (d) comparing both sample efficiency and wall-clock time. We evaluate our method on standard RL benchmark tasks, as well as a high-dimensional character control task with over 150 actuated degrees of freedom (some of tasks are shown in Figure []. We refer to our method as Short-Horizon Actor-Critic (SHAC), and our experiments show that SHAC outperforms state-of-the-ard policy learning methods in both sample-efficiency and wall-clock time.
tasks, (b) testing
2 RELATED WORK
Differentiable Simulation Physics- Physics-based simulation has been widely used in the robotics field (Todorov al., 2012 Coumans Bail 2016). More recently, there has been interest the con- struction of differentiable simulators, which directly compute the gradients of simulation outputs with respect to actions and initial conditions. These simulators may be based on auto-differentiation frameworks (Griewank & Walther, 2003) Heiden et al., [021] Freeman et al. (02T) or analytic gra- dient calculation (Carpentier & Mansard 2018; Geilinger et al. 2020 Werling et al.) 202I). One challenge for differentiable simulation is the non-smoothness of contact dynamics, leading many works to focus on how to efficiently differentiate through linear complementarity (LCP) models of
contact (Degrave al. 2019 de Avila Belbute-Peres al. 2018 Werling et al. (222) leverage a smooth penslty-based contact formulation (Geilinger et al. 2020: Xu et al. 2021).
Deep Reinforcement Learning Deep reinforcement lcarning has become a prevalent tool for learn- ing control policies for systems ranging from robots (Hwangbo et al.. 2019 OpenAl et 2019
Xu et al. 2019 Lee et al.] 2020 Andrychowicz et al. (0200] Chen et al. 202 to complex an- imation characters (Peng et al. 2018: 2021; Liu & Hodgins 2018: Lee al. 2019). Model-free RL algorithms treat the underlying dynamics as black box in the policy learning process. Among
them, on-policy RL approaches (Schulman man et al. 20155 2017) improve the policy from the experi-
ence generated by the current policy, while off off-policy methods (Lillicrap et al.] 2016 Mnih et al.
2

---

Published as a conference paper at ICLR 2022
Fujimoto et al. 2018 Haarnoja et al. 2018) leverage all the past experience a learning resource to improve sample efficieccy. On the other side, model-based RL methods (Kurutach et al,
2016
2018 Janner et al. 2019) have been proposed to learn approximated dynamics model from little experience and then fully exploit the learned dynamics model during policy learning. The underly- ing idea of our method shares similarity with some prior model-based methods (Hafner al. 2019] Clavera et al. (020) that both learn in an actor-critic mode and leverage the gradient of the simulator "model". However, we show that our method can achieve considerably better wall-clock time effi- ciency and policy performance using parallel differentiable simulation in place of learned model. A detailed note on comparison with model-based RL presented in Appendix AA.4.6
Differentiable Simulation based Policy Learning The recent development of differentiable simu- lators enables the optimization of control policies via the provided gradient information. Backprop- agation Through Time (BPTT) (Mozer 1995) has been widely used in previous work to showcase
differentiable systems (Hu et al. 2019 2020 Liang et al. 2019 Huang et al. 2021| Du al. 2021 However, the noisy optimization landscape and exploding/ Saaiin/g/aiiiinnggg gradients in long-horizon tasks make such straightforward first-order methods ineffective. A few methods have been proposed to resolve this issue. Qiao et al. (202T) present a sample enhancement method to increase RI. sample- efficiency for the simple MuJoCo Ant environment. However, as the method follows model-based learning framework, it is significantly slower than state-of-the-art on-policy methods such as PPO (Schulman al. 2017). Mooa al. (202T) propose interleaving trajectory optimization stage and an imitation learning stage to detach the policy from the computation graph in order to alleviate the exploding gradient problem. They demonstrate their methods on simple control tasks (e.g,, stopping pendulum). However, gradients flowing back through long trajectories of states can still create challenging optimization landscapes for more complex tasks. Furthermore, both methods require the full simulation Jacobian, which is not commonly efficiently available in everse-mode differ- entiable simulators. In contrast, our method relies only on first-order gradients. Therefore, it can naturally leverage simulators and frameworks that can provide this information.
3 METHOD
3.1 GPU-BASED DIFFERENTIABLE DYNAMICS SIMULATION
Conceptually, we treat the simulator as an abstract function 81+1 = F(s,, that takes a state s from time t + t 1, where is a vector of actuation controls applied during that time-step (may represent joint torques, or muscle contraction signals depending on the problem). Given a differen- ac the simulator backward pass computes:
tiable scalar loss function L, and its adjoint L
1 OC
Oc 06-(6: Oc
OF
OC
OF
3s:
0s:-1
es:
ta
Ōstt1
da,
Concatenating these steps allows us to propagate gradients through an entire trajectory.
To model the dynamics function F. our physics simulator solves the forward dynamics equations
Mq = JJ f('.(qqd) q) + c(q, 4) +Tq. a))
(2)
where q, q. q are joint coordinates, velocities and accelerations respectively, f represents external forces, с includes Coriolis forces, and represents joint-space actuation. The mass matrix M, and Jacobian J. are computed in parallel using one thread per-environment. We use the composite rigid body algorithm (CRBA) to compute articulation dynamics which allows us cache the resulting matrix factorization at each step (obtained using parallel Cholesky decomposition) for re-use in the backwards pass. After determining joint accelerations q we perform a semi-implicit Euler integra- tion step obtain the updated system state
(q. q).
For simple environments we actuate our agents using torque-based control, in which the policy outputs 7 at each time-step. For the more complex case of muscle-actuation, each muscle consists of a list of attachment sites that may pass through multiple rigid bodies, and the policy outputs activation values for each muscle. A muscle activation signal generates purely contractive forces (mimicking biological muscle) with maximum force prescribed in advance (Lee et al.) 2019).
Analytic articulated dynamics simulation can be non-smooth and even discontinuous when contact and joint limits are introduced, and special care must be taken to ensure smooth dynamics. To model contact, we use the frictional contact model from Geilinger al, (2020), which approximates Coulomb friction with linear step function, and incorporate the contact damping force formulation
3

---

Published as a conference paper at ICLR 2022
BPTT
SHAC
objective lass
48
A6
Figure 2: Landscape comparison between BPTT and SHAC. We select one single weight from policy and change its value by A&, E [- to plot the task loss landscapes of BPTT and SHAC w.r.t one policy parameter. The task horizon is H 1000 for BPTT. and the short horizon length for method is 32. As we can see. longer optimization horizons lead noisy loss landscape that are difficult optimize, and the landscape of our method can be regarded as smooth approximation of the real landscape.
from Xu et al. 2022 to provide better smoothness of the non-interpenetratton contact dynamics:
f. (-kin + kad) min(d, 0)n, f. vell mnin(k:llvall.:/lll
where fe, fr are the contact normal force and contact friction force respectively, d and d are the contact penetration depth and its time derivative (negative for penetration), n is the contact normal direction, V, is the relative speed at the contact point along the contact tangential direction, and kn, ka, hit, P are contact stiffness, contact damping coefficient, friction stiffness and coefficient of friction respectively.
To model joint limits, a continuous penalty-basc based force is applied:
flmiit = Kiimit (lower 9) q Qlower
Knimi (Yupper 9), > Qupper
where kimmi is the joint limit stiffness, and (Nower: [upper] the bound for the joint angle q.
We build our differentiable simulator on Py PyTorch (Paszke et al. 2019) and use a source-code trans- formation approach to generate forward and backward versions our simulation kernels (Griewank) & Walther, 2003; Hu et al.] (220)). We parallelize the simulator over environments using distributed GPU kernels for the dense matrix routines and evaluation contact and joint forces.
3.2 OPTIMIZATION LANDSCAPE ANALYSIS
Although smoothed physical models improve the local optimization landscape, the combination of forward dynamics and the neural network control policy renders each simulation step non-linear and non-convex. This problem exacerbated when thousands of simulation steps concatenated and the actions in each step are coupled by a feedback control policy. The complexity of the resulting reward landscape leads simple gradient-based methods to easily become trapped in local optima.
Furthermore, to handle agent failure (e.g., humanoid falling down) and improve sample efficiency, early termination techniques are widely used in policy learning algorithms (Brockman al. 2016). Although these have proven effective for model-free algorithms, early termination introduces addi- tional discontinuities the optimization problem, which makes methods based on analytical gradi- ents less successful.
To analyze this problem, inspired by previous work (Parmas al., 2018 we plot the optimization landscape in Figure 2 (Left) for humanoid locomotion problem with 1000-step task horizon. Specifically, we take a trained policy, perturb the value of single parameter 0g in the neural net- work, and evaluate performance for the policy variations. As shown in the figure, with long task horizons and early termination, the landscape discontinuous. In addition, the norm of the gradient a0 computed from backpropagation is larger than 106. We provide more landscape analysis in Appendix A.5 Thus, most previous works based on differentiable simulation focus on short-horizon tasks with contact-free dynamics and no early termination, where pure gradient- based optimization (e.g., BPTT) can work successfully.
the humanoid problem highly non-convex and Oc
3.3 SHORT-HORIZON ACTOR- CRITIC (SHAC)
To resolve the aforementioned issues of gradient-bast based policy learning, we propose the Short- Horizon Actor-Critte method (SHAC). Our method concurrently learns a policy network (i.e., actor) Ty and a value network (i.e., critic) Vg during task execution, and splits the entire task horizon into
4

---

Published as a conference paper at ICLR 2022
So
S1
Sp1-2
$11-2
Say-1
SH
ag
11
Q22
C
Q4
learning episode of horizon length H
Sg
Visi
Shr
7
Reset
Envs
Qg
Learning episode of short horizon length /t
Learning episode of short horizon length /t
Figure 3: Computation graph of BPTT and SHAC. Top: BPTT propagates gradients through an entire trajectory in each learning episode. This leads to noisy loss landscapes, increased memory, and numerical gradient problems. Bottom: SHAC subdivides the trajectory into short optimization windows across learning episodes. This results smoother surrogate reward function and reduces memory requiremenss, enabling parallel sampling of many trajectories. The environment reset upon early termination happens. Solid arrows denote gradient-presee preserving computations; dashed arrows denote locations at which the gradients are cut off.
several sub-windows of smaller horizons across learning episodes (Figure 3}. A multi-step reward in the sub-window plus terminal value estimation from the learned critic is used improve the policy network. The differentiable simulation is used to backpropagate the gradient through the states and actions inside the sub-windows provide an accurate policy gradient. The trajectory rollouts are then collected and used to learn the critic network in each learning episode.
Specifically, we model each of our control problems as finite-horizon Markov decision process (MDP) with state space S, action space A. reward function R, transition function F. initial state distribution Dx,- and task horizon H. At each step, an action vector a, computed by a feedback policy No (a, s.). While our method does not constrain the policy to be deterministic or stochastic, we use the stochastic policy in our experiments to facilitate extra exploration. Specifically, the action (uu((s)) (s:)). The transition function J is modeled by our differentiable simulation (Section 3.1 A single-step reward r = R(s,, a,) is received at each step. The goal of the problem is to find the policy parameters 9 that maximize the expected finite-horizon reward.
sampled by
Our method works in an on-policy mode as follows. In each learning episode, the algorithm samples N trajectories {T, of short horizon h < H in parallel from the simulation, which continue from the end states of the trajectories in the previous learning episode. The following policy loss is then computed:
loth-1
co= Nh Σ
Σ t=to
7t (()) + Vo(si,
(5)
where si and a; are the state and action at step of the i-th trajectory, and < is a discount factor introduced to stabilize the training. Special handling such as resetting the discount ratio conducted when task termination happens during trajectory sampling.
&Cg we treat the simulator as a differentiable layer (with To compute the gradient of the policy loss 30 backward pass shown in Eq. (1) in the PyTorch computation graph and perform regular backprop- agation. We apply reparameterization sampling method to compute the gradient for the stochastic policy. For details of gradient computation, see Appendix A. Our algorithm then updates the pol- icy using one step of Adam (Kingma& & Ba 2014). The differentiable simulator plays a critical role here, as it allows us to fully utilize the underlying dynamics linking states and actions, as well as op- timize the policy, producing better short-horizon reward inside the trajectory and more promising terminal state for the sake of long-term performance. We note that the gradients are cut off between learning episodes to prevent unstable gradients during mor-zooizon backpropagation.
After we update the policy Te, we use the trajectories collected in the current learning episode to train the value function Vg. The value function network is trained by the following MSE loss:
Lss =
E (+)) [V v
(6)
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: SHAC Horizon Actor-Critic) Policy Learning
Initialize policy Ty. value function Vg. and target value function Vy for learning episode +- 2...... - M do Sample N' short-horizon trajectories of length by the parallel differentiable simulation from
Vg.
the end states of the previous trajectories.
Compute the policy loss Lg defined in Eq. 5 from the sampled trajectories and Vgr-
Compute the analytical gradient &Ce and update the policy mo one step with Adam.
26 Compute estimated values for all the states in sampled trajectories with Eq. 7
Fit the value function Va using the critic loss defined in Eq. 6
Update target value function: V& aVo + (1 Va. end for
where (s)) is the estimated value of state s, and is computed from the sampled short-horizon tra- jectories through a td-À formulation (Sutton et al.) 1998), which computes the estimated value by exponentially averaging different k-step returns to balance the variance and bias of the estimation:
1-1-1
V(s:) -1 (1-1))
Σ
A x
+ +
where Gj = (E r'ret)) + 7 Vo((++*)) is the k-step return from time L. The estimated value
(s) is treated constant during critic training, in regular actor-critic RL methods. In other words, the gradient of Eq. doess not flow through the states and actions in Eq. 7
We further utilize the target value function technique (Mnih al. 2015) to stabilize the training by smoothly transitioning from the previous value function to the newly fitted one, and use the target value function Var to compute the policy loss (Eq. (5) and to estimate state values (Eq. 7.. In addition, we apply observation normalization as is common in RI algorithms, which normalizes the state observation by a running mean and standard deviation calculated from the state observations in previous learning episodes. The pseudo code of our method is provided in Algorithm !
Our actor-critic formulation has several advantages that enable it to leverage simulation gradients effectively and efficiently. First, the terminal value function absorbs noisy landscape over long dy- namics horizons and discontinuity introduced by early termination into a smooth function, as shown in Figure 2 (Right). This smooth surrogate formulation helps reduce the number of local spikes and alleviates the problem of easily getting stuck in local optima. Second, the short-horizon episodes avoid numerical problems when backpropagating the gradient through deeply nested update chains. Finally, the use of short-horizon episodes allows us to update the actor more fieqquenll, which, when combined with parallel differentiable simulation, results significant speed up of training time.
4 EXPERIMENTS
We design experiments to investigate five questions: (1) How does our method compare to the state- of-the-art RL algorithms on classical RL control tasks, in terms of both sample efficiency and wall- clock time efficiency? (2) How does our method compare to the previous differentiable simulation- based policy learning methods? (3) Does our method scale to high-dimensional problems? (4) the terminal critic necessary? (5) How important is the choice of short horizon length h for our method?
4.1 EXPERIMENT SETUP
To ensure a fair comparison for wall-clock time performance, we run all algorithms on the same GPU model (TITAN X) and CPU model (Intel Xeon(R) E5-2620). Furthermore, we conduct hy- perparameter searches for all algorithms and report the performance of the best hyperparameters for each problem. In addition, we report the performance averaged from five individual runs for each algorithm on each problem. The details of the experimental setup are provided in Appendix A.2 We also experiment our method with fixed hyperparameter setting and with a deterministic policy choice. Due to the space limit, the details of those experiments are provided in Appendix AA.4 A.4
4.2 BENCUMARK CONTROL PROBLEMS
For comprehensive evaluations, we select six broad control tasks, including five classical RL tasks across different complexity levels, as well as one high-dimensional control task with large action

---

Published as a conference paper at ICLR 2022
CartPole Swing Up
Ant
Humanoid
Humamoid MTU
-200
Rewward
500
Reward
-2003
3006
-2530
2000
1000
simudation (x109)
simulation sleps (x109)
simulation steps (x110)
simulla Lon sleps (x10*)
-590
3000
1000
2000
-1033
rean
Resward
Deward
Rowardd
-3631
senn
-3000
-2500
1880
-3890
wall-coock time (min)
wall-clock time (min)
wall-clock time (min)
wall-look time (h)
SHAC (Ours)
PPO
SAC
BFT7
PODS
No Crile
Figure 4: Learning curves comparison on four benchmark problems. Each column corresponds to a par- ticular problem. with the top plot evaluating sample efficiency and the boltom plot evaluating wall-clock time efficiency. For better visualization, time of our method (except for Humanoid MTU), and we provide the full plots in Appendix [A.4 Each curve is averaged from five random seeds, and the shaded area shows the standard deviation. SHAC is more sample cffi- cient than all baselines. Model-free baselines are competitive on wall -lock time on pedagogical environments such the cartpole, but are much less effective the problem complexity scales.
truncate all the curves up to the maximal simulation steps/wall-clock
space. All tasks have stochastic initial states to further improve the robustness of the learned policy. We introduce four representative tasks in the main paper, and leave the others in the Appendix.
We select CartPole Swing Up. Aut and Humanoid as three representative RL tasks, as shown in Figure [I] Their difficulty spans from the simplest contact-free dynamics (CartPole Swing Up), to complex contact-rich dynamics (Humanoid). For CartPole Swing Up, we use H 240 as the task horizon, whereas the other tasks use horizons of = 1000.
Classical Tasks:
To assess how our method scales to high-dimensional tasks, we examine the challenging problem of muscle-actuated humanoid control (Figure [ Right). We use the lower body of the humanoid model from Lee et al. (2019), which contains 152 muscle-tendon units (MTUs). Each MTU contributes one actuated degree of freedom that controls the contractile force applied to the attachment sites on the connected bodies. The task horizon for this problem is H 1000.
Humanoid MTU:
To be compatible with differentiable simulation, the reward formulations of each problem are de- fined as differentiable functions. The details of each task are provided in Appendix A.3
4.3 RESULTS
Comparison to model-free RL. We compare SHAC with Proximal Policy Optimization (PPO) (Schulman al. 2017) (on-policy) & Soft Actor-Critic (SAC) (Haarnoja et al., 2018) (off- policy). We use high-periormance implementations from RL games (Makovichhkk & Makoviychuk] 2021). To achieve state-of-the performance, we follow Makoviychuk al. (2021): all simula- tion, reward and observation data remain on the GPU and are shared as PyTore ToTch tensors between the RL algorithm and simulator. The PPO and SAC implementations are parallelized and operate on vectorized states and actions. With PPO we used short episode lengths, an adaptive learning rate, and large mini-batches during training to achieve the best possible performance.
As shown in the first row of Figure 4 our method shows significant improvements in sample ef- ficiency over PPO and SAC in three classical RI. problems, especially when the dimension of the problem increases (c.g., Humanoid), The analytical gradients provided by the differentiable simula- tion allow us to efficiently acquire the expected policy gradient through a small number of samples. In contrast, PPO and SAC have to collect many Monte-Carlo samples estimate the policy gradient.
Model-free algorithms typically have a lower per-iteration cost than methods based on differen- tiable simulation; thus, it makes sense to also evaluate wall-clock time efficiency instead of sample- efficiency alone. As shown in the second row of Figure the wall-clock time performance of PPO, SAC, and method are much closer than the sample efficiency plot. Interestingly, the training speed of our method is slower than PPO at the start of training. We hypothesize that the target value
7

---

Published as a conference paper at ICLR 2022
Figure 5: Humanoid MTU: A sequence of frames from a leamned running gait. The musele unit color indicates the activation level at the current state.
network in our method is initially requiring sufficient episodes to warm up. We also note that the backward time for our simulation is consistently around 2x that of the forward pass. This indicates that our method still has room to improve its overall wall-clock time efficiency through the devel- opment of more optimized differentiable simulators with fast backward gradient computation. We provide a detailed timing breakdown of our method in Appendix A.4
We observe that method achieves better policies than RL methods in all problems. We hy- pothesize that, while RL methods are effective at exploration far from the solution, they struggle to accurately estimate the policy gradient near the optimum, especially in complex problems. We also compare our method to another model-free method REDQ (Chen et al. 2021b) in Appendix A.4
Comparison with previous gradient-based methods. We compare our approach to three gradient- based learning methods: (1) Backpropagation Through Time (BPTT), which has been widely used in the differentiable simulation literature (Hu et al. 2019, Du et al. 1021). (2) PODS (Mora et al.) 2021] and (3) Sample Enhanced Model- Policy Optimization (SE-MBPO) QQia et al. 2021
BPTT: The original BPTT method backpropagates gradients over the entire trajectory, which results in exploding gradients as shown in Section We modify BPTT to work on a shorter window of the tasks (64 steps for CartPole and 128 steps for other tasks), and also leverage parallel differentiable simulation to sample multiple trajectories concurrently to improve its time efficiency. As shown in Figure 4 BPTT successfully optimizes the policy for the contact-free CartPole Swing Up task, whereas it falls into local minima quickly in all other tasks involving contact. For example, the policy that BPTT learns for Ant is a stationary position leaning forward, which is local minimum.
We compare to the first-order version of PODS, as the second-order version requires the full Jacobian of the state with respect the whole action trajectory, which is not efficiently available in moverse-mode differentiable simulator (including ours). Since PODS relies on a trajectory op- timization step optimize an open-loop action sequence, not clear how to accommodate early termination where the trajectory length can vary during optimization. Therefore, we test PODS per- formance only on the CartPole Swing Up problem. As shown in Figure 4 PODS quickly converges to a local optimum and is unable to improve further. This is because PODS is designed to be a method with high gradient exploitation but little exploration. Specifically, the line search applied in the trajectory optimization stage helps it converge quickly, but also prevents it from exploring more surrounding space. Furthermore, the extra simulation calls introduced by the line search and the slow imitation learning stage make it less competitive in either sample wall-clock time efficiency.
PODS:
SE-MBPO: Qiao et al. (202T) propose to improve a model-based RL method MBPO (Janner et al. 2019) by augmenting the rollout samples using data augmentation that relies the Jacobian from the differentiable simulator. Although SE-MBPO shows high sample efficiency, the underlying model-based RL algorithm and off-policy training lead to higher wall-clock time. As comparison, the officially released code for SE-MBPO takes 8 hours to achieve a reasonable policy in the Ant problem used by Qiao al. (02TT), whereas our algorithm takes less than 15 minutes acquire policy with the same gait level in our Ant problem. Aiming for a more fair comparison, we adapt their implementation work on our Ant problem in our simulator. However, we found that could not successfully optimize the policy even after considerable hyperparameter tuning. Regardless, the difference in wall-clock time between two algorithms obvious, and the training time of SE- MBPO is unlikely to be improved significantly by integrating it into our simulation environment. Furthermore, as suggested by Qiao et al. (202I), SE-MBPO does not generalize well to other tasks, whereas our method can be successfully applied to various complexity levels of tasks.
8

---

Published as a conference paper at ICLR 2022
Scalability to high-dimensional problems. We test our algorithm and RL baselines the Hu- manoid MTU example to compare their scalability to high-dimensional problems. With the large 152-dimensional action space, both PPO and SAC struggle to learn the policy as shown in Figure 4 (Right). Specifically, PPO and SAC learn significantly worse policies after more than 10 hours of training and with hundreds of millions of samples. This is because the amount of data required to accurately estimate the policy gradient significantly increases as the state and action spaces become large. In contrast, our method scales well due to direct access to the more accurate gradients pro- vided by the differentiable simulation with the reparameterization techniques. To achieve the same reward level as PPO, our approach only takes around 35 minutes of training and 1. 1.7M simulation steps. This results in over 17x and 30x wall-clock time improvement over PPO and SAC, respec- tively, and 382x and 170x more sample efficiency. Furthermore, after training for only 1.5 hours, our method is able to find policy that has twice the reward of the best-performing policy from the RL methods. A learned running gait is visualized in Figure 5) Such scalability to high- high-dimensional control problems opens up new possibilities for applying differentiable simulation in computer ani- mation, where complex character models are widely used to provide more natural motion.
Ablation study on the terminal critic. We introduce a terminal critic value in Eq. 5 to account for the long-term performance of the policy after the short episode horizon. In this experiment, we evaluate the importance of this term. By removing the terminal critic from Eq. S we get an algorithmic equivalent to BPTT with a short-horizon window and discounted reward calculation. We apply this no-critic variation on all four problems and plot the training curve in Figure denoted by "No Critic". Without a terminal critic function, the algorithm is not able to learn a reasonable policy, as it only optimizes short-horizon reward of the policy regardless of its long-term behavior.
Study of short horizon length h. The choice of horizon length h is important for the performance of our method. cannot be too small, as it will result in worse value estimation by íd-À (Eq. ( and underutilize the power of the differentiable simulator to predict the sensitivity of future performance to
3300
Cammmn O0 Time (m1
8200
Best Reward
5000
the policy weights. On the other hand, horizon length that is too long will lead frequent policy updates. Empirically, we find that short hori- zon length h = 32 with N 64 parallel trajectories works well for all tasks in our experiments. We conduct a study of short horizon length on the Ant task to show the influence of this hyperparameter. We run our algorithm with six short hori- zon lengths h- = 4, 8, 16, 32, 64, 128. We set the corresponding number of parallel trajectories N 512, 256, 128, 64, 32, 16 for the variant, such that each one generates the same amount of samples in single learning episode. We run each variant for the same number of episodes M = 2000 with 5 individual ran- dom seeds. In Figure 6 we report the average reward of the best policies from 5 runs for each variant, as well as the total training time. As expected, the best reward is achieved when h 16 or 32, and the training time scales linearly as increases.
noisy optimization landscape and less-
2300
3000
Short Horizon Length
Figure 6: Study of short horizon length h on Ant problem. A small h results in worse value estimation. A large h leads to iol-posed opti- mization landscape and longer training time.
5 CONCLUSION AND FUTURE WORK
In this work, we propose an approach to effectively leverage differentiable simulation for policy learning. At the core is the use of a critic network that acts as a smooth surrogate to approximate the underlying noisy optimization landscape. In addition, truncated learning window is adopted to alleviate the problem of exploding/vanishing gradients during deep backward paths. Equipped with the developed parallel differentiable simulation, method shows significantly higher sample efficiency and wall-clock time efficiency over state-of-the-a RL and gradient- based methods, espe- cially when the problem complexity increases. As shown in our experiments, model-free methods demonstrate efficient learning at the start of training, but SHAC is able to achieve superior perfor- mance after sufficient number of episodes. A compelling future direction for research is how to combine model-free methods with our gradient-basse both. Furthermore, in our method, we use a fixed and predetermined short horizon length h through- out the learning process; however, future work may focus on implementing an adaptive short horizon schedule that varies with the status of the optimization landscape.
method in order to leverage the strengths of
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their helpful comments in revising the paper. The majority of this work was done during the internship of Jie Xu NVIDIA, and we thank Tae- Yong Kim for his mentorship. This work is also partially supported by Defense Advanced Research Projects Agency (FA8750-20-C-0075),
REFERENCES
Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Ankur Handa, and Animesh Garg. Transferring Dex- terous Manipulation from GPU Simulation Remote Real-World TriFinger. arXiv preprint arXiv:2108.09779, 2021.
OpenAl: :
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal J6zefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, pp-
8234-8244, 2018.
Justin Carpentier and Nicolas Mansard. Analytical derivatives of rigid body dynamics algorithms. In Robotics: Science and systems (RSS 2018), 2018.
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Sth Annual Conference on Robot Learning, 2021a.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without a model. In International Conference on Learming Representa- tions, 2021b.
Ignasi Clavera, Yao Fu, and Pieter Abbeel. Model-augmented actor-critic: Backpropagating through paths. In International Conference on Learning Representations, 2020.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
Filipe de Avila Belbute-Peres Kevin Smith, Kelsey Allen, Josh Tenenbaum, and Zico Kolter. End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems, 2018
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, pp. 6, 2019.
Mare Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient ap- proach to policy search. Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'l1, pp. 465-472, 2011. ISBN 9781450306195.
Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojeiech Matusik. Diffpd: Differentiable projective dynamics. ACM Trans. Graph., 41(2), nov 2021.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax differentiable physics engine for large scale rigid body simulation. In Thirty-fifth Con- ference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1587-1596. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2022
Moritz Geilinger. David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for mulli-body systems with frictional contact. ACM Trans. Graph., 39(6), November 2020.
Andreas Griewank and Andrea Walther. Introduction to automatic differentiation. PAMM, 2(1):
45-49, 2003.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming. pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neural- Sim: Augmenting differentiable simulators with neural networks. Proceedings of the IEEE International Conference Robotics and Automation (ICRA), 2021.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simu- lator for soft robotics. 2019 International conference on robotics and automation (ICRA), pp-
IEEE, 2019.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelles and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020.
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Plasticinelab: soft-body manipulation benchmark with differentiable physics. In Interna- tional Conference on Learning Representations, 2021.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32:12519- 12530, 2019.
Michael I. Jordan and David E. Rumelhart. Forward models: Supervised learning with distal teacher. Cognitive Science, 16(3):307-354, 1992. ISSN 0364-0213.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988. 2014.
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1014, 2000.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020.
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Trans. Graph., 38(4). July 2019.
Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning, pp. 270-282, PMLR, 2018.

---

Published as a conference paper at ICLR 2022
Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. Advances in Neural Information Processing Systems, 32, 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster). 2016.
Libin Liu and Jessica Hodgins. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Trans. Graph., 37(4), July 2018.
Denys Makoviichuk and Viktor Makoviychuk. RL Games, 2021. URL httpt:://gttttttte com/ Derys88/rL_games:
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529 553-333, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 7805 7817. PMLR, 18- Jul 2021.
Michael Mozer. A focused backpropagation algorithm for temporal pattern recognition. Complex Systems, 3, 01 1995.
D.H. Nguyen and B. Widrow. Neural networks for self-learning control systems. IEEE Control Systems Magazine, 10(3):18-23, 1990. doi: 10.1109/37.55119
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng. Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's 'ss cube with a robot hand, 2019.
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-basee policy search robust to the curse of chaos. In International Conference on Machine Learning, pp. 4065 PMLR, 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. 2019.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 37(4): 143:1-143:14 14, July 2018. ISSN 0730-0301.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Trans. Graph., 40(4), July 2021.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C. Lin. Efficient differentiable simulation of articulated bodies. In ICMI., 2021.
12

---

Published as a conference paper at ICLR 2022
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-18888888 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approxi- mating dynamic programming. In Bruce Porter and Raymond Mooney (eds.), Machine Learning Proceedings 1990, pp. 216-224. Morgan Kaufmann, San Francisco (CA), 1990. ISBN 978-1- 55860- 141-3. doi:
1-55860- 141-3.50030-4.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and C. Karen Liu. Fast and Feature-Complete Differentiable Physics Engine for Articulated Rigid Bodies with Con- tact Constraints. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi:
Jie Xu, Tao Du Michael Foshey, Beichen Li, Bo Zhu, Adriana Schulz, and Wojeiech Matusik. Learning to fly: computational controller design for hybrid uavs with reinforcement learning. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojeiech Matusik, Shinjiro Sueda, and Pulkit Agrawal. An End-to-End Differentiable Framework for Contact-Aware Robot Design. In Pro- ceedings of Robotics: Science and Systems, Virtual, July 2021. doi: 2...6667RSS.2221..X2 008.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 POLICY LOSS GRADIENT COMPUTATION
We minimize the following policy loss to improve the policy network Ty.
toth-1
Co= Nh Σ Σ x R(s), + (o((s
E
(
(8)
ttto
where N is the number of trajectories, h is the short horizon length, to is the starting time of each trajectory, sj and ai are the state and actions time step of trajectory Tr.
To compute its gradient, we treat the differentiable simulator as a differentiable layer (with backward pass shown in Eq. in the computation graph for policy loss Lg, and acquiring the gradients ACe by PyTorch with its reverse-mode computation. Mathematically, we have the following terminal adjoints for each trajectory T;
.00
UCo
(V Nh
(9)
Ost to
as
reverse order: From the last time step to + h. we can compute the adjoints in the previous steps to S to in
to aR(s), at)
OF
OF dat
Ono (si)
+
(10)
Ōs
Nh.
Ost
0si+1
0s
ds;
aLe
RR(s), at)
ULo
OF
to
(11)
da
Nh.
da
Üs
Dat
From all the computed adjoints, we can compute the policy loss gradient by
Oc,
N
ta+h-1
(T:)):))
(12)
00
00
t=
To be noted, for our stochastic policy a, (s)(ss)), (s )) we use the reparamterization sam- pling method, which allows us to compute the gradients (ra(s) and (ma(s)
d0 8%:
A.2 EXPERIMENT SETUP
To ensure a fair comparison for wall-clock time performance, model (TITAN X) and CPU model (Intel Xeon(R) E5-2620). Furthermore, we conduct extensive hyperparameter searches for all algorithms and report the performances of the best hyperparameter settings on each problem. For our method, we run for 500 learning episodes for CartPole Swing Up problem and for 2000 learning episodes for other five problems. For baseline algorithms, we run each of them for sufficiently long time in order to acquire the policies with the highest rewards. We run each algorithm with five individual random seeds, and report the average performance.
all algorithms on the same GPU
A.2.1 BASELINE ALGORITHM IMPLEMENTATIONS
Policy and Value Network Structures We multilayer perceptron (MLP) as the policy and value network structures for all algorithms. We use ELU as our activation function for the hidden layers and apply layer normalization to each hidden layer. Since different problems have different state and action dimensionalities, we use different network sizes for different tasks.
Codebase and Implementations For PPO and SAC, we use pigh-peefo implementations of both methods available in RI games (Makoviichuk Makoviychuk] (00T). To achieve state-of- the-art performance we follow the approach proposed by Makoviychuk et al. (2021) where all the simulation, reward and observation data stays on GPU and is shared as PyTorch tensors between the RL algorithm and the parallel simulator. Both PPO and SAC implementations are parallelized and operate on vectorized states and actions. With PPO we used short episode lengths, an adaptive learn- ing rate, and large mini-batch sizes during training achieve the maximum possible performance.
For BPTT, the original BPTT requires to backprapagate gradients over the entire trajectory and results in exploding gradients problem. In order to make it work our long-horizon tasks with
14

---

Published as a conference paper at ICLR 2022
complex dynamies, modify BPTT to work shorter window of the tasks (64 steps for CartPole and 128 steps for other tasks). We further improve its wall-clock time efficiency by leveraging parallel differentiable simulation sample multiple trajectories concurrently.
For PODS, due to the lack released code, we implement our own version of it. Specifically, we implement the first-order version of PODS proposed by Mora al. (2021) since the second-order version requires the full Jacobian of the state with respect to the whole action trajectory, which is typically not efficiently available in reverse-mode differentiable simulator (including ours).
For SE-MBPO, we adapt the implementation released by Qiao et al. 2021 into our benchmark problem.
A.2.2 IMPLEMENTATION DETAILS OF OUR METHOD
Policy and Value Network Structures We use multilayer perceptron (MLP) as the policy for our method. We use ELU our activation function for the hidden layers and apply layer normalization to each hidden layer. Since different problems have different state and action dimensionalities, we different network sizes for different tasks. Basically, structures for problems with higher degrees of freedoms. Empirically we find the performance of our method is insensitive to the size of the network structures. We report the network sizes that used in our experiments for Figure 4 in Table I for better reproducibility. For other problems (i.e., HalfCheetah and Hopper), refer our released code for the adopted hyperparameters.
larger policy and value network
Table I: The network setting of our method on cach problem.
Problems
Policy Network Structure
Value Network Structure
CartPole Swing Up
[64, 64]
[64, 64]
Ant
[128, 64, 32]
[64, 64]
Humanoid
[256, 128]
[128, 128]
Humanoid MTU
[512, 256]
[256, 256]
A.2.3 HYPERPARAMETERS SETTINGS
Since our benchmark problems span across a wide range of complexity and problem dimentionality. the optimal hyperparameters in expected will be different across problems. We conduct an extensive hyperparameter search for all algorithms, especially for PPO and SAC, and report their best per- forming hyperparameter settings for fair comparison. For PPO, we adopt the adaptive learning rate strategy implemented RL games (Makoviichuk Makoviychuk (02T). The hyperparameters of PPO and SAC used in the experiments reported in Table and
Table 2: The hyperparameter setting of PPO on each problem.
Parameter names
CartPole
Ant
Humanoid
Humanoid MTU
short horizon length h
240
32
32
32
number of parallel environments, N
32
2048
1024
1024
discount factor
0.99
0.99
0.99
0.99
GAE À
0.95
0.95
0.95
0.95
minibatch size mini epochs
1920
16384
8192
8192
For method, we also conduct hyperparameter searches for each problem, including the short horizon length h, the number of parallel environments N, and the policy and critic learning rates. The learning rates follow a linear decay schedule over episodes. We report performance from the
15

---

Published as a conference paper at ICLR 2022
Table 3: The hyperparameter setting of SAC on each problem.
Parameter names
CartPole
Ant
Humanoid
Humanoid MTU
num steps per episode h
128
128
128
128
number parallel environments
32
128
64
256
discount factor y
0.99
0.99
0.99
0.99
learning rate
0.005
0.005
0.0002
0.0002
actor learning rate
0.0005
0.0005
0.0003
0.0003
critic learning rale
0.0005
0.0005
0.0003
0.0003
critic
0.005
0.005
0.005
0.005
minibatch size
1024
4096
2048
4096
replay buffer size
lc6
learnable temperature
True
number seed steps
2
best hyperparameter settings in Figure 4 and report the hyperparameters for each problem in Table
Table 4: The optimal hyperparameter setting our method on each problem.
Parameter names
CartPole
Ant
Humanoid
Humanoid MTU
short horizon length h
32
number of parallel environments N
64
policy learning rate
0.01
0.002
critic learning rate
0.001
0.002
0.0005
target value network
0.2
0.995
discount factor y
0.99
value estimation À
0.95
Adam (B1,82) (32)
(0.7, 0.95)
number of critic training iterations
16
number of critic training minibatches
number of episodes M
500
2000
Although the optimal hyperparameters are different for different problems due to the different com- plexity of the problem and we report the best setting for fair comparison against baseline algorithms, in general, we found the set of hyperparameters in T works reasonably well on all problems (leading to slightly slower convergence speed in CartPole Swing Up and Ant tasks due to large for target value network). More analysis is provided Appendix A.4.3
A.3 BENCHMARK CONTROL PROBLEMS
We describe more details of benchmark control problems in this section. We evaluate al- gorithm and baseline algorithms on six benchmark tasks, including five classical RL benchmark problems and high-dimensional problem. We define the reward functions be differentiable functions in order to incorporate them into our differentiable simulation framework.
A.3.1 CARTPOLE SWING UP
CartPole Swing Up is one of the simplest classical RL control tasks. this problem, the control policy has to swing up the pole to the upward direction and keeps it in that direction as long as possible. We have 5-dimensional observation space as shown in Table and 1-dimensional action to control the torque applied to the prismatic joint of the cart base.
The single-step reward is defined as:
R. -02 -
0.05x 0.1.²2
(13)
16

---

Published as a conference paper at ICLR 2022
Table 5: A general setting of hyperparametens of our method.
Hyperparameter names
Values
short horizon length h
32
number of parallel environments N
64
policy learning rate
0.002
critic learning rate
0.0005
discount factor y
0.99
value estimation X
0.95
target value network a
0.995
number of critic training iterations
16
number of critic training minibatches
4
Table 6: Observation vector of CartPole Swing Up problem
Observation
Degrees of Freedom
position of cart base:
velocity of the cart base:
sine and cosine of the pole angle: cin()),
2
angular velocity of pole: 0
The initial state is randomly sampled. The task horizon is 240 steps and there is no early termination in this environment that can be used the algorithms which are compatible with early termination strategy.
A.3.2 HALFCHEETAH
In this problem, two-legged cheetah robot is controlled to run forward as fast as possible. We have 17-dimensional observation space as shown Table 7 and G-dimensional action control the torque applied to each joint.
Table 7: Observation vector of HalfCheetah problem
Observation
Degrees of Freedom
height of the base: h
rotation angle of the base
linear velocity of the base: v
angular velocity of the base
joint angles
6
joint angle velocities
6
The single-step reward is defined as:
R U2 - 0
(14)
where er is the forward velocity.
17

---

Published as a conference paper at ICLR 2022
The initial state is randomly sampled. The task horizon is 1000 steps and there is early termination in this environment.
A.3.3 HOPPER
In this problem, jumping hopper is controlled to forward fast possible. We have 11- dimensional observation space as shown in Table g and 3-dimensional action to control the torque applied to each joint.
Table 8: Observation vector of Hopper problem
Observation
Degrees of Freedom
height of the base: h
rotation angle of the base
1
linear velocity of the base:
2
angular velocity of the base
joint angles joint angle velocities
The single-step reward is defined as:
R = Ry + Reeight + Rangle 0.1||a]|3,
(15)
where R. Urr is the forward velocity, Rheight is designed to penalize the low height state, defindd by:
Rheight 20004
2 b
A < 0
(16)
A D 2 0
A clip(h 0.3, 1, 0.3) and the Rangte is designed encourage the upper body of the hopper be as upward as possible, defined by
(17)
Rangle
(18)
30
The initial state is randomly sampled. The task horizon is 1000 steps and early termination is triggered when the height of the hopper is lower than -0.55m.
A.3.4 ANT
In this problem, four-legged ant is controlled to run forward as fast as possible. We have 37- dimensional observation space shown in Table 9 and 8-dimensional action to control the torque applied to each joint.
The single-step reward is defined as:
R Ry 0. Rheading Rheight:
(19)
where R, U, is the forward velocity, R.p = projection(upward direction) encourages the agent to be vertically stable, Rheading projection direction) encourages the agent to run straight forward, and Rheight h 0.27 the height reward.
The initial state is randomly sampled. The task horizon is 1000 steps and early termination is triggered when the height of the ant lower than 0.27m.
A.3.5 HUMANOID
In this problem, humanoid robot is controlled to forward as fast possible. We have 76- dimensional observation space as shown in Table 10 and 21-dimensional action vector to control the torque applied to each joint.
18

---

Published as a conference paper at ICLR 2022
Table 9: Observation vector of Ant problem
Observation
Degrees of Freedom
height of the base: h
rotation quaternion of the base
4
linear velocity of the base:
3
angular velocity of the base
joint angles
joint angle velocities
up and heading vectors projections
2
actions in last time step
8
Table 10: Observation vector of Humanoid problem
Observation
Degrees of Freedom
height of the torso:
rotation quaternion of the torso
4
linear velocity of the torso:
3
angular velocity of the torso
3
joint angles
21
joint angle velocities up and heading vectors projections
actions in last time step
21
The stnllestep reward is defined as:
R = R, + 0.1Rup + Rheading + Rheiiht - 0.002||a|]-
(20)
where R. = Us is the forward velocity, Rup is the projection of torso on the upward direction encouraging the agent to be vertically stable, Rheading is the projection of torso on the forward direction encouraging the agent to run straight forward, and Rheight is defined by:
2000 A.
A <0
Rheight
(21)
104 b:
A 0
A = clip(h - 0.84, 0.1
(22)
The initial state is randomly sampled. The task horizon is 1000 steps and early termination is triggered when the height of torso is lower than 0.74m.
A.3.6 HUMANOID MTU
This is the most complex problem designed to assess the scalability of the algorithms. In this prob- lem, lower body of the humanoid model from Lee al. (2019) is actuated by 152 muscle-tendon units (MTUs). Each MTU contributes one actation degree of freedom that controls the contractile force applied to the attachment sites the connected bodies. We have 53-dimensional observation space as shown in Table II and 152-dimensional action vector.
The single-step reward defined as:
R,, 0.1 + Rheading Rheight 0.001
(23)
where R, = U2 is the forward velocity, Rup = projection(upward direction) encourages the agent to be vertically stable, Reeading frojectionfforward direction) encourages the agent straight
19

---

Published as a conference paper at ICLR 2022
Table 11: Observation vector of Humanoid MTU problem
Observation
Degrees of Freedom
height of the pelvis: h
rotation quaternion of the pelvis
4
linear velocity of the pelvis:
3
angular velocity of the pelvis
joint angles
22
joint angle velocities
18
up and heading vectors projections
2
forward, and Rheight is defined by:
2200
An <0
Rheight
(24)
4 A. 2 0
A clip(h 0.51,
0.05)
(25)
The initial state is randomly sampled. The task horizon 1000 steps and early termination is triggered when the height of pelvis is lower than 0.46m.
A.4
MORE RESULTS
A.4.1
FULL LEARNING CURVE COMPARISON
In Section 4.3
provide the truncated version of the learning curves in Figure for better visu- alization. We provide the full version of the learning curves in Figure 7 where the curve for each algorithm continues to the end of the training of that algorithm. In addition, we include the experi- ment results for the other two classic RL environments, HalfCheetah and Hopper. From the figure, we can see that our method shows extreme improvements in sample efficiency over other methods and constantly achieves better policies given the same amount of training time.
A.4.2 WALL-CLOCK TIME BREAKDOWN OF TRAINING
We provide the detailed wall-clock time performance breakdown of single learning episode of our method in Table [2] Since we care about the wall-clock time performance the algorithm, we use this table for better analysis of which is the time bottleneck in our algorithm. The performance measured on a desktop with GPU model TITAN X and CPU model Intel Xeon(R) E5-2620 @ 2. 10.HHH As shown in the table, the forward simulation and backward simulation time scales up when the dimensionality of the problem increases, while the time spent in critic value function training almost constant across problems. As expected, the differentiable simulation brings extra overhead for its backward gradient computation. Specifically in our differentiable simulation, the backward computation time roughly 2x of the forward time. This indicates that our method still has room to improve its overall wall-clock time efficiency through the development of more optimized differentiable simulators with fast backward computation.
A.4.3 FIXED NETWORK ARCHITECTURE AND HYPERPARAMETERS
In the previous results, we fine tune the network architectures of policy and value function for each algorithm each problem to get their maximal performance for a fair comparison. In this section, we test the robustness of our method by using fixed network architecture and fixed set of hyperparameters (e.g. learning rates) to train on all problems. Specifically, architecture and the hyperparameters used in the Humanoid problem, and plot the training curves in Figure g
the same
20

---

Published as a conference paper at ICLR 2022
CartPole Swing Up
Ant
Humanoid
Humanoid MTU
0430
40.0
-1893
1000
Reward
+1.00
5000
Rew ard
Reward
4030
8000
3030
2000
-2203
2000
1800
simulation stops (x10")
simulation steps (x10"))
simulation steps (x110))
simulation steps (x100)
5000
0630
4093
-1300
5000
Reward
Reward
hore
Reward
Reward
8030
3800
-2500
2000
1890
Wall-clock time (minn
wall-olock time [min)
w wall-olock time (min)
wall-olock time (h)
HallCheetah
Hopper
5000
Reward
5000
Rew ard
saco
2000
simuletion steps (x10*)
simulation steps (x10%)
1000
1400
Reward
1000
Reiward
J00e
wall-olock time [min)
wall-olock time (min)
SHAC
Figure 7: Full learning curves of algorithms on six benchmark problems. Each curve is averaged from five random seeds
shaded area shows the standard deviation. Each problem is shown a column, with the plot on the top showing the sample efficiency and the curves on the bottom for wall-clock time efficiency. We run our method for 500 learning episodes on CartPole Swing Up. and for 2000 learning episodes for other three tasks. We run other algorithms for sufficiently long time for comprehensive comparison (either until convergence, up 2x of the training time compared hours in Humanoid MTU).
method first three problems, and up 20
Forward (s)
Backward (s)
Critic Training (s)
CartPole SwingUp
0.15
0.23
0.34
Ant
0.25
0.37
0.32
Humanoid
0.91
1.97
0.32
SNU Humanoid
0.82
1.66
0.33
Table 12: Wall-clock performance breakdown of a single training episode. The forward stage includes simulation, reward calculation, and observations. Backward includes the simulation gradient calculation and actor update. Critic training, which is specific to method, listed individually, and is generally small proportion of the overall training time.
A.4.4 DETERMINISTIC POLICY
Our method does not constrain the policy to be stochastic or deterministic. We choose to use the stochastic policy in the most experiments for the extra exploration that it provides. In this section, we experiment our method with the deterministic policy choice. Specifically, we change the policy of method in each problem from stochastic policy deterministic policy while keeping all
21

---

Published as a conference paper at ICLR 2022
CartPole Swing Up
Ant
Humanoid
Humanoid MTU
-400
0000
1000
Rowarrd
Roward
5ice
500
Reward
1840
-2000
3006
2030
2000
-2530
1004
1000
-2030
simosattio steps (x119)
simalation stops (x10f)
stamlatton staps (x110%)
stmalation steps (x109)
-500
3000
6490
-1830
reee
Rewsard
-1530
Reward
sxce
Reward
-2000
300
2002
100e
1080
wall clock time (mmn)
wall-clock
wall clock time (min)
wall clack time (min)
Wall clock time (b)
SHAC (oytimal awtling)
SHAC (Gasd setling)
Figure& 8: Learning curves of our method with fixed network architectures and learning rates. We use the same network architectures and learning rates used in Humanoid problem all other problems, and plot the training curves comparison with the ones using optimal settins.. The plot shows that our method still performs reasonably well with the fixed network and learning rates settings.
other hyperparameters such as network dimensions and learning rates the same as shown in Table 4 The training curves of the deterministic policy is plotted in Figure [ The results show that our method works reasonably well with deterministic policy, and sometimes the deterministic policy even outperforms the stochastic policy (e.g. Humanoid). The small performance drop on the Ant problem comes from one single random seed (out of five) which results in a sub-optimal policy.
CartPole Swing Up
Ant
Humanoid
Humamoid MTU
5300
Rewaadd
Rooware
5306
Reward
-2003
2000
-2500
-3091
simulation steps (x10P)
simuletion steps (x10*)
simulation steps (x109)
simulation steps (x10*)
-500
3000
1201
rean
tean
Roward
-3933
Resward
Rewaard
senc
-2800
-3531
1880
-2000
wall-book time (min)
wall-clock time (min)
wall-clock Lime (min)
well-clock time (b)
SHAC (stochotiic policy)
SHAC (dsterminiatic policy)
Figure 9: Learning curves of our method with deterministic policy. We test our method with deterministic policy choice. We use the same network sizes and the hyperparameters used in the stochastic policy and remove the policy output stochasticity. We run our method on each problem with five individual random seeds. The results show that our method with deterministic policy works reasonably well on all problems, and sometimes the deterministic policy even outperforms the stochastic policy (e.g.. Humanoid). The small performance drop on the Ani problem comes from one single seed (out of five) which results in a sub-optimu policy.
A.4.5 COMPARISON TO REDQ
In this section, we compare our method to another advanced model-free reinforcement learning algorithm, namely REDQ (Chen et al., 2021b). REDQ is an off-policy method and is featured by its high sample efficiency. We experiment the officially released code of REDQ on four of our benchmark problems: CartPole Swing Up. Ant, Humanoid, and Humanoid MTU, and provide the comparison between our method and REDQ in Figure 10 From the plots, we see that REDQ
22

---

Published as a conference paper at ICLR 2022
demonstrates its high sample efficiency in the CartPole Swing Up and Ant problems due its off- policy updates and the randomized Q function ensembling strategy. However, REDQ is extremely slow in wall-clock training time. For example, in Anr problem, REDQ takes 27 hours to achieve 5000 rewards while SHAC only takes 13 minutes to achieve the same reward. Such huge difference in wall-clock time performance renders REDQ less attractive method than SHAC in the settings where the policy is learned in a fast simulation. Furthermore, REDQ fails to train on our high- dimensional problems, Humanoid and Humanoid MTU problems.
A.4.6 COMPARISON TO MODEL. RL
In this section, we compare our method to model-based reinforcement learning algorithm. In typical model-based reinforcement learning algorithm, the learned model can be used in two ways: (1) data augmentation, (2) policy gradient estimation with backpropagation through differentiable model.
In the data augmentation category, the learned model is used provide rollouts around the observed environment transitions, as introduced in the DYNA algorithm (Sutton) 1990). The core idea being that the model can be used to provide additional training samples in the neighborhood of observed environment data. The core idea in DYNA was extended by STEVE (Buckman et al., 2018)) and MBPO (Janne et al.] 2019), which use interpolation between different horizon predictions compared to single step roll-out in DYNA. However, the performance of such approaches rapidly degrades with increasing model error. Notably, as noted by Janner et al. (019), empirically the one-step rollout is very strong baseline to beat in part because error in model can undermine the advantage from model-based data--ugmentttion.
In contrast the model can be used for valicy-value gradient estimation, which originates from
Nguyen & Widroww (1990) and Jordan & Rumelhart 1992). This idea has been extended multi-
ple times such as in PILCO (Deisenroth & Rasmussen 2011). SVG (Heess et al. (013). and MAAC
CClavera et al., 2020). Despite the core method being very similar to the original idea, the difference between recent methods such SVG and MAAC emerges in the details of what data is used in estimation. SVG uses only real samples, while MAAC uses both real and model roll-outs. But most of these methods still rely on model error being low for policy and value computation.
While SHAC resembles these gradient-base based methods, our model usage fundamentally differs from previous approaches. SHAC the first method that combines actor-critic methods with analyti- cal gradient computation from differentiable simulator model. The analytical gradient solves the model learning issues which plague methods both in model-based data aune--aattttoonn and direct gra- dient estimators. At the same time, SHAC prevents exploding/vanishigg gradients in BPTT through truncated roll-outs and a terminal critic function. SHAC offers improvement over prior model-based methods in both improved efficiency as well scale-up to higher dimensional problems, beyond the ones reported in these methods.
To further illustrate the advantage of our method over model-based RL, conduct the experiments to compare to Model-Based Policy Optimization (MBPO) ( (Janeer et al. 2019). We run the officially released code of MBPO on four of our benchmark problems: CartPole Swing Up, Ant. Humanoid, and Humanoid MTU, and provide the comparison between our method and MBPO in Figure I0] As shown in the figure, the MBPO with the default parameters does not work well mark problems except CartPole Swing Up. We hypothesize that it is because of the hyperparameter sensitivity of model-based RL method. Although a fair comparision can be further conducted by tuning the hyperparameters of MBPO, an implicit comparison can be made through the comparison with REDQ. In REDQ paper (Chen et al.) (2021b), it is reported that REDQ is up to 75% faster in wall-clock time than MBPO. This indirectly indicates that our method has significant advantages in the wall-clock time efficiency over MBPO.
bench-
A.5 MORE OPTIMIZATION LANDSCAPE ANALYSIS
In Figure 2
we show the original landscape of the problem and the surrogate landscape defined by SHAC. However the smoothness of the surrogate landscape sometimes does not necessarily results in smooth gradient when the evaluation trajectory is stochastic and the gradient is computed by averaging over policy stochasticity (Parmas et al., 2018). To further analyze the gradient of the landscape, we plot the comparison of the computed single-weight analytical gradient and the single-
23

---

Published as a conference paper at ICLR 2022
CartPole Swing Up
Ant
Humanoid
Humanoid MTU
1e91
1300
1400
Rewardd
-1500
Rewerdd
Roward
crea
1000
2403
-9891
stmulattio steps (x10%)
simolation steps (x101')
stmulation stops (x10*)
stmulation steps (x100)
0490
$300
-1893
Rewarrd
Rewert
Roward
3931
2493
2244
1003
-2800
wall-clock time (b)
wall clock time (h)
wall clock time (b)
wall clock time (h)
SHAC
REDO
MBPO
Figure 10: Comparison with REDQ and MBPO. We compare SHAC to REDQ and MBPO. While REDQ has very excellent sample efficiency, its wall-clock time efficiency very poor (102x slower than SHAC in CartPole Swing Up and 162x slower than SHAC in Ant), which renders it as a less attractive method than SHAC in the settings where the policy is learned in a fast simulation. The offical MBPO code works well on the CartPole Swing Up problem while works even worse than REDQ on other three tasks. Furthermore, high-dimensional problems, Humanoid and Humanoid MTU, with
both REDQ and MBPO fail to train default hyperparameters.
weight gradient computed by finite difference of the landscape (shown in Figure 2 Right) in Figure II The plot demonstrates that our computed gradient is well-behaved and close enough to the finite difference gradient.
Surrogate Landscape
Surrogate Gradient
fints difrewnnn grvfiert aralytical gradiest
A0
A6c
Figure 11: Single-weight landscape and gradient the SHAC loss. the left figure, we select one single weight from a policy and change its value by A0, E [-1, 1] to plot the task loss landscapes of SHAC w.r.t one policy parameter (same as FigureDrightt, The short horizon length is h = 32. Each policy variant is evaluated by 128 stochastic trajectories and we set the same random seed before the evaluation of cach policy variant. In the right figure, we plot the finite difference gradient and the analytical gradient along that single weight axis. shows that the analytical gradient bell-behaved and close enough to the finite difference gradient.
Furthermore, we compare the gradient of the BPTT method and the gradient of SHAC in Figure 12 where we picked six different policies during the training for the Humanoid problem, and plot the gradient value distribution of BPTT method and SHAC method for each of them. From Figure 12 we can clearly see that the gradient explosion problem exists in the BPTT method while the gradient acquired by SHAC is much more stable throughout the training process.
To better visualize the landscape, we also compare the landscape surfaces in Figure 13 Specifically, experiment on the landscapes for the Humanoid and Ant problems. For Humanoid, we select policy during training, and for Ant we use the optimal policy from training for study. We randomly select two directions 0j and 6g in the policy network parameter space, and evaluate the policy vari- ations along these two directions with weights 0' 0 + k k + k202, ki, k2 E 1-1, 1]. The figure shows that the short-horizon episodes and the terminal value critic of SHAC produce a surrogate landscape which is an approximation of the original landscape but is much smoother.
24

---

Published as a conference paper at ICLR 2022
BPTT
SHAC
Episode 0
-100
-30
100
150
gradkent
grodent
Episode
-2002
-1300
3002
2000
-033
-2.
-03
0.0
01
0.
03
camaity
Episode 800
-455
200
-30
-20
10
seaatt
Episode 1200
750
grodient valat
grodeet vaha
denett
Episode 1600
gruduet value
grudact
cendrr
Censty
Episode 2000
20030
grudiret cahua
Figure 12: Gradient distribution evolving over training process. We select six policies from different episodes during the training process for the Hunanoid problem and plot the gradient value distribution of BPTT (left) and SHAC (right). Note the z-axis scales the plots. The plots show that the gradient computed from BPTT suffers from gradient explosion, whereas the analytical gradient computed from our method much more stable.
25

---

Published as a conference paper at ICLR 2022
BPTT
SHAC
Humanoid
Ant
Figure 13: Landscape surfaces comparison between BPTT and SHAC. We study the landscapes for the Humanoid and Ant problems. For Humanoid, choose policy during the optimization and for Ant choose the best policy from the learning algorithm. each problem, we randomly select two directions the policy parameter space and evaluate the policy variations along these two directions. The task horizon H = 1000 for BPTT, and the short-horizon episode length for our method is h = 32. As we can see, longer optimization horizons lead to noisy loss landscape that are difficult to optimize, and the landscape of our method approximates the real landscape but is much smoother.
26


Reviews:
[{"review_id": "tMN_P29pV-", "reviewer": "Reviewer_4PEt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Idea is conceptually simple and provides strong results on a spectrum of established benchmark environments as well on one very high dimensional task.\n* Shows the potential of backward differentiation through time.\n\nWeaknesses:\n* Could be run on other benchmark environments, which might highlight more the limits of the proposed approach.\n* The analysis of why and whether the loss landscape is smoother could be more elaborate. Additional figures like for example a histogram of the gradient distribution evolving over time would, in my opinion, help the reader to have better insight than Figure 2.\n\nQuestions:\n* How important is the structure of the reward function for this approach?\n* Have you found out on which kind of environments the proposed method fails?\n\nAdditional Remarks:\n* Typo in the introduction: high-dementional\n* I found the truncation of the curves in Figure 4 visually misleading at first.\n* If section 3.2 would be more elaborate and the experiment would include more environments, this paper could be further improved.", "summary_of_the_paper": "This paper demonstrates the effective use of differentiable simulators for significantly speeding up the training of policy gradient methods. The main idea is to backpropagate through the simulator only on shorter partitions of the trajectory, which seems to lead to a smoother loss landscape and better gradient signal. To avoid getting stuck in local minima, a critic is used to incorporate a terminal cost. As the implementation can be parallelized and the simulation run on the GPU, the experiments show drastic improvements not only in sample efficiency but also in run time.", "summary_of_the_review": "This paper shows that differentiable simulators can be leveraged to significantly speed up training even on very challenging high-dimensional tasks, making it an interesting empirical and conceptual contribution. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "rMO1bWh85-", "reviewer": "Reviewer_Hcaa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper introduces an interesting idea of leveraging simulator derivative information to achieve fast and sample efficient reinforcement learning. The idea is well presented and I found the empirical results very convincing. However I do find a few shortcomings that I hope the authors can address.\n\n\nPros\n\n\n1. The empirical results of the proposed method are very strong. It can be clearly seen that the proposed method outperforms baselines significantly. I consider this as the main advantage of this paper, since the proposed method and the differentiable simulator could serve as good foundations for future work.\n\n\n2. The PyTorch GPU based differentiable simulator that supports complex environments is another important contribution of the paper. If released, it could be impactful for the field as it enables researchers to obtain controllers for complex systems much faster.\n\n\n3. The paper is well written. The overall structure of the paper is well organized and the presentation of the main contribution is easy to follow.\n\n\nCons\n\n\n1. I’m not convinced about the novelty of the proposed method. The proposed method is mostly the same as the method introduced in [1], where the only difference seems to be that the proposed method uses exact gradient from the simulator instead of gradient of a learned dynamics model. Although the proposed method is not new, a major novelty of this paper could come from the creation of the differentiable simulator that supports complex dynamics models such as the Humanoid MTU used in this paper. However unfortunately the simulator implementation is not discussed in detail in this paper.\n\n\n2. The sample efficiency of off policy reinforcement learning methods has also improved a lot in recent years, where some model free methods can even achieve sample efficiency close to that of model based methods ([2]). It would be important to compare to some of these methods, because this comparison would tell us how much improvement can we get from using the derivative information of the simulator.\n\n\nGiven these limitations, I cannot recommend acceptance of this paper at its current state. I highly encourage the authors to continue improving the paper by providing more discussion regarding the simulator and adding the comparison to more recent reinforcement learning methods.\n\n\nReferences\n\n\n[1] Clavera, Ignasi, Violet Fu, and Pieter Abbeel. \"Model-augmented actor-critic: Backpropagating through paths.\" arXiv preprint arXiv:2005.08068 (2020).\n\n\n[2] Chen, Xinyue, et al. \"Randomized ensembled double q-learning: Learning fast without a model.\" arXiv preprint arXiv:2101.05982 (2021).\n\n\n## Update after Authors' Response\nThe authors have updated the paper to include more description of the simulator and the missing references. Focusing more the the differentiable simulator, the manuscript now reflects the contribution of the work more accurately and therefore I believe it is a valuable addition of the community. Now I recommend acceptance of this paper.\n", "summary_of_the_paper": "This paper focuses on the setting of reinforcement learning with differentiable simulation, where the authors assume access to a simulator which can compute the derivatives of the next state and reward with respect to the current state and action exactly. Under this setup, the authors propose an actor-critic style reinforcement learning algorithm that leverages the simulator derivative information to achieve better training sample efficiency and wall-clock time. Specifically, the authors expand the value function of the policy in h steps, and combine the sum of the simulator gradient of these h steps and the gradient of the learned value function at the h + 1 step to obtain the overall gradient for training the policy. The value function is then trained in the same way as in other actor-critic style methods.\n\nThe authors evaluate the proposed algorithm empirically on a few domains and the experiment results suggested that the proposed method achieves superior sample efficiency and wall-clock time compared to prior methods. \n\n", "summary_of_the_review": "The paper presents an intuitive reinforcement method for leveraging derivative information from a differentiable simulator. The methods significantly outperforms prior methods in terms of both sample efficiency and wall-clock time. However, the proposed method is not novel as it is largely the same as that introduced in a prior work, and it is missing an important baseline comparison. Therefore I cannot recommend acceptance of the paper in its current state.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L72ig3zxfB", "reviewer": "Reviewer_5Q2y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The experimental results are impressive, and show the utility of using differentiable models.\n- The method is sensible and well-motivated.\n- Clarity is generally good, and method is explained in sufficient detail.\n- The experimental procedures seemed well done, and they included ablation studies on the necessity of the critic and the used short horizon length.\n\n**Weaknesses**\n- Novelty is somewhat low.\n- Discussion of gradient explosion seems inaccurate.\n- Some discussion of related work is missing, particularly about related works in the model-based RL literature.\n- Sometimes they are stating things as facts without providing evidence.\n- There was no detailed discussion of how the results with the current simulator relate to other recent simulators such as Brax. Are the wallclock times similar? The results seem mixed, but having a detailed discussion would have been useful.\n- The simulator is fast, so it may have been better to do more than 4 environments.\n\n**Recommendation**\n\nI recommend accepting the paper as I have not seen differentiable simulators used for tasks of the difficulty considered here (while also\ntaking advantage of differentiating through the simulator). I was considering a score of 6 or 8, but gave 6 for now.\n\n**Discussion of points brought up**\n\nNovelty:\nIt seems the contribution is primarily one of engineering, and they don't propose any surprising new idea. The idea of truncated backpropagation is old. Moreover, the policy training scheme resembles that of the the Dreamer algorithm (Hafner et al. 2019). Dreamer also uses short horizon rollouts together with a terminal value function, and backpropagates through these short horizons to optimize the policy. The differences are only: Dreamer uses lambda return weighting during the short horizons (why didn't you use this?), and the method of constructing the sequences is different (Dreamer samples start states from a replay buffer and performs model rollouts from these states, while the current work splits an episode into chunks). I am surprised that Dreamer was not discussed when explaining the methodology. Certainly it should be mentioned that there are prior works using a similar policy optimization procedure (with slight variations).\n\nDiscussion of gradient explosion:\nThere are other earlier works that do a more detailed job of discussing the gradient and landscape issues, such as PIPPS (Parmas et al. 2018), which should have been cited and discussed (moreover your methodology was very similar to these previous works). In your work, if gamma were 1, the value function were perfect and the policy were deterministic, the gradient you compute with your method should be exactly the same as the gradient that is computed using BPTT. From this point of view, your discussion is insufficient, as you do not explain why the loss landscape and gradients end up being smooth despite this fact. I can think of two possible reasons: 1. The value function is an approximator that ends up being smooth because of limited capacity to model the complicated landscape. 2. You are using a stochastic policy, and this stochasticty smooths out the landscape that the value function is estimating; hence it becomes smooth. However, one of the points brought up by Parmas et al. (2018) was that even if the landscape is smooth (due to averaging over policy or model stochasticity) the gradients computed by backpropagtion can be ill-behaved and lead to an explosion of the gradient variance.\n\nFor the other points see the raw notes at the bottom of this section.\n\nParmas, P., Rasmussen, C. E., Peters, J., & Doya, K. (2018,\nJuly). PIPPS: Flexible model-based policy search robust to the curse\nof chaos. In International Conference on Machine Learning\n(pp. 4065-4074). PMLR.\n\n**Questions**\n\nQ1. In section 3.1 you write that you built the simulator. But from section A.3.1. it seems you just used Isaac Gym. So which one is it: did you make the simulator or did you use Isaac Gym?\n\nQ2. In Figure 2, did you use a deterministic or stochastic policy? Was this the same policy as was used during training the value function? If the policy was stochastic, then how did you evaluate the landscape? This would require sampling many trajectories with the same policy and averaging. Are the scales on the left and right figures the same?\n\nQ3. Did you do any ablation study of the policy noise? Does the method still work when the policy is deterministic? How much does the performance drop?\n\n**Additional notes made during reviewing**\n\n\"for systems ranging from robots (e.g., Cheetah, Shadow Hand) to\ncomplex anima- tion characters (e.g., muscle-actuated humanoids) using\nonly high-level reward definitions.\"\nPlease provide references.\n\n\"A differentiable simulator provides accurate first-order gradients of\nthe task performance reward with respect to the control inputs.\"\nThis is speculation. You provide no evidence. Problems with accuracy\ncan arise when the task performance depends on a sampled initial\nstate (so that the gradients are inherently stochastic).\nPerhaps change to \"may provide\".\n\n\"However, despite the availability of differentiable simulators, it\nhas not yet been convincingly demonstrated that they can effectively\naccelerate policy learning in complex high-dementional and\ncontact-rich tasks, such as some traditional RL benchmarks.\"\nWhile this may be the case for differentiable simulators, there are\nseveral model-based RL works that showed effective learning (e.g.,\nDreamer. The difference with a simulator is just that\nthe model does not have to be learned. I think the claim here is\ndownplaying such previous contributions. Also, \"dementional\" should be\n\"dimensional\".\n\n\"There are several reasons for this: 1.(), 2.(), 3.()\"\nThese reasons are stated as facts, while they are speculations.\nPerhaps, \"possible reasons\" would be better. At least points 1 and 3\nshould be the same for model-free RL, so are they really the reason?\nNo references were provided.\n\n\"Because of these challenges, previous work has been limited...\"\nHow do you know that those were the challenges that limited the\napplicability of the previous methods? The publications themselves\ndo not seem to note your reasons as the reason why they limited their\nexperiments. For example, the PODS paper says their method overcomes\nthe issues of exploding gradients. \n\n\"In addition, we propose a truncated learning window to shorten the\nbackpropagation path to address problems with vanishing/exploding\ngradients and reduce mem- ory requirements.\"\nThis is known as \"truncated backpropagation\". It is plagiarism to\nclaim that you \"proposed\" this.\n\nIn Equation 1, please provide the definition of the Jacobian's and\ngradients. Usually, the gradient is a row vector, whereas you are\nusing a column vector.\n\n\"This makes the reward function smoother...\"\nWhat do you mean by this? The reward function is the same in all cases.\n\n\"In addition, we apply state normalization as is common in RL\nalgorithms.\"\nPlease explain what \"state normalization\" is.\n\n\"First, the terminal value function absorbs the discontinuity of long\ndynamics horizons and early termination into a smooth function, as\nshown in Figure 2 (Right).\"\nThis explanation is incomplete. If you had no discount factor,\na perfect value model and a deterministic policy, your computed\npolicy gradient would be exactly the same as that of BPTT.\n\n\"Finally, the use of short horizons allows us to update the actor more\nfrequently, which, when combined with parallel differentiable\nsimulation, results in a significant speed up of training time.\"\nDo you have an ablation study showing that it speeds up?\n\n\"In contrast, our method scales well due to direct access to the true\ngradients from differentiable simulation.\"\nYou don't have access to true gradients when you are using a\nstochastic policy. It may be better to explain this by referring to\nthe fact that reparamterization gradients are often more accurate\nwhen computing gradients of smooth functions.\n", "summary_of_the_paper": "They implement a differentiable simulator based on PyTorch (actually it was a bit unclear to me whether they implemented it, or just used Isaac Gym; I have a question about this below, and hope to receive an answer from the authors), as well as a method to utilize the differentiable simulation to optimize the policy more efficiently. The method operates as follows:\n1. It computes a batch of episodes each of length H using a stochastic policy.\n2. It splits the episodes into subsequences of length h.\n3. It adds a terminal value function to each sequence, as well as discount factors on the rewards (starting from no discounting at the beginning of each subsequence), and computes policy gradients via backpropagation for optimization.\n4. The value function is learned by computing lambda return targets (note that lambda returns are not used in the policy optimization though).\n\nThere is also some discussion about exploding gradients that hinder optimization through long horizons, which is why they truncated their episodes into subsequences.\n\nThey perform experiments on cartpole swingup, ant, humanoid and humanoid MTU (a humanoid actuated by 152 muscle tendon units), and compare to PPO, SAC, Backpropagation through time and PODS. The proposed method (SHAC) outperformed the other methods both in terms of number of steps (by a large margin) as well as computation speed (though PPO was faster in terms of wall-clock time in the early stage of learning after which SHAC overtook it). Particularly on the humanoid MTU task, the final performance was much better than the other methods. In terms of wall-clock time, the required computation time was a few minutes for cartpole and ant, and around 1h for the humanoid tasks.\n\nThey also performed ablation studies looking into the necessity of the terminal value function (it was necessary) and also the length of the subsequences (intermediate length sequences were optimal).\n\nRecently some similar simulators, e.g. Brax or Isaac Gym have been made, but these works have not yet shown impressive results by taking advantage of differentiating through the model.", "summary_of_the_review": "The main convincing result from the paper are the empirical results showing that using a differentiable model speeds up learning by a large margin in terms of step count on challenging tasks. I have not seen such results using differentiable simulators before, so I think it warrants publishing. The work itself seems primarily an engineering contribution: the methodology and ideas are not particularly novel, but executed in a sensible way. The quality of the work was generally good. Some more related work could be discussed, and the writing could be improved to be more precise.\n\n______________________________________________________________\n**Update**\n\nI have increased my score to 8, novelty & significance score to 3, correctness to 4, and confidence to 4.\n\nThe authors have done a good job of addressing the reviewer concerns, and have made several changes to the manuscript, including adding more references; running more ablation experiments, e.g. experiments with a deterministic policy; added 2 new environments, etc.\n\nI also note that their approach by differentiating through the trajectory and adding a terminal value function is not exactly the same as previous methods, such as Dreamer or MAAC. These previous works sample start states from a replay buffer of real data, and perform short rollouts from these states. The current work, on the other hand, operates on-policy, and it samples the start states from the end of the previous short rollout (this process is repeated until the total length of the short rollouts exceeds the episode length, and then the episode is restarted).  The policy is updated after each rollout, resembling a more typical truncated backpropagation procedure, but while adding a terminal value function. I think such an approach is tailored to using a simulator, because with a model-based approach, the predictions may diverge from the real trajectory when the task horizon is too long. On the other hand, if using a replay buffer based approach, such as is used in Dreamer or MAAC, the simulations to generate the replay buffer may be a waste of computational resources, so the approach presented in this paper seems like a natural idea to use in the setting when one has a simulator, and is not a direct copy of the previous works. The authors have also added a discussion of previous model-based RL methods compared to their work into Appendix A.4.6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3IzY97_Ggc", "reviewer": "Reviewer_nSwn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The paper is clearly written and well organized.\n* The differentiable simulator introduced will likely have great value to researchers, as not only does it have the ability to be differentiable, but being built on PyTorch means that it is likely more interpretable and customizable than similar methods [1], and is also natively parallelizable, which will result in massively reduced experimentation times.\n* The authors show very convincing performance of their introduced SHAC method on this suite of benchmarks compared to competitive methods, including non-analytical policy gradient approaches (PPO/SAC), as well as other methods that leverage differentiable environment dynamics.\n\nCons:\n* Key design elements and motivations in the SHAC method have appeared in different guises in the literature before, and some of these aren't referenced. For instance, in MAAC [2], they too have an analytical policy gradient (APG) (due to differentiability of the world model), and then prevent issues of long-horizon BPTT by including a truncating value function. Going further back, in [3] the idea of terminating rollouts using a value function is also used/introduced (in their case also to improve computation time). Indeed, the paper cited by the authors [4] which uses 'enhancement' can be viewed as a one-step analytical policy gradient with termination. Perhaps the authors could describe why having a state-value terminal function (instead of an action-value function which supports action gradients) is beneficial in their setting? Also I am somewhat concerned by the use of different architectures for each task; could the authors run some sensitivity analysis by keeping architecture fixed for each task to demonstrate the robustness of SHAC to this?\n* Similarly, given the relative simplicity of the SHAC method, it's somewhat disappointing that the authors only introduced 4 environments, and no manipulator-based contact environments (which would surely be a good fit for the goal of smoothened contact physics). In my opinion, focus should have been placed more on either the method or the simulator, but at this point for me neither display the requisite gravitas. I would like to see some additional environments introduced, such as a Cheetah and Hopper derivatives, and this would help the adoption of such an environment to a wider RL audience.\n* Are the smoothness assumptions made in the simulator actually transferable to real life robotics? There should be discussion as to whether or not the modelling choices made are realistic (e.g., smooth joint limits, choice of differentiable contact physics). As has been pointed out in the past, non-APG methods can be successfully deployed on real robots without requiring differentiability [5], whereas APG approaches will need to to get over the issue of `sim2real`. I can see this in fact being further exacerbated by the differentiability of the simulator, which would allow the policy to very effectively exploit the simulator to obtain great solutions that don't transfer (for instance, see [6] where the solutions are able to exploit deficiencies the MuJoCo simulator). In this case, perhaps the humanoid MTU solutions of SAC aren't as optimal in the simulator, but could transfer better IRL. Is there a way the authors could show that this exploitation is avoided in their simulator?\n* I found the single weight perturbation method to demonstrate smoothness slightly odd as there is abundant literature that aims to visualize loss surfaces, such as [7]. Why did the authors elect to use their single weight perturbation approach?\n\nRefs:\n\n[1] Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation: Freeman et al., arXiv:2106.13281\n\n[2] Model-Augmented Actor-Critic: Backpropagating through Paths: Clavera et al., arXiv:2005.08068\n\n[3] Value Function Approximation and Model Predictive Control: Zhong et al., ADPRL ‘13\n\n[4] Efficient Differentiable Simulation of Articulated Bodies: Qiao et al., arXiv:2109.07719\n\n[5] Soft Actor-Critic Algorithms and Applications: Haarnoja et al., arXiv:1812.05905\n\n[6] On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning: Zhang et al., arXiv:2102.13651\n\n[7] Visualizing the Loss Landscape of Neural Nets: Li et al., arXiv:1712.09913\n", "summary_of_the_paper": "In this paper, the authors introduce a novel differentiable simulator (built on PyTorch) that focuses on enabling smooth gradients with respect to expected return, especially with a view to ameliorating discontinuities in contact dynamics. In order to leverage this differentiable engine as effectively as possible, they also introduce SHAC, which is an analytical policy gradient (APG) method that features a terminal value function to truncate the value gradient calculation through time. The authors' show this results in a smoother objective function with lower training time, which empirically gives rise to faster/improved convergence in policy learning.", "summary_of_the_review": "Overall the paper introduces a novel differentiable environment suite, and aims to ameliorate a key issue in analytical policy gradients. However I believe there are deficiencies in both these things that I have listed above, so recommend a weak rejection for now.\n\nHaving said this I believe this paper has great promise, and think these issues are not major, and am happy to raise to an accept if they are sufficiently addressed.\n\n--------------- POST REBUTTAL ------------------\n\nThe authors have addressed most of my concerns, including the sensitivity of their approach to architecture, and included new environments aligned with canonical continuous control problems. Furthermore they focused more on the simulator in their updates, which is where I believe most of the impact of this paper lies, and produced additional interesting analysis about loss landscape smoothness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is consensus in the reviews that this paper convincingly demonstrates strong acceleration of policy learning using differentiable simulation in tasks involving contact-rich dynamics. The authors are encouraged to explore where the smoothness assumptions made in the simulator actually transfer to real robots. The paper may be further strengthened through more complex benchmarks involving contact rich manipulation.
Decision: Accept (Poster)