Complex Locomotion Skill Learning via Differentiable Physics
Authors: Jiancheng Liu
Yu Fang
Mingrui Zhang
Jiasheng Zhang
Yidong Ma
Minchen Li
Yuanming Hu
Chenfanfu Jiang
Tiantian Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: differentiable physics
locomotion skill learning
physical simulation

Abstract: Differentiable physics enables efficient gradient-based optimizations of neural network (NN) controllers. However, existing work typically only delivers NN controllers with limited capability and generalizability. We present a practical learning framework that outputs unified NN controllers capable of tasks with significantly improved complexity and diversity. To systematically improve training robustness and efficiency, we investigated a suite of improvements over the baseline approach, including periodic activation functions, and tailored loss functions. In addition, we find our adoption of batching and a modified Adam optimizer effective in training complex locomotion tasks. We evaluate our framework on differentiable mass-spring and material point method (MPM) simulations, with challenging locomotion tasks and multiple robot designs. Experiments show that our learning framework, based on differentiable physics, delivers better results than reinforcement learning and converges much faster. We demonstrate that users can interactively control soft robot locomotion and switch among multiple goals with specified velocity, height, and direction instructions using a unified NN controller trained in our system.

Under review as a conference paper at ICLR 2022
COMPLEX LOCOMOTION SKILL LEARNING VIA DIFFERENTIABLE PHYSICS
Anonymous authors Paper under double-blind review
ABSTRACT
Differentiable physics enables efficient gradient-based optimizations of neural network (NN) controllers. However, existing work typically only delivers NN controllers with limited capability and generalizability. We present practical learning framework that outputs unified NN controllers capable of tasks with sig- nificantly improved complexity and diversity. To systematically improve training robustness and efficiency, we investigated suite of improvements over the base- line approach, including periodic activation functions, and tailored loss functions. In addition, we find our adoption of batching and modified Adam optimizer ef- fective in training complex locomotion tasks. We evaluate our framework on dif- ferentiable mass-spring and material point method (MPM) simulations, with chal- lenging locomotion tasks and multiple robot designs. Experiments show that our learning framework, based differentiable physics, delivers better results than reinforcement learning and converges much faster. We demonstrate that users can interactively control soft robot locomotion and switch among multiple goals with specified velocity, height, and direction instructions using unified NN controller trained in our system.
INTRODUCTION
MLS-MPM
Mass Spring
Agents
Running (at controllaule velocity)
Jumping (at controllable height)
Tasks
Figure 1:
Our learning system is robust and versatile, supporting various simulation methods, robot designs, and locomotion tasks with continuously controllable target velocity and heights.
Differentiable physical simulators deliver accurate analytical gradients of physical simulations, opening up promising stage for efficient neural network (NN) controller training via gradient descent. Existing research demonstrates that on simple tasks, learning systems via differentiable physics can effectively leverage simulation gradient information and converge orders of magnitude faster (see, e.g., de Avila Belbute-Peres et al. (2018); Hu et al. (2019) than reinforcement learning.
However, the capability of existing differentiable physics based learning systems is relatively lim- ited. Typically, optimized controllers can only achieve relatively simple single-goal tasks (e.g., moving in specific direction, in ChainQueen Hu et al.] (2019)). Those learned controllers often have difficulty generalizing the task to perturbed version.
In this work, propose learning framework for complex locomotion skill learning via differen- tiable physics. The complexity of our tasks comes from two aspects: first, the degrees freedom of our soft agents significantly higher compared with the rigid-body ones; second, the agents expected to learn multiple skills at the same time. We systematically propose a suite enhance- ments (Fig. 4 over existing training approaches (such Hu et al. (2020): Huang al. (202 to

---

Under review as a conference paper at ICLR 2022
significantly improve the efficiency, robustness, and generalizability of learning systems based on differentiable physics.
In addition, investigated the contributions of cach enhancements training in detail by series of ablation studies. We also evaluated our framework through comparisons against Proximal Policy Optimization (PPO) [Schulman et al. (017), state-of-the-art reinforcement learning algorithm. Results show that our system is simple yet effective and has a much-improved convergence rate compared PPO.
We demonstrate the versatility of our framework via various physical simulation environments (mass-spring systems and the moving least squares material point method, MLS-MPM (uu et al. (2018), robot designs (structured and irregular), locomotion tasks (moving, jumping, turning around, all controllable velocity). To the best of our knowledge, this is the first time when neural network can be trained via differentiable physics to achieve tasks of such complexity (Fig. [ Our agent simultaneously trained with multiple goals, the learned skills can be continuously in- terpolated. For example, we can control both the velocity and height of an agent while it runs. The trained agent can also controlled real-time, enabling direct applications soft robot control and video games. In summary, our key contributions are listed below:
To the best of our knowledge, we proposed the first differentiable physics based learning frame- work for multiple locomotion skills learning soft agents using single network.
We developed an end-to-end differentiable physical simulation environment for deformable robot locomotion skills learning, which supports mass-spring system and material point method as dy- namic backends.
We systematically investigated the key factors contributing to the differentiable physics based learning framework. We believe investigation inspire further researches explore the possibilities of differentiable physics for more complex learning tasks.
2
DIFFERENTIABLE SIMULATION ENVIRONMENTS
2.1 SIMULATION SETTINGS
Our learning framework supports multiple types of differentiable physically-based deformable body simulators. The simulator can be seen as a black box that outputs the states for the next time step given the actuation signal and states from the current time step. The NN controller takes the in- formation given by the simulator as part of its input features and decides the actuation signal for the simulator the next time step. With our automatic differentiation system, we can evaluate the gradient information to guide the training of our NN controllers. We use the mass-spring systems and the moving least squares material point method (MLS-MPM) as our simulation environments. The design choices for each environment can be found in the appendix.
2.2
AGENT STRUCTURE AND ACTUATION MODE
Figure 2: 3D agents collection. These agents are designed with simple stacked cubes or complex handcrafted meshes.
Our framework supports a variety of differently shaped agents as shown in Fig. and 3
Our actua-
tion signal exerts forces along with the "muscle" directions of the simulated agents. This signal falls within the interval [-1, +1], where its sign determines whether an agent wants to contract (negative sign) or relax (positive sign) muscle and its absolute value determines the magnitude of the force. In mass-spring systems, the muscle directions are represented by the spring directions, we change the rest-length of springs to generate forces. The actuation signal used scale the rest-length up
2

---

Under review as a conference paper at ICLR 2022
Alpaca
Monster Figure 3: 2D agents collection.
HugeStool
Stool
to some limit (usually at 20%). In addition, only activated springs, i.c. actuators (springs marked red or blue shown in Fig. [T). are able to generate forces according to control signals. In material point methods, the muscle directions mentation, we simply modify the Cauchy stress the vertical direction of material space to apply this force. Again, the force is scaled user-defined bound which can be adjusted for different simulations.
always along the vertical axis in material space. In our imple-
3 LEARNING FRAMEWORK
Time Step
Time Step
Time Step N-1
SIREN-
SIREN-
SIREN-
TaskO Loss
Rest Pose
Enhanced Controller
Differentiable
Aimulattion
Enhanced Controller
Differentiable
Aimulatiion
Enhanced Controller
Difierentiable Simulation
Task1 Loss
Task2 Loss
Extract
Extract Features
Extract Features
Continutus State
Features
Task' Loss
State
State
State N-1
1... Fredifakk
Figure 4: Framework overview. Simulation instances are batched and executed in parallel on GPUs. The whole system is end-to-end differentiable, and we use gradient- gradient-based algorithms to op- timize the neural network controller weights. Each time step involves evaluating NN controller inspired by SIREN (Sitzmann et al. (2020), and differentiable simulator time integration imple- mented using Difi'aichi Hu et al. (2020). Simulation states are fed back to the initial state pool to improve the richness of training sets and thereby the robustness of the resulted NN controller. A tailored loss function (as shown in section 3.2) is designed for each task.
In this section, we describe our training framework based on differentiable physics (Fig. 4). In summary, we develop differentiable simulator with embedded NN controller. In each time step, the program performs an NN controller inference and differentiable simulator time inte- gration. After few hundred time steps, the program pnoo-propagates gradients end-to-end reverse-mode automatic differentiation, and updates the weights of the neural network using the Adam optimizer Kingma Ba (2014).
3.1 TASK REPRESENTATION
Given an agent A, its position and corresponding velocity are denoted x (x, y. and (u, v, w). We presents three tasks (running, jumping and crawling) for 2D agents, two (running and rotating) for 3D. whole simulation with total steps T. an agent is expected to achieve all goals in goal sequence G = [G,. G.......IIn2 goal, is the number of goals For each goal G. there are multiple tasks in it. Each task is encoded using a target value which instructs the controller to drive the agent. For example, G {gv, gh represent performing running and jumping simultaneously, the target velocity and height are ge and Ih. The agent is expected switch between multiple goals during one simulation.
T). where P is the time period for an agent to achieve
The running task is defined as velocity tracking problem. Given time step t, the center mass is used to represent the position of the agent, its velocity defined as the averaged velocity over time period P,.. (P, < P). Then the agent is expected to achieve velocity v close the target velocity do.
Running.
v(t)
P, (c(t) c(t P.))
3

---

Under review as a conference paper at ICLR 2022
Jumping. The jumping task is defined as reaching given height 94 in time period P. Due the gravity, it is not possible for a running robot to stay away from the ground. Therefore, the jumping height hy (t) of an agent defined by the maximum vertical position its lowest point in a certain period P. To be more formally, given S set of nodes (or points) that constitute the agent, h is function that can extract the vertical position given a node (or point), the jumping height over the n-th period is defined
hy (n)
max min h +P) te|u,P| sES
(2)
where n. is the index of the period in whole simulation.
Crawling. The crawling task is defined as lowering the highest point as much as possible. The target of crawling 9r is defined as an indicator function. The agent is controlled to crawl g. = otherwise if 9c O. Since crawling is status that needs to be maintained, the crawling height is defined as the vertical position of its highest point at each time step.
(t) max ((s t) sES
(3)
3.2 LOSS FUNCTIONS
Consider locomotion task where an agent is instructed to move at specified velocity, we think proper loss function should satisfy the following requirements:
Periodicity. The agent is expected to move periodically following predefined cyclic activa- tion signal. Therefore, the loss function should encourage periodic motions.
2. Delayed evaluation. Due inertia, it takes time for the agent start running adjust running velocity. Therefore, an ideal loss function should take this delay into consideration.
3. Fluctuation tolerance. During a single running cycle, requiring the center of mass to move a constant velocity at each time step may induce highly fluctuated losses. To avoid that, our loss function should be smooth during one motion cycle.
Task loss. For each task, we defined tailored loss shown in equation 4 according to the require- ments above. The loss for running C, is defined as the accumulation of the difference between the target and the agent's velocity from start to current time step. For jumping, define a sparse loss L,. which only evaluate once for each period. The crawling loss L. defined as the accumulation of the crawling height if the loss is applicable.
p
P ΣV ge (n) +PP) (4)
C= AM Σ Σ (v(t) - (n))) +Ar RET
(m) - -m))))))((()) +hc
nET
RET RET 1=0
The Any Ab, de are weights for losses of dfferent tasks. Our loss function accumulates the contribu- tions of all steps in a "sliding window".. This prevents the loss function from being too small as well the vanishing gradient problem. We find that having an accumulated loss evaluated every step works much better when we want to control the speed of the agent explicitly.
Regularization. Since the agents may keep shaking when target velocity or height are given, we introduce an actuation loss as regularization term. The intuition is to penalize comparatively large actuations when small goals value is given:
C, Σ Σ (A(t) (ll..
(5)
RET (E).
where A(t) is the actuation output of NN, μ is normalizer. The actuation loss balances the impor- tance of input channels the NN, i.e., different input channels produce similar contributions to the first layer of the trained NN.

---

Under review as a conference paper at ICLR 2022
3.3 NETWORK ARCHITECTURE
We use two fully connected (FC) layers with sine activation function as the neural network. The network input vector consists of three parts:
1. Periodic control signal of the same period with different phases, encourage periodic actu- ation. In this work, we use sin waves as the periodic control signal.
State feature vector extracted from the current state of the agent. Using the positions and velocities of all the vertices of the simulated object as the input feature is not good idea since these quantities are neither translationiivariinn pose" to remove global translation and rotation information. To be more specific, position the center of mass (CoM) in both 2D and 3D cases to remove the global translation. In 3D cases, also compute the rotation around the vertical axis of the agent to remove its global rotation.
nor rotation-invariant. We use a "centralized
subtract the
3. Targets that encode the task information such as running velocity, jumping height, and ori- entation for 3D agents. Different tasks different channels represents their target values. During the training, we assign random values to these targets. When validating the model, we al- ways test the agent values to control the agent's s motion. To amplify the importance of the targets, we duplicate these channels multiple times, try to make them comparably important among the other input features.
fixed of target values. In interactive settings, assign various target
3.4 TRAINING
In the training process, we run one simulation in one training iteration. The duration of the sim- ulation divided into several periods as mentioned in the section 3.1 each P, a goal G contains several tasks is assigned. For example, G = (ge- 9h represents that the agent is expected to perform both running and jumping in this period. The goal and its target values are generated randomly in uniform distribution. They are kept fixed inside one period P, but varied between different periods.
4 EXPERIMENTS AND ANALYSIS
this section, we systematically investigate the key factors that contribute to the training, evalu- ate our framework through ablation studies and comparisons against PPO Schulman et al. (2017). a tf-t-e-rrtttttttt reinforcement learning algorithm. All experiments the ablation studies are per- formed at least three times. The results of ablation studies summarized in Table I and Fig.
Ablation Study on Alpaca
Batch Size Ablation
Normlized Validation Task Loss
1.00
Normlized Validation Task Loss
1.00
Batch size
Batch size 32
0.95
Batch 64
Difitaichi
0.90
Rull-8S
0.90
Rull
0.85
Rul-TG
Rull-VV
Full-PS
0.80
0.80
Full-LD
0.75
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Iteartions
Iteartions
Figure 5: Summary of the ablation study. Here we show the summary of ablation study on agent Alpaca. Each iteration indicates training iteration, which is composed of 1000 steps of physical simulation and one step of update on weights of the controller. The result show that the Full method achieves the best performance. For more results of ablation studies on other agents, please check the appendix.
5

---

Under review as a conference paper at ICLR 2022
Training Setting
Norm. Valid. Loss (Alpaca)
Name
OP
AF
BS
PS
SV
TG
LD
Run.
Jump.
Task
Full
Adam
sin
32
v
0.58+00.01
0.79++0.01
0.74+0.001
Difftaicii*
SGD
tanh
v
V
V
V
0.98+0.00
0.95+0.01
0.96++0.01
Full-BS
Adam sin
V
V
/
V
0.82+0002
0.85+0.01
0.84+0.02
Full-OP
SGD
sin
32
V
V
V
V
0.97
0.98
0.98
Full-AF
Adam
tanh
V
V
0.67+00.02
0.81
0.77
Full-PS
Adam
sin
32
X
V
0.97
0.99
0.99+0.00
Full-SV
Adam
sin
V
X
V
0.54
0.84#0.01
0.77#0.01
Full-TG
Adam
sin
32
V
V
0.99
0.95#0.00
0.97
Full-LD
Adam
sin
32
X
0.99+0.001
0.74+00.01
0.82+0.01
Table Ablation summary on agent Alpaca. this table, we show the Task, Run and Jump val- idation loss of proposed method and its ablated versions. Full represents the proposed method. The abbreviations under training setting represent name of components remove. To be more specific, OP: Optimizer, AF: Activation Function, BS: Batch Size, PS: Periodic Signal, SV: State Vector, TV: Targets, LD: Loss Design. *Difftaichi an implementation of et al.] (2020), enhanced with our loss design. It can be observed that the Full method achieves the best performance Task loss among models.
4.1 VALIDATION METRIC
In validation, test all the trained agents fixed set of goals. These goals represented by combination of targets targets values, which are uniformly sampled by interpolating between the lower and upper bound of all targets appeared in training.
4.2 DIFFEENTIABLE PHYSICS GRADIENT ANALYSIS
One concern in differentiable physics based learning is the that the gradient may explode during the pack-propagation through long simulation steps. Here we visualize the distribution of gradients norm of whole training process in Fig. 6 It can be observed that most values of the gradients are distributed inside interval [-10, 10] in log scale, which indicates that our learning approach provides stable gradients during training.
Alpaca
Monster
HugeStool
Stool
Probwlliyy Densby
6.80
0.75
Prebabilly Dendoy
0.25
2.0.5
0.18
0.20
0.20
6.20
0.
0.06
0.0
0.25
0.00
-12 Sum Gradents Norm (ing scale)
Sum Gradleres Norm
9um of Gradieets Norm Oog scaln)
-3
Sum Graderts Norm 0og scale)
Figure 6: Gradient Analysis. The plots show the gradicnt distribution of different agent. The x- axis represent the sum of gradients norm. The values drawn in log scale for clear visualization. We record the sum gradients norm 10000 iterations for one training. For each agent, the experiments are performed at least three times, i.e., there least 30000 gradient samples for one agent.
Adam vs SGD. Hu et al. (202)) shows that stochastic gradient descent (SGD) with differentiable physics can achieve satisfying results for tasks with single goals, e.g. running along one direction. However, this longer holds for tasks with multiple goals. To investigate the differences, perform a learning task with multiple goals on a 2D mass spring system. In this experiment shown in the left of Fig. SGD (curve Full-OP) makes minor progress while modified Adam optimizer with reduced momentum drops the validation obviously within 10000 iterations. One key difference between Adam and SGD is that Adam utilises the momentum while vanilla SGD doesn't To further

---

Under review as a conference paper at ICLR 2022
81
0.968
0.9
0.82
0.68
0.43
32
0.999
0.791
0.799*
0.764
0.783
0.781
0.99
0.763
0.795
0.786
0.757
0.756
0.968
0.773
0.776
0.788
0.772
0.773
0.9
0.799
0.745
0.760
0.783
0.817
0.68
0.804
0.756
0.777
0.767
0.776
Table 2: Grid search for Adam hyperparameters. The values in the table represent the normal-
ized validation loss for Task. We chose different values for 31 and Bg in Adam, which are sampled in logarithmic scale. For each setting, the experiments are repeated for multiple times. *The default values for Adam hyperparameters are $ = 0.9 and 32 = 0.999.
investigate the the effectiveness of the momentum hyperparameters B, perform a grid search for both 31 and 82 whose default values are 0.9 and 0.999 respectively. The results presented in table 2 show that reducing 3g 0.9 delivers the best performance. We therefore use this Adam optimizer with reduced momentum in all our experiments.
4.3 LEARNING TECHNIQUES
Batching We adopt batching into differentiable physics and perform a series experiments as shown in the right of Fig. to verify its effectiveness, with batch size 1, 8. 32, and 64. It can be observed that without batching, there is a large variance in training loss and hardly converges. Batching with proper size helps reduce the variance and effectively improve the training perfor- mance. We also find that an overlarge batching size does not help training obviously. For example, batching with size 64 only improve the performance marginally.
Loss design We compare our tailored loss function with a naive loss function. The naive loss function defines the velocity as the position difference of the center of mass between consecutive time steps, i.e., requiring the agent to move at a constant velocity at each time step. The experiments show that the agent under the naive loss design can not be properly trained run, i.c. there are almost no progress in rumning loss dropping.
Activation Function Ablation
Activation Runction Ablation Alpara
Activation Runction Ablation Alpara
Normizzed Validation Taak Lovs
Aormlized Vilitation Rum. Loss
Nomilized Valldattoo Jump. Loss
0.55
teb
0.90
27
0.80
2.5
2000
13050
4000
4050
10003
4000
6000
16:00
Training Beaations
Training Iteartions
Training Reaations
Figure 7: Activation functions ablation study. The figure show the validation loss between choices of different activation functions. From left to right, the subplot shows the results of Task, Run and Jump loss, respectively.
Activation function To validate the effectiveness of our sin activation function, we replaced for hidden layer with various popular activation functions tanh, relu, gelu and sigmoid. For the output layer, the output actuation expected to be limited in the range -1 to 1 due the physical constraints of an actuator, e.g., spring should not be overly compressed or stretched. Therefore we made some modifications for the activation functions in the output layer. To be more specific, we clamp the values larger than or smaller than -1. In addition, for activation functions whose values are constantly zero (or very small) when input is negative, we modify the relu and gelu by subtracting the value by 1, for sigmoid we map the its value from [0, 1] to [-1, 1]. The results in Fig show that using sin as activation functions gives the best performance among all three losses. For more results of activation functions ablation studies, please check the appendix
7

---

Under review as a conference paper at ICLR 2022
4.4
IMPORTANCE OF INPUT FEATURES
Inpur rotation on Apas
Insue Twat.re Noreter
Irpur Easture Abarton Iisge
Itper Fastume Abtation
1.00
0.95
00.5
5.3
6.20
TU1TG
Tul-C 1V
Bimbet
0.33
1y1.5 SV 1U
8.00
Ful-72
Fdl:59
Tul-15
0.30-
0.75
5
2000
4002
6035 2000 12000
0
2003
4005
5500
5000
32000
2002
4200
5100
8900 10020
2000 4200
6002
5020 12000
Figure 8: Input features ablation study, Full-PS, Full-SV and Full-TG indicate the models trained without Periodic Signal, State Vector or Targets respectively.
Ablation on input features The input features consist of three parts: periodic control signal, state vector, and targets, as mentioned in section 3.3 We investigate each part of the contribution to training by ablation studies as shown in Fig The results show that each of the three parts serves as a necessary component for converged training. The periodic control signal serves as an important role. The agents can hardly move without this signal as the corresponding task loss does not drop along the training (shown by the red curves in Fig M The targets are another essential part, without which the optimizer can hardly work after a minor progress as shown by the yellow curves in Fig Although the state vector is less important in terms of reducing the loss, the overall task performance degraded without the state vector as shown by the green curves in Fig
4.5
BENCHMARK AGAINST REINFORCEMENT LEARNING
"Alpaca"
"Monster"
"HugeStool"
"Stool"
Nomllzed volldsaton Loss
1.0
Normlzed Vallestion Lass
lormized Usllidttion Lost
0.9
0.9
0.8
0.8
Vallddation
0.8
0.7
0.7
0.6
0.6
0.6-
0.
0.4
0.4
0.5
2500
5000
10000
2500
7500
19000
2500
5000
10000
2500 5000
10609
Training Itera tions
Training Iterations
Training Iterations
Training Iterations
Ours task
Ours running loss
Ours jurpping less
PPO task loss
PPO running loss
PPO jumpinn less
Figure 9: Comparison on different agents. Both our method and PPO run on GPUs, The solid and dashed lines show the validation loss of our method and PPO, respectively. The agent design is also shown in the figure. Springs marked in red are actuators.
We implement the standard proximal policy optimization (PPO) benchmark in multiple goals set- tings as shown in Fig. 9 The PPO agent can make progress in single goal learning and learn certain locomotion patterns to move toward the goal. Yet as the task gets complicated, PPO often gets trapped in the local minima. We try our best to fine-tune the PPO tends to overfit to goal but fails to find a balance between different goals. It can be observed in Fig. g where the agents trained by PPO can learn to jump but struggle to run.
but still find
5 RELATED WORK
Differentiable simulation Differentiable physical simulation is getting increasingly more atten- tion the learning community. Two families methods exist: the first family uses neural networks to approximate physical simulators (Battaglia et al., 2016 Chang et al. 2016 Mrowca et al. 2018
Li et al. 2018). Differentiating the approximating NNs then yields gradients of the (approximate) physical simulation. The second family is more accurate and direct: many of these methods ing differentiable programming (specifically. reverse-mode automatic differentiation) to implement physical simulators (Degrave al. 2016 de Avila Belbute-Peree al.] 2018 Schenck & Fox 2018
8

---

Under review as a conference paper at ICLR 2022
Heiden et al. 2019 Hu et al. for explicit time integrators, but when it comes to implicit time integration, people often adopt the adjoint methods (Bern et al.] 2019; Geilinger et al., 2020). LCP (de Avila Belbute- Pelbute-Peress et al. 2018) and QR decompositions (Liang et al. 2019; Qiao et al. 2020). In this work, we leverage Diff- Taichi (Huu et al.) 2 an automatic differentiation system, create high-performance parallel differentiable simulators and the built-in NN controllers.
2019 2020: Huang et al. 2021
Automatic differentiation works well
Locomotion skill learning Producing physically plausible characters with various locomotion
skills is a challenging problem. One classic approach is to manually design locomotion controllers subject to physics laws of motion order to generate walking Ye Liu (2010) Coros et al. (2010) or bicycling Tan et al. (2014) characters. These physically-baser controllers often rely on a complex set of control parameters and are difficult to design and conemmonu to optimize. Another line of research suggests using data to control characters kinematically. Given a set of data clips, controllers can be made to select the best fit clip to properly react to certain situations Safonova & Hodgins (2007) Lee et al. (2010); Liu et al. 2010). These kinematic models use the motion clips build a state machine and add transitions between similar frames in adjacent states. Although being able produce higher quality motions than most simulation-based methods, the kinematic methods lack the ability to synthesize behaviors for unseen situations. With the recent advances in deep learning techniques, attempts have been explored to incorporate reinforcement learning (RL) into locomotion skill learning Peng et al. (2155) Liu & Hodgins (2017); [Pegg et al. (2018); Park et al. (2019). These modern controllers gain their effectiveness either from tracking high-quality reference motion clips or from cleverly designed rewards to imitate the reference. However, the exploration space for RL is usually prohibitively large to achieve complicated target motions. Carefully designed early termination strategies Ma et al. (202I); Won et al. (2020), better optimization methods Yang & Yin (202T), and adversarial RL schemes Peng et al. (2021) enable these RL-based methods to achieve richer behaviors. However, is still time- coneeminnuingg to get well-trained RL model in complex tasks such as soft robot control. Our method, on the other hand, leverages the differentiable simulation framework and can achieve much better convergence behavior compared to RL-based methods.
LIMITATIONS AND CONCLUSIONS
Currently, difficulty of the learning tasks depends on robot design and physical parameters. The training performance may be degraded with improper robot designs. For instance, our stool case where unnecessary actuators on its body are allowed, it is more difficult to achieve good results. Offloading the physical parameter tuning and robot designing to an automatic pipeline will be an interesting future research direction.
To summarize, we have presented an effective end-to-end differentiable physics based learning framework for soft robot complex locomotion skills learning. We systematically enhance classical differentiable physics learning systems with a suite of techniques and investigated the key factors contributing the training in detail. We show that our framework without explosion during a simulation with hundreds of steps. Together with batching, the modi- fied Adam dramatically outperforms the stochastic gradient descent (SGD) on complex tasks with multiple goals. Benefited by our tailored loss functions, the network take advantages of the three parts of the input features. We found that the periodic control signal dominates the actuation signal, while state vector and task goals have weaker effects. The doe-nntnddteed periodic signal may in- duce freeueryyyncy noises. To balance the importance between the input features, we additionally pose an actuation loss as a regularization term. It can effectively suppress the noises, which help the agents move more smoothly. addition, we compare our method with the state-of-theeart re- inforcement learning proximal policy optimization. Our method shows ad vdnatages in both training performance and convergence efficiency on tasks with multiple goals.
provide stable gradients
Our framework enables users to flexibly design robots and to teach them locomotion skills. The trained agents can be manipulated to smoothly switch locomotion tasks such as running, jumping, and crawling with different speeds and orientations, interactively. To the best of our knowledge, this is the first time a learning system based on differentiable physics can deliver controllers with such robustness, flexibility, and efficiency.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Baraff and Andrew Witkin. Large steps in cloth simulation. In Proceedings of 25th annual conference on Computer graphics and interactive techniques, pp. 43-54, 1998.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502-4511 4510, 2016.
James M Bern, Pol Banzet, Roi Poranne, and Stelian Coros. Trajectory optimization for cable-driven soft robot locomotion. In Robotics: Science and Systems, 2019.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. ICLR, 2016.
Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. Generalized biped walking control.
ACM Transactions On Graphics (TOG), 29(4):1-9, 2010.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances Neural Information Processing Systems, pp. 717877777770 7188-711,,881 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. arXiv preprint arXiv: 1 2016.
Moritz Geilinger. David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. arXiv preprint arXiv:2007.00900 2020.
Eric Heiden, David Millard, Hejia Zhang, and Gaurav S Sukhatme. Interactive differentiable simu- lation. arXiv preprint arXiv: 10706, 2019.
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4): 2018.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum. William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simu- lator for soft robotics. In 2019 International Conference on Robotics and Automation (ICRA), pp- 6265-6271. IEEE, 2019.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan KellnyKelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. ICLR, 2020.
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with diffcrentiable physics. arXiv preprint arXiv:2104.03311 2021.
Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cell method. ACM Transactions Graphics (TOG), 34(4):1-10, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14/2.6980 2014.
llya Kostrikov. Pytorch implementations of reinforcement learning algorithms. github. com/ htt.:/otttooo///oorrttteeeeeeee Pelldc-pppacckee gail 2018.
https://
Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović. Motion fields for interactive character locomotion. In ACM SIGGRAPH Asia 2010 papers, pp. 1-8. 2010.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv: 1810.0010 2018.
10

---

Under review as a conference paper at ICLR 2022
Junbang Liang, Ming C Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse prob-
lems. Advances in Neural Information Processing Systems, 2019.
Libin Liu and Jessica Hodgins. Learning to schedule control fragments for physics-based characters using deep q-learning. ACM Transactions on Graphics (TOG), 36(3):1-14, 2017.
Libin Liu, KangKang Yin, Michiel van de Panne, Tianjia Shao, and Weiwei Xu. Sampling-basee contact-rich motion control. In ACM SIGGRAPH 2010 papers, pp. 1-10. 2010.
Li-Ke Ma, Zeshi Yang, Xin Tong, Baining Guo, and KangKang Yin. Learning and exploring motor skills with spacetime bounds. arXiv preprint arXiv:2103.1680) 2021.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physics prediction. arxiv preprini
0xXiv:8806000047 2018.
Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. Learning predict-and simulate policies from unorganized human motion data. ACM Transactions on Graphics (TOG), 1816))1-11, 2019.
Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. Dynamic terrain traversal skills using reinforcement learning. ACM Transactions on Graphics (TOG), 34 2015.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. arXiv preprint arX1v:2104.02188, 2021.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.02/68, 2020.
Alla Safonova and Jessica K Hodgins. Construction and optimal search of interpolated motion graphs. ACM SIGGRAPH 2007 papers, pp. 106-es. 2007.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamics for deep neural networks. arXiv preprint arXiv: 06094, 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Jie Tan, Yuting Gu, C Karen Liu, and Greg Turk. Learning bicycle stunts. ACM Transactions on Graphics (TOG), 3314):1-12, 2014.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. ACM Transactions on Graphics (TOG), 39(4):
2020.
Zeshi Yang and Zhiqi Yin. Efficient hyperparameter optimization for physics character ani- mation. Proceedings of the ACM on Computer Graphies and Interactive Techniques, 4(1):1-19, 2021.
Yuting Ye and C Karen Liu. Optimal feedback control for character animation using abstract model. In ACM SIGGRAPH 2010 papers, pp. 1-9. 2010.
11

---

Under review as a conference paper at ICLR 2022
A REINFORCEMENT LEARNING SETTING
Environment. We use the open open-source implementation of PPO Kostrikov 2018 in our environ- ments. Part of important Hlpee--paametten are listed in the table 3
Table 3: PPO ppper-paramterrs
Parameter
Values
learning rate entropy coef
2.5e
4
0.01
value loss coef number of processes
0.5
number of simulation steps.
1000
Reward Design. We design reward function for PPO based on our loss functions. Recall the velocity part of equation
We define the total velocity reward as:
R, A, E Σ NET NE
gu(n)² (v(t)
(6)
If we directly split the reward into each time steps by L, the rewards of first P, time steps would be zero and it is too difficult for PPO to learn the policy. Therefore, we modified the reward function to tackle this issue. By equation
!
we have
R2 = A Σ
Do(n)²
[
P., (c(t) c(t - P.)) (n(
1
EET LE[P,
A Σ
Σ
(n) (c(( PP) (rgg
(8)
22
nET (E(P,..)
We can define a function (t',', t) = (Prg - (c(t P,) + (n)) and substitute
back to equation g The reward can be re-written as
A
3 (n(1,t
R,
(9)
P2
nET PEPP
Refer to that for any t E P, we have fa Pr, t) = 0, which means
R2 =
p2 Σ Σ Σ
fa(t At,t) - fn(t - -0------- 1,8) 1,t)
(10)
RET
P. ArE(O.P.)
= At
(in(t'PP,) Σ
R2
SalC,e'' + -1) - 11(++
(11)
Ph
nET ('E(
AL
min(L'
A
Σ
-lo((') - -cc(l' + (1-PP)) P,) Prgr
P2
NET ['E
A1=
(c(t' - 1) (c(t' At P
+ (n))
(12)
So split the whole reward into each time step
min(t.P.)
R.(t) = Σ
[c(t 1) - (c(( At - -P,) + 2r))
At=0
(c(t) (c(t At P,) + (r))())))
(13)
12

---

Under review as a conference paper at ICLR 2022
Recall that the goal of a jumping task is to reach specified height goal gh, the intuition for jumping reward is to encourage the agent to improve its maximum height until reaching the height.
Rx
A
(min h(, t nP) (n))² (h)(n) (n)))
(14)
#ES
EE
where the min.es (ss, (+PP) is the agent height at current time step as mentioned in 3.1 and (n) is the max height record during one task period.
B NETWORK ARCHITECTURE
Actuation
FC2 with sime
Hidkln Layer (64)
FCI with sime
Input Features
Periodic Control Sygnal
State Vectoo
Tangets
Figure 10: Network architecture.
Our network architecture is shown in Fig. 10 It is two layers fully connected network with 64 channels hidden layer. The number of input and output channels varies according different agent designs.
C SIMULATION SETTING
Mass-spring systems We adopt the classic Hookean spring model to represent the clastic force and use dashpot damping Baraff & Witkin (1998) the damping force simulate mass-spring system. We found that the drag damping model used by Hu et al. Hu et al. (2020) damps the gradient of the NN controller while the dashpot damping model is able to generate vividly changing gradients. Adopting dashpot damping model makes our agent more flexible.
Material point methods We MI.SMPM Hu et al. (2018)
material point method simu- lator. We further applied the affine particle-in-cell method Uiang et al. (2015) to reduce the artificial damping. Due the nature MPM hybrid Lagrangian- Bulerian method, it always requires background grid during the simulation. Naive MPM implementations fix the position of the back- ground grid, hence limit the moving range of the agent. We apply dynamic background grid that follows the agent all the time to overcome this problem.
D INTERACTIVE CONTROL
Locarwtiee
Figure 11: Left: Our system allows the user to control the agent's motion interactively. Right: the trajectory of walking quadruped agent under user control.
Smooth interpolation between different tasks is key advantage of our approach. In interactive settings, users can smoothly control the agent's motion via input devices such as a keyboard as shown in Fig. [1 Please refer to supplemental video for more details.
13

---

Under review as a conference paper at ICLR 2022
E 3D RESULTS
In addition to 2D cases, our framework can be applied to 3D cases seamlessly. In 3D space, an agent is additionally expected control its orientation on a plane, i.e., rotate. We design several 3D agents (shown in Fig.2) and train them run and rotate. These trained agents can achieve specified goals given control signals after trained for several hundred to a few thousand iterations. Please check the supplemental videos for more details.
Loss function in 3D The agents in 3D are trained with the loss function below. The loss for running L, inherited from 2D settings. The rotation loss L, is defined as the accumulation of centralized point-wise distance between current position and target rotated position.
P ΣE
C A Σ RET Pr
(v(t) g.(n))) +Ar P=
E ((s(t) c(t)) R(s(t P,) c(t P,)))²
(15)
Cos (Prow)
0
sin(Prgw)
0
1
(16)
sin (Prg))
0
(or(P
where Iw is the target angular velocity assembled in input features. The format of R is derived by rotation matrix along XZ-plane.
Note, for solving running tasks, the rotation loss also plays an important role. A significant issue of 3D agents running is accumulated orientation error. Utilizing identity rotation matrix R helps adjust agent's 's posture. In turn's 's of rotation tasks, the rotation center should be maintained still which is equivalent to zero velocity. These two losses complement with each other to achieve the final goals.
Friction model. real world, agents running is driven by static friction between "feet" and ground. We configure and train agents with different contact models in our experiments. With zero friction applied, training loss shows no evidence decreasing and the agent not able to move no matter what actions patterns it learns. With a fully sticky surface, where only upwards vertical velocity is kept and all other components of velocities are projected to zero after contact, agents work well. However, in 3D, it turns out that the agent tends to stuck. In practice, simulation which involves both kinetic friction and static friction. These two categories of contact determined by how much pressure force is applied on the surface and can be treated like a branched function numerically which can be handled by our auto-differential system. Please the supplemental video for visual comparison between slip and sticky friction models.
adopt
the classic slip with friction model from physically-beit
1.00
sticky
sip 0.4 zero friction
Training loss
0.60
0.40
Traininig Iterations
Figure 12: Left:
Training loss for friction, .4 friction and sticky contact models. Middle: Agent suffers from sticky surface and cannot move further. Right: Agent moves smoothly towards right.
14

---

Under review as a conference paper at ICLR 2022
F GENERALIZATION TO MANIPULATION TASKS
Here we provide results on simple manipulation tasks additional to locomotion tasks. In these tasks, the agent expected to manipulate a object (marked purple) to hit target point (marked in green). This task has similar periodicity property like locomotion. We designed two scenarios: Juggle and Dribbble and Shot shown Fig The visual results are also shown in the video in supplemental materials.
Juggle In this scenario, the target point appears in the sky above the the agent. The agent is expected to "juggle' the object to the target point.
Dribble and Shot In this scenario, the target point keeps moving toward left. The agent is expected carry the object and shot it to the target point.
Tagett Point
[4] Ohjject
Taget Point
Olject
Figure 13: Manipulation task showcase. The upper plot shows the snapshots of scenario 'Juggle' and the lower one shows that of the 'Dribble and Shot'.
0.16
Training
0:34
6.32
202
400
600
002
Training lleration
Figure 14: Training curve the 'Juggle' task. The loss is defined as the distance between center of the object and the target point.
The experiments indicate that our method has the possibility to generalize to tasks beyond locomo- tion.
G DETAILED RESULTS ON FURTHER EXPERIMENTS
Here show more detailed results further experiments, including ablation studies different agents, validation loss on different tasks for agents and more network weights visualization results.
15

---

Under review as a conference paper at ICLR 2022
Table 4: Ablation Summary on Monster.
Training Setting
Norm. Valid. Loss (Monster)
Name
OP
AF
BS
PS
SV
TG
LD
Run.
Jump.
Task
Full
Adam
sin
32
V
v
v
0.46+
0.77=#0.00
0.71+
Full-OP
SGD
sin
32
/
/
V
0.99110.01
0.99110.00
0.99110.01
Full-AF
Adam
tanh
32
/
0.65+10.01
0.84+0.02
0.79:1.02
Full-PS
Adam
sin
32
X
V
V
0.93+0.00
0.99#0.00
0.98
Full-SV
Adam
sin
32
0.51+0.01
0.86+0.01
0.78
Full-TG
Adam
sin
32
0.99+00.00
0.97+0.01
0.98+00.01
Full-LD
Adam
sin
32
0.99+00.00
0.90++0.06
Table 5: Ablation Summary on HugeStool.
Training Setting
Norm. Valid. Loss (HugeStool)
Name
OP
AF
BS
PS
SV
TG
LD
Run.
Jump.
Task
Full
Adam
sin
32
V
v
0.40
0.58+0.01
0.53+
Full-OP
SGD
sin
32
/
d
d
0.97+10.01
0.9610.01
0.96:10.01
Full-AF
Adam
tanh
32
V
/
0.49:10.01
0.59+0.01
0.56:+0.01
Full-PS
Adam
sin
32
X
/
V
0.99++0.00
0.99#0.00
0.99:00.00
Full-SV
Adam
sin
32
V
X
V
0.401:0.01
0.62:10.01
0.57+10.00
Full-TG
Adam
sin
32
0.99+00000
0.90+00.00
0.93+00.00
Full-LD
Adam sin
32
V
0.99+-0.01
0.70++0.01
Table 6: Ablation Summary on Stool.
Training Setting
Norm. Valid. Loss (Stool)
Name
OP
AF
BS
PS
SV
TG
LD
Run.
Jump.
Task
Full
Adam
sin
32
V
V
V
0.50
0.58+
0.56
Full-OP
SGD
sin
32
/
V
V
0.78+0.02
0.76+10.02
0.76:10.02
Full-AF
Adam tanh
32
/
0.60++0.01
0.63+00.33
0.62++0.04
Full-PS
Adam sin
32
/
0.98+10.01
0.96+00.00
0.96:+0.02
Full-SV
Adam
sin
32
V
X
V
0.50+00.3
0.64
0.61
Full-TG
Adam
sin
32
0.98+10.0
0.87+0.01
0.8910.01
Full-LD
Adam sin
32
V
0.99+0.01 x
0.83+0.05
Hidden
sin
tanh
relu
gelu
sigmoid
Ouput
sin
0.745
0.771
0.765
0.803
0.810
tanh
0.753
0.805
0.783
0.799
0.849
relu
0.780
0.770
0.806
0.775
0.889
gelu
0.767
0.780
0.773
0.795
0.829
sigmoid 0.754
0.767
0.800
0.799
0.860
Table 7: Full ablation studies for activation functions. The values in the table represent the normalized validation loss for Task. For each setting, the experiments are repeated for multiple times. It can be observed that the model using sin for both hidden and output layer achieves the best results.
16

---

Under review as a conference paper at ICLR 2022
Validation Loss "Alpaca'
Ours task loss
0.08 [m0.1
06,
1,00
0.04 Fw
0.02 hw01
+-0.2 1-0,1
hw0.
Our running loss
0:3-
Ours jumping loss
0.
12:0 1980
18601
1930
uce 18912
1201 19321
VidE
Lnkee
sdea 18912 hwo.
0.09 1m6.1
0.06
0.02 hw01
+-002 1-0
+0006
hw0
1/8
15
3.0
875
64-
850
0.5-
24
12:0
1990
11601
10:00
1990
Viee 18911
Vic
19ke
1201
19131
Vide
tnies
3000 19903
3..0..1 0.2
V: 06, 0:02
v.-0..4 0z
v.-0.00 h:02
v.0.2 5:02
v-5.0) h:02
0.0.6 h:02
h-02
122
1.00
0.-
150
0:00
5300
30300
5000
10690
5300
33000
3006
38003
5006
1000
5000
30000
5006
1008
3003
38009
Validation Loss
"Monster"
Ours
V- 0.000 1-0,1
--00.0 2..5 1w0.1
y- 0.04 1-3,1
Viu 2,08 1w0.1
0-0.02 heo
y-0.044 h-0
V+0 05.05
Outs rinning loss
26-
Outs jumping loss
65
vin 1r691
vin trees
Vic 19rr
1891 1900
Vie 1090
side 1889n
1991
seds 19502
v-0000 0.05 1-211
0.26 1w0.1
y- 0.04 1w3,
ye 2.00 1w0.1
0-0.02 he0d
1-0.06 h=0
V-0 05, hw0.1
V=0.00 h-0.1
30
10
10-
1.00
35-
4.5-
18691
19t
(rees
Vine 199re
1991 1900
1990
1899
1291
sids 19993
40.0.0 5:32
v:-4.36, h202
v--0.44 5:3.2
Vi-5.02 8:
V.0000 h:02
V.0.1.h 0:
V.0.0..
V:00.00, h:02
140
2.00
835
6.0
6.00
ESc-
622
620
825
5006
10602
5300
1060
5000
20006
2000
1000
28000
5006
20000
5003
20000
5043
28000
Validation Loss
"HogeStool"
Our task loss
1.0-
1--0.05 h-01
0.09 hw0.1
2.04
heg,2
1/-
y- 0.02 he01
v-0.22 he011
y-2.04 he0.1
3.8
2-000
running
Out jumping loss
8.5
6.5
0.6
2.0
10000
1993 10ore
1952 10000
5963
40-0.00 08,
0000.66 hic.1
h.
hu01
v.n6.2, hun.1
vud.64 hi6
bud))
00,00 hin1
1.0
3.8
2.8
0.3-
0.5
8.0
0.0
0.8
08,
h.0
02.0 h202
V..02, hu0z
vud.04
00000..00
3.00
18C
1.40
1.00
6.55
0.35
6.75
0.59
0.50
025
325
0.25
6:25
Validation Loss
"Stool"
Ours task loss
3.00
300
Outs running loss
5.19
Oum jumping loss
625
30093 J.
La3
0.2
c5
0.50
022
23000
2006 12000
5000 20690
5200 Locee
5000 1300e
28303
3.00
6:5-
30
3.0
355
151
025
5003
23300
20230
10030
2000
13000
5200
20603
5350
10060
5000
32005
5300 22303
Figure 15: Validation loss on different agents and targets. The plot shows the validation loss on different targets combinations for agents. The target velocity ranges from -0.08 to 0.08 and the target heights are 0, 0.15 and 0.20. Each subplot shows the task, running and jumping loss for one agent given a pair of specified target velocity and height.
17

---

Under review as a conference paper at ICLR 2022
ulinn Stady Alooca
Atlatinn Study Norsler
Alr Atreton HugeStoo
Abration Stady
Taring Marters
Tyaniig baarbrr
Tran licatiors
#blation Study alaaca
Jubrion St. Wonder
AMarron Stity in HappeStod
Stuty Stal
live
Stady Aipeta
Morsiet
S.ddy MageStoo
Suad
Drtach
200
40%
900
20A
BU00
LO00
200
2000
2000
420
000
000
10000
Figure 16: Summary of the ablation study for agents. The subplots show the normalized valida- tion loss for task (weighted summation of different losses), running and jumping from top to bottom row. The subplots different columns show the results of different agents.
18


Reviews:
[{"review_id": "W9uAbJKoLgh", "reviewer": "Reviewer_kPtv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) This paper combines differentiable simulation with many carefully chosen design decisions, which enables effective learning of soft body locomotion.\n2) The analysis and ablation study provides useful and practical guidance to practitioners in the field of reinforcement learning and locomotion control.\n\nWeakness:\n1) Technical contribution is lean. The Taichi differentiable simulator is an existing work. This paper seems to use it without lots of changes. The second claimed contribution \"end-to-end differentiable simulation environment, including mass-spring and material point\" is not well supported. The description of simulation is high level or in Section 2. It is hard to understand the scope of this contribution.\n\n2) I hope that the paper can demonstrate more impressive and diverse results. For example, a soft character navigating uneven terrains, or a soft robot manipulating objects. A soft body character running on flat ground, as shown in this paper, has been demonstrated in several prior works, such as DiffTaichi. Even though this paper demonstrates multiple locomotion skills (turning, different speed and jumping) in one policy, the improvement over the prior work is incremental.\n\nAdditional questions and comments:\n1) The paper claims in Section 2.1 that \"Any physically-based deformable object simulator can be applied to our learning framework\". I thought that the simulator needs to be differentiable. So only a small subset of simulation will work. Did I miss anything?\n\n2) It is unclear to me how applicable the results of the ablation study are to other tasks. For example, the sine actuation function and the periodic signal may not be as important for manipulation tasks. If the application of this paper is only locomotion, its contributions are quite limited.\n", "summary_of_the_paper": "This paper proposes to use differentiable simulators for learning locomotion skills for deformable characters. The key contribution is the experiments and analysis of a large number of design choices, including activation function, reward design, optimizer, batch size, etc. The paper finds that by combining differentiable simulator with the right design choices, the character can learn general and complex skills with one policy. The paper also shows that this combination can significantly outperform PPO.", "summary_of_the_review": "The paper demonstrates good results for learning soft body locomotion. Although the technical novelty is lean, I find that the analysis and ablation of various design decisions are helpful, and has practical values to guide further development in this direction. For this reason, I am slightly leaning towards accepting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VhDQTdsYTUk", "reviewer": "Reviewer_2nYD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Based on existing differentiable physics engines, this paper further gives empirical recipes on how to use the physics gradients to train powerful controllers, which can help the community to develop techniques related to differentiable physics. \n\nThis method divides long-horizon tasks into smaller periods with different objectives. To avoid sparse and non-smooth rewards, objective functions are manually designed for subtasks. Periodic control input is also added to the input to actuate the controller. When training the network, the authors use batching, regularization terms, and sine activation layer to accelerate and stabilize the convergence.\n\nWhen reading this paper, I have several concerns,\n1. In Equation 5, how is $g_v(n)$ subtracted from $A(t)$? Do they have the same dimension? $g_v(n)$ as a velocity should have a dimension of 3 (or 6 if rotation is included). The dim of  $A(t)$ should be the DOF of actuation. Maybe you mean $||g_v(n)||$. Why only target velocity is considered, how about the height $g_h(n)$? Why “periodic control signal may induce unwanted high-frequency noises”？\n2. The paper mentions \"a modified Adam optimizer\". I did not find how Adam is modified.\n3. In Figure 4, what is the definition of one iteration? Is it one step of network update or one step of simulation? \n", "summary_of_the_paper": "This paper provides a practical guide on how to train an NN-based controller for locomotion tasks with robustness and efficiency. \n\nThe tasks are categorized into motion primitives including running, jumping, and crawling, each with a manually designed objective function. Considering the periodic nature of locomotion, the author proposed to use sine activation layers and periodic signals in different phases as input. The experiments show that a larger batch size can lead to faster convergence. Compared to SGD, this paper points out that Adam is better for the tasks. The supplementary video presents that the trained networks enable real-time control of complex skills.\n", "summary_of_the_review": "This paper gives us a good example of how to train a network using differentiable physics. Their video of interactively controlling 2D and 3D robots looks interesting. Although some of the techniques like staging, batching, since activation, and periodic signals are intuitive and existing tricks, it is good to see that a combination of them can successfully train a controller.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T1fp36YNzaF", "reviewer": "Reviewer_cp7c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n* +++ The paper shows great results of interactively controlling soft robot locomotion and switching among different goals.\n\n### Weakness\n* --- The overall technical contribution is limited. The proposed contribution on loss designs is limited to the locomotion tasks, and it is not clear whether these losses would generalize to other robots and the real world [1, 2].\n* -- The use of the sine activation function is not well supported by the experiments. In table 1, sine function shows marginal improvement over tanh, and in table 5 of the appendix, using tanh shows much better performance over sine. Furthermore, tanh tends to become saturated during backpropagation of a long sequence. I would suggest the author also compares with ReLU.\n* -- Presentation in the paper can be improved:\n     1. The modified Adam optimizer is listed as one of the technical contributions. However, it is not stated clearly what is the modification and how does it compare to the original Adam optimizer. \n     2. The name of the robots are presented in Fig. 7 of Sec. 4.5 but are referenced multiple times in previous sections and table 1. It would be helpful to clarify the name of the agents early on. \n     3. Reference of Fig. 5 at the end of Sec. 4.2 seems to be incorrect, as no comparison of SGD and Adam is shown in Fig. 5.\n\n[1] Lee, Joonho, et al. \"Learning quadrupedal locomotion over challenging terrain.\" Science robotics 5.47 (2020).\n[2] Zhao, Allan, et al. \"RoboGrammar: graph grammar for terrain-optimized robot design.\" ACM Transactions on Graphics (TOG) 39.6 (2020): 1-16.", "summary_of_the_paper": "The paper proposes to learn neural network controllers in a differentiable simulator for locomotion tasks. A set of technical improvements are made and make the controllers more robust and require fewer samplers to learn. These include: using periodic activation functions, design of loss functions, and using large batch size. The paper shows learned locomotion skills of running, jumping, and crawling for different soft robots, which can be interactively controlled. ", "summary_of_the_review": "The paper shows interesting results of locomotion tasks, but the overall technical contribution is limited as presented. The writing of the paper could also be further improved. I am willing to raise my score if the listed concerns can be addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BiuL5uskVBY", "reviewer": "Reviewer_38Wd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n* The loss function design is clear and well explained, it is specifically tailored for cyclic motion which is the main task addressed by the paper.\n* The use of SIREN to process the input and output a periodical signal is quite interesting. It suits well to tasks requiring cyclic motion.\n* The comparison to PPO seems fair and thorough, I appreciate that the authors put efforts into having a well-tuned baseline, something which is often overlooked.\n* The control of complex mass spring system is impressive\n\n## Weaknesses\n\n* The claim: \"To the best of our knowledge, we proposed the first differentiable physics based learning frame-\nwork for complex locomotion tasks using single network.\" is a bit fishy.\nMany RL algorithms handle locomotion tasks of humanoid-like robots in simulation with a single neural network. I understand the current framework use differentiable physics for mass-spring systems, and thus is not directly applicable to classical RL benchmarks without a differentiable physics simulator. Such a claim could be made if it solved classical RL locomotion tasks + harder ones for example.\n\n* The experiment section is not really convincing and hard to follow. \n** Table 1 is barely commented, there is only a bold number in one column. Only tanh activation function is compared to sin, while relu, gelu, sigmoid could be used for hidden layers with an activation function outputting a value in [-1, 1] for the final layer of the network. Given that using SIREN seems to be an important component, the paper would benefit from a better ablation study. \n** The importance of features section is really unclear. \nThere are sentences like \"The results show that each of the three parts serves as a necessary\ncomponent for a converged training. The periodic control signal serves as an important role. The\nagent can hardly move without this signal\" without reference to a table or a figure, so I do not know what to look to confirm the statement.\nFigure 6 is also hard to read and I am not sure if the section is super insightful. To my understanding it validates the design choice of the input feature which is a bit minor compared to other components of the model.\n* In the abstract, a modified Adam optimizer is mentioned \"we find our adoption of batching and modified Adam optimizer effective in training complex locomotion tasks\". It seems to be important according to the authors but there is no experiment on vanilla Adam vs the proposed one with reduced momentum so we cannot asses how useful it is.\n\n## Other comments\n\n* The abbreviations of Table 1 could be put as caption rather than footnote for readability. The back-and-forth between table and footnote is not great for the reader.\n", "summary_of_the_paper": "This work proposes to learn locomotion skills on soft robots made of springs. It proposes to leverage differentiable physics along with a NN controller based on SIREN, which allows to directly learn policies by minimizing loss functions defined on trajectories. The methods allows to learn locomotion and jump behaviors on a various set of 2D and 3D mass-spring systems.", "summary_of_the_review": "The paper is interesting, proposing a clean framework to perform locomotion tasks on mass-spring systems. The benefit of using differentiable physics is clearly highlighted and the use of SIREN to learn periodic motions is on point. However the experimental section is weak. For this reason I only grade it as weak-reject but would be inclined to bump it if this part is strengthened.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A nice paper and very close to being good.  But the focus on hyperparameter tuning of the optimisation method is really not novel, and the experimental validation is not strong enough.  With both theory and experimental just being marginal improvements, the paper is not considered quite ready yet.  Strong suggestion to improve on the weaknesses of the paper and resubmit – next time you'll have a clear acceptance.
Decision: Reject