Universal Controllers with Differentiable Physics for Online System Identification
Authors: Michelle Guo
Wenhao Yu
Daniel Ho
Jiajun Wu
Yunfei Bai
Karen Liu
Wenlong Lu
Venue: ICLR 2022 Submitted (2022)

Abstract: Creating robots that can handle changing or unknown environments is a critical step towards real-world robot applications. Existing methods tackle this problem by training controllers robust to large ranges of environment parameters (Domain Randomization), or by combining ``Universal'' Controllers (UC) conditioned on environment parameters with learned identification modules that (implicitly or explicitly) identify the environment parameters from sensory inputs (Domain Adaptation). However, these methods can lead to over-conservative behaviors or poor generalization outside the training distribution. In this work, we present a domain adaptation approach that improves generalization of the identification module by leveraging prior knowledge in physics. Our proposed algorithm, UC-DiffOSI, combines a UC trained on a wide range of environments with an Online System Identification module based on a differentiable physics engine (DiffOSI). We evaluate UC-DiffOSI on articulated rigid body control tasks, including a wiping task that requires contact-rich environment interaction.
Compared to previous works, UC-DiffOSI outperforms domain randomization baselines and is more robust than domain adaptation methods that rely on learned identification models. In addition, we perform two studies showing that UC-DiffOSI operates well in environments with changing or unknown dynamics. These studies test sudden changes in the robot's mass and inertia, and they evaluate in an environment (PyBullet) whose dynamics differs from training (NimblePhysics).

Under review as a conference paper at ICLR 2022
UC-DIFFOSI: UNIVERSAL CONTROLLERS WITH DIFFERENTIABLE PHYSICS FOR ONLINE SYSTEM IDENTIFICATION
Anonymous authors Paper under double-bling brind review
ABSTRACT
Creating robots that can handle changing or unknown environments a critical step towards real-world robot applications. Existing methods tackle this problem by training controllers robust to large ranges of environment parameters (Domain Randomization), or by combining "Universal" Controllers (UC) conditioned on environment parameters with learned identification modules that (implicitly or ex- plicitly) identify the environment parameters from sensory inputs (Domain Adap- tation). However, these methods can lead to over-conservative behaviors or poor generalization outside the training distribution. In this work, we present domain adaptation approach that improves generalization of the identification module by leveraging prior knowledge in physics. Our proposed algorithm, UC-DifIOSI, combines a UC trained on a wide range of environments with an Online System Identification module based on a differentiable physies engine (DiffOSI). We eval- uate UC-DiffOSI on articulated rigid body control tasks, including a wiping task that requires contact-rich environment interaction. Compared to previous works, UC-DiffOSI outperforms domain randomization baselines and is more robust than domain adaptation methods that rely on learned identification models. addition, we perform two studies showing that UC-DiffOSI operates well in environments with changing or unknown dynamics. These studies test sudden changes the robot's mass and inertia, and they evaluate in environment (PyBullet) whose dynamics differs from training (NimblePhysics).
INTRODUCTION
In order for robots to shine in real-world applications, they need to handle ever-changing and un- predictable situations in real environments. For instance, a robot waiter should be able to serve type of dish without spilling food, and autonomous vehicle should take person safely to an unvisited destination. Creating artificial agents that can operate in changing and unknown environments is longstanding problem in the robotics community.
The collective wisdom of the robotic research community in recent years indicates that enabling learning agents to work in changing and unknown environments is not about making one big break- through, but rather making many small but informed decisions. One general approach advances control policies such that they can operate more robustly TTan et al., 2018) more adaptively (Cully et al.) 2015) in testing environments. However, these methods usually exhibit sub-optimal task performance or require additional fine-tuning in the target environment. Alternative approaches advance simulation techniques to bring the training environment closer to the testing one prior to learning control policy, such as training dynamics model (Jiang et al. 202T) or identifying sim- ulation parameters (Tan et al. 2018) from data. These methods are often used in offline settings as learning or identifying an accurate simulation model can be time consuming. Thus, they cannot han- dle changing environments naturally. Work such Yu et al. (201)) have also investigated methods that learn models to perform online system identification. However, a learned model often does not generalize well to unknown environments those not seen during training.
Recent developments in differentiable physics simulation potentially offer a more effective way to address these challenges by advancing both control and simulation techniques. By utilizing fast

---

Under review as a conference paper at ICLR 2022
computation of analytical gradients, one can devise more computationally and sample efficient opti- mal control and system identification algorithms. Recent differentiable physics simulators, such as NimblePhysicss (Werling et al. (02I), provide fast computation of analytical gradients in the face of constraint satisfaction and hon-differentiable contact handling. These enable generie gradient-basee optimizers to solve contact-rich optimal control problems. While promising, differentiable physics simulation does not solve the fundamental problem of multiple local minima due to ill-conditioned cost functions, often exacerbated by long-horizon and highly nonlinear differential equations.
This paper introduces a new approach for creating resilient and adaptive agents by combining dif- ferentiable physics simulation for online system identification and reinforcement learning for offline policy training. Online system identification can be formulated as a tion problem, must be solved fast. This plays to the strength of differentiable physics simulation which provides analytical gradients efficiently, while avoiding the pitfalls of poor cost function land- scapes. On the other hand, for challenging control problems with long-horizon cost functions, we resort to reinforcement learning approach leveraging samples ("rollouts") generated offline at scale to train a control policy. We explore many possible situations the agent might encounter when oper- ating in the testing environment by varying the simulation parameters during training and learning a Universal Controller (UC) conditioned on the simulation parameters. At test time, we use differ- entiable physics simulation to continuously optimize the simulation parameters based the most recent history of observations (DiffOSI). The optimal simulation parametens will "modulate" the universal policy to output the optimal action for the currently identified environment.
local optimiza-
We evaluate our approach on two robotic control tasks, a cartpole balancing problem and robot arm table wiping task involving rich contact phenomena. We show that our proposed approach combin- ing a Universal Controller and a Differentiable physics- based Online System Identification module (UC-DiffOSI) can outperform pure learning- -ased or traditional system identification methods. Fi- nally, we demonstrate that our approach can be applied to environments with changing dynamics or un-modeled effects.
2 RELATED WORK
Deep Reinforcement Learning and Domain Randomization. Deep reinforcement learning has been proven to be effective in learning complex motor skills for simulated robots, such as run- ning (Yu et al., 2018), parkour (Heess et al.] [017]. and dressing (Clegg al.] 2018). However, these controllers often perform poorly on real robot hardware due to the discrepancies between the simulated and real environment, also known as the sim-to-real gap (Neunert al., 2017). Do- main randomization of the simulation physics parameters has been extensively explored to help the simulation-train trainedd controller transfer to different target environment, where robust control policy is trained to perform well for wide variety of simulated environments (Peng et al.) 2017 Tan al. 2018 Hwangbo et al. 2019: Exarchos et al. 2020 OpenAl et al. 2019). However, policies trained from domain randomization often exhibit consevvvvvveee behaviors, leading to sub-optimal per- formance (Tan et al. 2018). Different from these methods, we develop a domain adaptation approach by training adaptive controllers that can adjust behavior for different environments using estima- tion of the environment parameters. This enables our controller to achieve better performance than a domain randomization controller.
Domain Adaptation. To achieve better task performance in novel situations, researchers have de- veloped adaptive controllers that can adjust behavior for different environments. [Szita et al. (2003)
showed that Q-learning, using wvent-leaming, can find near-optimal policies in varying environ- ments. Heess al. (2015) demonstrated that control policies modified use recurrent networks are also capable of dealing with unknown kinematic parameters such as link lengths. Xu et al. (2020) presented deep reinforcement learning method that encodes the dynamic context online to achieve stable non-planar pushing task controller. Yu et al. (017)) proposed a system using Universal Policy and Online System Identification (OSI) function to explicitly incorporate model parameters to adapt to varying environments. These methods usually identify the environment parameters (ex- plicitly or implicitly) and then adjust the controllers to adapt to the new environment.
Differentiable Simulation. In recent years, researchers have built more efficient and feature- complete differentiable physics engines. These engines support 3D rigid body and contact con- straints between spheres and planes (Degrave al. 2016), analytic differentiation of a linear com-
2

---

Under review as a conference paper at ICLR 2022
UC
Tt
falaz, Tel
f
DiffOSI
( Fi)
Figure Overview of our method, UC-DiRIOSL. The Universal Controller (UC) takes as input the current robot states and the dynamics parameters fo identified by the differentiable physics engine (DiffOSI), to generate optimal control actions Tt-
plementarity problem (de Avila Belbute- al. 2018 modeling soft bodies via differentiable real-time differentiable Material Point Method (Hu et al. 2018). support differentiable cloth sim- ulation (Liang al. 2019), optimize for large numbers objects and contact interactions (Qiao et al. (0200) and support articulated rigid bodies with contact (Werling et al. (021). Prior work, such as Toussaint et al. (2018) and Heiden et al. (2019), also showed that differentiable physics can be integrated for end-to-end controller learning, in addition to parameter learning. Patavallabhula jet al. (202I) further integrated differentiable rendering to remove dependency on 3D vision in an end-to-end g pipeline.
METHODS
Our goal is to design a system that can handle changing or unknown dynamics in the environment. The true dynamics in the target environment can be described by #t+1 = fy ( Tz), where (q, q) denotes the robot's sensed states and their time derivatives, and denotes the control actions. fu evolves the system from timestep t to + with dynamics parameters μ.. We aim to predict optimal controls T² which maximize task performance. The controls are predicted in the first part of The second part is a differentiable physics
system, the universal controller (UC): (z, 14) engine that performs online system identification (DiffOSI): {(z, I++1, T:)) } + μ.. The overview of the system is shown in Figure I unknown or changing environment dynamics.
Together, they form robust controller capable of handling
3.1 LEARNING A ANIVERSAL CONTROLLER
Universal Controller (UC) augments a regular robotic controller by conditioning it on parameters of the environment μu, such friction coefficient robot payload. This information is crucial for the controller to select appropriate actions for different environments, yet are non-trivial to infer directly from sensory input. By providing this additional information the UC, we expect it to outperform a regular policy given the true environment parameters.
A successful UC should perform near-optimally for wide range of p's. Given that the best way to obtain control policy can be different across tasks, the training of UC largely depends on the task be performed. In this work, we tailor the training of UC to two control tasks of interest: cart pole balancing and table wiping. For the cart-pole balancing problem, we want to obtain controller that directly sends torque commands to the robot at high-frequency. Thus, we directly apply reinforcement learning approach to obtain Universal Control Policy done in Yu et al. (2017). On the other hand, for the table wiping problem, we adopt a hierarchical control structure where the learned UC needs to modulate the parameters of low-level admittance controller per wiping motion. black-bor optimization technique more suitable for this frequency problem. More details on how train UPs can be found in Section
Note that we train our UC with a set training environments 9p. which approximates the target environment fp- la and ja need not represent the same of parameters, and neither do fa and
3

---

Under review as a conference paper at ICLR 2022
Ga need to represent the same model, as the exact governing equations of fa is usually unknown. We aim to use a Ir diverse enough to train a robust UC and expressive enough to approximate all possible trajectories evolved with fj.
3.2 DIFFERENTIABLA PHYSICS ONLINE SYSTEM IDENTIFICATION
We use differentiable physics to identify some unknown physics parameters ja R that parameter- ize the dynamics of the system. The numerical modeling, #+++1 = 9p (#r, 7:), in the differentiable physics engine should be the same as in UC training approximating the target environment dynam- ics, #++1 = fa ( T²). This way, the nature of the UC's imputs stays consistent.
To perform system identification, DiffOSI requires first collecting a small number of samples X {æ,, } from target environment fa. DiffOSI uses these samples to optimize for a f that minimizes the differences between the resulted trajectory (#+, f:) and the target state-action history (æ,, F.). DiffOSI requires minimum of two samples, but if the problem is nondeterministically under- constrained (c.g., in the presence contact), more samples (e.g.. 30-50) may be required to exercise all dynamics of the system.
At the beginning (first iteration = 0), we initialize j = o4l (e.g., mean of expected distribution). For each iteration of DiffOSI optimization k execute the UC for certain number of steps Th < [ using actions predicted with (C ftx). With the collected samples Xg, we use DiffOSI to optimize for the fgg that minimizes the following objective function:
((Xx) = E 6(9 + +(Q+11.(++))
(1)
1EXx
where (dr+1, d(+1) gau (g:, qt., Tz)
and Φ is any differentiable distance function.
diffcrentiable physics engine enables the computation of gradients of with respect to the un- known parameters J.:
ac(X;)
(3)
ajp
Our system agnostic the choice the differentiable physics engine. We use the Nimble dif- ferentiable physics engine (based on DART) by Werling et al. (2021), which has the advantage of being able to handle articulated rigid bodies and differentiate through contact.
EXPERIMENTS
4.1 TASK EVALUATION OVERVIEW
We compare proposed algorithm (UC-DifIOSI) six baseline methods:
1. Domain Randomization (DR): Optimize controller in an environment where dynamics parameters are randomized.
2. UC-Random: UC given random parameters input.
3. UC-Average: UC given the middle parameter of the training range input,
4. UC-MLP (Yu et al.): UC given parameters predicted with an MLP.
5. UC-CMA-ES: UC given parameters optimized using CMA-ES (Hansen 2016).
6. UC-Oracle: UC given ground truth parameters as input.
to address the following questions:
Does UC-DiffOSI outperform Domain Randomization (DR)?
Is DiffOSI more robust than MLP, in generalization new environments?
Is DiffOSI more efficient than CMA-ES?
4

---

Under review as a conference paper at ICLR 2022
Table 1: Results on the Cartpole task. We report errors on the parameter estimation of μ (lower is better), as well as the overall task performance (higher is better). Mean and standard deviation are reported. Results are averaged over models and 10 episodes. Task performance defined as the number of simulation steps where the state of the cartpole is within certain thresholds.
Approach
Mean Abs. Error (1))
Mean Abs. Rel. Error (μ)
Task Performance
DR
N/A
N/A
277.77 + 169.63
JC-Random
0.45 = 0.11
1.42 + 1.50
169.63 = 127.92
UC-Average
0.39 + 0.15
0.50 + 0.00
215.83 + 126.89
UC-MLP-Narron
0.32 + 0.12
0.60 0.13
242.77 + 157.96
UC-MLP
0,09 = 0.03
0.38 0.43
309.23 = 72.55
UC-DiffOSI (Ours)
0.09 0.07
0.13 0.05
390.77 = 95.67
UC-Oracle
0.00 0.00
0.00 + 0.00
451.33 + 48.59
We compare our proposed algorithm with baseline methods in both task performance (total toward over one episode) as well as prediction accuracy for the environment parameters (mean absolute error and mean absolute relative error).
4.2 CARTPOLE
Task Description. We first evaluate our algorithm on the cartpole task from OpenAl Gym (Brock- man et al. 2016). where the goal is to balance pole on a cart-on-track. The state space of the cartpole task consists of the position of the cart and pole
(x, 0), as well their velocities (+;, 0). To create different environment variations, we offset the center of mass of the pole from its geometric center by a random displacement (j, 0.2pu) in 2D, with uniformly sampled within the range [-0.6m, 0.6m]. This is mimic attachments of different objects on the pole We ran- domly initialize the and of the cart and the pole, with x E [-0.05, 0.05], & E [-0.55, 0.05], 0 [-0.05, 0.05]. & E [-0.05, 0.05]. The controller can apply an impulse of [-500N, 500 to the cart at 50 Hz. We define the task performance metric as the number of simulation steps where |0| S 2/2 (where 0 0 means the pole is upright) and |x|| < 2.5 are satisfied.
Controller Details. As classic control problem, there a numerous ways to design a controller for the cartpole task. In this work, we leverage reinforcement learning-basee approach. Specifically, we use Proximal Policy Optimization (PPO) (Schulman et al. 2017) to train a control policy (MLP with 2 layers, 64 hidden units each) that takes the cartpole state (q,4) input, and predicts the appropriate control force applied to the cart.
For the domain randomization (DR) baseline, we train the control policy with j randomly selected at the beginning of each training episode. We then augment the policy input with ja to obtain the universal controller (UC). By providing this critical information about the environment, the policy can better decide the optimal action to take, which is demonstrated in our evaluation results.
System Identification. For all methods, we use the observations containing (qr, q: Tt, q-+1 (.+1). For the UC-MLP baseline, we train an MLP (3 hidden layers with 256, 128, and 64 hidden units respectively) predict f from the states and actions in the history. We use dataset collected by executing + = UC(x, (a) in the cartpole simulation environment 9a for uniformly sampled μ. Both UC-CMA-ES and UC-DiffOSI optimize j minimize the MSE between the result and target trajectories.
Results. Results for the cartpole task can be found in Table I Our approach achieves better results than the baseline methods in terms of task performance and is close to the upper bound (UC-Oracle). This suggests our algorithm can successfully infer the environment parameters from the input his- tory of past observations and actions. This is further supported by the low prediction error of our algorithm. We observe that UC-MLP can achieve low mean absolute error in predicted fá, yet per- forms worse in relative prediction error. This is possibly due to the model learning focus regions where the true ç has higher magnitude, which also leads to inferior task performance. This can possibly be mitigated by tuning the loss function training data further, yet these are task spe-
5

---

Under review as a conference paper at ICLR 2022
100
100
400
Rowaro
300
0..00
0:31
0.10
120041
11.00
11020
10255
0:133
Timestops
lon)
Timestops
(a) Evaluation curves in the Nimble physics engine (averaged across 20 runs).
(b) Sim-to-Sim Transfer results in the Bullet physics engine (averaged across 20 runs).
Figure 2: Evaluation curves for the DR and UC-Oracle models on the Cartpole task.
cific, non genereeeralli processes. In addition, we also evaluate MLP- which trains MLP model to predict the environment parameter, but with j sampled from a narrower range ((-0.2, 0.2).. As shown in Table T MLP-Narrow does not generalize parameters outside the training range.
4.3 HALF CHEETAI
In this section, we evaluate our method on 7-link, 6-DoF 2D half cheetah task (Figure 3ap.
Task Description. We adapt the half cheetah environment from OpenAI Gym (Brockman et al. 2016 and replace the simulator with the NimblePhysies simulator. The mass of the half cheetah's torso is unknown, and uniformly sampled within the range [1, 50) kg (where the original torso mass 4.9 kg).
Controller Details. Similar to cartpole, we use the PPO algorithm to train a control policy (MLP with 2 layers, 256 hidden units each). We train 6 different seeds for each approach, DR and UC, where UC is trained with ground truth mulrue, corresponding to the UC-Oracle results.
System Identification. Both the MIP and DiffOSI methods for system identification use the same inputs as those used in the cartpole task in Section 4.2 The MLP (2 hidden layers with 512 and 128 hidden units respectively) trained on a dataset of 600K examples generated from rolling out trajectories from the trained UC-Oracle models over uniformly sampled μ. The DiffOSI module uses NimblePhysics as its simulator, and optimizes p (initialized to the mean of the distribution) to minimize the L2 norm between the predicted and target next state.
Results. In Figure 3b we show that UC-Oracle able to outperform pure domain randomization. This shows that observation of physical parameters is beneficial for this task. Evaluations of various approaches using the trained DR and UC policies are shown in [able[] We show that UC-DiffOSI able to perform comparably to the upper bound, UC-Oracle, due its low parameter estimation errors. UC-MLP has, on average, higher parameter estimation errors than UC-DifrOSI, but lower parameter estimation errors than random or average baselines (Figure 3c). However, we found that for some trajectory rollouts, UC-MLP errors tend to magnify over time (Figure 3d is one such example). This shows the weakness of UC-MLP the UC depends on perfect MLP for system identification, and any slight mistake the MIP makes can quickly cause errors to compound over time. These examples can lead an even larger scale of failure at the task performance level (relative to the scale of μ estimation errors). Thus the overall task performance of UC-MLP is worse than UC-Randon and UveAvere
We also analyze the runtime of UC compared to MPC baseline. To achieve reasonable task performance, MPC baseline needs to have a planning horizon of more than 200 timesteps and replan every 40 timesteps. Each trajectory optimization in MPC takes 100-300 milliseconds to solve while evaluating UC for 40 timesteps only takes roughly 40 milliseconds. As such, MPC is not a suitable method for setting like ours where realtime performance is important.

---

Under review as a conference paper at ICLR 2022
UC-Oralle
4020
3500
Reward
2000
2000
1020
2.0
Terreslegs Million))
(a) The 7-link, 6-DoF 2D half cheetah task, simu- lated in NimblePhysics.
(b) Evaluation curves comparing DR and UC-Oracle seeds).
25
Rancom
50
Random
40
DiffOSi
atzolltte enoor (ma)
stzzolute entor (mai
MM
-00
20
10
10
0
200
400
E00
800
1000
200
400
600
800
1000
episode timestep
episode timestep
(c) Average absolute estimation error acrOss episode timesteps (3 seeds).
(d) Absolute estimation error on single example across episode timesteps.
Reward and estimation results for the half cheetah task. Results are averaged over models for each approach.
Figure 3:
Approach
Mean Abs. Error (μ)
Mean Abs. Rel. Error (1)
Task Performance
DR
N/A
N/A
2546.51 t 1593.81
UC-Random
19.11 12.97
2.27 3.42
3567.45 588.88
UC-Average
17.97 + 2.85
2.24 + 2.63
3596.58 + 802.73
UC-MLP
9.25 + 10.20
1.27 2.25
3102.02 2202.76
UC-DifFOSI
0.06 + 0.14
0.01 + 0.02
3930.97 + 801.29
UC-Oracle
0.0 + 0.0
0.0 + 0.0
3936.30 + 609.14
Table 2: Results on the Half Cheetah task (trained on 6 seeds each for DR and UC, each model evaluated on 3 random seeds).
4.4 TABLE WIPING
Task Description. In this task, wipe tabletop using wiping tool attached robot arm's end effector, as shown in supplementary Figure The wiping tool joint has unknown stiffness and damping coefficients μ {kpp. ka)). To perform a wipe, we track a scripted wiping trajectory {q. tra.) with an admittance controlled robot arm. The goal is to maintain target contact force on the contact normal direction fz.gout during the whole wiping process, i.c., to minimize the MSE between f fr,sensed and f fz.goal for all observed states when the wiping tool in contact with the tabletop. The system overview can be found in supplementary Figure 5b with details follows.

---

Under review as a conference paper at ICLR 2022
Table 3: Results on the wiping task. We report errors on the parameter estimation of μ (lower is better), as well as the overall task performance (lower is better). Mean and standard deviation are reported. Results are averaged over 10 runs.
Approach
Mean Abs, Error (p)
Mean Abs. Rel. Error (p)
Task Performance
DR N/A
N/A
1.69 1.58
UC-Random
39.038 22.157
3.607 7.874
1.07 + 0.78
UC-Average
22.296 = 12.333
2.120 + 4.129
0.92 + 0.73
UC-MLP
36.875 + 20.324
3.508 + 6.569
0.93 + 0.75
UC-CMA-ES
0.418 + 0.502
0.029 + 0.037
0.50 + 0.32
UC-DiffOSI
0,001 + 0.001
4.3e-05 + 4.0e-05
0.51 + 0.32
UC-Oracle
0.000 + 0.000
0.000 + 0.000
0.51 + 0.32
Controller Details. The task controller is defined by the combination of the admittance control law and the admittance control parameters 0 (ky, kg))
For the DR baseline, we optimize for the optimal admittance control parameters in the simulation environment 9p
The goal of the UC is to apply admittance control with the optimal control parameters mapped from the identified ja. We collect dataset (85 examples) to train the ANN-parameterized UC (2 hidden layers, 512 units each with ReLU activation functions and a linear FC layer). The dataset contains paired examples of juruue and corresponding optimal control parameters 0" (k* kg)-
System Identification. Our goal with system identification for the wiping task is to predict dynamics parameters i to condition the UC. For all approaches with UC, we first collect a trajectory in the environment (parameterized by some unknown (i) using canonical admittance control parameters Ó. From the collected trajectory, we uniformly sample 50 segments from the history perform system identification, where each segment contains (g,, 9++1 q: dras).
Both UC-CMA-ES and UC-DiffOSI optimize for the f that minimize the MSE between the resulted and target sampled segments from the collected trajectory.
For UC-MLP, we feed the same 50 segments to the MLP and predict f.. The MLP (same architec- ture UC) is trained dataset mapping trajectory segments to Perue- The Perue is uniformly sampled, and the trajectories are generated by executing the UC with Parue + 0.
We evaluate all baselines on the wiping task and report our results in Table For reference, also include UC using random or average ý as baselines. UC with average j can be seen as an offline system identification method. For simulation, we use timestep of 1e-3s. For all our experiments, we set Fz.goel = 3N and uniformly sample µ [1, 100]. DR performs worse than all UC-based approaches, which indicates that UCs provide useful chysics-awarent for the wiping task. UC-MLP performs slightly better. Both UC-CMA-ES and UC-DifIOSI outperform all baselines and are close to UC-Oracle.
Results.
Evaluating Higher Number Dimensions. To further compare UC-CMA-ES and UC-DifFOSI, we evaluate the same wiping task where ş is higher dimensional. We increase the number of DoFs with unknown stiffness and damping coefficients. The convergence curves can be found in supple- mentary Figure 6 In both the 2D and 3D optimization cases, UC-DiffOSI converges more quickly than UC-CMA- ES.
4.5 ROBUSTNESS TO VARYING DYNAMICS
In this example, we evaluate whether our proposed algorithm can handle varying dynamics through- out an episode. During each episode, we change f at t 250 (halfway through the episode) to randomly sampled 4í' ~ |-0.6, 0.6]. As shown Table 4 our algorithm achieves notably better prediction accuracy than UC-MLP. However, the task performance is not good the oracle ver- sion. This is because in order to detect abrupt changes in the system dynamics, we need to collect sufficient data with the new system, which leads to delay in identifying the correct parameters. For
8

---

Under review as a conference paper at ICLR 2022
Table Results on the Cartpole task with changing μ. We report errors on the parameter estimation of je (lower is better), as well as the overall task performance (higher is better). Mean and standard deviation are reported. Results are averaged over 3 episodes.
Approach
Mean Abs. Error (1)
Mcan Abs. Rel. Error (p)
Task Performance
UC-MLP
0.13 0.06
0.92 0.84
192.33 160.02
UC-DiffOSI
0.02 + 0.01
0.27 = 0.18
222.33 + 89.09
UC-Oracle
0.00 = 0.00
0.00 = 0.00
435.00 91.92
Table 5: Results sim-to-sim transfer (Nimble to Bullet) the Cartpole task. We report emror on the parameter estimation of μ (lower better), as well as the overall task performance (higher is better). Mean and standard deviation are reported. Results averaged over 20 models and 100K timesteps.
Approach
Mean Abs. Error (1)
Mean Abs. Rel. Error (/4)
Task Performance
DR
N/A
N/A
297.26 + 197.85
UC-DiffOSI
0.02 0.04
0.03
0.06
434.10 110.88
UC-Oracle
0.00 + 0.00
0.00 = 0.00
451.00 + 73.81
abrupt and large changes in fa such as the one we used, the task performance can be sensitive to the accuracy of the prediction.
4.6 ROBUSTNESS TO NOVEL DYNAMICS MODELING
To evaluate the ability of our approach to generalize environment variations beyond the training range, we apply our cartpole controller to the same task implemented in PyBullet physics engine. Due to differences in how the two physics engines solves the equation of motion and performs integration, a policy trained in UC-DifiOSI (Nimble) does not transfer directly to PyBullet. As shown in Figure 2b method still outperforms baselines. The gap is even larger between method and DR, which suggests that in cases where the dynamics gap is larger, UC-DiffOSI can offer larger improvements.
5 CONCLUSION
UC-DiffOSI a learning-based approach for training control policies that can operate changing and unknown environments. Our method combines domain adaptation with differentiable physics simulator by first training Universal Controller that is conditioned on the environment param- eters and then using differentiable physies engine, NimblePhysics, parameters from a recent history of robot sensory inputs. By using a differentiable physics engine, achieve efficient and generalizable system identification compared to prior methods based on learned models or traditional system identification. We evaluate our method on two robotic control problems: cartpole balancing and table wiping with a robot arm. Our method achieves superior performance than the baseline methods and is able to handle changing or un modeled dynamics.
identify the environment
There are promising directions that further extend our work. For example, our algorithm currently assumes that the Universal Controller can optimally handle different environments given an accurate estimation of the environment parameters. However, in real applications, the robot might encounter situations that are beyond the capability of pre-trained UC. Determining how to efficiently fine- tune UC with limited data is thus important future work, where differentiable physics engines could play pivotal role. In addition, this work focuses rigid body environments, while real-world environments are filled with objects that can deform. An interesting direction could be to extend approach to handle deformable objects while retaining high efficiency in the online system identification. Finally, we plan to apply our approach to real robot hardware, requiring bridging the sim-to-real gap and running the end-to-end control pipeline in real time.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To maximize reproducibility. we describe our methodology in detail Section 3 and our experi- mental setup in Section 4 The code, based upon on the open-source physics simulator (Nimble-
Physics (Werling et al. 2021 ) will be released upon publication to facilitate future research.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Alexander Clegg, Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. In SIGGRAPH Asia 2018 Technical Papers, pp. 179, ACM, 2018.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Saan-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503, 2015.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiable physics for learning and control. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (ods.), Advances in Neural Information Processing Systems. volume 31. Curran Asso- ciates, Inc., 2018 URL https::
neur.ss...e.usso.. wttp:///.....cpppe////////////// pdf
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis Wyffels. differentiable physics en- gine for deep learning in robotics. CoRR, abs/161 1.01 101 2016. [tll:ttt///arxxi.orrr
abs/16 01652
Ioannis Exarchos, Yifeng Jiang, Wenhao Yu, and C. Karen Liu. Policy transfer kinemati domain randomization and adaptation. CoRR, (bs/2011.01891, 2020. URL httpss//aaxxiv.orr
abs/2011. 01891
Nikolaus Hansen. The CMA evolution strategy: A tutorial. CoRR, abs/1604.00772,, 2016. URL
/ttt:://axivy wwwpiiiorr//..oo
Nicolas Heess, Jonathan Hunt, Timothy P. Lillicrap, and David Silver. Memory-based control with recurrent neural networks. CoRR, abs/1512.00455,, 2015. URL attp://arxiv.ort
abs/1512.04455
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours rich environ- ments. arXiv preprint
2017.
Eric Heiden, David Millard, Hejia Zhang, and Gaurav S. Sukhatme. Interactive differentiable simu- lation. CORR, hhts//90...n. 10706, 2019. URL. http:://axxvv.or////aa /190.. 10706
Yuanming Hu, Jiancheng Liu, Andrew Spielberg. Joshua B. Tenenbaum, William T. Freeman, Ji- ajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical /bbs/
simulator for soft roboties. CoRR, 1000000,,,, 2018. URL
1810.0100 01054]
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. CoRR, abs/1901.08652. 2019.
wwtp:ii.or///b//1//010g0 08652
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differentiable sim- ulation for system identification and visuomotor control. CoRR, abs/2104.02646, 2021. URL
https: http:s////xxxioor///aasss..
02646
Yifeng Jiang, Tingnan Zhang, Daniel Ho, Yunfei Bai, C Karen Liu, Sergey Levine, and Jie Tan. Sim- gan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. arXiv preprint arXiv:2101.0600 2021.
Junbang Liang, Ming Lin, and Vladien Koltun. Differentiable cloth simulation for inverse prob- lems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL ht httpss::/ppooceedaggggg wwwp:eeee...ee.eeei..e.ee....... hct.pp/...c//pprr///////11 238. htt.::////ppeeeccooo..eeeeeeeeep

---

Under review as a conference paper at ICLR 2022
Michael Neunert, Thiago Boaventura, and Jonas Buchli. Why off-the-shelf physics simulators fail in evaluating feedback controller performance-a case study for quadrupedal robots. In Advances in Cooperative Robotics, pp. 464-472. World Scientific, 2017.
OpenAl, Ilge Akkaya, Marcin Andrychowicz. Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07113, 2019. URL
htte:://arxxiv.rr//ab///// .07113
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer 2017. URL http:
of robotic control with dynamics randomization. CoRR, // www....ogg.aa//aa.....aa
Yi-Ling Qiao, Junbang Liang. Vladlen Koltun, and Ming C. Lin. Scalable differentiable physics for learning and control. CoRR, abs/2007.02166 2020. URL https://arxiv..rg//3/
0007702118
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
István Szita, Bálint Takács, and András Lörincz. E-mdps: Learning in varying environments.
J. Mach. Learn. Res., 34m111):445-174, March 2003. ISSN 1532-4435. doi: 10.1162/ 153244303768966144 48. URLpttps:://de wrt.or.g///......... 10. 12/111/111 112116662333338816111111
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. CoRR, abs/1 1044 10332, 2018. URL/tttp://axxiy
10332
Marc A Toussaint, Kelsey Rebecca Allen, Kevin A Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
Keenon Werling, Dalton Omens, Jeongscok Lee, Ioannis Exarchos, and C. Karen Liu. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. CORR, abs/2103.16021, 2021.
wwwpr///.rrr//////////// abs/ 03. 16021
Zhuo Xu, Wenhao Yu, Alexander Herzog, Wenlong Lu, Chuyuan Fu, Masayoshi Tomizuka, Yunfei Bai, C. Karen Liu, and Daniel Ho. COCOL: contact-aware online context inference for general- izable non-planar pushing. CoRR, abs/2011.11270, 2020.
201
Wenhao Yu, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. CoRR, abs/1702.02453, URL
abs/ 02453
Wenhao Yu, Greg Turk, and C. Karen Liu. Learning symmetry and low-energy locomotion. CoRR, abs/1801.08093, 2018. URL http:://ax....rgg/////o wtw.////////.....o.///// 08093
12

---

Under review as a conference paper at ICLR 2022
DifIOSI
UC DifIOSI (N-3)
UC DIfOS! (N-5)
absolute error (mu)
0.4
0.2
0.0
-0.2
1000
episode timestep
Figure 4: Analyzing the effect of the number of samples (window size from history) used by UC- DiffOSI vs. the absolute error prediction for the half cheetah task. N denotes the context window size, and the number of samples required is N 1.
A LATENCY
We additionally analyze the effect of the number of sampled used by DiffOSI on the prediction accuracy of få. Results are shown in Figure 4 We compare different context window sizes (2, 3, and 5) and plot the absolute μ error as a function of the episode timestep for the half cheetah task. We find that single sample (corresponding
single step, with context window size of 2 containing states before and after the step) is sufficient to obtain an accurate prediction μı; more samples does not help. single optimization step of DiffOSI takes 2-3 milliseconds.
B WIPING
In this section we show additional diagrams and results the wiping task. Figure 5a and Figure 5b depict the wiping task as well as an overview of the system we implemented. Figure 0 shows comparison between CMA-ES and DifiOSI terms of absolute error y predictions.
13

---

Under review as a conference paper at ICLR 2022
Wiping Universal Control
Admittance Control
T
fa Ti.
Arg.
Universal Control
fe
backdrop
Wiping Canonical Control
T
Ap cry Ti
iP
Loss
x
DiffOSI
Wiping Canonical Control
Admittance Control
fa Ti)
(K+ &xg
Target Environment Sample
(b) Overview applying our method, UC-DiffOSI, on wiping task. From bottom to top, we first collect samples with a canonical wiping controller. Then, DiffOSI optimizes for the en- vironment parameters with the same controller. Finally, pre-trained DNN DNN- parameterized UC takes the identified ç and generates optimal wiping control.
target environment
(a) Visualization of the wiping task performed in NimblePhysics simu- lation.
CMA 65
CMA-F5
17.5
040004 (201
23.5
3/mOGI (30)
35.0
33.5
20.0
200
2000
3.2
300
aa) 2D Optimization.
(b) 3D Optimization.
Figure 6: Comparison of CMA-ES and DifTOSI on the parameter estimation task. Results averaged over 3 runs and all dimensions.
14


Reviews:
[{"review_id": "el4dTBUh_7z", "reviewer": "Reviewer_S8au", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis paper takes advantage of advances in differentiable physics (e.g., Werling et al. (2021)) and combines this with the idea of Universal Controllers (Yu et al., (2017)) to build controllers that are more robust. The proposed approach outperforms baselines in two benchmarks.\n\nWeaknesses:\n\nThis paper seems to only differ from Yu et al., 2017 in that it replaces the MLP for OSI with a differentiable physics simulator. Given that the paper easily combines already published ideas, I would expect rich experimental results. They use two tasks of cartpole balancing and tabletop wiping. Other works have more complex domains such as 4-link, 6-DoF 2D hopper and the 18-DoF Unitree Laikago quadruped in Jiang et al., (2021)  and Fetch robot in Peng et al., (2017).\n\nMinor:\n\nSome figures are not referenced in the text (e.g., Figure 3 and 4, in Figure 2 only 2b is referenced).", "summary_of_the_paper": "The paper tackles the problem of learning robot controllers that can handle changing or unknown environments. It proposes to use differentiable physics for online system identification and reinforcement learning for offline policy training. The differentiable physics module estimates simulation parameters from robot history and feeds this to the controller that is parameterized by these simulation parameters. They use domain randomization to ensure the universal controller conditioned on simulation parameters is robust to changing environments (simulation parameters). At test time, the differential physics simulation is used to estimate the simulation parameters to bias the controller to output controls for the 'correct' simulation parameters.\n\nThe proposed approached is evaluated against several benchmarks on the cartpole and a tabletop wiping tasks and has been shown to outperform the baselines, which include domain randomization and some domain adaptation approaches.", "summary_of_the_review": "Given minimal modifications from existing work, this work is missing comprehensive experiments. Experiments on real robots as mentioned as part of future work would strengthen the paper significantly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dPwf6Xqw-j", "reviewer": "Reviewer_3zvY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong:\n\n- The paper applies the recently developed differentiable physics and succeeds in applying it in a wiping task that requires contact-rich environment interaction. This will encourage the development of differentiable physics in the future.\n- Combining differentiable physics with UCOSI is a natural idea and performs well in practice.\n\nWeak:\n\n- Applying differentiable physics for system identification has been seen in the previous work. Combining it with UCOSI method is rather straightforward. The paper's novelty is limited.\n- Changing the environment from NimblePhysics to PyBullet doesn't mean good Sim2Real performance. The current paper only includes a very limited set of environments (cartpole and wiping), which is not enough to prove its effectiveness in practice.\n- The number of parameters to identify in both environments is small. In this case, using finite-difference to compute gradients would be a not bad replacement for a differentiable physics engine. It would be better if the authors can consider environments with more dofs. For example, they can consider identifying the Inertia matrices of the robot arms.", "summary_of_the_paper": "In this paper, the authors propose UC-DIFFOSI, which combines a differentiable physics simulator for system identification, and a universal controller which takes the identified parameters and passes them into the neural network to output actions. The method achieves better performance than domain randomization baselines and other system-identification methods.", "summary_of_the_review": "The paper proposes a nice framework that combines differentiable physics with UCOSI. However, employing differentiable physics for system identification has only limited novelty. The experiments are a little bit weak with only 2 simulation tasks and no real-world environments. I think the paper's contribution doesn't pass the bar for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b6EZZTKRFw4", "reviewer": "Reviewer_hY3g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is the usage of a differentiable physics engine for online system identification, which supplements a universal controller(UC) trained offline. \n\nThe main concern is in its current form, the paper is not clear about the trade-off of the proposed approach. The results are not surprising: 1) Doing system ID is better than domain randomization and 2) Using a differentiable physics engine is better than using MLP prediction / blackbox optimization (e.g. CMA-ES) for system identification. Since either using UC nor using a differentiable physics engine for system ID is novel, while the combination of the two is what the paper is proposing, it can be clearer on the trade-off this combination offers. The paper could improve its quality if the following questions can be answered (1 is the major one while 2 and 3 are minor):\n\n1. Latency: how fast is DiffOSI? Is there a trade-off between number of samples versus accuracy? How sensitive the task performance is with regard to latency (time spent on) in system ID?\n2. Why are the number of runs/models different between table 1 (3 models) and figure 2 (20 runs).\n3. Figure 5: 3 runs are not statistically significant. It's also a bit weird seeing CMA-ES achieves similar error around 200 evaluations for both 2D and 3D cases.", "summary_of_the_paper": "The paper proposed to use differentiable physics engine for online system identification, which is then combined with controllers trained in a wide variety of tasks for robotic control problems in environment with unknown dynamics. ", "summary_of_the_review": "The idea of combining  a differentiable physics engine for online system identification with a universal controller(UC) is interesting but the paper could improve its quality by adding more details on how the two modules work for continuous control tasks (e.g. regarding latency).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IJRG2perXod", "reviewer": "Reviewer_8qJx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper idea is overall interesting and fits very well with the current developments of differential simulation systems. The main strengths are:\n1. It empirically shows that domain randomization is insufficient to solve complex control problems with varying dynamics. This is a surprising finding, which could help future research in robotics.\n2. It shows that learning a universal control with a simple MLP architecture is possible if conditing it on robot parameters.\n3. The paper is well written and easy to follow.\n\nIn my opinion, the major weakness of the current submission are:\n1. The experimental results are not sufficient to back up the claims made in the introduction. Most of the experiments are done on the cart-pole system, which has been a benchmark of the control community for decades and is very simple to model and control. In the single experiment with a manipulator, the performance is slightly inferior to a baseline. Therefore, I would recommend performing experiments on a more diverse set of robots and tasks. The most interesting experiments are the ones in which the environment suddenly changes, where I think the proposed approach could have advantages over existing systems. Therefore, I would argue that sections 4.4 and 4.5 could be vastly extended.\n2. There are no state-of-the-art model-based reinforcement learning baselines. They are very similar in their problem formulation (identifying a model and then using it for control via online or offline optimization). Therefore, I believe them to be valid baselines. Possible examples are PETS [R1] or Dreamer [R2]. As mentioned above, they would probably need many samples to adapt to changes, so this could be a selling point for the approach.\n\nOther comments are:\n* The approach used for table wiping is very similar to the one used for object cutting (in the real world!) proposed by [R3]. I think that this deserves at least mention and possibly quantitative comparison. \n* It would be nice to have some demonstrator where differential simulators shine, e.g., for deformable objects.\n* What would happen if the policy input were very high dimensional, for example, images? What would be the changes required to make the approach work? \n* What about using a model-predictive controller (e.g., MPC) instead of a universal controller defined by an MLP? In theory, the MPC would require the same input of the MLP, i.e., the platform state and the robot parameters (i.e., the model). This could be very easily implemented, at least for the cartpole.\n\n[R1] Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models, Chua et al.\n[R2] Learning latent dynamics for planning from pixels, Hafner et al.\n[R3] DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting, Heiden, et al.\n", "summary_of_the_paper": "This paper proposes an algorithm to control robots with a universal controller conditioned on the robot parameters, that are identified online using differential simulation. The idea is simple yet interesting: giving the controller explicit information about the system could definitely help performance. Estimating those parameters with differential simulation definitely makes sense. The approach could be valuable for helping robots handle internal or external changes during operation. The approach is evaluated on a series of rigid body control tasks, where it is on pair or better than previous solutions.", "summary_of_the_review": "The paper presents an interesting idea, but more experiments on different systems and tasks, and possibly a more diverse set of baselines, could significantly strengthen the submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the learning of robot controllers for changing or unknown environments. It makes use of differentiable physics for online system identification and of reinforcement learning for offline policy training. A universal controller is trained on a distribution of simulation parameters in order to ensure its robustness. Differentiable physics is used to estimate the simulation parameters from the recent observation history. These parameters are fed to the controller so as to modulate the policy. This approach is evaluated on three benchmarks (2 + 1 added during the rebuttal).

The main originality of the paper is the use of differentiable physics for the identification of the parameters in the context of varying environments. The topic is of interest and in line with the recent developments for robotics. However, the novelty is limited, and all evaluators were concerned about the limited experimental contribution. The authors have added a new experiment during the rebuttal but this was not considered sufficient to change the evaluation. Overall this is considered as a promising contribution, but the experimental setting should be largely improved with additional problems and comparisons with SOTA methods from the recent RL literature.
Decision: Reject