What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
Authors: Peide Huang
Xilun Zhang
Ziang Cao
Shiqi Liu
Mengdi Xu
Wenhao Ding
Jonathan Francis
Bingqing Chen
Ding Zhao
Venue: CoRL 2023 Poster (2023)
Keywords: sim-to-real gap
reinforcement learning
causal discovery

Abstract: Training control policies in simulation is more appealing than on real robots directly, as it allows for exploring diverse states in an efficient manner. Yet, robot simulators inevitably exhibit disparities from the real-world \rebut{dynamics}, yielding inaccuracies that manifest as the dynamical simulation-to-reality (sim-to-real) gap. Existing literature has proposed to close this gap by actively modifying specific simulator parameters to align the simulated data with real-world observations. However, the set of tunable parameters is usually manually selected to reduce the search space in a case-by-case manner, which is hard to scale up for complex systems and requires extensive domain knowledge. To address the scalability issue and automate the parameter-tuning process, we introduce COMPASS, which aligns the simulator with the real world by discovering the causal relationship between the environment parameters and the sim-to-real gap. Concretely, our method learns a differentiable mapping from the environment parameters to the differences between simulated and real-world robot-object trajectories. This mapping is governed by a simultaneously learned causal graph to help prune the search space of parameters, provide better interpretability, and improve generalization on unseen parameters. We perform experiments to achieve both sim-to-sim and sim-to-real transfer, and show that our method has significant improvements in trajectory alignment and task success rate over strong baselines in several challenging manipulation tasks. Demos are available on our project website: https://sites.google.com/view/sim2real-compass.

What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
Peide Huang', Xilun Zhang', Ziang Caol*, Shiqq Liul*
Mengdi Xu', Wenhao Ding', Jonathan Francis?, Bingqing Chen?, Ding Zhao!
'Carnegie Mellon University, "Bosch Center for Artificial Intelligence, "equal contribution
{peideh, xilunz, ziange, shiqiliu, mengdixu, wenhaod, dingzhao @andrew. cmu, edu {jon. fjoncfrancis, bingqing.char chen chen) @uss bosch. com
Abstract: Training control policies simulation is more appealing than on real robots directly, as it allows for exploring diverse states in an efficient manner. Yet, robot simulators inevitably exhibit disparities from the real-world dynamies, yielding inaccuracies that manifest the dynamical simulation-to-realit (sim-to- real) gap. Existing literature has proposed to close this gap by actively modifying specific simulator parameters to align the simulated data with real-world obser- vations. However, the set of tunable parameters is usually manually selected to reduce the search space in a case-by-case manner, which is hard to scale up for complex systems and requires extensive domain knowledge. To address the seal- ability issue and automate the parameter-tunin process, we introduce COMPASS: which aligns the simulator with the real world by discovering the causal relation- ship between the environment parameters and the sim-to-real gap. Concretely, our method learns differentiable mapping from the environment parameters the differences between simulated and real-world robot-object trajectories. This mapping is governed by a simultaneously learned causal graph to help prune the search space of parameters, provide better interpretability, and improve general- ization on unseen parameters. We perform experiments achieve both sim-to- sim and sim-to-real transfer, and show that our method has significant improve- ments in trajectory alignment and task success rate strong baselines in sev- eral challenging manipulation tasks. Demos are available our project website: https://sites, http:://sites.gggle..coo google. www.://wwwwim///aalooooooo/ compass..
Keywords: sim-to-real gap, reinforcement learning, causal discovery
1
Introduction
Training control policies directly on real robots poses challenges due to the sample complexity of deep reinforcement learning (RL) algorithms. Therefore, training in simulation is often necessary to perform diverse exploration of the state-action space an efficient manner [1, 2, 4, 5, 6, 7J. How- ever, robot simulators are constructed based on simplified models and are thus approximations the real world. For example, dynamics such as contact and collision are notoriously difficult simulate with simplified physics [8, 9]. Even if the dynamics could be simulated accurately, not all physical parameters can be precisely measured in the real world and specified in simulation, e.g., friction coefficients, actuation delay, etc. As result, robot that is trained in a biased simulator could have catastrophic performance degradation in the real world [10, 11, 12, 13J. It is, therefore, critical to use simulators that closely mimic real-world dynamics to reduce this sim-to-real gap [14, 15, 16, 17].
Existing literature has proposed to close the sim-to-real gap by adjusting the parameters of the simu- lator align the simulated data with the observed real data. To facilitate this, robot simulators such
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Causality- Domain Randomization
Causality
Gradien backprop
Rollout sim trajectories
Env parametor
Learn differentiable causal model
Optimize env params
Sim-to-real traj, dill. d,
Train agent policy
frittin
Ineriial
puck1
Rollout trajectories 2 ((c*) (")"))))
puck2
pusk2 d,
Figure 1: Overview of the COMPAS, framework.
as robosuite [18] provide APIs to modify over 400 different environment parameters. Unfortu- nately, the search space grows exponentially with the dimension of the environment parameters. To mitigate this issue, various existing methods attempt to modify the parameters more efficiently, us- ing gradient-based or gradient-free sampling-bast based techniques [19, 20]. For example, in quasi-static manipulation tasks, such as sorting pegs or opening drawers, Chebotar al. [19] chose to modify parameters related to the size, positions, and compliance; in dynamic manipulation tasks, Muratore et al. [20] chose to modify mass, friction, and restitution coefficients, etc.
However, this parameter selection process is typically carried out on a case-by-case basis, necessi- tating substantial domain knowledge to restrict the scope of environment parameters. This becomes challenging when dealing with simulations involving multiple interacting objects [21]. Furthermore, most existing methods lack the capability to offer explicit insights that effectively guide the fu- ture deployment of more complex systems. They fail to offer direct answers to the question "What went wrong with my simulator" or, more specifically, "What simulator parameters should I tune to reduce the sim-to-real gap" without post hoc analysis of the final parameters. In contrast, humans are good at analyzing complex events, identifying and eliminating irrelevant factors, and uncovering crucial cause-and-effect relationships. Such causal discovery capability enables an efficient and terpretable search [22, 23, 24] for differences between two systems, providing promising direction to bridge the gap between simulation and reality.
In this work, we propose method that aims to align the simulator with the real world by discovering the çausality between environment parameters and the sim-to-real gap (COMPASS) as illustrated in Figure 1. COMPASS learns a differentiable mapping, from the simulation environment parameters to the differences between simulated and real-world trajectories of dynamic robot-ohject interactions, governed by simultaneouslly learned causal graph. With the differentiable causal model fixed, COMPASS mack-propagates gradients to optimize the simulation environment parameters in end- to-end manner to reduce the domain gaps. Beyond the interpretability, the causal graph also helps to prune the parameter search space, thus improving the efficiency of domain randomization as well as the scalability. We summarize our contributions as follows:
1. We propose novel causality-guided parameter estimation framework close the sim-to-real gap and improve agent performance in the real world.
2. We design a dullediifferentiable model that explicitly embeds the causal structure to provide better interpretability, prune the search space of parameters, and improve generalization.
3. We empirically evaluate method in both the simulation and the real world, which outperforms baselines in terms of trajectory alignment and task success rate with the same sample size.
2 Related Work
Closing the sim-to-real gap robotic control tasks often approached through domain randomiza- tion (DR) [10, 25, 26, 27, 28, 29, 30, 31]. Although DR proves successful in numerous applications, particularly when access to real environments data collected therein is unavailable, it is recognized

---

that vanilla DR may result in overly conservative policies when the range of randomization is broad [32, 33). As an alternative approach for achieving sim-to-real transfer, system identification aims to estimate the parameters of the environment through limited interactions with real environments [20, 34, 21, 35, 36, 37] and has been combined with DR methods. We draw inspiration from this line of work in developing our parameter estimation framework, which learns causal relationships from dynamic robot-object interactions in order to facilitate the sim-to-real transfer.
Gradient-free Parameter Estimation for Sim-to-Real Transfer. Gradient-free free parameter estima- tion methods typically utilize sampling-based methods to update the simulation environment param- eters. Chebotar et al. [19] propose the SimOpt framework, which iteratively alters the distribution of environment parameters in simulation to mirror real enviromment rollouts via the cross-entropy method. Moving away from the assumption that the distribution of environment parameters follows Gaussian distribution, adopted in [19], Ramos al. [38] develop BayesSim, which uses Gaus- sian mixture model and optimizes the parameter distribution from a Bayesian perspective. Muratore et al. [20] propose Neural Posterior Domain Randomization (NPDR) which further removes assump- tions on the environment parameter distribution by utilizing neural likelihood inference methods and could handle correlated parameters. is worth noting that scaling up sampling-base -bsed methods becomes challenging when dealing with a large number parameters. In contrast, COMPASS PSSS learns a difference- preeeccccioriiiiionn model, leverages gradients to adjust the simulation parameters, and further improves scalability with learned causal structures.
Gradient-based Parameter Estimation for Sim-to-Real Transfer. Gradient- based methods typ- ically employ a neural network to encapsulate the gradient landscape of parameter differences [34, 21, 35] or to model environment dynamics [36]. TuneNet [34] uses neural network to predict the discrepancies in parameters based on the observations derived from two distinct environments. The Search Parameter Model (SPM) [35] is binary classifier, with rollouts and parameters as input, which predicts whether set of parameters is higher lower than the target ones. Unlike TuneNet or SPM, we opt predict observation differences using environment parameters as inputs. Allevato et al. [21] further expand the capabilities of TuneNet from handling single parameter to managing model with parameters. In contrast, we demonstrate COMPASS is able to scale up to a 64-paramete system, a capacity notably larger than existing works. EXI-Net [36] implements a dynamics pre- dictive model, conditioned on environment parameters, and identifies the most suitable parameters via back paek-propagation. While EXI-Net strives to model a broad spectrum of environments by sepa- rately modeling the known/explicit and implicit dynamics parameters, we aim to enhance sim-to-real transfer efficiency by capitalizing on the progressively discovered causal structure.
3 Methodology
3.1 Problem formulation: Markoy Decision Processes and sim-to-real gap
A finite-horizon Markov Decision Process (MDP) is defined by M = (S, P. R. po, T), where and AA are state and action spaces, P S x A Ax + R4 is a state- transition probability function or probabilistic system dynamics, R: A - IR a reward function, po : - R4 an initial state distribution, 7 is a reward discount factor, and T fixed horizon. Let T (5o, op..... , ST, or) be a trajectory of states and actions and R(r) = ΣT= 7 ('R(( (st. at) is the trajectory reward. The objective of RL is to find parameters 0 of a policy ny(a that maximize the expected discounted reward over trajectories induced by the policy: EE,,, [R(r), where sp ~ S1+1 P ($141/81, at), and 04 Re (or|st).
In our work, we assume that the simulator's ' system dynamics are conditioned on environment pa- rameters e E RIeT, i.e., P S x A x x RIEl + R., where is the set of all tunable environment parameters and measures the cardinality of the set. Given a simulator parameterized by e, the agent is optimizing F [.])) When the simulation dynamics are very close to the real-world dy- namics, one can expect the trajectory rollouts in the simulator be close to that in the real world as well. Hence, an optimal agent trained in the simulator would expect near-optimal performance in the real world [39]. However, due to unmodeled dynamics and inaccurate environment parameters, the
3

---

simulation dynamics are different from the real world (i.e., there exists sim-to-real gap), resulting in different trajectory rollouts and thus degradation in real-world performance [1, 2, 10, 11, 15].
For better interpretability, we assume a factorized state space, i... S (Si x x... x Sk }. with SR,t E Sk representing the k-th factorized state time 1. Each component usually has explicit semantic meanings (i.c., event object's property) [23], which holds through state and action abstraction in general [40, 41, 42]. For example, in the case of pick-and-place, the state space can be factorized to the 3D position and orientation of the object and end effector. Similar to Chebotar et al. [19], we then define a factorized trajectory difference function:
dy (Tsim: Treal)
sk.t,,im - Sk.t.realll2 for k: 1.2,..,, K
The trajectory difference function is then d = [di....,ddl dx]. and the trajectory difference d, is the output of d to measure the sim-to-real gap between of pair of trajectories, (Tsim- Treai). In this work, we aim to find a simulation environment parameter e that minimizes the expectation of trajectory differences d, under the same policy.
3.2 Learning causality between environment parameters and trajectory differences
To model the causality, COMPASS learns a causal model fa(e, a; 9) mapping the environment param- eter e and action sequence a = [a....., 2r] to the trajectory difference d,. This model contains a causal graph G, whose nodes represent the variables to be considered and the edges represent the causal influence from one node to another node. We jointly learn the model parameter ó and discover the underlying causal graph in fully differentiable manner.
The causal graph G plays a crucial role in the model by providing interpretability, pruning the search space of parameters, and improving the generalization on unseen parameters. Since we focus on the influence of environment parameter e to the trajectory difference d,, we can represent the graph with a binary adjacency matrix of size |E| K, where 1/0 indicates the exis- tence/absence of edge from the environment parameter to the trajectory difference. Motivated by previous works [43, 44, 45] that formulate the combinatorial graph learning into continuous optimization problem, we design a sample-efficient pipeline by making the optimization of G differ- entiable. We sample elements of the graph G from a Gumbel-Softman distribution [46], parametrized by & E [o, 1|| i.e., Gi ~ Gumbel Sof tmax (max(1 T = 1), where T is the softmax temperature. We denote the parameterized causal graph as Gg. All elements (i.j) are initialized to ones to ensure the causal graph is fully connected at the beginning.
Causal Graph.
Structural Causal Model. Since the causal graph only describes the connection between variables, we also need a parameterized model to precisely represent how the causes influence the effects. We design an encoder-decoder structure in S. with G as a linear transformation applied to the intermediate features. First, the encoder operates on each dimension of e independently to generate features Ze E IR Elxd, Then the causal graph is multiplied by the features to generate the inputs for the decoder, i.e., Ie 2 g e Rdd, x where d, is the dimension of the feature. Similarly, the action sequence is passed through the encoder and transformation produce the feature of the action sequence 9a Rd, x Finally, 9c + 9a is passed through the decoder to output the prediction d,.
Differentiable Causal Discovery. Given a dataset D == {ETH, am d mm the optimization objective to discover the underlying causal model consists two terms:
1 M Σ lIfole". am: +
Lo.e
where the first term is the mean squared error between the predicted trajectory differences and the real differences, and the second is regularization term that encourages the sparsity of G ([lv) I is the entry-wise p-norm of e with positive scalar À to eliminate the influence of irrelevant environment parameters. The detailed architecture of this causal model can be found in Appendix A.

---

Algorithm Causality between envirOnMent PArameterS and the Sim-to-real gap (COMPASS)
1: Input:
2: 60 E [R|E;; initial guess of environment parameters,
3: (: threshold for sim-to-real gap,
4: Sim(-): simulator with controllable environment parameters
5: Output: Eg, n
6: Initialize agent policy πe
7: Initialize fa(E, a: Gw) with dh
g|*K
8: for pari00
1, 0, 1......... Maxlter do Train Th in Sim(c;)
9:
10:
[T In= f Rollout N trajectories using Vy in Sim(c;)
[Teal Rollout N trajectories using अe in the real environment
12:
Stop the iterations if AVERAGE(d(7M)) T -.alls.... (aal)----,d(si))
d( - =(al) S
13:
D 3-6
14:
for (1, (1,...,.N] N) do
This loop in parallel
15:
{ m=1...M. f CAUSLITGUUDDDDOHHRRNOOO.TTOTOO
for m (L...., .....,,,... do
17:
sim f- Rollout To in Sim(E
d d( m real
19:
D d real
Jointly optimize model parameter & and causal graph parameter & of fole, a; Gw) + UPDATEENNPARAMME fo (ei a; Gv))
Eq. b Eq.
21:
Algorithm Causality-Guided Domain Randomization
1: function CUUSLLTTCCGUBDDDDAAAANAANOOAAATT vb) for r- 10,r=1,2,-... .,2,...,ff do
2:
3:
if max (bx) Threshold then {ETTT /m=1....M UNIFORM(E, -r, 8,)
bh, is the r-th row of ei
4:
D E, the r-th dimension of
5:
return {ETM Jmm1,...MM
3.3
Closing the sim-to-real gap via differentiable causal discovery
The main algorithm is shown in Algorithm 1. We highlight two parts of the algorithm here.
Cuuddl Domain Randomization. The learned causal graph used to prune the search space. Since each element of the causal graph parameter và indicates the probability of edge from the environment parameter the trajectory difference, we to randomize particular environment parameter not. For instance, if the learned 1 indicates that there is no causal relationship between the torsional friction and the trajectory difference, the torsional friction will be excluded from randomization in the subsequent iteration, which enhances efficiency of DR. The randomized environment parameters are sampled uniformly from certain ranges according to the current environment parameters. In this way, COMP ASS automatically reduces the search space by orders of magnitude without human supervision domain knowledge. The detail of causality-guns guided domain randomization is shown in Algorithm 2.
té to determine whether
Environment Parameter Optimization. Owing to the full diffcrentiability of our model, we can back- back-propagate the gradient information directly to the environment parameters to minimize the predicted sim-to-real gap:
I Iian(e,a
c-e-rVa
(3)
where fa.k is the k-tth dimension of the output. We use the real action sequences and apply Eq. multiple times until convergence or reaching the maximum step. The sparse causal graph G could improve the robustness against noisy trajectory data and generalization unseen environment pa- rameter values during parameter optimization.
5

---

4 Experimental Results
4.1 Experimental setups
Mini-Air-Hocker with Obstacle. We design a challenging task of playing air hockey with robot arm. To reach the goal, the agent needs to consider pusher-to-puck, puck-to-puck, puck-to-wall colli- sions, and surface properties of the hockey table. The task is to ma- nipulate the pusher to hit the first puck, colliding with the second, which in turn needs to avoid the obstacle to reach the goal position by bouncing against the wall. Similar to Evans al. [47], the ac- tion space includes the starting position of the pusher, hitting angle, and velocity. The state space includes the position of the two pucks (K' = 2). The pusher, puck, and goal have a radius of 3cm, 2.55cm, and 15cm, respectively. This task requires precise actuation since objects interact multiple times, propagating and compounding sim- to-real mismatches such that the agent can experience a significant
Pusher
Back wall
Pücki
Puck2
oot wall
Leff wall
Obstactl
Goal
Front wall
Figure 2: Experimental setup. drop in success rate. Additionally, methods need to be robust against noisy and unmodeled dynam- ics. For instance, the floating force, in reality, is generated by a fan placed at the center of the table, and thus is non-uniform and stochastic. There are 64 tunable environment parameters in experi- ments (|E) = 64), and we use parameter notations in the format of object@ param jype @ param. We use the Kinova Gen 3 robot arm in the real-world test bench and simulate in the robosuite [18] environment with MuloCo physics engine [8]. Experimental details and two supplementary experi- ments, Boabble-Buuning-BBal and Push-I, are available in Appendix B, E and F, respectively.
Baselines. As baselines, we select the Hate-of-the-ar gradient-free sampling baselines NPDR [20], and two gradient-based baselines, TuneNet [34] and EXI-Net [36]. discussed in Section 2.
4.2 Sim-to-sim trajectory alignment with known target environment parameters
In this experiment, we verify whether COMPASS can align trajectories between two different envi- ronments. We conduct experiments in simulation so that the ground truth environment parameters are known to us. Among the two environments, one is treated as the "real" (target) environment we want to align with, the other is the simulation environment we will fine-tune. We collect rollouts with a scripted stochastic policy. For each method except Tune-Net, we Maxlter 10, N 10, M 64. For Tune-Net, we collect dataset of size Maxlter N M 6400 to train the regression model. More implement details and ablation study are presented in Appendix C and D.
The learned causal graph parameters ei after iterations (i = 0, 1, 2) are shown in Figure. 3. We ob- serve that the learned causal graph is very sparse, reducing the search space by orders of magnitude without extensive domain knowledge. Our method is able automatically discover different types of relevant environment parameters such actuation, sensing, and dynamics. It is worth noting that only the damping of the right wall out of 4 sides has causality discovered with the trajectory
pusict
puoia
pooit-
and syrist goon
wirigh veccely
>
and wwogermmethen J-
unvilt Buuneee 2
Sestoooo onto by-
pud maaigmmm weghoee eith iar
pasil padiigayy grston Joatoe
caaann alcry-
wakkgehmiii reths ine-
sodddre Di oilaa it-
pust dy weth ior pusaig g to by-- eo gelanenr
Frod soggrreeee pe- ge
tollirtennd indificto ficdin shilig- yorr, Fort wet t the goursarr
book Sall ougilii hedten Jostree
back BE OETO Senat watu io-- back sea
back an www.eggaaannnno taE angerratt withe ier
"it nght wo wonngggttnn weterirt retle yo
(NPt de wir.tretee nipit gin antha iz-
lng-
ngg Bo wetss ier lst wiggtrr wnts iz
iit do or orlla ln-- lut, riggommn
enta ixc-
of wolggdy oid dooo skiti lef wod & gittii ditciion to byp tgeerngggann geaareen
dostcce didnggi onia iar-
hort aegggrr wette
taek Meneeii with
cbeaer deeee wy gi netie
craae do Godmmaa ertia
fort wn wat ficiio
it wr oyoomee berannn
sushingg
Beyy rody
pukk
pussiggón
chass
Figure 3: Learned causal graph parameters v. Darker colors present values closer to 1. We use parameter
notations in the format of @bjecr@param.yypecc parem param.

---

COMPASS
ExI-Net
NPDR
Tune Not
Ground Truh
PuCk 10 www.aaaaaamgngoog
Soct@ve died!
1.0
0.9
0.8
-12
12
-1
0.7
1
4 5
2
4 8
lleration
Iteration
Herraton
lteration
pock1@dyraa dyrrraaalllla addding
0.04
0.01
0.05
0.08
0.03
0.04
0.04
0.02
-0.01
0.01
-0.02
0.03
0.08
-0.09
0.00
-0.04
5 1teration
Noration
Itaration
Meration
Figure 4: Environment parameters optimization. We show 8 parameters with discovered causality (as shown in Figure 3). The final mean absolute percentage errors (MAPE) are 0.22, 0.95, 0.29, 3.85 for COMPASS, EXI- Net, NPDR, Tune-Net, respectively. The solid lines represent the value across 5 random seeds, and the shaded area represents the standard deviation. damping negative the sign convention MuJoCo.
COMPASS
NPDR
EXI-Nct
Puck1 de
-11
-41
1ea7
1p407 (a) NPDR
Bx2 past
Herrtion
Puck2 de
pudc
2.11
B47 22
lleration
(b) COMPASS
(c) Trajectory difference
Figure 5: (a) and (b) visualize sampled trajectory-aligning results using NPDR and COMPASS ASS, respectively. The overlapping of trajectories and the obstacle due the camera bias (c) shows the mean trajectory
differences throughout 10 iterations, evaluated with random seeds 10 pairs rollouts per seed.
difference because initialized the position of puck1 and puck2 in such way that they almost only collided with the right wall. Though occasionally pucks did collide with the front wall or the obstacle, the speed at contacting is usually low and has minimal effects on the trajectory; therefore COMPASS placed low attention to such causality. Compared with the baseline methods, our proposed method provides unique interpretability that could guide the sim-to-real transfer in other systems.
The optimization process of 8 environment parameters with discovered causality is shown in Fig- ure 4. We observe that COMPASS optimizes the environment parameters approach the ground truth gradually, while baseline methods struggle to converge, especially for damping. Since Tune-Net trains a regression model to predict the difference between the current and the target set of environ- ment parameters given trajectories, it does not work well when most the environment parameters have negligible effects on the trajectories. Note that our method does NOT necessarily converge the ground-truth environment parameters; rather, it aims to find combination of environmental parameters to minimize the trajectory difference (more discussion in Section 5). The trajectory- aligning results shown in Figure. 5())(b). It is observed that method outperforms all the base- lines in terms of the trajectory difference using the same number of "real" and simulation rollouts.
7

---

Table 1: Trajectory difference (averaged between Puckl and Puck2) agents' performance in the real environment. "L" represents the standard deviation. We evaluate the results using 5 policies generated from independent runs and collect 10 trajectories for cach run.
Low fan speed NPDR
High Fan speed NPDR
Nominal
COMPASS
Nominal
ASS
Trujectory difference min (J)
3.68 0.07
2.81 0.16
2.37 + 0.10
2.23 0.37
3:05 + 0.46
1.41 0.25
Trajectory difference max (1)
10.77 + 0.05
7.34 +: 1.41
4.21::0.23
9.84 = 0.56
10.36 + 0.82
8.17 :: 1.34
Trajectory difference mean (3)
7.60 4 0.03
5.18 0.77
4.02 0.07
6.07 = 0.40
5.63 0.5
3.97 0.45
Puck2 final dist. goal center (L)
0.35 0.04
0.18 0.05
0.12
0.29
0.15
0.13
Success rate (f)
0.00 + 0.00
0.39 0.33
0.80 + 0.09
0.20
0.07
0.47 0.41
0.75 =: 0.22
4.3 Sim-to-real with policy optimization in the loop
In this experiment, first trained the agent in the initial simulation environment parameters with Soft Actor-Critie Critic (SAC) [48]. Then, we applied COMP ASS and the best-performing baseline in the sim-to-sim experiment, NPDR, to update the environment parameters and retrained the agent in the new simulation environment parameters. Finally, we deployed the agent in the real environment and reported the evaluation statistics. We used a fixed set of real trajectories instead of collecting new ones in every iteration. We effectively set up two real environments by powering the electric fan at different speeds, referred as low fan speed and high fan speed.
Upon inspecting Table. 1, we first observe that (i) COMPASS consistently outperforms the nominal simulator and the NPDR baseline in terms of trajectory difference and success rate. Notably, COMPASS improved success rate by 105. 1% and 59.6% compared with NPDR in the low and high fan speed settings, respectively. We hypothesize that COMPASS is more robust to the noisy and unmodeled dynamics in the real environment owing to the sparse causal model learned during the model learning and parameter optimization process. We also observe that (ii) the agent's real-world performance positively correlates with the trajectory alignment performance. This is expected given that previous works have proved that the policy performance degradation is bounded by the difference in transition distributions between two systems [39]. Similar patterns are observed in the supplementary experiments as well (Appendix E).
Discussion and Conclusion
In conclusion, COMPASS is novel causality-guided framework to identify simulation enviromment parameters that minimize the sim-to-real gap. It has three salient features. Firstly, COMPASS requires less domain knowledge the randomized environment parameters, enabling a more automated pro- cess for sim-to-real transfer. Secondly, COMPASS learns an interpretable causal structure, providing better generalization during environment parameter optimization and robustness against observa- tional noise in real rollouts. Lastly, COMPASS employs fully differentiable model to update the environment parameters, which mitigates the efficiency issue of the existing sampling-base bbaed meth- ods. Through both simulation and real-world experiments, we verify that our proposed method outperforms the existing gradient-free and gradient- based parameter estimation methods in terms of trajectory aligmment accuracy and the agent's 's success rate, while offering interpretability.
Limitations. With all the advantages of COMPASS discussed, some limitations of method also suggest directions for future work. Similar to the existing gradient-based methods [34, 36, 35, 21), COMPASS could converge to local minima since the identifiable set of parameters may be coupled [49, 50J, which would result in multiple local minima in the parameter space. Indeed, if the purpose is to find a specific combination of parameters that minimize the sim-to-real gap, it becomes less important whether converges to the global optimum or not [34]. addition, COMPASS finds single combination of environment parameters rather than distribution of them. Nevertheless, method can maintain several particles of environment parameters as empirical distribution [51, 52, 53, 54, 55] without extensive modifications to the core algorithm.

---

Acknowledgments
The authors gratefully acknowledge the support from the National Science Foundation under grants CNS-2047454.
References
[1] X. B. Peng. M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[2] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Reseurch, 39(1):3-20, 2020.
[3] Francis, B. Chen, S. Ganju, S. Kathpal, J. Poonganam, A. Shivani, S. Genc, Zhukov, M. Kumskoy, A. Koul, et al. Learn-torrace challenge 2022: Benchmarking safe learning and cross-domain generalisation in autonomous racing. arXiv preprint arXiv:2205.02955 2022.
[4] H. Park, G. Lee, J. Seo, M. Bhat, M. Kang, J. Francis, A. Jadhav, P. Liang, and L.-P. Morency. Diverse and admissible trajectory forecasting through multimodal context under- standing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pages 282-298. Springer, 2020.
[5] G. Tatiya, J. Francis, L. Bondi, I. Navarro, E. Nyberg, J. Sinapov, and J. Oh. Knowledge-driven scene priors for semantic audio-visual embodied navigation. arXiv preprint arXiv:22 1135, 2022.
[6] J. Francis, B. Chen, W. Yao, E. Nyberg, and J. Oh. Distribution aware goal prediction and con- formant model-based planning for safe autonomous driving. arXiv preprint arXiv:22/2.08729, 2022.
[7] J. Herman, J. Francis, S. Ganju, B. Chen, A. Koul, A. Gupta, A. Skabelkin, Zhukov, M. Kum- skoy, and E. Nyberg. Learn-to-race: A multimodal control environment for autonomous rac- ing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9793-9802, 2021.
[8] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control, In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[9] B. Ellenberger. Pybullet gymperium. https://ggttab..cc//eeeddd/////p 2018-2019.
[10] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEF, 2017.
[11] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters, 5(4):6670-66) 6677, 2020.
[12] M. Xu, P. Huang, F. Li, J. Zhu, X. Qi, K. Oguchi, Huang. H. Lam, and D. Zhao. Scalable safety-critical policy evaluation with accelerated rare event sampling. In 2022 IEEE/RSI ternational Conference on Intelligent Robots and Systems (IROS), pages 12919 12926. IEEE, 2022.
[13] M. Xu, Z. Liu, Huang, W. Ding, Cen, B. Li, and D. Zhao. Trustworthy reinforce- ment learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.00 200.00020000 2022.

---

[14] S. Shah, D. Dey, C. Lovett, and A. Kapoor. Airsim: High-fidelity visual and physical simula- tion for autonomous vehicles. In Field and Service Robotics: Results of the I Ith International Conference, pages 621 621-635. Springer, 2018.
[15] J. Francis, N. Kitamura, F. Labelle, X. Lu, I. Navarro, and J. Oh. Core challenges in embodied vision-language planning. Journal of Artificial Intelligence Research, 74:459-515 2022.
[16] S. Carpin, J. Wang, M. Lewis, A. Birk, and A. Jacoff. High fidelity tools for rescue robotics: results and perspectives. In RoboCup 2005: Robot Soccer World Cup IX 9, pages 301-311. Springer, 2006.
[17] E. Heiden, D. Millard, E. Coumans, Y. Sheng, and G. S. Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 9474-9481. IEEE, 2021.
[18] Y. Zhu, J. Wong, A. Mandlekar, Martin-Martin A. Joshi, S. Nasiriany, and Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[19] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973-8979. IEFE, 2019.
[20] F. Muratore, T. Gruner, F. Wiese, B. Belousov, M. Gienger, and J. Peters. Neural posterior domain randomization. In Conference on Robot Learning, pages 1532-1542. PMLR, 2022.
[21] A. Allevato, M. Pryor, and L.. Thomaz. Multiparameter real-world system identification using iterative residual tuning. Journal of Mechanisms and Roboties, 13(3), 2021.
[22] W. Ding, L. Shi, Y. Chi, and D. Zhao. Seeing is not believing: Robust reinforcement learning against spurious correlation. arXiv preprint arXiv:2307.07907, 2023,
[23] W. Ding, H. Lin, B. Li, and Zhao. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. arXiv preprint drXIv:2207.00081. 2022.
[24] W. Ding, H. Lin, B. Li, and D. Zhao. Causalaf: causal autoregressive flow for safety-critical driving scenario generation. In Conference on Robot Learning, pages 812-823. PMLR, 2023,
[25] B. Mchta, M. Diaz, F. Golemo, C. Pal, and L. Paull. Active domain randomization. In Conference Robot Learning, pages 1162-1176. PMLR, 2020.
[26] X. Chen, J. Hu, C. Jin, L. Li, and L. Wang. Understanding domain randomization for sim-to- real transfer. arXiv preprint arXiv:21 10..x::2100.3:: 10.0339, 2021.
[27] Q Vuong, S. Vikram, H. Su, S. Gao, and H. I. Christensen. How pick the domain random- ization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint arXiv:190..11774 2019.
[28] F. Muratore, C. Eilers, M. Gienger, and J. Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters, (22)911-918, 2021.
[29] A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza. Deep drone racing: From simulation to reality with domain randomization. IEEE Transactions Robotics, 36(1):1-14, 2019.
[30] Tobin, L. Biewald, R. Duan, M. Andrychowicz, A. Handa, V. Kumar, B. McGrew, A. Ray, J. Schneider, P. Welinder, al. Domain randomization and generative models for robotic grasp- ing. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3482-3489. IEEE, 2018.
10

---

[31] P. Huang, M. Xu, F. Fang, and D. Zhao. Robust reinforcement learning as stackelberg game via adaptively-reguulaizee adversarial training. arXiv preprint arXiv:2202.095/ 2022.
[32] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar. Multi- manipulation via locomotion using hierarchical sim2real. arXiv preprint (a00:00208.02 2019.
[33] M. Xu, P. Huang, Y. Niu, V. Kumar, J. Qiu, C. Fang, K.-H. Lee, X. Qi, H. Lam, B. Li, et al. Group distributionally robust reinforcement learning with hierarchical latent variables. In International Conference Artificial Intelligence and Statistics, pages 2677-2703, PMLR, 2023.
[34] A. Allevato, E. S. Short, M. Pryor, and A. Thomaz. Tunenet: One-shot residual tuning for system identification and sim-to-real robot task transfer. Conference on Robot Learning, pages 445-455, PMLR, 2020.
[35] Y. Du, O. Watkins, Darrell, P. Abbeel, and D. Pathak. Auto-tuned sim-to-real transfer. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 1290-1296. IEEE, 2021.
[36] T. Murooka, M. Hamaya, F. von Drigalski, K. Tanaka, and Y. Ijiri. Exi-net: Explic- itl/implicitly conditioned network for multiple environment sim-to-real transfer. In Confer- ence on Robot Learning, pages 1221-1230. PMLR, 2021.
[37] W. Yu, V. C. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In 2019 ieee/rsj international conference on intelligent robots and systems (iros), pages 3503- 3510. IEEE, 2019.
[38] F. Ramos, R. C. Possas, and D. Fox. Bayessim: adaptive domain randomization via probabilis- tic inference for robotics simulators. arXiv preprint
2019.
[39] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[40] D. Abel. theory of abstraction reinforcement learning. arXiv preprint arXiv:2033.03397, 2022.
[41] M. Shanahan and M. Mitchell. Abstraction for deep reinforcement learning. arXiv preprint arXiv:2202.05839, 2022.
[42] D. Abel, D. Arumugam, L. Lehnert, and M. Littman. State abstractions for lifelong reinforce- ment learning. In International Conference on Machine Learning, pages 10-19, PMLR, 2018.
[43] Brouillard, Lachapelle, A. Lacoste, S. Lacoste-Julien, and A. Drouin. Differentiable causal discovery from interventional data. Advances in Neural Information Processing Sys- tems, 33:21865-21877 2020.
[44] Y. Yu, Chen, T. Gao, and M. Yu. Dag-gnn: Dag structure learning with graph neural net- works. In International Conference Machine Learning, pages 7154-7163. PMLR, 2019.
[45] S. Lachapelle, P. Brouillard, T. Deleu, and S. Lacoste-Julien. Gradient- based neural dag learn- ing. arXiv preprint arvv 1906.0226, 2019.
[46] E. Jang. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: [oo::1611.0144 144, 2016.
[47] B. Evans, A. Thankaraj, and Pinto. Context is everything: Implicit identification for dynam- ics adaptation. In 2022 International Conference Robotics and Automation (ICRA), pages 2642-2648. IEEE, 2022.

---

[48] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[49] P. K. Khosla and T. Kanade. Parameter identification of robot dynamics. In 1985 24th IEEE conference on decision and control, pages 1754-1760. IEEE, 1985.
[50] N. Fazeli, R. Tedrake, and A. Rodriguez. Identifiability analysis of planar rigid-body frictional contact. Robotics Research: Volume 2, pages 665-682, 2018.
[51] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L.. Paull. Active domain randomization. In I.. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learn- ing, volume 100 of Proceedings of Machine Learning Research, pages 1162-1176. PMLR, 30 Oct-01 Nov 2020. URL https: http://poocee.....e//e/e/. mlr. preess/v100/mmttaa200.. http://........ee//////eeeee..ea
[52] P. Huang, M. Xu, J. Zhu, L. Shi, F. Fang, and D. Zhao. Curriculum reinforcement learn- ing using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35:10656-10670, 2022.
[53] P. Klink, C. D' D'Eramo, J. Peters, and J. Pajarinen. On the benefit of optimal transport for curriculum reinforcement learning. arXiv preprint arXiv:2309. 14099,, 2023.
[54] D. Cho, S. Lee, and H. J. Kim. nutcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.11741 2023.
[55] J. Kim, D. Cho, and H. J. Kim. free autonomous reinforcement learning via implicit and bidirectional curriculum. arXiv preprint urXiv:2305.09944 2023.
[56] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, beghhperformance deep learning library. Advances in neural information processing systems, 32, 2019.
[57] A. Tejero-Cantero, J. Boelts, M. Deistler, J.-M. Lueckmann, C. Durkan, P. Gonçalves, D. S. Greenberg, and J. H. Macke. Sbi a toolkit for simulation-based inference, 2020.
[58] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu. Stable baselines. https: / //ggttttuuu com/hill-ap watllaaaasaaaebbbaaaaaaee basel ines, 2018.
[59] G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
[60] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. Robotics: Science and Systems, 2023.
[61] Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26):caa5587,, 2019.
[62] A. Kumar, Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv: 107.04034, 2021.
[63] A. Kumar, Z. Li, J. Zeng, Pathak, K. Sreenath, and Malik. Adapting rapid motor adapta- tion for bipedal robots. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1 1161-1169 IEEE, 2022.
[64] L. Smith. J. C. Kew, X. B. Peng. Ha, Tan, and Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 International Conference on Robotics and Automation (ICRA), pages 1599-15999 IEEE, 2022.
[65] M. Mozian, C. G. Higuera, D. Meger, and G. Dudek. Learning domain randomization dis- tributions for training robust locomotion policies. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 127166117. IEEE, 2020.
12

---

[66] S. Liu, M. Xu, P. Huang, X. Zhang, Y. Liu, K. Oguchi, and D. Zhao. Continual vision- based reinforcement learning with group symmetries. In 7th Annual Conference on Robot Learning, 2023.
13

---

A Model details
In standard fully-connected multilayer perceptron (MLP), the input is treated as a whole and input into the first linear layer. blends information into the feature of subsequent layers, making difficult to separate the cause and effects. highlight the difference between our model and traditional MLP, we plot the detailed model architecture of the COMPASS in Figure.
More specifically, we design encoder-decoder structure in f, with G as linear transformation applied to the intermediate features. First, the encoder operates on each dimension of indepen- dently to generate features 2e E RIE|xds, Then the causal graph is multiplied by the features to generate the inputs for the decoder, i.e., Ir = g E Rd,*K, where d2 is the dimension of feature. Similarly, the action sequence is passed through the encoder and transformation to produce the feature of the action sequence Ja Rd.xK, Finally, Ie + Ga passed through the decoder to output the prediction d,.
graph
0o0 tion embedding
Encoder
postiion embedding
Fri param
Enocder
multiplication
Decorder
Trajectory Differenze
Enooder
Decuder
Actions
transfammation
Enooder
Figure 6: COMPASS model architecture.
B Experimental Details
B.1 Experimental Configurations for Robot Simulation Setup
simulator was developed to mimic a simulated air hockey table with dimensions matching the real-world setup using robosuite [18]. This allowed us to gather rollout trajectories and training data within the simulated environment. The specific dimensions of the objects within the simulation, such the table size, are detailed in Table Additionally, for the sim-to-sim experiment config- uration, default simulation parameters and the ground-truth simulated real environment parameters are presented in Table 3.
State and action space. The observation space for the RL agent comprises 6 dimensions. It includes initial position of puck1 (3D), and the initial position of puck2 (3D). The RL agent's action space consists of 4 dimensions: the initial position the pusher in the x-direction, the initial position of the pusher in the y-direction, the shooting angle of the pusher, and the pushing velocity of the pusher. We fixed the initial position of puck1 and puck2. Note that the shooting angle is relative to the line connecting the center of the pusher and puck1.
The reward is calculated as -10 [[[xpuck1 3puck1 [pack1] (goott Mgoal; =g22 To provide additional incentive for reaching the goal, the distance penalty term is divided by Furthermore, terminal reward is given if the hockey stays within the success region the last time step. It is important to mention that the reward is not accumulated through- out the horizon. Instead, it only considers the final Euclidean distance between puck2 and the goal center. For further numerical details and specifications, please refer to Table 4.
Reward function.
14

---

Table Mujoco Simulation Environment Setup
(m)
Y (m)
Z (m)
Radius(m)
Air hockey tahle
0.0
0.0
0.8
[0.45, 0.9, 0.035]
Puck1
-0.15
0.0
0.0255
Puck2
-0.055
-0.075
0.8
0.0255
Goal point
0.43
0.0
0.8
0.15
Obstacle bar
0.1
0.0
0.8
[0.025, 0.18, 0.025]
Table 3: Sim-to-Sim Eny Parameters
Default Simulation Env Parameters
Simulated "Real" Env Purameters
Env param
pusher ausherr@actuationn@ vel discount
0.75
0.85
pusher@dyna@ @ammping
-10.0
-6.0
puckl @ dyna@ damping
-10.0
-6.0
puckl @ dyna@ friction_sliding
0,05
0.04
puck2 dyna damping
-10.0
-6.0
puck2 @ dyna@ friction sliding
0.05
0.03
front_wall, back_wall, eftl, wall, obstacle dyna@ damping
-10.0
-6.0
env @camera @ bias_x
0.0
+0.03
env @cammer @ bius
0.0
-0.02
B.2 Experimental Configurations for Real Robot Setup
This is a top-down view of the mini air hockey table we used to collect real trajectories. The dimensional attributes of each component are annotated in Figure while green dashed lines distinetly demarcate the designated goal area. We used Kinova Gen 3 robot platform and installed a top-down Intel RealSense D345f RGB camera to track the position of puck1 and puck2.
18 m
0.49
0.15
0,215 m
0.45
m
Figure 7: Top-Down View of Table Air Hockey.
C Implementation Details
We reproduced the baseline implementation EXI-Net [36], NPDR [20] and Tune-Net [34] based on the papers and released code base. We utilized software packages such as PyTorch [56], shi [57], StableBaseline3 [58], and OpenCV [59]. We report the hyperparameters used for all algorithms,
Table 4: Simulation Environment Setup
Simulation Parameters
low: [-0.24, 0.065, -0.157, 0.3] high: [-0.21, 0.085, 0.157, 0.5] low: [-inf] [inf 2.25
Action space
Observation space
Terminal reward
Simulation horizon
50 time steps 0.05 s
Simulation timestep
15

---

Table Soft Actor Critic Hyperparameters
Parameters Name
Values
learning.rate
3e-4
gradient_stepp
32
batch_size
32
train_freq
8
ent_coef
0.005
net net_arch
[32, 32]
policy env env_number
"MIpPolicy"
64
buffer-size
1,000,000
learning_starre tau gamma action_noise
100
0.005
0.99
None
stats_window size
100
including the proposed COMPASS method, in Table 7, and which provide a comprehensive list of the parameters we utilized reproduce the results.
Soft Actor-Critic hyperparameters. We use SAC implementation in StableBaseline3 [58] to train the RL agents. The training hyperparameter is shown in Table 5.
Table 6: COMPASS hyperparameters
Description
value
variable name
Shared Hyperparameters
Number of iterations
10
n.round
Retrain in each iteration
True
retrain scratch
(if False, keep using model trained in the first iteration)
Number of rollouts in each iteration
640
samples per round n_emd _actin
Number of command actions in each iteration
10
Number of epochs
4000
n_epochs
Batch size
64
batch size
Learning rate
0.001
learning rate
Setorithm-Speciffe Hyperparameters
Network encoder dimension
32
emb_dim hidden_dim causal dim sparse_weight ww_discount loss function
Network hidden dimension
[256, 256]
Causal dimension
32
Sparsity weight of the loss function Sparsity weight discount
0,003
0.5
Loss function Optimizer
MSE + Sparsity
Adam
optimizer
16

---

Table 7: EXI-Net hyperprrmeeess
Description
value
variable name
Shared Hyperparameters
Number of iterations
10
round
Retrain in each iteration
True
retrain from scratch
False, keep using the model trained in the first iteration)
Number of rollouts each iteration
640
nasamlees per round
Number of command actions each iteration
10
n_emd_action
Number of epochs
4000
n_epochs
Batch size Learning rate
64
batch size
0.001
learning rate
Algorithm-peeificc Hyperparameters
Network hidden dimension
[256. 256]
hidden dim loss function
Loss function Optimizer
MSH
Adam
optimizer
Table 8: NPDR hyperparameters
Deseription
value
variable name
Shared Hyperparameters
Number of iterations
10
n. round
Retrain in each iteration
True
retrain wwwmofooooooooooooo scratch
False, keep using model trained in the first iteration)
Number of rollouts in each iteration
640
n n_samples ppersppppeeeereeeeeeer round
Number of command actions in each iteration
10
n cmd action
Seporithm-Specilee Hyperparameters
Prior distribution lype Inference model type
Uniform maf
prior
inf model
Embedding net type Embedding downsampling factor
LSTM
embedding sttredding.struct struct
2
downsampling. factor
Posterior hidden features Posterior number of transforms Normalize posterior
100
hidden
10
transforms normalize .potteii num num.esochs learning rate
False
Density estimator training epochs Density estimator training rate Early stop epochs once posterior converge Use combined loss for posterior training Discard prior samples
3e-4
20
crop_atee_epochh use combined loss
True
False
Sampling method
MCMC
sample_with
MCMC thinning factor
2
thin
Table 9: Tune-Net hyperparameters
Description
value
variable name
Shared Hyperparameters
Number of iterations
n_round
Retrain in each iteration
False
retrain from scratch
(if False, keep using the model trained in the first iteration)
Number rollouts in each iteration
6400
samples round round
Number command actions in each iteration
cndddd action
Number epachs
4000
n_epochs
Batch size
size
Learning rate
0.001
learning rate
Algorithm-Specifee Hyperparameters
Network input dimension
(2,304)
(dim (dim-pair, dim_state)
(Pair of Trajectory and Action dimension)
Network output dimension
64
dim zeta
(Tunable env param dimension)
Env param update iteration
Network hidden dimension
[256, 256]
hidden dim
Loss function
MSE
loss_nn
Optimizer
Adam
optimizer
17

---

D Supplementary Ablation Study for Air-Hockey Experiment
D.1 Different Real Rollout Size N and Sim Rollout Size M
We investigated the effect of the real rollout size N and simulation rollout size M by testing values of N 5, 10, 20 M 64 and M 32, 64, 128 at N 10. The optimization processes for environment parameters. given these sizes, are illustrated in Figure 8 and 9. Notably, for given N, M, COMPASS demonstrates superior convergence in comparison to NPDR.
Further insights are offered in Figure 10a and 10c, which depicts the trajectory differences over it- erative steps, and Figure 10b and 10d, showing the final trajectory discrepancies for different N, M values. These visualizations highlight that as the rollout budget NN, M diminishes, the performance gap between COMPASS and other benchmark methods widens. This can be attributed to the fully- differentiable nature of COMPASS. Reinforeing our key contributions, COMP ASS consistently outper- forms the baselines, maintaining superiority under identical counts real and simulation rollouts.
COMPASS (N-S)
COMPASS (N-10)
COMPASS (N-20)
NPDR (N-5)
NPOR (N-10)
NPOR (N~20)
Ground Truth
Rushen2tact8- disct
puck1 Badyna@ddammpp
puck2@4yn@@ammmpi
-8
-8
-8
-8
0.8
-10
-10
-10
-12
-12
-12
0.7
0
4
2
4
6
2
3
5
lleration
lleration
lleration
llerstion
BuccAB-yaagiiccioo alding
0.04
0.01
1..05
0.00
0.53
0.00
0.52
-0.02
0.01
-0.03
Haration
Itoration
Iteratoon
Haraton
Figure 8:
Environment parameters optimization with different real rollout sizes N. We show 8 parameters with discovered causality (as shown in Figure 3). The solid lines represent the mean value across 3 random seeds, and the shaded area represents the standard deviation.
COMPASS (M+32)
COMPASS (M+64)
COMPASIS (M-
NPDR (M+32)
NPOR (M+64)
NPDR (M+125)
Groundd Trath
Snttev dest
pasher @ayn
-4
-6
0.8
-10
-12
0.7
5
5
Iteation
Berstion
lleration
Iteration
(rx22@yrr@@rrdii slidng
0.04
0.01
@ddrr
0.65
0.05
envgeamen@eess.
0.00
0.04
0.04
0.00
-0.01
0.01
-0.02
0.03
0.03
0.00
-0.03
-0.04
2
4
Resaiin
Bersion
lerrsooo
leraaioo
Figure 9: Environment parameters optimization with different sim rollout sizes M. We show 8 parameters
with discovered causality (as shown in Figure 3). The solid lines represent the mean value across 3 random sceds, and the shaded area represents the standard deviation.
18

---

COMPASE (N-5)
NP3R (4-6)
COMPASS (0-37)
kPOR (66322)
COMPASS IN-10
NP3R (4-10)
COMPABS
COMPASS W066
kPoit (9m04)
COMPASS (N:20)
NP3R (4-200
COMPASS (0-120)
hPoot 19e
Posh LG
Pocst d:
tlenation
rollect
Beraloon
Puni
64
128
Hlnstion
Roal rollout size
Ho ution
8 lout aas
(a)
(b)
(d)
Figure 10: (a)(b) Trajectory difference for diffcrent real rollout size N. (c)(d) Trajectory difference different sim rollout size M.
D.2 Different Initial Environment Parameters
Pucki
To investigate the sensitivity of COMPASS to initialization, we conducted experiments with randomly initialized environment parameters. The en- vironment parameter optimization processes are illustrated in Figure. 12, while Figure. presents the trajectory discrepancies.
2
3
5 Ileration
9
A key observation from both figures is that the starting point-is i.e. the parameter intialization-dde influence the complexity of aligning the trajectories. Despite these variances initialization, COMPASS consis- tently demonstrates capability to minimize the trajectory difference across different initial conditions.
Puck2
3 5 Neration
Figure 11: Trajectory dif- ference.
D.3 Different Sparsity Weight
The learned causal graph parameters, denoted as vb are illustrated in Figure. 13 across various sparsity weights: X = 0.001, 0.005, 0.01. In the absence of regularization (specifically when À = 0), the graph tends to be denser, devoid of any constraints to minimize edge count. As À increases, the
puckt @orragaaamminn
pusher Seccivvv diedt
0.9
-6
-6
-10
-10
purk 1edgn@Blieddr slidgg
Heration
Rusiggonagmmeon siding
0.05
0.08
env@rarere@@oss
0.64
0.04
0.02
0.00
0.03
0.08
0.00
+0.00
0.62
-0.02
4
2
5
6
1
4 4
Beration
lleraioon
lteraion
lleraioo
Figure 12: Environment parameters optimization with randomized initialization. initialization.
We only show one run per
19

---

pucki
puck2
pucki
puck2
(a) À 0.001
puck2
pucki
puck2
puckt-
puck2
(b) À = 0.005
puckt
puck2
pucki-
puck2
3
da geneo po to-
Sireent b7 pusii cogeryy
pentroal
pack dngiggi Jry
pag
g ghyy prrcction skueg
punt Run punkt proye di gireett Jyy
puet puckagggnnn giriction roling
pellpptyeeee See 1o0k froit
back wall gey gir restion roiing
bark well Body @ perrriitttt fifcctioe sttnng
www.geonnggnnooocooo..mm gi Wiccctinn 2e nestonal back a0ll2-0m
bark um ladyyr da ltx back well wel.gggoeeeeeoooo back weligntm
night niiit peitt wellooyneeeett grmmetti Soorg
right wall go rection rolling
right ave th pirmee Jyy
left en aesllggyaaa genooaaaaaanii genario, iit Hitt wall lo wergggannnnn
gennritta byy
loft jen lady pimertte jes- wet wnl oyn gifie san cenumnl
grectttiiiiiininn gircttinn sottiintoon Sidng
gea giraction collig
tett wettiddmm dettmmprn pellggggne
obotscicggnn get fiction rolling
sutharg o ganaena
p oy- ginamio
parto wwg.ggggggetiieee
mont an 1 don
puckt puckit peggee
mbeec whe garty wit wall
parti gely
park
foot avali pullgrynnd
lort eellladyyn
front frnntsali
frost
obitec
(c) = 0.01
Figure 13:
Learned causal graph parameters &i with different sparsity weight A. Darker colors present values
closer to 1. We use parameter notations in the format of object@ object@ppram..yye papem_type @aram.
graph becomes progressively sparser. However, it still maintains the most influential edges; omitting them would significantly elevate the prediction error.
20

---

E Sim-to-Real Double-Bouncing-Bal Experiment
E.1 Double-houncing-hal experimental setup
Figure 14 illustrates the experimental setup. Building upon the experimental design presented by Allevato et al. [34], we've raised the bar by allowing the ball to bounce twice, rather than just once, before landing in the goal basket. The goal basket has a diameter of 13.0 cm, and the ball measures 6.8 cm in diameter. Successful task completion is characterized by the ball's accurate entry into the hoop from above.
The task's 's intricacy hinges on achieving seamless alignment between the real-worke dynamics and the simulation. By releasing the ball at a specific height, it needs to bounce first off inclined plate 1, followed by plate 2, and finally enter into the goal basket. The ball's motion is constrained within a 2-D plane with properly aligned plates. As such, the action space is represented by a scalar the ball's release height. Simultancously, the state space corresponds to the ball's 2-D positional coordinates within this plane (K 1). This experiment encompasses 82 environment parameters (18) 82).
ball
arm
plate 2
goal
plate
Figure 14: Double- soailee-oounn---ee ball setup.
E.2 Learned causal graph
Figure 15 depicts the learned causal graph parameter, denoted as v. Starting from a fully-comnected causal graph, COMPASS efficiently narrows down the parameter search space from 82 dimen- sions to a mere dimensions. Specifically, these are ball@ dyna@ damping, ball dyna mass, platel @ dyna damping. and plate2 dyna damping. Remarkably, this refinement is achieved within just two iterations.
E.3 Sim-to-real trajectory alignment results
As depicted in Figure 16, it's evident that COMPASS effectively aligns simulated trajectories closely with their real-world counterparts. Given the extensive scale of environment parameters in this con-
-A (occol jouu
6' d'aocuueii uf Buss
-aape @2-03
soo
Me wa
-
of te dhe-
to the aoe a
for to aoe
in coroee dn moooonee
mnmaee toe i tne of suaaii
if the for the
dh ti rooi doon moee ooher luug tiae in do
in the in tith co munddoon
har di aadee
of the no in
the seceeigge
codeent tooule coutte di
the di the sondee
of woleet
(9 (90---9-000 4idh ied
a 10000 tnd
o
dee cu oud malg
ex
Figure 15: Learned causal graph parameters vi for double bouble-bouncing-bal experiments. Darker colors present
values closer to 1. We parameter notations in the format of abject@param.cc stpe @
21

---

text, sampling- based techniques like NPDR become notably inefficient. This is primarily because they necessitate an immense sample size to learn a posterior distribution accurately.
10
(a) NPDR
14 Parl
Ber
Bar
26
(b) COMPASS
Figure 16: Visualization of sim-to-real trajectory alignment.
E.4 Real-world agent performance
Table 10 presents the real-world performance of agents after policy optimization in the adjusted environment, compared with nominal agents (those trained in the original environments). Consistent with the trajectory aligmment outcomes, agents trained using COMPASS demonstrate superior success rates compared to those trained using NPDR and the nominal approach.
(a) Nominal
(b) NPDR
(c) COMPASS
Figure 17: Real-world policy rollouts.
Table 10: Agents' performance in the real environment. represents the standard deviation. We evaluate the results using policies generated from independent runs and collect 10 trajectories for each run.
Nominal
NPDR
COMPASS
Success rate (1) 0.30 + 0.14 0.58 + 0.15 0.86 = 0.10
22

---

Sim-to-Sim Push-1 Experiment
F1 Experimental setup
We conducted further evaluations using an experiment inspired by the Push-T experiment described by Chi et al. [60]. In this experiment, illustrated in Figure. 18, the agent controls the robot arm to strategically move a slender I-shaped cube towards a specific position and alignment. The task spans horizon of T 30. The state space comprises the 3-D coordinates and orientation of the cube, the target, and the end effector. The action space is defined by the 3-D velocity of the end effector and the gripper's s action (either open close). The states of interest are factorized into the position, roll, pitch, and yaw of the cube (K 4). This setup involves 67 environment parameters (E) 67).
yaw
roll
pitch
Figure 18: Push-I task. The green cube represents the target.
F.2 Learned causal graph
The learned causal graph parameter is shown in Figure. 19. Starting from fully-connected causal graph, COMPAS ASS efficiently narrows down the parameter search space from 67 dimensions to only 3 dimensions within 2 iterations. Specifically, they are cube @ dyna @ mass, cube @ dyna @ damping, cube @ dyna fiiiiiinnniiiiisiiiinn
Figure 19; Learned causal graph parameters s for Push-I experiments.
23

---

COMPASS
EXI-Net
NPDR
Ground Truh
20.00
Brrerrrrrreeeeen.... aldingg
-17.5
1.0
-15.0
0.5
-30
-12.5
04
-100
0.8
-50
gripprr
Meralion
Ileration
Iteration
Iteration
Figure 20: Environment parameters optimization. The solid lines represent mean value across random seeds, and the shaded area represents the standard deviation.
WW
9.5
0.30
3.44
0.23
0.3
0.55
lend mm
Tandan
Henaaon
Iend
Figure 21: Trajectory difference optimization results. The solid lines represent the value random seeds, and the shaded area represents the standard deviation.
F3 Environment parameter optimization and trajectory alignment
Figure.20 depicts the optimization of environment parameters across iterations. We observe that COMPASS optimizes the environment parameters to approach the ground truth gradually, while base- line methods struggle to converge. The trajectory difference over the iterations presented in Fig- ure. 21, while the visualization of trajectory alignment shown in Figure. 22, 23, 24, 25. At first glance, the trajectory difference in the roll angle doesn't seem show much reduction. However, closer examination of Figure.22 22 reveals that the roll angle has minimal changes during the whole trajectory regardless of the environment parameters. Given the elongated shape of the cube (illus- trated in Figure. 18), it's reasonable to see only minor variations in this direction. Overall, these figures highlight COMPASS's efficiency in aligning the simulator with the real world.
F.4 Sim-to-sim agent performance
Table 10 presents the "real" performance of agents after policy optimization adjusted environ- ment, compared with nominal agents (those trained the original environments). Consistent with the trajectory alignment outcomes, agents trained using COMPASS demonstrate higher success rates compared to those trained using NPDR and the nominal approach.
Table Agents' performance the "real" environment. "4" represents the standard deviation. evaluate the results using 3 policies generated from independent runs and collect 10 trajectories for each run.
Nominal
NPDR
EXI-Net
COMPASS
Success (1) 0.10 + 0.22 0.40 0.21
0.50 + 0.22 0.70 + 0.28
24

---

240
204P
Ume 210p
2009
Emn
504p
EXI-Net
2173
0ime STmp
Uiee
time step NPDR
Eime
Bime Step
COMPASS.
Figure 22: Trajectory alignment results of EXI-Net, NPDR, and COMPASS.
300
2000
S100
EXI-Net
22'
2012
J00
1171 Mlep
[me slap
10th step
Litee step
Urte step
NPDR
km1
200
Unc
COMPASS.
Figure 23: Trajectory alignment results of EXI-Net, NPDR, and COMPASE ASS.
25

---

pltch 5.E
pos
019
3055
stop
210p
EXI-Net
2013
1013
0/90 Stop
[ms 2069
1ime 20p
Eme step
Emm S1op
(b) NPDR
kerk
(c) COMPASS.
Figure 24: Trajectory alignment results EXI-Net, NPDR, and COMP ASS.
300
500
ST0p
Stop
EXI-Net
117e 1100p
Uree
Bime slep
Liir
NPDR
200
pHb
Emc
COMPASS.
Figure 25: Trajectory alignment results of EXI-Net, NPDR, and COMPASS.
26

---

G Additional Literature Review
Adaptive Policy in Locomotion and Manipulation. The challenge of sim-to-real transfer has been central the field of locomotion tasks, and has recently demonstrated remarkable success [61, 62, 63, 64, 65]. Rapid Motor Adaptation (RMA) [62] proposed solution to bridge the sim-to- real gap by effectively learning the relationship between dynamic-affecting parameters and historical contexts. More recently, Kumar et al. [63] introduced Adapting- RMM (A-RMA) to further refine the base policy of RMA using model-free reinforcement learning (RI.) techniques. Typically, RMA- based methods approach the sim-to-real challenge generalization problem. They tend to assume an appropriate range and set of parameters that influence testing performance, along with a sizable randomized training budget, to ensure successful operation. These assumptions present inherent challenges due to the requisite domain expertise and training time. manipulation, Liu et [66] approached the adaptive policy from a continual RL perspective, cultivating policy for each group of tasks rather than an individual task to solve unseen tasks in seen groups in zero-shot manner. In contrast, this paper focuses aligning simulators with real-world dynamics. Our approach involves the automated identification of simulation environment parameters that minimize the sim- to-real dynamics gap. While there are similar studies, such as the work by Mozian et al. [65], which on searching for the environment parameter distributions that are challenging yet not excessively adversarial to learn, our emphasis is on sim-to-real applications with novel causality-based system identification.
27


Reviews:
[{"review_id": "j1-rthlAOr6", "reviewer": "Reviewer_UgPf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the well known problem of the sim2real gap, that is the difficulty of implementing a simulator such that robot-object trajectories in simulation exactly match the real world. One common method for reducing this gap is to tune various simulated parameters. However this often requires extensive domain knowledge and tends to scale poorly with the number of potential tunable parameters. To address this, the authors propose COMPASS, a method for discovering causal relationships between tunable simulator parameters and the sim2real gap. The causal relationship graph is constructed so as to encourage pruning of relevant parameters, and consequently increases the efficiency of domain randomization and enables targeted environment parameter optimization. The method is evaluated on a mini-air hockey task in which the robot needs to control a puck in order to send a 2nd puck to a goal, avoiding an obstacle.  ", "summary_of_recommendation": "Overall a strong paper. Well written and clearly presented - a pleasure to read. The benefits of COMPASS are demonstrated convincingly and the experimental analysis is insightful. I think it will be relevant and interesting to the research community. All my suggestions are to further improve a paper that I think already passes the acceptance bar.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In addition to some suggestions and questions in the previous section, please see below for some further questions.\n* Table 2: The Puck2 final dist. to goal center for NPDR and COMPASS appear quite similar (0.15 vs 0.13 on average) however the average success rates are quite different (0.47 vs 0.75). Why is this the case?\n* What about interactivity between objects?\n   * For example, perhaps the accuracy of tracking the 1st puck affects where the 2nd puck is hit and thus indirectly affects the accuracy of the 2nd puck track. Do you see any systematic differences in weight between parameters associated with the 1st vs 2nd puck?\n   * If there were many pucks (e.g. 5) would you expect to see COMPASS place successively less weight on the parameters associated with later pucks in the causal chain?\n* Why was EXI-NET chosen as the comparison method for visualizing trajectory aligning results in Figure 5 (a) instead of the best baseline, NDPR?\n* How sensitive is this method to the initial simulator parameter values? It would be interesting to see a variation of Figure 4 with a variety of different parameter initializations.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality**\n* Overall high technical quality. The experiments clearly assess the effectiveness of COMPASS relative to baseline methods.\n* Experiments are described in sufficient detail.\n* Insightful analysis assessing the method in a number of different ways. E.g. Fig 3 which highlights how the sparsity of the causal graph changes from iteration to iteration, Fig 4 which assess how well COMPASS finds the correct parameter values vs ground truth and prior methods and finally Fig 5 which assess COMPASS’ performance on the end-to-end trajectory tracking task.\n* Well chosen baselines\n* Some improvement of COMPASS over the baseline methods\n* Nice analysis of the way COMPASS functions (Fig 3)\n* Good related work section which clearly situates COMPASS in the broader research context and highlights how COMPASS differs.\n* To further improve quality it would be nice to see this evaluated on another task and in a setting where there are more than 2 states in the factorized state space\n\n**Clarity**\n* Extremely clear and well written.\n* Well motivated.\n* One minor suggestion - line 127 - a couple of examples of a factorized state would help a reader build a mental model of how one might factorize a state space.\n\n**Originality**\n* The use of a causal graph to prune simulated parameters to optimize is interesting and novel\n\n**Significance**\n* It is hard to assess.\n* On the one hand this paper presents an original approach to a well known problem, the sim2real gap in robot learning.\n* However a competing and simpler strategy is to make use of careful system id combined with extensive domain randomization. The difficulty of working on a variety of heterogeneous hardware often leads to working on a single robot system or a small number of different robots, and under those circumstances a time consuming system id is often worthwhile.\n* Given the additional complexity involved in this method it is difficult to assess how widely this might influence the research community. Regardless, I think it is interesting to the community.\n\n**Minor**\n* Table 1: Nominal max std deviation overlaps with COMPASS so I think this should be bolded.\n* Table 2: High fan speed NPDR has standard deviation overlaps with COMPASS so I think this should be bolded.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "gR-LxfK6l-Z", "reviewer": "Reviewer_7MJV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method to learn causal relationships between simulation parameters and states. The output of the method is a differentiable causal graph that can be used in various ways: First, it provides insights to the system by indicating the importance of different parameters to the system’s behavior. Second, the graph can be used to devise targeted domain randomization that narrows down the number parameters to randomize. Third, since the graph is differentiable, one can directly optimize simulation parameters with gradient based optimization by minimizing the difference between simulated and real trajectories. Forth, the graph can be used as a part of sim-to-real policy optimization by iteratively collecting data from a real robot, fitting the simulation parameters, and optimizing a new policy in simulation. The proposed algorithms are evaluated on table top air hockey both in sim-to-sim and sim-to-real setting, and compared to three prior methods.", "summary_of_recommendation": "While the proposed approach to learn causal graphs is sound, evaluation and comparison to prior works is limited, making it difficult to asses the benefits and potential use-cases. Better evidence to support the claims made in the paper are needed to justify acceptance.\n\nAfter the rebuttal phase, I have increased my recommendation score from Weak Reject to Weak Accept. The authors have provided detailed answers to my concerns. However, I remain slightly unsure of the practical significance of the work. This may be due to my limited knowledge of related work, which is reflected in my confidence score.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "* Please elaborate on the importance/practicality of Casuality-Guided Domain Randomization. Perturbing parameters that don’t change rollouts does not hurt, and if we leave out such perturbations, we cannot learn potential causal relations on the next iteration. How is the perturbation magnitude $\\delta_r$ for each parameter chosen?\n* Evaluations are too limited. Please justify the choice of the hyperparameters for each algorithm and, in case a different sets of hypers work (clearly) better for different algorithms, report the results in terms of the number of rollouts needed on the real robot. \n* If possible, please consider a more challenging domain with higher dimensional action and parameter space. Alternatively, please include a naive baseline that uses finite-difference gradients for tuning the simulation parameters, and discuss (or show) whether the policy could be optimized or fine-tuned directly on hardware with the same data budget.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and easy to read. Overall the method of learning a causality graph is interesting as it sheds light on how different parameters affect simulation. Air hockey is a good domain for illustrating and studying the approach as the dynamics are intuitive. On the other hand, for the same reason, the chosen domain does not properly demonstrate the benefit of providing interpretability to complex systems nor improvements over other methods. A substantially more complex (simulated) evaluation would help make the claims of the paper stronger.\n\nThe paper proposes Causality-Guided Domain Randomization that selects simulation parameters to be perturbed based on a learned causal graph: dimensions that do not affect simulated trajectories are ignored, and DR is applied to the other dimensions only. However, these ignored dimensions would not affect the simulation anyway, so skipping them seems unnecessary. Instead, choosing the perturbation magnitude ($\\delta_r$) is often more challenging and requires manual tuning. Perhaps one could use the graph to automate that too. Moreover, if at iteration $i$, we don’t perturb a given parameter, then the causality graph will not find the related connection for the next iteration, even if such connection should emerge due to change in the policy.\n\nMy other concern is regarding limited evaluations. In Algorithm 1 line 16, the number rollouts collected from the simulation (roughly) matches the number of simulation parameters. One could estimate the gradient via finite difference and use line search with roughly the same amount of compute. Perhaps this approach will break for more complex domains, but it seems a natural thing to try in this case. Second, the task has only four actions (starting position of the pusher, hitting angle, and velocity), which might be directly learnable (or fine-tunable) on a robot with the same data budget. Therefore, the task serves as a great test domain for illustrating the method, but does not help understand how the method scales to harder problems. For example a more complex sim-to-sim example could help understand scalability of the method better. Third, the paper uses the same hyperparameters ($MaxIter$, $N$, $M$) for all baselines, and no justification is given for the particular choice. Instead, it would be better to tune the baselines separately and report performance as a function of environment rollouts (both sim and real separately). \n\nMinor comment:\n* I think there should not be equality on line 135 between $d_\\tau$ and $f_\\phi$. This is what we optimize for, but is never exactly true.  \n* Typos in Algorithm 1 and 2: casualty → causality.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "OYrn4nEDL9S", "reviewer": "Reviewer_mBeP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes COMPASS (Causality between environment parameters and the sim-to-real gap). COMPASS aims to tackle the dynamics sim-to-real gap by learning a differentiable mapping from the simulation environment parameters to the differences between sim and real robot trajectories. This is done by a simultaneously learned casual graph to optimize the simulation parameters so that we end up with a simulation that is as close to the real world as possible. These parameters can then be used in combination with domain randomization to transfer plaices from sim to real. \n\nExperiments are performed in both a sim-to-sim scenario, and a sim-to-real scenario (Mini air hockey with obstacle).", "summary_of_recommendation": "Currently the paper is missing crucial baselines that are used in the quadruped sim2real community. To name but a few:\n\n- Hwangbo  et al. Learning Agile and Dynamic Motor Skills for Legged Robots\n- Kumar et al., Adapting Rapid Motor Adaptation for Bipedal Robots", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How would this approach compare to more recent work in learning quadruped locomotion?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n- The analysis in the sim-to-sim section is well executed.\n- The paper is well written and easy to follow.\n\n\n## Weaknesses\n\n- The biggest weakness lies in the results. There seems to be a missing simpler baseline: using a recurrent based policy in combination with wide-domain randomization; e.g. a recent example is: Kumar et al., Adapting Rapid Motor Adaptation for Bipedal Robots. In general, sim2real for quadruped locomotion has come a long way, and a lot of the related work in that domain seems to be missing. It’s not clear to me why iterating back and forth between sim and real in order to tune the physics simulation is needed.\n- Having more than 1 task in the real world would increase the confidence that the findings are applicable to a broad range of tasks. \n- The abstract and introduction does not state that the focus of this paper is on dynamics sim-to-real, rather than visual sim-to-real. This is quite imprint to state early, as there are many statements in the first 2 pages that are true for dynamics, but not vision sim2real work; e.g. “…set of tunable parameters is usually manually selected to reduce the search space in a case-by-case manner”.\n- There is a contradiction in the motivation for the work; in the first sentence of the introduction, the paper states that “training control policies directly on real robots poses challenges due to potentially unsafe exploration…”; however, the method that the paper presents requires that we roll out a potentially dangerous trajectory in the real world in order to perform the causality-guided domain randomisation. You may want to reconsider this entrance given the presented method. \n- Details of the network architecture/hyperparams are missing.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "9dMKvq1wJk", "reviewer": "Reviewer_Ltn4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper discusses the challenges associated with training robot control policies in simulations that \nneed to transfer to the real world, a problem often termed the simulation-to-real transfer gap. \nThis gap is often a result of disparities between simulators and the real world, which can produce \ninaccurate results. To address these issues, the authors introduce an automated method to tune simulator \nparameters. This is achieved by learning the causal relationships between the environment parameters and \nthe sim-to-real gap. The method works by learning a differentiable mapping from the environment parameters \nto the differences between simulated and real-world robot-object trajectories. A causal graph is learned \nsimultaneously to prune the search space of parameters, provide better interpretability, and improve generalization. ", "summary_of_recommendation": "This paper introduces COMPASS, a causality-guided framework designed to minimize the simulation-to-real gap. COMPASS identifies and adjusts simulation environment parameters by learning a differentiable mapping from these parameters to the differences between simulated and real-world robot-object trajectories.  The optimization objective includes a mean squared error term for accuracy and a regularization term. The ideas presented here are novel and interesting and the efficacy of the method is demonstrated by performing a sim2real experiment.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Missing baseline : I think there are some missing baselines including [1] which you should cite. They are also learning a domain randomization distribution, just with a different methodology. Even though they are not doing sim vs real trajectory alignment.\n\n- How does COMPASS handle high dimensional spaces and the so-called \"curse of dimensionality\"? Could you further explain how the differentiable model helps in mitigating this issue?\n\n- How does COMPASS interact with the reinforcement learning (RL) components in the system? Does the process of learning the causal graph have any impact on the RL policy optimization? As far as I understood this process is currently disjoint. I'm wondering if the authors tried learning the parameters and the policy at the same time.\n\n- How does the regularization term encourage the elimination of irrelevant environment parameters in the objective function? I did not see any ablation study of the effect of this term.\n\n- I got a little bit lost when reading the encoding part where the method requires the parameters and the action sequence. Are these actions interpreted as interventions ? And are these, so to speak, the correct actions ? The actions that successfully take you to the next desired state?\n\n[1] Learning Domain Randomization Distributions for Training Robust Locomotion Policies - https://ieeexplore.ieee.org/abstract/document/9341019", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces `COMPASS`, a method designed to align simulators more closely with the real world by discovering the causality \nbetween environment parameters and the sim-to-real gap. COMPASS achieves this by learning a differentiable mapping from simulation environment parameters to differences between simulated and real-world robot-object interaction trajectories. This mapping process is governed by a concurrently learned causal graph.\n\nAfter the causal model is established, COMPASS employs back-propagation to optimize simulation environment parameters \nin an end-to-end manner. The causal graph helps the model by enhancing interpretability, pruning the parameter search space, and thereby improving the efficiency of domain randomization and scalability.\n\nThe causal graph focuses on the influence of environment parameters ($\\epsilon$) on the trajectory difference ($d\\tau$). \nIt is structured with predefined directed edges and represented by a binary adjacency matrix of size $|E| \\times K$, \nwhere 1 and 0 indicate the existence and absence of an edge, respectively.\n\nInspired by previous works, the authors formulate combinatorial graph learning into a continuous optimization\nproblem, making the optimization of `G` differentiable. They designed a pipeline where elements \n of the graph `G` are sampled from a `Gumbel-Softmax` distribution, parameterized by $\\psi$.\n\nThe causal graph is denoted as $G \\psi$, with all elements (i, j) initialized to ones to ensure the graph is fully \nconnected at the start. In addition to the causal graph, the authors underscore the need for a parameterized model to accurately \nrepresent the influence of causes on effects. They design an encoder-decoder structure (`f`) \nin which the causal graph (`G`) is applied as a linear transformation to intermediate features.\n\nThe encoder processes each dimension of the environment parameters independently to generate \nfeatures $z_\\epsilon$. The causal graph then multiplies these features to create inputs for the decoder $(g\\epsilon = z\\epsilon T G)$, \nwhere dz is the dimension of the feature. The action sequence (a) follows a similar process through the \nencoder and transformation, yielding the feature of the action sequence ($ga$). Finally, the sum of $g\\epsilon$ and $ga$ is passed through the decoder to output the predicted trajectory difference ($dˆ \\tau$). \nThis process  captures how changes in environment parameters and action sequences can influence the \ndifference between simulated and real-world trajectories.\n\n**Experiments**\n\nThe robotics task is a `Mini Air Hockey with Obstacle`. To reach the goal, the agent needs to consider pusher-to-puck, puck-to-puck, puck-to-wall collisions, and surface properties of the hockey table. Baselines are `Neural Posterior Domain Randomization (NPDR)`, `EXI-Net` and `Tune-Net`.\n\n- Sim-to-sim trajectory alignment with known target environment parameters\nThis experiment verifies whether COMPASS can align trajectories between two different envionments.\n\n- Sim-to-real with policy optimization in the loop\nFor the policy learning, Soft Actor-Critic (SAC) is used and COMPASS is applied to update the environment parameters and retrained the agent in the new simulation environment parameters.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to CoRL 2023. The reviewers have spent considerable time working with this paper and the current version is recommended for acceptance.

Please follow up with promised improvements, experiments and additional suggestions from the discussion phase. These include in particular the extended evaluation and comparison against simpler baselines.
Decision: Accept (Poster)