Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a method to learn causal relationships between simulation parameters and states. The output of the method is a differentiable causal graph that can be used in various ways: First, it provides insights to the system by indicating the importance of different parameters to the system’s behavior. Second, the graph can be used to devise targeted domain randomization that narrows down the number parameters to randomize. Third, since the graph is differentiable, one can directly optimize simulation parameters with gradient based optimization by minimizing the difference between simulated and real trajectories. Forth, the graph can be used as a part of sim-to-real policy optimization by iteratively collecting data from a real robot, fitting the simulation parameters, and optimizing a new policy in simulation. The proposed algorithms are evaluated on table top air hockey both in sim-to-sim and sim-to-real setting, and compared to three prior methods.
review: The paper is well-written and easy to read. Overall the method of learning a causality graph is interesting as it sheds light on how different parameters affect simulation. Air hockey is a good domain for illustrating and studying the approach as the dynamics are intuitive. On the other hand, for the same reason, the chosen domain does not properly demonstrate the benefit of providing interpretability to complex systems nor improvements over other methods. A substantially more complex (simulated) evaluation would help make the claims of the paper stronger.

The paper proposes Causality-Guided Domain Randomization that selects simulation parameters to be perturbed based on a learned causal graph: dimensions that do not affect simulated trajectories are ignored, and DR is applied to the other dimensions only. However, these ignored dimensions would not affect the simulation anyway, so skipping them seems unnecessary. Instead, choosing the perturbation magnitude ($\delta_r$) is often more challenging and requires manual tuning. Perhaps one could use the graph to automate that too. Moreover, if at iteration $i$, we don’t perturb a given parameter, then the causality graph will not find the related connection for the next iteration, even if such connection should emerge due to change in the policy.

My other concern is regarding limited evaluations. In Algorithm 1 line 16, the number rollouts collected from the simulation (roughly) matches the number of simulation parameters. One could estimate the gradient via finite difference and use line search with roughly the same amount of compute. Perhaps this approach will break for more complex domains, but it seems a natural thing to try in this case. Second, the task has only four actions (starting position of the pusher, hitting angle, and velocity), which might be directly learnable (or fine-tunable) on a robot with the same data budget. Therefore, the task serves as a great test domain for illustrating the method, but does not help understand how the method scales to harder problems. For example a more complex sim-to-sim example could help understand scalability of the method better. Third, the paper uses the same hyperparameters ($MaxIter$, $N$, $M$) for all baselines, and no justification is given for the particular choice. Instead, it would be better to tune the baselines separately and report performance as a function of environment rollouts (both sim and real separately). 

Minor comment:
* I think there should not be equality on line 135 between $d_\tau$ and $f_\phi$. This is what we optimize for, but is never exactly true.  
* Typos in Algorithm 1 and 2: casualty → causality.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: While the proposed approach to learn causal graphs is sound, evaluation and comparison to prior works is limited, making it difficult to asses the benefits and potential use-cases. Better evidence to support the claims made in the paper are needed to justify acceptance.

After the rebuttal phase, I have increased my recommendation score from Weak Reject to Weak Accept. The authors have provided detailed answers to my concerns. However, I remain slightly unsure of the practical significance of the work. This may be due to my limited knowledge of related work, which is reflected in my confidence score.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper addresses the well known problem of the sim2real gap, that is the difficulty of implementing a simulator such that robot-object trajectories in simulation exactly match the real world. One common method for reducing this gap is to tune various simulated parameters. However this often requires extensive domain knowledge and tends to scale poorly with the number of potential tunable parameters. To address this, the authors propose COMPASS, a method for discovering causal relationships between tunable simulator parameters and the sim2real gap. The causal relationship graph is constructed so as to encourage pruning of relevant parameters, and consequently increases the efficiency of domain randomization and enables targeted environment parameter optimization. The method is evaluated on a mini-air hockey task in which the robot needs to control a puck in order to send a 2nd puck to a goal, avoiding an obstacle.  
review: **Quality**
* Overall high technical quality. The experiments clearly assess the effectiveness of COMPASS relative to baseline methods.
* Experiments are described in sufficient detail.
* Insightful analysis assessing the method in a number of different ways. E.g. Fig 3 which highlights how the sparsity of the causal graph changes from iteration to iteration, Fig 4 which assess how well COMPASS finds the correct parameter values vs ground truth and prior methods and finally Fig 5 which assess COMPASS’ performance on the end-to-end trajectory tracking task.
* Well chosen baselines
* Some improvement of COMPASS over the baseline methods
* Nice analysis of the way COMPASS functions (Fig 3)
* Good related work section which clearly situates COMPASS in the broader research context and highlights how COMPASS differs.
* To further improve quality it would be nice to see this evaluated on another task and in a setting where there are more than 2 states in the factorized state space

**Clarity**
* Extremely clear and well written.
* Well motivated.
* One minor suggestion - line 127 - a couple of examples of a factorized state would help a reader build a mental model of how one might factorize a state space.

**Originality**
* The use of a causal graph to prune simulated parameters to optimize is interesting and novel

**Significance**
* It is hard to assess.
* On the one hand this paper presents an original approach to a well known problem, the sim2real gap in robot learning.
* However a competing and simpler strategy is to make use of careful system id combined with extensive domain randomization. The difficulty of working on a variety of heterogeneous hardware often leads to working on a single robot system or a small number of different robots, and under those circumstances a time consuming system id is often worthwhile.
* Given the additional complexity involved in this method it is difficult to assess how widely this might influence the research community. Regardless, I think it is interesting to the community.

**Minor**
* Table 1: Nominal max std deviation overlaps with COMPASS so I think this should be bolded.
* Table 2: High fan speed NPDR has standard deviation overlaps with COMPASS so I think this should be bolded.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall a strong paper. Well written and clearly presented - a pleasure to read. The benefits of COMPASS are demonstrated convincingly and the experimental analysis is insightful. I think it will be relevant and interesting to the research community. All my suggestions are to further improve a paper that I think already passes the acceptance bar.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes COMPASS (Causality between environment parameters and the sim-to-real gap). COMPASS aims to tackle the dynamics sim-to-real gap by learning a differentiable mapping from the simulation environment parameters to the differences between sim and real robot trajectories. This is done by a simultaneously learned casual graph to optimize the simulation parameters so that we end up with a simulation that is as close to the real world as possible. These parameters can then be used in combination with domain randomization to transfer plaices from sim to real. 

Experiments are performed in both a sim-to-sim scenario, and a sim-to-real scenario (Mini air hockey with obstacle).
review: ## Strengths

- The analysis in the sim-to-sim section is well executed.
- The paper is well written and easy to follow.


## Weaknesses

- The biggest weakness lies in the results. There seems to be a missing simpler baseline: using a recurrent based policy in combination with wide-domain randomization; e.g. a recent example is: Kumar et al., Adapting Rapid Motor Adaptation for Bipedal Robots. In general, sim2real for quadruped locomotion has come a long way, and a lot of the related work in that domain seems to be missing. It’s not clear to me why iterating back and forth between sim and real in order to tune the physics simulation is needed.
- Having more than 1 task in the real world would increase the confidence that the findings are applicable to a broad range of tasks. 
- The abstract and introduction does not state that the focus of this paper is on dynamics sim-to-real, rather than visual sim-to-real. This is quite imprint to state early, as there are many statements in the first 2 pages that are true for dynamics, but not vision sim2real work; e.g. “…set of tunable parameters is usually manually selected to reduce the search space in a case-by-case manner”.
- There is a contradiction in the motivation for the work; in the first sentence of the introduction, the paper states that “training control policies directly on real robots poses challenges due to potentially unsafe exploration…”; however, the method that the paper presents requires that we roll out a potentially dangerous trajectory in the real world in order to perform the causality-guided domain randomisation. You may want to reconsider this entrance given the presented method. 
- Details of the network architecture/hyperparams are missing.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Currently the paper is missing crucial baselines that are used in the quadruped sim2real community. To name but a few:

- Hwangbo  et al. Learning Agile and Dynamic Motor Skills for Legged Robots
- Kumar et al., Adapting Rapid Motor Adaptation for Bipedal Robots
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper discusses the challenges associated with training robot control policies in simulations that 
need to transfer to the real world, a problem often termed the simulation-to-real transfer gap. 
This gap is often a result of disparities between simulators and the real world, which can produce 
inaccurate results. To address these issues, the authors introduce an automated method to tune simulator 
parameters. This is achieved by learning the causal relationships between the environment parameters and 
the sim-to-real gap. The method works by learning a differentiable mapping from the environment parameters 
to the differences between simulated and real-world robot-object trajectories. A causal graph is learned 
simultaneously to prune the search space of parameters, provide better interpretability, and improve generalization. 
review: This work introduces `COMPASS`, a method designed to align simulators more closely with the real world by discovering the causality 
between environment parameters and the sim-to-real gap. COMPASS achieves this by learning a differentiable mapping from simulation environment parameters to differences between simulated and real-world robot-object interaction trajectories. This mapping process is governed by a concurrently learned causal graph.

After the causal model is established, COMPASS employs back-propagation to optimize simulation environment parameters 
in an end-to-end manner. The causal graph helps the model by enhancing interpretability, pruning the parameter search space, and thereby improving the efficiency of domain randomization and scalability.

The causal graph focuses on the influence of environment parameters ($\epsilon$) on the trajectory difference ($d\tau$). 
It is structured with predefined directed edges and represented by a binary adjacency matrix of size $|E| \times K$, 
where 1 and 0 indicate the existence and absence of an edge, respectively.

Inspired by previous works, the authors formulate combinatorial graph learning into a continuous optimization
problem, making the optimization of `G` differentiable. They designed a pipeline where elements 
 of the graph `G` are sampled from a `Gumbel-Softmax` distribution, parameterized by $\psi$.

The causal graph is denoted as $G \psi$, with all elements (i, j) initialized to ones to ensure the graph is fully 
connected at the start. In addition to the causal graph, the authors underscore the need for a parameterized model to accurately 
represent the influence of causes on effects. They design an encoder-decoder structure (`f`) 
in which the causal graph (`G`) is applied as a linear transformation to intermediate features.

The encoder processes each dimension of the environment parameters independently to generate 
features $z_\epsilon$. The causal graph then multiplies these features to create inputs for the decoder $(g\epsilon = z\epsilon T G)$, 
where dz is the dimension of the feature. The action sequence (a) follows a similar process through the 
encoder and transformation, yielding the feature of the action sequence ($ga$). Finally, the sum of $g\epsilon$ and $ga$ is passed through the decoder to output the predicted trajectory difference ($dˆ \tau$). 
This process  captures how changes in environment parameters and action sequences can influence the 
difference between simulated and real-world trajectories.

**Experiments**

The robotics task is a `Mini Air Hockey with Obstacle`. To reach the goal, the agent needs to consider pusher-to-puck, puck-to-puck, puck-to-wall collisions, and surface properties of the hockey table. Baselines are `Neural Posterior Domain Randomization (NPDR)`, `EXI-Net` and `Tune-Net`.

- Sim-to-sim trajectory alignment with known target environment parameters
This experiment verifies whether COMPASS can align trajectories between two different envionments.

- Sim-to-real with policy optimization in the loop
For the policy learning, Soft Actor-Critic (SAC) is used and COMPASS is applied to update the environment parameters and retrained the agent in the new simulation environment parameters.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper introduces COMPASS, a causality-guided framework designed to minimize the simulation-to-real gap. COMPASS identifies and adjusts simulation environment parameters by learning a differentiable mapping from these parameters to the differences between simulated and real-world robot-object trajectories.  The optimization objective includes a mean squared error term for accuracy and a regularization term. The ideas presented here are novel and interesting and the efficacy of the method is demonstrated by performing a sim2real experiment.
review_title: null

Meta-review: Thank you for your submission to CoRL 2023. The reviewers have spent considerable time working with this paper and the current version is recommended for acceptance.

Please follow up with promised improvements, experiments and additional suggestions from the discussion phase. These include in particular the extended evaluation and comparison against simpler baselines.
Decision: Accept (Poster)