Flow-based Maximum Entropy Domain Randomization for Multi-step Assembly

Authors: Aidan Curtis, Eric Li, Michael Noseworthy, Nishad Gothoskar, Sachin Chitta, Hui Li, Leslie Pack Kaelbling, Nicole E Carey
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Domain Randomization, Uncertainty, Assembly, Planning

Abstract:
Domain randomization in reinforcement learning is an established technique for increasing the robustness of control policies learned in simulation. By randomizing properties of the environment during training, the learned policy can be conformant to uncertainty along the randomized dimensions. While the environment distribution is typically specified by hand, in this paper we investigate the problem of automatically discovering this sampling distribution via entropy-regularized reward maximization of a neural sampling distribution in the form of a normalizing flow. We show that this architecture is more flexible and results in better robustness than existing approaches to learning simple parameterized sampling distributions. We demonstrate that these policies can be used to learn robust policies for contact-rich assembly tasks. Additionally, we explore how these sampling distributions can be used for out-of-distribution detection in the context of an uncertainty-aware multi-step manipulation planner.

Decision: reject
Reviewer scores: 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730582450385

thread_type:
official_review

<thread_object>
creator: reviewer_kdab

<content>
summary:
This paper presents GoFlow, a novel method to domain randomization for reinforcement learning. In robotics, domain randomization is a common technique used to enhance the robustness and performance of sim-to-real reinforcement learning. In recent years, sim2real RL methods have enabled quadrupeds and humanoids to achieve impressive locomotion and manipulation skills [1][2][3]. However, often domain randomization procedures require extensive human engineering. The proposed method seeks to automate the tuning of domain randomization parameters using a normalizing flow architecture and maximum entropy principle.

The method improves the robustness of control policies learned in simulation by adaptively randomizing simulation physical properties during training. It produces policies that can handle uncertainties more effectively. The authors demonstrate the capabilities of this approach with one multi-step real-world contact-rich assembly robotic task. Experiments in the paper suggest that it outperforms existing methods in terms of success rate in both simulated and real-world settings.

However, in its current form, I cannot recommend acceptance due to the following reasons:
- experimental evaluations are limited, and cannot support the claims of the core contributions
- real robot results are impressive in video, but not presented quantitatively or clearly in the paper 

[1]: UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers, Huy Ha et al, CoRL 2024.
[2]: Takahiro Miki et al. ,Learning robust perceptive locomotion for quadrupedal robots in the wild.Sci. Robot.7,eabk2822(2022).DOI:10.1126/scirobotics.abk2822
[3]: Expressive Whole-Body Control for Humanoid Robots, Cheng et al, RSS 2024

strengths:
Originality:
- While prior works such as DORAEMON have explored automating domain randomization using maximum entropy principles to enhance the generalization capacity and performance of RL policies, this is the first work that combines normalizing flow and maximum entropy to perform domain randomization for RL, to the best of my knowledge. The authors also demonstrated the effectiveness of their method on a challenging robotics task: multi-step contact-rich assembly, which is new to this line of literature.

Quality:
- The proposed algorithm, GoFlow, is shown to outperform prior works on 4 simulation tasks and 1 real robot task. While the presented success rate is seemingly low (ranging from 0.1-0.2), the real robot video seems convincing.

Clarity:
- The paper is overall well written and easy to flow. The authors provided a comprehensive overview of related works, background, and method sections. Graphics are helpful for understanding the motivation and strength of the proposed method. Specifically, figure 2 provides a good visualization of the multimodal ability of the normalizing flow approach and figure 5 demonstrates how GoFlow can be used to perform belief space planning for multi-step gear insertion tasks on a real robot.

Significance:
- From the perspective of more effective domain randomization for sim2real RL, this work shows it outperforms prior methods. From the angle of multi-step manipulation, this work also shows it is able to plan over belief space and perform contact-rich assembly tasks.

weaknesses:
1. Experimental evaluations are limited:
- For the comparisons against baseline methods, the range of tasks is too narrow (only 4 simulation tasks).

2. Sim results are hard to interpret
- Out of the 4 sim tasks, it is difficult to assess the effectiveness or significance of GoFlow's improvements or better performance, especially when all methods and tasks have relatively low success rates. For instance, in quadcopter, it is hard to draw a conclusion that 0.02 achieved by GoFlow is indeed better than baseline methods in a statistically significant manner.   
- For cartpole and gears, GoFlow performs similarly to some baseline methods.

3. Real robot results are also limited
- Although I truly appreciate the effort to design and perform such a challenging real robot experiment, it is hard to find the success rates or other evaluation metrics for the multi-step gear insertion task presented in the paper. Thus, it is difficult to assess the effectiveness of this method quantitatively, despite the cool results from the supplementary video.

questions:
1. If the aim is to make domain randomization easier or better for RL, the experimental results need to demonstrate that on a wider range of tasks, such as tasks that are highly relevant to real robotic systems, such as quadrupeds / humanoid locomotion tasks, manipulation tasks, etc, or simulation benchmarks that contains more than 4 tasks.

2. Related to the current experimental results, why are the success rates lower than the numbers presented in prior works and baseline papers (for example Doraemon)? It would really help if the authors could explain the difference in evaluation protocols and metrics in more detail.

3. The proposed method seems to have a high variance in success rate and low success rate from Figure 3. Could authors elaborate on this?

4. Could the authors perform more experimental evaluations on the real robot task? For example, it is hard to tell the success rate for the overall multi-stage task, or the success rates for each stage of the multi-step task.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675650056

thread_type:
official_review

<thread_object>
creator: reviewer_xq1i

<content>
summary:
The authors propose a method (GoFlow) for the automatic design of domain randomization distributions to facilitate sim-to-real transfer in absence of real-world data. In particular, GoFlow finds the right balance between a maximum entropy distribution and good policy performance within this distribution, while employing normalizing flows for extreme flexibility in capturing feasible patterns. The results demonstrate superior generalization performance when policies are trained with GoFlow.

strengths:
- The method extends current state-of-the-art algorithms for Domain Randomization by employing flexible distribution representations through normalizing flows.
- The method demonstrates that normalizing flows can be learned in this context to autonomously capture complex patterns among unobservable parameters, such as circular shapes in Fig. 2.
- The authors successfully deploy a probabilistic pose estimation model that allows guiding the agent to seek additional information in partially observable settings.

weaknesses:
- Poor description of the method and contextualization relative to existing works:
  - The authors present the method as a novel approach for learning DR distributions. However, to my understanding, the presented opt. problem in Eq. 5 has been recently proposed by Tiboni et al. [1] (aka DORAEMON) under an equivalent formulation, which, in turn, took inspiration from the self-paced curriculum learning opt. problem [2] (See Eq. 3 and Sec. B in [1], and see Eq. 12 and Eq. 20 in [2]). All these methods involve a joint opt. problem between the policy performance and an entropy/KL objective from a target distribution. However, the authors fail to make connections to these existing works, while proposing an equivalent formulation. The authors shall explain which parts of the method are novel, and which come from existing works in literature.
  - The final objective for optimizing the sampling distribution in line 9 of Algorithm 1 includes a third term beyond the entropy and the KL term which is not discussed in the main text. It's also unclear why this term is added. Perhaps more importantly, this term was introduced by LSDR [3] but no connection/citation is reported in the manuscript by the authors. In fact, the final algorithmic implementation in Algorithm 1 appears to be equivalent to LSDR, but extended to normalizing flows.
  - DORAEMON [1] and LSDR [3] provide clear motivations behind their choices of the respective objective functions. However, the authors here do not discuss how the opt. problem described in Eq. 5 translates to Algorithm 1. As a result, the algorithmic implementation in Algorithm 1 appears disconnected from the motivation and the general problem described by Eq. 5.

- Validity of the experimental evaluation:
  - DORAEMON [1] recently showcased successful deployment of FullDR, ADR, and DORAEMON in the cartpole and other locomotion tasks. However, in this paper these methods seem to score critically low in the same environments. This raises questions on the validity of the experimental evaluation, if not properly analyzed and discussed. E.g. why are the success rates so low across all methods?
  - Are baseline methods such as LSDR, ADR and DORAEMON trained with privileged value functions as well? This is known to drastically affect the results, and should be implemented for all baselines for a fair comparison.
  - ADR and DORAEMON assume that the initial sampling distribution is a feasible one, i.e. a near-optimal policy may and should be learned before ever stepping the sampling distribution. Is this assumption satisfied across the experimental evaluation? If not, it should be made clear that this assumption is violated.
  - How does the entropy change for different methods across the training process? What is the starting distribution/entropy for each method? No insight on the varying training distributions is reported, making it harder to draw conclusions on why each method is performing the way it does.
  - Hyperparameter curves for ADR and Doraemon seem to only include one value in some plots of Fig. 9, 10, and 11.

- Lack of proper citing or outdated citations:
  - Sec. 1: "too broad a distribution leads to unstable [...] while too narrow a distribution leads to poor real-world generalization". Citation required, this claim is not demonstrated in this paper.
  - Sec. 1: "many esisting methods rely on real-world rollouts from hardware experiments to estimate dynamics parameters". DROPO [4] and NPDR [5] are recent state-of-the-art methods that compare and beat the cited and older papers BayesSim and SimOpt. They should likely be cited as well.
  - Sec 1: "As shown in this paper and elsewhere". "Elsewhere" should be accompanied by one or more references.


[1] Tiboni, G. et al. "Domain Randomization via Entropy Maximization." ICLR 2024.

[2] Klink, Pascal, et al. "A probabilistic interpretation of self-paced learning with applications to reinforcement learning." Journal of Machine Learning Research 22.182 (2021): 1-52.

[3] Mozian, Melissa, et al. "Learning domain randomization distributions for training robust locomotion policies." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

[4] Tiboni, G., Arndt K., and Kyrki V. "DROPO: Sim-to-real transfer with offline domain randomization." Robotics and Autonomous Systems 166 (2023): 104432.

[5] Muratore, Fabio, et al. "Neural posterior domain randomization." Conference on Robot Learning. PMLR, 2022.

questions:
- What is the starting distribution and entropy of GoFlow? How does it vary during the algorithm? E.g., it starts wide and then it shrinks to focus on the feasible regions, or viceversa.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695929434

thread_type:
official_review

<thread_object>
creator: reviewer_h6d7

<content>
summary:
This paper proposes a method to automatically discover a good sampling distribution for domain randomization in reinforcement learning to learn conformant policies. This is done by using normalizing flows to learn a neural sampling distribution that maximizes the return and entropy of the sampling distribution. The approach is validated in simulation and a gear-assembly task. Finally, the learned sampling distribution is used for out-of-distribution detection to enable uncertainty aware manipulation planning.

strengths:
- This paper addresses an important problem of improving robustness of control policies learned in simulation and would be of interest to the robotics community.
    
- Experimental comparison with prior methods shows that GoFlow learns a better sampling distribution.
    
- Using the learned distribution to estimate the belief space precondition is a useful contribution.
    
- Experiments on real robots on a challenging gear assembly task indicate the efficacy of the approach.

weaknesses:
- The main novelty of using normalizing flows to learn the parameter sampling distribution is somewhat limited. However, it has been shown to be quite effective.
- The related work section is brief. I would like to see a more thorough discussion of the baselines and of methods that learn the domain randomization distribution.
- Error in estimating the belief space precondition can lead to incompleteness in planning.

questions:
- What is the true uncertainty distribution in the gear assembly task? Is it artificially induced or due to to uncertainty in perception and actuation? How is it simulated during training?
    
- How sensitive is the method to the regularization coefficient hyperparameter? The scale of the expected return J depends on the reward function. Does one need to tune the coefficient for every reward function?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732496179172

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Thank you to the reviewers and AC for their work in helping improve the quality of this paper. We appreciate the recognition of the importance of our work in improving control policy robustness through normalizing flows showing how they can be integrated into uncertainty-aware planning frameworks. Below, we summarize the key changes we made to address all of the reviewers' concerns:
- We updated the related work and method section to include all relevant citations and more clearly explain the meaning of each term in the algorithm and how it connects to the original optimization problem.
- We elaborated on the details of the baseline implementations and the root cause of their limitations and poor performance in comparison to our method.
- We included entropy statistics during training in the appendix for increased interpretability of the results.
- We expanded the set of simulated domains by including a humanoid task with randomized link masses.
- We performed real-world experiments comparing performance across baselines on a gear insertion task. Footage of these experiments have been uploaded as supplemental materials.
- We performed statistical tests on the simulated and real-world results to confirm the significance of our results.

More detailed responses to each reviewer is provided in the comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a method combining normalizing flows and maximum entropy principles for domain randomization in reinforcement learning. The reviewers recognized the significance of the problem and the effort to tackle it, particularly with a challenging real-world robotic assembly task. However, they noted significant weaknesses in experimental evaluation, theoretical contributions, and connection to prior work.

Strengths:

- The work addresses a critical problem in sim-to-real transfer for RL.

- Reasonable use of normalizing flows for domain randomization.

- Demonstrated applicability in a challenging real-world tasks.

Weaknesses:

- Limited theoretical novelty compared to recent works like DORAEMON and LSDR.

- Experimental design and baseline comparisons raise concerns about validity and fairness.

- Insufficient quantitative evaluation of real-world results.

While the reviewers acknowledged the importance of the problem, the concerns about the rigor of the experiments, fair comparisons to baselines, and limited theoretical insights outweigh the strengths. Addressing these concerns would significantly enhance the paper’s impact. Given these considerations, I recommend rejection at this stage.