Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Authors: Aidan Curtis
Eric Li
Michael Noseworthy
Nishad Gothoskar
Sachin Chitta
Hui Li
Leslie Pack Kaelbling
Nicole E Carey
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Domain Randomization
Uncertainty
Assembly
Planning
Primary area: reinforcement_learning->planning
TL;DR: A normalizing-flow approach to domain randomization yields robust RL policies for multi-step robot manipulation under uncertainty.

Abstract: Domain randomization in reinforcement learning is an established technique for increasing the robustness of control policies learned in simulation. By randomizing properties of the environment during training, the learned policy can be robust to uncertainty along the randomized dimensions. While the environment distribution is typically specified by hand, in this paper we investigate the problem of automatically discovering this sampling distribution via entropy-regularized reward maximization of a neural sampling distribution in the form of a normalizing flow. We show that this architecture is more flexible and results in better robustness than existing approaches to learning simple parameterized sampling distributions. We demonstrate that these policies can be used to learn robust policies for contact-rich assembly tasks. Additionally, we explore how these sampling distributions, in combination with a privileged value function, can be used for out-of-distribution detection in the context of an uncertainty-aware multi-step manipulation planner.

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Aidan Curtis' Eric Li"
Michael Noseworthy'
Nishad Gothoskar'
Sachin Chitta? Hui Li
Leslie Pack Kaelbling' Nicole Carey?
Abstract
ments (Kober et al., 2013). Learning in simulation elimi- nates some of these problems, but introduces new issues in the form discrepancies between the training and real- world environments (Valassakis et al., 2020).
Domain randomization in reinforcement learning is established technique for increasing the ro- bustness of control policies trained in simulation. By randomizing environment properties during training, the learned policy can become robust to uncertainties along the randomized dimensions. While the environment distribution is typically specified by hand, in this paper we investigate automatically discovering sampling distribution via reguaay-re reward maximization of normalizing-flol flow- based neural sampling distri- bution. We show that this architecture is more flexible and provides greater robustness than ex- isting approaches that learn simpler, parameter- ized sampling distributions, as demonstrated in six simulated and one real-world robotics domain. Lastly, explore how these learned sampling distributions, along with privileged value func- tion, can be used for out-of of--iisibiittiionn detection in an uncertainty-aware multi-step manipulation planner.
Successful RL from simulation hence requires efficient and accurate models of both robot and environment during the training process. But even with highly accurate geometric and dynamic simulators, the system can still be only con- sidered partially observable (Kober et al., qualities, inertial properties, perception noise, contact and force sensor noise, manufacturing deviations and tolerances, and imprecision in robot calibration all add uncertainty to the model.
To improve the robustness of learned policies against sim-to- real discrepancies. it is common to employ domain random- ization, varying the large set of environmental parameters inherent to task according to a given underlying distribu- tion (Muratore al., 2019). In this way, policies are trained to maximize their overall performance over diverse set of models. These sampling distributions are typically con- structed manually with Gaussian or uniform distributions on individual parameters with hand-selected variances and bounds. However, choosing appropriate distributions for each of the domain randomization parameters remains a delicate process (Josifovski al., 2022); too broad a distri- bution leads to suboptimal local minima convergence (see Figure 3), while too narrow distribution leads poor real- world generalization (Mozifian et al., 2019; Packer et al., 2018) 8). Many existing methods rely on real-world rollouts from hardware experiments to estimate dynamics parame- ters (Chebotar et al., 2019; Ramos et al., 2019; Muratore et al., 2022). However, for complex tasks with physical parameters that are difficult to efficiently or effectively sam- ple, this data may be time-consuming produce, or simply unavailable.
1. Introduction
Reinforcement learning (RL) is powerful tool robotics because it can be used to learn effective control policies for systems with highly complex dynamics that are difficult to model analytically. Unlike traditional control methods, which rely on precise mathematical models, RI. learns di- rectly from simulated or real-world experience (Luo & Li, 2021; Zhu et al., 2020; Schoettler et al., 2020).
However, RL approaches can be inefficient, involving slow, minimally parallelized, and potentially unsafe data- gathering processes when performed in real environ-
An alternative strategy is to learn an environment distribu- tion during training with the aim of finding the broadest possible training distribution that can feasibly be solved in order to maximize the chances of transferring to an unknown target environment. Automating updates to parameter dis- tributions during the training process can remove the need
MIT CSAIL ² Autodesk Research. Correspondence Aidan Curtis <curtisa mit.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
for heuristic tuning and iterative experimentation (Mozi- fian al., 2019; OpenAl et al., 2019; Tiboni et al., 2024). In this paper, we present Goflow, a novel approach for learned domain randomization that combines actor-critic reinforcement learning architectures (Schulman et al., 2017; Hаarnoja et al., 2018) with a neural sampling distribution to learn robust policies that generalize to real-world settings. By maximizing the diversity of parameters during sampling, we actively discover environments that are challenging for the current policy but still solvable given enough training.
enhancing performance in partially observable settings.
2. Related Work
Recent developments in reinforcement learning have demon- strated that policies trained in simulation can effectively transfer to real-world robots, even for contact-rich robotic tasks (Zhang et al., 2024: Tang et al., 2023b; Noseworthy et al., 2024; Jin et al., 2023). A key innovation enabling this transferability is domain randomization (?Peng et al., 2017), where environment parameters are sampled from predefined distributions during training, enabling learned policies to generalize to environmental uncertainties upon deployment.
As proof of concept, we investigate one real-world use case: contact-rich manipulation for assembly. Assembly is a criti- cal area of research for robotics, requiring a diverse set of high-contact interactions which often involve wide force bandwidths and unpredictable dynamic changes. Recently, sim-to-real RL has emerged potentially useful strategy for learning robust contact-rich policies without laborious real-world interactions (Noseworthy et al., 2024; Tang et al., 2023b; Zhang et al., 2024). We build on this work by testing our method on the real-world industrial assembly task of gear insertion.
Traditionally, domain randomization requires manually defining these sampling distributions, which can be labor- intensive. To address this, recent work has explored meth- ods to automatically learn these distributions, aiming for maximal generalization with minimal manual effort. One notable approach involves constructing adversarial distribu- tions that challenge the current policy, ensuring coverage of the environment parameter space (Mehta et al., 2020; Wang et al., 2025). While adversarial methods can outperform uni- form randomization, they typically assume all environments sampled are solvable, limiting their effectiveness in highly uncertain scenarios.
Lastly, we extend this classical gear insertion task to the setting of multi-step decision making under uncertainty and partial observability. As shown in this paper and else- where (Tiboni et al., 2024; Mozifian et al., 2019; OpenAI et al., 2019), policies trained in simulation have an upper bound on the environmental uncertaintics that they can be conformant to. For example, a visionless robot executing an insertion policy can only tolerate so much in-hand pose error. However, estimates of this uncertainty can be used to inform high level control decisions through task-oriented in- formation gathering (Liang et al., 2020; Curtis et al., 2023). For example, looking closer at objects can result in more accurate pose estimates or tracking objects in the hand to detect slippage. By integrating a probabilistic pose estima- tion model, we can use the sampling distributions learned with GoFlow as an distribbuioo detector to determine whether the policy is expected to succeed under its current belief about the world state. If the robot has insufficient information, it can act to deliberately seek the needed infor- mation using a simple belief-space planning algorithm.
An alternative line of research adopts a curriculum-bast ap- proach, progressively expanding the complexity of the train- ing distribution while maintaining policy success. Specific curricular methods include minimizing divergence from a target distribution using multivariate Gaussians (Mozifian et al., 2019), maximizing entropy through independent beta distributions (Tiboni et al., 2024), and incrementally ex- panding uniform sampling distributions via boundary sam- pling (OpenAl et al., 2019). In this work, we propose a novel learned domain randomization technique leveraging normalizing flows (Rezende & Mohamed, 2015) as neural sampling distributions. This approach offers increased flexi- bility and expressivity over existing parametric methods.
Beyond training robust policies in simulation, learned sam- pling distributions can be tied to the real-world environmen- tal conditions under which policies are likely to succeed. Previous works have integrated domain randomization with real-world interactions for more informed training distri- butions (Ramos et al., 2019; Sagawa & Hino, 2025; Ajay et al., 2022; Muratore et al., 2020) or to find the maximally effective real-world strategy (Yu et al., 2018; Ren et al., 2023). However, these methods often necessitate expen- sive policy retraining or dlta-inntessiv evolutionary search based on real-world feedback. posing challenges for real- time applications. Instead, we utilize our learned sampling distribution as an -ut-of-distribution detector within a multi- step planning framework, enabling fast and data-efficient
Our contributions are as follows: We introduce GoFlow, a novel domain randomization method that combines actor- critic reinforcement learning with learned neural sampling distribution. We show that GoFlow achieves higher domain coverage than fixed and other learning- based solutions to domain randomization on a suite of simulated environments. We demonstrate the efficacy of GoFlow in a real-world contact-rich manipulation fask-gear insertion and extend it to multi-step decision-making under uncertainty. By inte- grating a probabilistic pose estimation model, we enable the robot to actively gather additional information when needed,
2

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
information gathering in the real world.
environments sampled from p(E). In our implementation, we employ the Proximal Policy Optimization (PPO) algo- rithm (Schulman al., 2017), an on-policy policy gradient method that optimizes a stochastic policy while ensuring stable and efficient learning.
3. Background
3.1. Markov Decision Process
A Markov Decision Process (MDP) is a mathematical frame- work for modeling decision-making. Formally, an MDP is defined as a tuple (S, A, R. 7), where S is the state space, A is the action space, P x A x + [0. 1] is the state transition probability function, where P(s') 8, s,a) denotes the probability of transitioning state s' from state after taking action a, R S x A - R is the reward function, where R(s, a) denotes the expected immediate reward received after taking action a in state s, E [0, the discount factor, representing the importance of future rewards.
To further stabilize training, we pass privileged information about the environment parameters the critic network. The critic network, parameterized by ti. estimates the state- value function:
(st ETT
reth se,
(2)
where 8, is the current state, T are future rewards, and 7 is the discount factor. By incorporating & the critic can provide more accurate value estimates with lower vari- ance (Pinto et al., 2018). The actor network no (ax)(s)) does not have access to E, ensuring that the policy relies only on observable aspects of the state.
A policy S x A > [0, 1] defines probability distribu- tion over actions given states, where <aa s) is the prob- ability of taking action a in state The goal is to find an optimal policy x² that maximizes the expected cumulative discounted reward.
3.3. Normalizing Flows
3.2. Domain Randomization
Normalizing flows are a class of generative models that transform a simple base distribution into a complex tar- get distribution using a sequence of invertible, differen- tiable functions. Let = ~ pz(z) be a latent variable from base distribution (e.g., a standard normal distribution). A normalizing flow defines invertible transformation fe Rd - R parameterized by neural network parameters Φ such that = fø(=), aiming for x to follow the target distribution.
Domain randomization introduces variability into the envi- ronment by randomizing certain parameters during training. Let E denote the space of domain randomization parame- ters, and let & E E be specific instance of these parameters. Each & corresponds to a different environment configuration or dynamics.
We can define a parameterized family of Markov Decision Processes (MDPs) where each Mg- = AS, Pe, Rg, 7) has transition dynamics Pg and reward function Rg dependent on &. The agent interacts with environments sampled from a distribution over E. typically denoted as p(E).
The density of x is computed using the change of variables formula:
@55 (x)
The objective is to learn policy S + A that maximizes the expected return across the distribution environments:
px pz)(f=(x))) det
(3)
dr
For practical computation, this is often rewritten as:
[] (1)
[(T]= Be-ple)
BAnPeix
Σ 1=0
x (g(s)
log (r) = log P2(2) - log det (0f((z)
(4)
Oz
where 7 = {(s0, ao, $1, )) denotes a trajectory gen- erated by policy æ in environment ç. Domain randomization aims to find a policy xL such that: n" arg max, (T).
where az)(2) the Jacobian of So at z. By composing dz
multiple such transformations fo fex o... for each parameterized by neural network parameters dz. normaliz- ing flows can model highly complex distributions.
In deep reinforcement learning, the policy π is neural net- work parameterized by 0, denoted as ag. The agent learns the policy parameters f through interactions with simulated
In our work, we employ neural spline flows (Durkan et al., 2019), a type of normalizing flow where the invertible trans- formations are constructed using spline-basee based functions. Specifically, the parameters é represent the coefficients of
Thii problem can also be thought of as POMDP where the observation space is S and the state space is product of and = as discussed (K won et al., 2021).
3

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
~~ 10,,1
& E~ppp po(E)
Flow fz
Episode Start
Each Step
Training Only
O0
Critic
ot, Tt, E
Simulator
V (Ot, St) Å
Observations q, q7, F Ot
Ot, T'
Actor
TO at 0t)
at
at
Figure 1. An architecture diagram for our actor-critic RL training setup in the gears domain. At the start of cach training cpisode, an environment initialization privileged critic. At each time step, the agent receives observations including joint positions joint velocities q. and force measurements from which the actor policy generates actions. The critic takes observations ot, reward rt, and environment initialization to estimate the value. This estimated value is used to compute advantage Á which is then used to train the actor. Only the actor is used during testing and real-world deployment, but the critic and flow can help determine when policy will be successful discussed in section 6.
PE(E) is sampled via the normalizing flow model. This variable conditions both the simulator and the
the splines (e.g., .., knot positions and heights) and the weights and biases of the neural networks that parameterize these splines.
space for greater generalization. We take inspiration from these works to form a joint optimization problem:
4. Method
max ([e=pp)e(TT] + aH (Poall)))) P.s
(5)
In this section, we introduce GoFlow, a method for learned the flow by adaptively
where is the sampling distribution over environment pa- rameters p(E). H(p) is the differential entropy of p(E), DKL (pllpou) is the divergence between the current and previous sampling distributions, and a > 0, 3 > 0 are reg- ularization coefficients that control the trade-off between generalizability, training stability, and the expected reward under the sampling distribution.
domain randomization that goes with the adjusting the domain randomization process using normal- izing flows.
In traditional domain randomization setups, the distribution p({) is predefined. However, selecting appropriate p({) is crucial for the policy's performance and generalization. Too broad a sampling distribution and the training focuses on unsolvable environments and falls into local minima. In contrast, too narrow a sampling distribution leads to poor generalization and robustness. Additionally, rapid changes to the sampling distribution can lead to unstable training. To address these challenges, prior works such as (Klink et al., 2021) have proposed self-paced learner, which starts by mastering a small set of environments, and gradually expands the tasks to solver harder and harder problems while maintaining training stability. This strategy has subsequently been applied domain randomization in (Mozifian al., 2019) and (Tiboni et al., 2024), where terms were included for encouraging spread over the sampling
Other learned domain randomization approaches propose similar objectives. (Mozifian et al., 2019) maximizes reward but replaces entropy regularization with a KL divergence to a fixed target distribution and omits the self-paces KI. term. (Tiboni et al., 2024) includes all three objectives but frames the reward and self-paced KI. terms as constraints, maxi- mizing entropy through a nonlinear optimization process that is not easily adaptable to neural sampling distributions. We compare GoFlow to these methods in our experiments to highlight its advantages. To our knowledge, GoFlow is the first method to optimize such an objective with a neural sampling distribution.
The GoFlow algorithm (Algorithm 1) begins by initial-

---

Flow- baw-based Domain Randomization for Learning and Sequencing Robotic Skills
Ground Truth
LSDR
ADR
DORAEMON
GoFlow (a=0.1)
Goflow (a=1.0)
Figure 2. An illustrative domain showing the learned sampling functions over the space of unobserved parameters for the tested basclines. Compared to other learning methods, GoFlow correctly models the multimodality and vater-variable dependencies of the underlying reward function. This toy domain, along with other domains in our experiments, violates some of the assumptions made by prior works, such as the feasibility of the center point of the range.
Algorithm GoFlow
first estimates the policy's performance on the sampling distribution using the previously sampled rollout trajecto- ries (Line 7). The entropy of the sampling distribution (Line 8) and divergence from the previous sampling distri- bution (Line 9) are estimated using newly drawn samples. Importantly, compute the reward and entropy terms by importance sampling from uniform distribution rather than from the flow itself. This broad coverage helps prevent the learned distribution from collapsing around a small region of parameter space. Derivations for these equations can be found in Appendix A.2. These terms are combined to form a loss, which is differentiated to update the parameters of the sampling distribution (Line 10). An architecture diagram for this approach can be seen in Figure 1.
Require:
Initial policy parameters 0, flow parameters di training steps N, network updates K, entropy coeffi- cient a, similarity coefficient 3, and learning rate no
1: for n = 1 to N do
2: Sample (ctran) fi=1 ~ JE=1
pp((), {
M==1
u())
3: Train To with ghrin initializations
4: Estimate (TO) via policy rollouts
5: Save current flow distribution as Pors (E)
6: for k to K do
7:
R
- _ Ee
(o((test)
Jepare (TT)
8:
A - - EI
Bgg mu(E)
(E) log (c)
9:
DKL - Renpen((5) (5)
loggoou (5) log (E)
o o Φ + no Vo RR (R+af- 3Dxx)
10:
5. Domain Randomization Experiments
11: end for
12: end for
Our simulated experiments compare policy robustness in raagee of domains. For full details on the randomization parameters and bounds, see Appendix A.3.
izing both the policy parameters 0 and the normalizing flow parameters Φ In each training iteration, GoFlow first samples a batch of environment parameters {E from the current distribution modeled by the normalizing flow (Line 2). These sampled parameters are used to train the policy no (Line 3). Following the policy update, expected returns Jg. (TT) are estimated for each sampled environment through policy rollouts, providing a measure of the policy's s performance under target uniform distribution w(f) after being trained on p(E) (Line 4).
5.1. Domains
First, we examine the application of GoFlow to an illustra- tive 2D domain that is multimodal and contains intervariable dependencies. The state and action space are in IR.. The agent is initialized randomly in bounded energy function is defined by a composition of Gaussians placed in a regular circular or linear array. The agent can observe its position with Gaussian noise proportional to the inverse of the energy function. The agent is rewarded for guessing its location, but is incapable of moving. This task is infeasible when the agent is sufficiently far from any
plane. An
After sampling these rollouts, GoFlow then performs K steps of optimization on the sampling distribution. GoFlow
5

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
0.15
Coverage
0.06
0.015
0.4
0.06
0.10
0.04
0.010
0.2
0.04
0.2
0.05
0.02
0.005
0.3
0.02
0.00
0.00
0.000
0.01
0.00
2.5
7.5 X10'
Timesteps x12
Timestops
X23*
Timesteps
X26*
Timesteps
Timesteps
x137
Timesteps
GOFLOW
FUllDR
NoDR
DORAEMON
LSDR
ADR
Figure 3. The coverage ratio over the target distribution across five random seeds for cach of the environments. The bands around cach curve indicate the standard error.
of the Gaussian centers, so a sampling distribution should come to resemble the energy function. Some example func- tions learned by GoFlow and baselines from Section 5.2 can be seen in Figure 2.
identical neural architecture and PPO implementation. The success thresholds along with other hyperparameters are in Appendix A.5.
We evaluate on the following baselines. First, we com- pare to no domain randomization (NoDR) which trains on a fixed environment parameter at the centroid of the parameter space. Next, compare to full domain ran- domization (FulIDR) which samples uniformly across the domain within the boundaries during training. In addition to these fixed randomization methods, we evaluate against some other learning-bast based solutions from the literature: ADR (OpenAl et al., 2019) learns uniform intervals that expand over time via boundary sampling. It starts by oc- cupying an initial percentage of the domain and performs "boundary sampling" during training with some probability. The rewards attained from boundary sampling are com- pared to thresholds that determine if the boundary should be expanded or contracted. LSDR (Mozifian et al., 2019) learns a multivariate gaussian sampling distribution using reward maximization with a KL divergence regularization term weighted by an a hyperparameter. Lastly, DORAF- MON (Tiboni et al., 2024) learns independent beta distribu- tions for each dimension of the domain, using a maximum entropy objective constrained by an estimated success rate.
Second, we quantitatively compare GoFlow existing base- lines including Cartpole, Ant, Quadcopter, Quadruped, and Humanoid in the IsaacLab suite of environments (Mittal et al., 2023). We randomize over parameters such as link masses, joint frictions, and material properties.
Lastly, we evaluate our method on a contact-rich robot ma- nipulation task of gear insertion, a particularly relevant problem for robotic assembly. In the gears domain, we randomize over the relative pose between the gripper and the held gears along three degrees of freedom. The prob- lem is made difficult by the uncertainty the robot has about the precise location of the gear relative to the hand. The agent must learn to rely on signals of proprioception and force feedback to guide the gear into the gear shaft. The action space consists of end-effector pose offsets along three translational degrees of freedom and one rotational degree around the dimension. The observation space consists of history of the ten previous end effector poses and velocities estimated via finite differencing. In addition to simulated ex- periments, our trained policies are tested on a Franka Emika robot using IndustReal library built on the f frankapy toolkit (Tang et al., 2023a; Zhang et al., 2020).
We compare coverage across environments sampled from a uniform testing distribution within the environment bounds. Our findings in Figure 3 show that GoFlow matches or out- performs baselines across all domains. We find that method performs particularly well in comparison to other learned baselines when the simpler or more feasible regions of the domain are off-center, irregularly shaped, and have inter- peter-parameter dependencies such as those seen in Figure 2. We intentionally chose large intervals with low coverage to demonstrate this capability, and we perform a full study of how baselines degrade significantly with increased param- eter ranges while GoFlow degrades more gracefully (see Appendix A.6). Additionally, coverage should not be mis- taken for success-rate, as these policies can be integrated into a planning framework that avoids the use of infeasi- ble actions or gathers information to increase coverage as
5.2. Baselines
In our domain randomization experiments, we compare to a number of standard RL baselines and learning-bag based ap- proaches from the literature. In our quantitative experiments, success is determined by the sampled environment passing certain performance threshold JT that was selected for each environment. We measure task performance in terms of coverage, which is defined as the proportion of the total sam- pling distribution for which the policy receives higher than Jr reward. Coverage is estimated via policy rollouts on 4096 uniformly selected environment initialization samples from the total range of parameters. All baselines are trained with
6

---

blswd Domain Randomization for Learning and Sequencing Robotic Skills
t=
t=
==3
3
t=
x
x x&y x xay x8y
1t
300
you
y0W
yau
y0w
50W
Figure 4. A multi-step manipulation plan using probabilistic pose estimation estimate and update beliefs over time. The three show the robot state st, the observation os, and the robot belicf by at cach timestep. The red dotted line in the belief indicates the marginal entropy thresholds for the x, y. and yaw (rotation around z) dimensions determined by the learned normalizing flow. A belief with entropy surpassing the threshold line indicates the policy will likely fail. For full visualizations of the belief posteriors, flow distributions, and value maps, see Figure 14.
discussed in the following sections.
is important take actions to gain information.
In addition simulated experiments, additionally eval- uated the trained policies a real-world gear insertion task. The results of those real-world experiments show that GoFlow results in more robust sim-to-real transfer as seen in Table and in the supplementary videos.
Rather than operating the primitive action level, belief- space planners often make use of high-level actions An, sometimes called skills or options. In our case, these high- level actions will be discrete set of pretrained RL poli- cies. These high-level actions come with a belief-xpace precondition and a belief-space effect, both of which are subsets of the belief space B (Kaelbling & Lozano-Pers Perez, 2013; Curtis et al., 2024). Specifically, a high-level action t E An is associated with two components: precondi- tion Pre, C B, representing the set of belief states from which the action can be applied, and an effect Eff, S B, representing the set of belief states that the action was de- signed or trained to achieve. the precondition holds-that is, the current belief state b satisfies E Pre.-then applying action a will achieve the effect Eff, with proba- bility at least v) E [0, 1]. More formally, if b E Pre.. then Pr (be+1 E Eff, b,, T) n. Depending on the planner, may be set in advance, or calculated by the planner function of the belief.
6. Application to Multi-step manipulation
While reinforcement learning has proven be valuable technique for learning short-horizon skills in dynamic and contact-rich settings, it often struggles to generalize to more lonizhorizon and open ended problems (Sutton & Barto, 2018). The topic of sequencing short horizon skills in the context bigher-level decision strategy has been of in- creasing interest to both the planning (Mishra et al., 2023) and reinforcement learning communities (Nasiriany al.,, 2021). For this reason, examine the utility of these learned sampling distributions as out-of-distribution detec- tors, or belief-space preconditions, in the context of multi- step planning system.
This formalization allows planners to reason abstractly about the effects of high-level actions under uncertainty, which can result in generalizable decision-male making in long-horizon problems that require active information gathering or risk- awareness.
6.1. Belief-space planning background
Belief-space planning is framework for decision-nie -maiing under uncertainty, where the agent maintains a probability distribution over possible states, known as the belief state. Instead of planning solely in the state space S, the agent operates in the belief space B, which consists of all possible probability distributions over S. This approach is partic- ularly useful in partially observable environments where there is uncertainty in environment parameters and where it
6.2. Computing preconditions
In this section, we highlight the potential application of the learned sampling distribution Ps and privileged value function V4 as useful artifacts for belief-space planning. In particular, we are interested in identifying belief-space

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Po
V (20,, E)
precondition
belief (t=0)
belief (t=4)
20
-2.5
60
y offset (mm)
0
logie
value
logi0
-2
-4
-12
-20
20
-20
20
-20
-20
20
-20
x (mm)
x (mm)
x (mm)
x (mm)
x (mm)
y offset (mm)
20
-2.5
60
-6
log(p)
value
log(p)
-0
-2.5
2.5
-2.5
2.5
-2.5
0.0 yaw (rad)
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
yaw (rad)
yaw (rad)
yaw (rad)
yaw (rad)
Figure 5. A visual example of the precondition computation described in Section 6.2 for the gear assembly plan shown in Figure 4. The two rows show two different projections of the 3D sampling space (x position vs position in the top row and position vs yaw rotation in the bottom row). We apply threshold the sampling distribution remove low oowprobabillity regions (column 1). Additionally, we filter the value function by retaining only the regions where the expected value exceeds predetermined threshold (column 2). The intersection of these regions defines the belief-space precondition, indicating where the policy is likely succeed (column 3). Comparing the precondition to the beliefs, we can see that the belief is not sufficiently contained within the precondition at = (column 4), but passes success threshold y at after closer inspection 4 (column 5).
preconditions of a set of trained skills.
6.3. Updating beliefs
One point of leverage we have for this problem is the privi- leged value function Vo (s. {). which learned alongside the policy during training. One way to estimate the belief- space precondition is to simply find the set of belief states for which the expected value of the policy is larger than Jr with probability greater than under the belief:
Updating the belief state requires a probabilistic state esti- mation system that outputs a posterior over the unobserved environment variables, rather than single point estimate. We probabilistic object pose estimation framework called Bayes3D infer posterior distributions over object pose (Gothoskar et al., 2023). For details on this, see Ap- pendix A.4.1.
Pre, = {b B E, [Avo(+55)2>)]] n).
(6)
The benefit of this approach contrast to traditional rendering- pose estimation systems, such as those pre- sented in (Wen al., 2024) (Labbé al., 2022), is that pose estimates from Bayes3D indicate high uncertainty for distant, small, or occluded objects as well uncertainty stemming from object symmetry. Figure 14 shows the pose beliefs across the multi-step plan.
However, practical issue with this computation is that the value function is likely not calibrated in large portions of the state space that were not seen during policy training. To address this, we focus on regions of the environment where the agent has higher confidence due to sufficient sampling, for threshold This enables to integrate the value function over the belief distribution b( f) and the trusted region within E:
i.e., where (e(
6.4. A simple belief-space planner
While the problem of general- purpose multi-step planning in belief-space has been widely studied, in this paper we use simple BFS belief-space planner to demonstrate the utility of the learned sampling distributions as belief-space precon- ditions. The full algorithm can be found Algorithm 2.
Pre, {b E B Es [[, I > m} (7)
Here, lower bounds the probability of achieving the de- sired effects Eff, (or value greater than Jr) after executing ı in any belief state in Pre.. Figure 5 shows an example precondition for single step of the assembly plan.
An example plan be seen in Figure 4. The goal is to assemble the gear box by inserting all three gears (yellow, pink, and blue) into the shafts on the gear plate. Each gear insertion is associated with a separate policy for each color
8

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
trained with GoFlow. In addition the trained policies, the robot is given access to aejeectpprammetriized inspection action which has no preconditions and whose effects are a reduced-variant
from a threshold into cost in the belief space planner, which would remove one point of manual tuning. However, removing the parameter may prove more difficult, as it would require uncertainty quantification of the neural value function. Despite these challenges, we hope this work in- spires further research on integrating short-horizon learned policies into broader planning frameworks, particularly in contexts involving uncertainty and partial observability.
pose estimate attained by moving the camera closer to the object. The robot is initially uncertain of the x, y, and yaw components of the 6-dof pose based on probabilistic pose estimates. Despite this uncertainty, the robot is confident enough in the pose of the largest and closest yellow gear to pick it up and insert it. In contrast, the blue and pink gears require further inspection to get better pose estimate. Closer inspection reduces uncertainty along the x and axis, but reveals no additional information about yaw dimension due to rotational symmetry. Despite an unknown yaw dimension, the robot is confident in the insertion because the flow Ps indicates that success invari- ant the yaw dimension. This is due to the fact that success in the insertion task is defined by the distance between the bottom center of the gear and the base of the gear shaft, which is independent of gear rotation. For visualizations of the beliefs and flows at each step, see Appendix A.4.
A Acknowledgements
We gratefully acknowledge support from NSF grant 2214177; from AFOSR grant 1495550-2--102455 from
ONR MURI grants N00014-22-1 1-2240 and N00014-24-1- 2603; from the MIT Quest for Intelligence; and from the Robotics and Artificial Intelligence Institute. Aidan Curtis was supported by the NSF Graduate Research Fellowship. This work was partially conducted during a summer intern- ship at Autodesk.
Impact Statement
7. Conclusion and discussion
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
In this paper, we introduced GoFlow, a novel approach to domain randomization that uses normalizing flows to dy- namically adjust the sampling distribution during reinforce- ment learning. By combining actor-critic reinforcement learning with a learned neural sampling distribution, we enabled more flexible and expressive parameterization of environmental variables. leading to better generalization in complex tasks like contact-rich assembly. Our experi- ments demonstrated that GoFlow achieves higher coverage than traditional fixed and learning- -baedd domain randomiza- tion techniques across a variety of simulated environments, particularly in scenarios where the domain has irregular de- pendencies between parameters. The method also showed promise in world robotic tasks including contact-rich assembly.
References
Ajay, A., Gupta, A., Ghosh, D., Levine, S., and Agrawal, P. Distributionally adaptive meta reinforcement learning, In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871085
Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., and Fox, D. Closing the sim-to- real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973-8979. IEEE, 2019.
Moreover, we extended GoFlow to multi-step decision- making tasks, integrating it with belief-space planning to handle long-horizon problems under uncertainty. This ex- tension enabled the use of learned sampling distributions and value functions as preconditions leading to active infor- mation gathering.
Curtis, A., Kaelbling, L., and Jain, S. Task-direeted explo- ration in continuous pomdps for robotic manipulation of articulated objects. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 3721-3728, 2023. doi: 10.1 1111110AAAA99911111
Although GoFlow enables more expressive sampling dis- tributions, it also presents some new challenges. One lim- itation of our method is that it has higher variance due to occasional training instability of the flow. This instability can be alleviated by increasing 3, but at the cost of reduced sample efficiency (see Appendix A.5). In addition, using the flow and value estimates for belief-space planning require manual selection and tuning of several thresholds which are environment specific. The 1) parameter may be converted
Curtis, A., Matheos, G., Gothoskar, N., Mansinghka, V., Tenenbaum, J., Lozano-Pérez, Pérez, T., and Kaelbling, L. P. Partially observable task and motion planning with un- certainty and risk awareness, 2024. URL https:/ arxiv. arxiv.org/eae abs 220/.2203... 2400.10454.
Del Moral, P., Doucet, A., and Jasra, A. Sequential monte

---

Flow- bese- Domain Randomization for Learning and Sequencing Robotic Skills
carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, (8(3):411-4336 2006.
Liang, J., Saxena, S., and Kroemer, O. Learning active task-oriented exploration policies for bridging the sim-to- real gap. CoRR, abs/2066.01952, 2020. URL https: /ttpxii.ogg/abs///// wttpo//ggabb/...ooooo/ 01952.
Durkan, C.,, Bekasov, A., Murray, I., and Papamakarios, G. Neural spline flows. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Luo, J. and Li, H. A learning approach to robot-agnostic force-guided high precision assembly. In 2021 IEEE/RSJ Interaational Conference on Intelligent Robots and Sys- tems (IROS), pp. 2151-2157 IEEE, 2021.
Gothoskar, N., Ghavami, M., Li, E., Curtis, A., Noseworthy, M., Chung, K., Patton, B., Freeman, W. T., Tenenbaum, J. B., Klukas, M., et al. Bayes3d: fast learning and inference in structured generative models of 3d objects and scenes. arXiv preprint ar arXiv:2312.081
Mehta, B., Diaz, M., Golemo, F., Pal, C. J., and Paull, L. Active domain randomization. In Conference on Robot Learning, pp. 1162-1176. PMLR, 2020.
2023.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Mishra, U. A., Xue, S., Chen, Y., and Xu, D. Genera- tive skill chaining: Long-horizon skill planning with dif- fusion models, 2023. URL https: http///axxvvvooo.ee/e abs/2401. 03360.
Mittal, M., Yu, C., Yu, Q. Liu, J., Rudin, N., Hoeller, D., Yuan, J. L.. Singh, R., Guo, Y., Mazhar, H., Man- dlekar, A., Babich, B., State, G., Hutter, M., and Garg, A. Orbit: A unified simulation framework for inter- active robot learning environments. IEEE Robotics and Automation Letters, 86)):3740-3747, 2023, doi: 10.1 10.11099RRA
Jin, P., Lin, Y., Song, Y., Li, T., and Yang, W. Vision- force-fused curriculum learning for robotic contact-rich assembly tasks. Frontiers in Neurorobotics, 17:1280773, October 2023. doi: 10.3389/finboo.2223. 1280773.
Josifovski, J., Malmir, M., Klarmann, N., Zagar, B. I.., Navarro-Guerrero, N., and Knoll, A. Analysis of random- ization effects on sim2real transfer in reinforcement learn- ing for robotic manipulation tasks. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp. 10193-10200. IEEE, 2022.
Mozifian, M., Higuera, J. C. G., Meger, D., and Dudek, G. Learning domain randomization distributions for transfer of locomotion policies. CoRR, htts/1906.004/ 2019. 1909906.00 00410.
URL http://arxiv.or
Kaelbling, L. and Lozano-Perez, T. Integrated task and motion planning in belief space. The International Jour- nal of Robotics Research, 32:1 194-19222 08 2013. doi: 10.1 10.117702233664114444772
Muratore, F., Gienger, M., and Peters, J. Assessing trans- ferability from simulation to reality for reinforcement learning. IEEE transactions on pattern analysis and ma- chine intelligence, 43(4): 2019.
Klink, P., Abdulsamad, H., Belousov, B., D' E'Ermoo, C., Peters, J., and Pajarinen, J. A probabilistic interpretation of self-paced learning with applications to reinforcement learning. J. Mach. Learn. Res., 22(1), January 2021. ISSN 1532-4435.
Muratore, F., Eilers, C., Gienger, M., and Peters, J. Bayesian domain randomization for sim-to-real transfer. CoRR, hls/2003.024477 2020. URL https:://axii.....// abs3 02471.
Muratore, F., Gruner, T., Wiese, F., Belousov, B., Gienger, M., and Peters, J. Neural posterior domain randomization. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceed- ings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1542-1542. PMLR, 08-11 Nov 2022.
Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11) 12181)22221224, 2013.
Kwon, J., Efroni, Y., Caramanis, C., and Mannor, S. RI for latent mdps: regret guarantees and lower bound. In Proceedings of the 35th International Conference Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 978171 7781713845393.
Nasiriany, S., Liu, H., and Zhu, Y. Augmenting reinforce- ment learning with behavior primitives for diverse manip- ulation tasks. 2022 International Conference on Robotics and Automation (ICRA), pp. 7477-7484, 2021.
Labbé, Y., Manuelli, L., Mousavian, A., Tyree, S., Birch- field, S., Tremblay, J., Carpentier, J., Aubry, M., Fox, D., and Sivic, J. Megapose: 6d pose estimation of novel objects via render compare, 2022. URL https: /arxivv org/ abs /212. 6870.
Noseworthy, M., Tang, B., Wen, B., Handa, A., Roy, N., Fox, D.,, Ramos, F., Narang, Y., and Akinola, I. Forge: Force- guided exploration for robust contact-rich manipulation under uncertainty, 2024. URL https: // //aaxiivvvo. /rrg/ abs/2408. 04587.
10

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plap- pert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q. Zaremba, W., and Zhang, I.. Solving rubik's cube with a robot hand. CoRR, abs/1910.07113, 2019. URL http:
Tang, B., Lin, M., Akinola, I., Handa, A., Sukhatme, G., Ramos, F., Fox, D., and Narang, Y. Industreal: Trans- ferring contact-rich assembly tasks from simulation to reality. 07 2023a. doi: 10.
Tang, B., Lin, M. A., Akinola, I., Handa, A., Sukhatme, G. S., Ramos, F., Fox, D., and Narang, Y. Industreal: Transferring contact-rich assembly tasks from simulation to reality, 2023b. URL https: /ttp///rxvvv////////vv///////aaa org worg/aaa abs// 2305 17110.
www.aiioorg/aae.e....oooooooeeee httg//bb/1990 191019111..0 07113.
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., and Song, D. Assessing generalization deep reinforcement URL http:
learning. CoRR, abs/1810. 12282, 2018.
//arxiv. .rgg abs 111//1110011 12282.
Tiboni, G., Klink, P., Peters, J., Tommasi, T., D'Eramo, C., and Chalvatzaki, G. Domain randomization via entropy maximization, 2024. URL https://arxiv..
Peng, X. B., Andrychowizz, M., Zaremba, W., and Abbeel, P. Sim-to-real transfer of robotic control with dynam- ics randomization. CoRR, abs/1710.06537, 2017. URL http://aaxlv..rrg/as//700....0
0ttp/2331.0018855
Valassakis, E., Ding, Z., and Johns, E. Crossing the gap: deep dive into zero-shot sim-to-real transfer for dynamics. In 2020 IEEE/RSJ International Conference Intelli- gent Robots and Systems (IROS), pp. 5372-5379. IEEE, 2020.
Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., and Abbeel, P. Asymmetric actor critic for image-based robot learning. RSS, 2018.
Ramos, F. Possas, R. C., and Fox, D. Bayessim: adap- tive domain randomization via probabilistic inference for robotics simulators. CoRR, abs/1906.01729, 2019. URL http:/ /ttp:///aavv.///////a/eeeeraaa. wwexiiv.org//.. abs 1tt///996..011 017288.
Wang, C., Lv, Y., Mao, Y., Qu, Y., Xu, Y., and Ji, X. Robust fast adaptation from adversarially explicit task distribu- tion generation, 2025. URL https: arxiv..rg.. abs 0288/2407.15222
Ren, Z., Dai, H., Burchfiel, B., and Majumdar, A. Adapt- sim: Task-driven simulation adaptation for sim-to-real abs/
Wen, B., Yang, W., Kautz, J., and Birchfield, S. Founda- tionpose: Unified 6d pose estimation and tracking of novel objects, 2024. URL https://arxiv.or abs/2312 08344.
transfer, 2023. URL https:://ariiii 2302. 04903.
Rezende, J. and Mohamed, S. Variational inference with normalizing flows. In Proceedings of the 32nd Interna- tional Conference on Machine Learning (ICML). PMLR, 2015.
Yu, W., Liu, C. K., and Turk, G. Policy transfer with strategy optimization. CoRR, abs/ 010.005511 2018. URL http: /
1aw.bb/s1111.00 05751.
Zhang, K., Sharma, M., Liang, J., and Kroemer, O. A
Rozet, F. et al. Zuko: Normalizing flows in pytorch, 2022. URL htt.t::/// /ttp://yypp.////// wttgproojecttooooooooooooooooooe zuko.
modular robotic control stack for research: Franka- interface and frankapy. arXiv:2011 2xX1:2011.002998. 2020.
Sagawa, S. and Hino, H. Gradual domain adaptation via normalizing flows. Neural Computation, 37(3):522-568, 2025. doi: 10. 162/neco_a_01733
Zhang, X., Tomizuka, M., and Li, Bridging the sim-to-
real gap with dynamic compliance tuning for industrial insertion. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4356-4363, 2024. doi: 10. 1..110//CAA51147..00000 106110070..1077.7
Schoettler, G., Nair, A., Luo, J., Bahl, S., Ojea, J. A., Solowjow, E., and Levine, S. Deep reinforcement learn- ing for industrial insertion tasks with visual inputs and natural rewards. In 2020 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. IEEE, 2020.
Zhu, H. Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., and Levine, S. The ingredients of real- world robotic reinforcement learning. arXiv preprint ar04v:2004 200:20004./2200 /2570, 2020.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, 0D0/17770063477 2017. URL http: //ar:v/aaaivvv// org/ bogg/bs///e 1alss/170...0 06347.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
A. Appendix
Hence
A.1. Code Release
/ po(E) log
[L u(E)
po(e) a(S)
log po(() de.
The codebase for the project can be found here.
If | is the measure of under u(E). then u(E) = 1/| So the Monte Carlo estimate for J pp(() log (E) de becomes
A.2. Importance Sampling Proofs
Here we show how we obtain the importance-sampled esti- mates for both the reward term R and the entropy term H. in CoFlow(Algorithen 1). Specifically, we sample from a uni- form distribution u(E) over E, rather than from the learned distribution (C) directly, in order avoid collapse onto narrow regions of the parameter space.
B Σ (e() log loga)(Et).
B
with & ~ u(E). Multiplying by -1 yields the differential entropy:
H(ps) - / pol() log de
A.2.1. REWARD TERM: R
We want an unbiased estimate of
Be-pa (8) [Je(TT] = (po(e) Je(TT) dg.
B Σ PolE:) log pe()).
B
Let u(E) be a uniform distribution over E. Since
which is exactly what we implement in Line of Algo- rithm 1.
(E) JE(T) po(E) m(E) u(E) (T), can rewrite:
Remark A.I. Using uniform sampling u(E) to approximate these terms provides global coverage of E, helping prevent the learned distribution ps from collapsing around a small subset of parameter space. By contrast, one sampled from (5) itself for these terms, the distribution might fail to expand to other promising regions once it becomes peaked.
(E) [ m(() (+( u({) Je((T) dg.
Hence, sampling & ~ u(E) and averaging
Poll))
u(E.) Jg, (T)
gives an unbiased Monte Carlo estimate of Ep,, (=)
If E has finite measure |E], then u({) = 1/ Thus
(E (E) E|. u{E)
A.3. Domain Randomization Parameters
Below describe the randomization ranges and parameter names for each environment. We also provide the reward success threshold (Jr) and cut the max duration of some environments in order to speed up training (mmar). J, was chosen to be below the optimal performance under no envi- ronment randomization. We verified that the trained policy still exhibited qualitatively successful performance at the target reward threshold. Lastly, we slightly modified the Quadruped environment to only take fixed forward com- mand rather than the coal-conditioned policy learned by default. Other than those changes, the first four simulated environments official IsaacLab implementation.
Therefore, the empirical estimate becomes
R 1
(Es) (Et) EI) Je.
B
Σ p (E (T).
B
where (6) ~ u(E). This matches Line 7 in Algorithm 1.
A.2.2. ENTROPY TERM: H
Similarly, to compute the differential entropy of (E), we have
Cartpole parameters (Jr = 50, tmax 2s):
Pole mass: Min 0.01, Max 20.0
H(ps) = -
{ - po(E) log (c( dg.
-
Cart mass: Min 0.01, Max = 20.0
Slider-Cart Friction: Min Bound 0.0, Max
Again, we apply the same importance sampling trick via (E)). We write:
Bound = 1.0
Ant parameters (JT = 700, bmaz = 2s):
Po(E) u({)
p)(E) lo)
u(E) log (gE)..
Torso mass: Min = 0.01, Max 20.0
12

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Quadcopter parameters (Jr 15, mmax 29)):
where i and j are pixel row and column indices, C is the set of valid pixels returned by the renderer, Digb and by are hy- perparameters that control posterior sensitivity to the color and depth channels, and Pour is the pixel outlier probability hyperparameter. For an observation Orghd, we can sample from Prrx: 4. Orgba) x P(Ortu Tas L. y) to recover the object pose posterior with tempering exponential factor o to encourage smoothness. We first find the maximum a pos- teriori (MAP) estimate of object pose using coarse-to-fin sequential Monte Carlo sampling (Del Moral et al., 2006) and then calculate posterior approximation using grid centered at the MAP estimate.
Quadcopter mass: Min 0.01, Max 20.0
Quadruped parameters (Jr 1.5, max 55):
Body mass: Min 0.0, Max 200.0
Left front hip friction: Min = 0.0, Max = 0.1
Left back hip friction: Min 0.0, Max 0.1
Right front hip friction: Min 0.0, Max 0.1
- Right back hip friction: Min 0.0, Max 0.1
Humanoid parameters (Jr 1000, tmax 5s):
The benefit of this approach in contrast to traditional rendering-basete pose estimation systems, such as those presented in (Wen et al., 2024) or (Labbé et al., 2022), is that pose estimates indicate high uncertainty for distant, small, occluded, or non-visible objects as well dimensions along which the object is symmetric. A visualization of the pose beliefs at different points in the multi-step plan can be seen in Figure 14 in the Appendix.
Torso Mass: Min 0.01, Max 25.0
Head Mass: Min 0.01, Max 25.0
Left Hand Mass: Min = 0.01, Max 30.0
Right Hand Mass: Min 0.01, Max 30.0
Gear parameters (Jr 50, mar 4s):
Grasp Pose x: Min -0.05, Max = 0.05
Grasp Pose y: Min -0.05, Max = 0.05
A.S. Hyperparameters
Grasp Pose yaw: Min -0.393, Max 0.393
Below we list out the significant hyperparameters involved in each baseline method, and how we chose them based on our hyperparameter search. We run the same seed for each hyperparameter and pick the best performing hyper- parameter the representative for larger quantitative experiments in figure 3. The full domain randomization (FullDR) and no domain randomization (NoDR) baselines have no hyperparameters.
A.4. Multi-Step Planning Details
A.4.1. UPDATING BELIEFS VIA PROBABILISTIC POSE ESTIMATION
Updating the belief state b requires probabilistic state estimation system that outputs a posterior over the state space S, rather than single point estimate. Given a new observation o, we use probabilistic object pose estimation framework (Bayes3D) to infer posterior distributions over object pose (Gothoskar al., 2023).
1.5.1. GoFLow
We search over the following values the a hyperparam- eter: [0.1, 0.5, 1.0, 1.5, We search over the follow- ing values of the 3 hyperparameters [0.0, 0.1, 0.5, 1.0, 2.0). Other hyperparameters include number of network updates per training epoch (K = 100), network learning rate (ns le - 3), and neural spline flow architecture hyper- parameters such as network depth (E = 3). hidden features (64), and number of bins (8). We implement our flow using the Zuko normalizing flow library (Rozet et al., 2022).
The pose estimation system uses inference in probabilis- tic generative graphics model with uniform priors on the translational æ, and rotational yaw (or r2) components of the 6-dof pose (since the object is assumed to be in flush contact with the table surface) and an image likelihood P(Orgh Tay TTy I. 3). The object's geometry and color infor- mation is given by mesh model. The image likelihood is computed by rendering latent image im!ell with the object pose corresponding to (ra, F, y) and calculating the per-pixel likelihood:
A.5.2. LSDR
Similary to GoFlow search over the following values
P(Orgbd Tz, F, v) x
of the aj hyperparameter: [0.1. 0.5, 1.0, 1.5, 2.0). Other hyperparameters include the number of updates per train- ing epoch (T=100), and initial Gaussian parameters: μ (Emaz + Emin)/2.0 and Σ diag (Emax /min/10)
II 1,EEC
Pout + (1 Poot) Pin Po 2,
rghul
(8)
Pun (o
1r, 2. 2. xc
|
im 1
reb
1108 img
(9)
exp
bebb
13

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Method
FullDR
NoDR
DORAEMON
LSDR
ADR
GoFlow
Success Rate
6/10
3/10
5/10
5/10
5/10
9/10
Table Real-world experimental results with the statistically significant results bolded.
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
FulIDR
0.060:1.002
0.000=10.000 0.004
0.012.100077
0.195:0.004
0.000110.000
NoDR
0.0001100000 0.005
0.000=10000
0.086=000008
0.001
0.008:10.003
DORAEMON
0.015:10.005 0.0001100000
0.000=100000
0.029#00008
0.000=100000
0.012:10.004
LSDR
0.103:+00013 0.000:10.000
0.007=0.003
0.009=#0008
0.176=10.822
0.000=#0.000
ADR
0.006:1:0001
0.000:10.000
0.000=00000
0.000=#0.000 0.070=#00066
0.008=10.044
GOFLOW
0.138:10.005
0.056:10.033
0.010=0.006
0.275=
0.274=1.076
0.035:#0.019
Table Mean and standard error (SDE) of the final reward value, with entries in bold indicating the pest-perforning method or methods not statistically significantly worse than the best (one-tailed z test, 0.05).
A.5.3. DORAEMON
dependent on Jr, GoFlow largely dominates the coverage distribution for wide ranges of Jr.
We search over the following values of the hyperparame- ter: [0.005, 0.01, 0.05, 0.1, 0.5). After fixing the best eD for each enviromment, we additionally search over the success thrshold aD: [0.005, 0.01, 0.05, 0.5)..
A.8. Real-world experiments
In addition to simulated experiments, we compare GoFlow against baselines on real-world gear insertion task. In particular, we tested insertion of the pink medium gear over trials for each baseline. To test this, we had the robot perform 10 successive pick/inserts of the pink gear into the middle shaft of the gear plate. Instead of randomizing the pose of the gear, we elected to fix the initial pose of the gear and the systematically perturb the end-effector pose by a random #0.01 meter translational offset along the x dimension during the pick. We expect some additional grasp pose noise due to position error during grasp and object shift during grasp. This led to a randomized in-hand offset while running the trained insertion policy. Our results show that GoFlow can indeed more robustly generalize to real-world robot settings under pose uncertainty.
A.5.4. ADR
In ADR, we fix the upper threshold be the success threshold + Jr as was done in the original pa- per and search over the lower bound threshold t- [0.1tt, 10.25* 0.5t+, 0.75tt, 0. The value used in the original paper was 0.5tH- Other hyperparameters in- clude the expansion/contraction rate, which we interpret to be a fixed fraction of the domain interval, A 0.1 * [Smux {min]. and boundary sampling probability ps = 0.5.
A.6. Coverage vs Range Experiments
We compare coverage vs. range scale in the ant domain. We adjust the parameter lower and upper bounds outlined in Appendix A.3 and see how the coverage responds to those changes during training. The parameter range is defined rel- ative nominal midpoint set to the original domain pa- rameters: (m- [m-(mm lowerl The results of our experiment are shown in Figure 12
A.9. Statistical Tests
We performed statistical analysis of the simulated results reported in Figure 3 and the real-world experiments in Ta- ble 1. For the simulated results, we recorded the final do- main coverages across all seeds and performed pairwise t-tests between each method and the top-performing method. The final performance mean and standard deviation are re- ported in Table 2. Any methods that were not significantly different from the top performing method (p < 0.05) are bolded. This same method was used to test significance of the real-world results.
m+ (upper-m)**ccn }c
A.7. Coverage vs. Threshold Experiments
In this experiment, assess the sensitivity of coverage to the performance threshold J, for single training seed. We experiment with threshold multipliers between 0 and 1.2- The results of this experiment can be seen in Figure 13. These results demonstrate that although coverage is highly
14

---

flow- Domain Randomization for Learning and Sequencing Robotic Skills
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
0.6
0.10
0.15
0.075
Coverage
0.10
0.050
0.05
0.029
0.02
0.2
0.00
0.000
0.00
0.0
0.00
x23*
X264
X10
x13"
k20
Timesteps
Timesteps
Timesteps
Timesteps
Timesteps
a-2.0
2=1.5
-=10
0=0.55
8=0.1
Figure 6. GoFlow hyperparameter sweep results for a
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
0.5
0.15
0.10
Coverage
0.4
0.10
0.000
0.05
0.02
0.2
0.00
0.000
3.0
0.00
0.0
0.3
0.0
0.00
Timestops
V12'
Timestops
x02
Timesteps
X06
Timesteps
X10'
Timesteps
Timesteps
p=2.0
p=1.0
8-0.5
p=0.1
p-0.0
Figure 7. GoFlow hyperparameter sweep results for / after fixing the best for each environment
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
0.05
0.0075
0.05
0.0075
0.6
0.006
Coverage
0.04
6.0050
0.0050
00044
0.02
0025
0.2
0.00
0.0
Timestops
Timesteps
Timesteps
S16
Timesteps
X16
Timestops
+1.7
Timosteps
K000
02.0.0
a=15
1=0
0000.5
a0
Figure 8. LSDR hyperparameter sweep results for az
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
0.94
0.10
0.003
Coverage
0.002
0.00
0.00
0.05
0.001
0.02
0.90
1.5
0.00
0.00
Timesteps
Timesteps
X22*
Timesteps
X064
Timesteps
X201
Timesteps
x127
Timesteps
K204
0.5
5000.05
4.00.01
5000.00
Figure 9, DORAEMON hyperparameter sweep results for ep
15

---

Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
0.94
0.30
0.003
Coverage
0.002
0.00
0.00
0.05
0.001
0.02
0.90
-0.05
-0.05
0.00
0.00
X23*
X264
X10"
x137
Timesteps
Timesteps
Timesteps
Timesteps
Timesteps
0.50.5
2p-0.1
20
2000.01
Figure 10. DORAEMON hyperparameter sweep results for o,p after fixing the best ép
Cartpole
Quadcopter
Quadruped
Humanoid
0.05
Coverage
0.02
0.90
0.01
0.002
0.0
Timesteps
'.0'
Timesteps
Timesteps
Timestops
Timestops
100
Timesteps
K20P
1=0.99
(=0075
("=025
100.1
Figure 11. ADR hyperparameter sweep results for t
DORAEMON
Coverage
10-2
10-2
10-3
30
10
10
10.4.
10
10-4
10
10-4
3.0
2.5
155
Timesteps
X1
Timesteps
Timesteps
Timesteps
Timesteps
Timesteps
K10
scale-0.1
scale-0.75
scalo-0.
Figure 12, Coverage vs range experiment results discussed in Section A.6
Cartpole
Quadcopter
Quadruped
Humanoid
0.6
Coverage
0.4
0.05
0.2
0.00
0.0
0.9
0.0
122
0.4
0.4
0.8
Multiplier
Multiplier
Multiplier
Multiplier
Multiplier
Multiplier
GOFLOW
FulDR
NoDR
DORAEMON
LSDR
ADR
Figure 13, Coverage vs threshold experiment results discussed in Section A.7
16

---

Plow-based Domain Randomization for Learning and Sequencing Robotic Skills
Cartpole
Ant
Quadcopter
Quadruped
Humanoid
Gears
FulIDR
-920.132
362.454
-2.286
0.108
657.414
8.627
NoDR
-949.225
133.331
-3.807
-4.062
180.734
12.697
DORAEMON
-930,417
86.742
-1.767
-2.422
235.516
11.378
LSDR
-867.347
367.612
-1.870
-0.052
532.531
9.289
ADR
-953.498
64.815
-3.13
-2.926
246.297
9.671
GOFLOW
-820.510
198.302
-1.000
-2814
376.652
14.209
Table CVaR computed the mean of rewards falling below
10% percentile (VaR).
yellow segment
pink segment
blue segment
yellow inspect
pink inspect
blue inspect
y transiation (mm)
5.0
7755
10..0
-20
-20
0
20
-20
20
-20
20
-20
20
-20
20
-20
20
12.5
translation (mm)
translation (mm)
translation (mm)
translation (mm)
translation (mm)
translation (mm)
transition (mm)
0.0
7.0
2.5
7.5
8.0
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
angle (rad)
angle (rad)
angle (rad)
angle (rad)
angle (rad)
yaw angle (rad)
trannsattonn (mm)
2.5
6.5
0.0
7.0
-2.5
1.b
-2.5
0.0
-2.5
0.0
-2.5
2.5
-2.5
2.5
-2.5
2.5
-2.5
0.0
2.5
yaw angle (rad)
yaw angle (rad)
yow angle (rad)
yaw angle (rad)
yaw angle (rad)
yaw angle (rad)
Figure 14. A visualization of the beliefs over the object under the initial image (first three columns) and after closer inspection (last three columns) as generated from the posterior of model described Section 6.3. The colormap corresponds the log probability the posterior pose estimate, All plots centered around the most likely pose estimate under the image model.
17

---

based Domain Randomization for Learning and Sequencing Robotic Skills
Algorithm 2 Belief- Belief-Space Planner Using BFS
Require: Initial belief state bp. goal condition C B, set of skills An, success threshold
1: Initialize the frontier F < {bo}
2: Initialize the visited set V
3: Initialize the plan dictionary Plan mapping belief states to sequences of skills
4: while F is not empty do
5: Dequeue from F
6: if E G then
7: return Plan[b]
b Return the sequence of skills leading to b
8: end
9: for all skills क E An do if b Pre, given 7) then V' sample(Bff,) if D' & V then Add 1/ to F and V Update Plan[6'] + Plan[b] + [T]
12:
13:
14:
15:
end 16: end
end for
18:
end while
19:
return Failure
b. No plan found
18


Reviews:
[{"review_id": "KzsHnJmGnr", "reviewer": "Reviewer_aKt5", "summary": "The paper proposes updating environment parameters for policy training by learning a sampling distribution parameterized as a normalizing flow. Normalizing flow is known to be capable of representing more expressive distributions. The distribution is trained to maximize policy performance, maximize its marginal entropy, and minimize the change from the previous distribution. The method is validated on a few simulated tasks showing improved coverage (percentage of the parameters where the policy performance is higher than some threshold) over the baselines, as well as being applied as a OOD detector for a real-world robot manipulation planner.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed insights on experimental design and parameter selection, which have greatly helped us clarify our approach. Below we address each of your comments and questions.\n\n> … how are the initial distribution and target distribution decided? How are $\\alpha$ and and $\\beta$ chosen in Fig. 3?\n\nThe target distribution is the same across all methods, as it is a uniform distribution over a set of physical properties such as link masses, joint frictions, and object poses. We attempted to select ranges for these parameters that were large enough to capture all possible physically realistic parameter settings. \nThe initial distribution depends on the method. For GoFlow, the initial distribution is defined by the random initialization of the network.\n$\\alpha$ and $\\beta$ were chosen through a hyperparameter selection process detailed in Appendix A.5.\n\n> More critically, it was not clear how $J_t$, the threshold for reaching target performance at any environment, is chosen. More discussion/justification (or additional results on varying $J_t$) should be provided.\n\n$J_t$ was chosen to be slightly below the optimal performance under no environment randomization. We verified that the trained policy still exhibited “qualitatively successful” performance at the target reward threshold. For example, we verified that the ant still exhibits running behavior at the chosen target threshold. We have updated the manuscript to describe this selection process. \nAdditionally, we performed an experiment showing how coverage changes with $J_t$. Although we cannot update the manuscript during the rebuttal process, new results in appendix section A.7 show that while the coverage is highly dependent on the choice of $J_t$, GoFlow outperforms baseline methods across almost all choices of $J_t$.\n\n> I think the line of work around task-driven system identification [1,2,3] is very relevant\n\nWe agree that these are relevant papers, and we have updated our related work [1,2] and introduction [3] to include them.\n\n> I recommend putting GoFlow as either the first or the last method in the legend in Fig. 3.\n\nWe have moved GoFlow to be the first method in Figure 3.\n\n> Do you have intuition why quadruped works the best with \\beta=0 as Fig. 7 shows?\n\nWhile $\\beta$ can help with training stability, it can also cause the distribution to converge more slowly. The quadruped domain specifically was less sensitive to large swings in the sampling distribution, and therefore did not benefit from larger $\\beta$.\n\n> Some of the figures in the paper can be improved. I don't understand the \\theta part of Fig. 4. It also took me quite a while to understand Fig. 5, especially top row vs. bottom row. I think the caption can be vastly improved to provide more context and detailed explanations.\n\nThank you for this suggestion. We have expanded on the captions to figures 4 and 5 to improve clarity.\nFor figure 4, we modified the figure to remove $\\theta$ and replace it with yaw, which is how it was described in the caption and elsewhere in the paper. We also defined it in the caption.\nFor Figure 5, we rewrote the entire caption to clearly describe the meaning of each column/row of subfigures. The new caption reads as follows: \n\n> \"A visual example of the precondition computation described in Section 6.2 for the gear assembly plan shown in Figure 4. The two rows show two different projections of the 3D sampling space (x position vs y position in the top row and y position vs yaw rotation in the bottom row). We apply a threshold $\\epsilon$ to the sampling distribution to remove low-probability regions (column 1). Additionally, we filter the value function by retaining only the regions where the expected value exceeds a predetermined threshold $\\eta$ (column 2). The intersection of these two regions defines the belief-space precondition, indicating where the policy is likely to succeed (column 3). \n    Comparing the precondition to the beliefs, we can see that the belief is not sufficiently contained within the precondition at $t=0$ (column 4), but passes the success threshold $\\eta$ at after closer inspection at $t=4$ (column 5).\"", "claims_and_evidence": "The paper's main claim is that the learned sampling distribution improves the overall performance in testing environments. The results are mostly shown in Fig. 3. While the results themselves look convincing, many details of the experiment setup seem missing (or should have been provided in the main text). For example, how are the initial distribution and target distribution decided? How are $\\alpha$ and $\\beta$ chosen in Fig. 3?\n\nMore critically, it was not clear how $J_t$, the threshold for reaching target performance at any environment, is chosen. I imagine the choice can largely affect the behavior of the curves shown in Fig. 3. More discussion/justification (or additional results on varying $J_t$) should be provided.", "methods_and_evaluation_criteria": "I think the set of tasks considered in the environments is fairly comprehensive, and the study on a real-world manipulation task is well-appreciated. However, I do feel the toy problem in Fig. 2 is too contrived given the nature of flow matching vs. the baselines.", "theoretical_claims": "The paper has a small amount of theoretical claims in Appendix A.2, and they look correct to me.", "experimental_designs_or_analyses": "Again, I think the paper is missing important details on the experimental designs as discussed above.", "relation_to_broader_scientific_literature": "The paper studies the effect of environment parameters for training effective control policies. Such study is fitting as we consider more generalizable policy. However, I do think the paper lacks more discussion on how the approach can be applied in more realistic real-world tasks (for improved performance instead of just for OOD detection).", "essential_references_not_discussed": "I think the line of work around task-driven system identification [1,2,3] is very relevant, also trying to identifying relevant parameters for improving policy performance.\n\n[1] Muratore et al., Data-efficient domain randomization with bayesian optimization\n\n[2] Ren et al., Adaptsim: Task-driven simulation adaptation for sim-to-real transfer\n\n[3] Liang et al., Learning active task-oriented exploration policies for bridging the sim-to-real gap", "other_strengths_and_weaknesses": "Some of the figures in the paper can be improved. I don't understand the $\\theta$ part of Fig. 4. It also took me quite a while to understand Fig. 5, especially top row vs. bottom row. I think the caption can be vastly improved to provide more context and detailed explanations.", "other_comments_or_suggestions": "I recommend putting GoFlow as either the first or the last method in the legend in Fig. 3.", "questions_for_authors": "Do you have intuition why quadruped works the best with $\\beta=0$ as Fig. 7 shows?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0QwvgyrnG", "reviewer": "Reviewer_CXok", "summary": "This paper proposes a normalizing flow based approach to learn sampling parameters for domain randomization. Instead of doing naive sampling for domain randomization hyper-params, the paper uses a more principled way (which has been proposed before). But different from previous works, this paper proposes using a normalizing flow based model to learn the sampling distribution and iteratively improve the policy by sampling domain parameters from this distribution. Experimental results in simple domains shows the effectiveness of the proposed approach. The proposed approach is also combined with a belief space planner to show how the proposed approach can be combined with traditional belief space planning to accomplish long horizon tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful analysis and positive review. Below we address each of your comments and questions. \n\n> For the other mujoco experiments it was not immediately clear how the base sampling distributions were chosen\n\nWe attempted to select ranges for these parameters that were large enough to capture all possible physically realistic parameter settings. \n\n> I guess one big concern would be how realistic are some of these assumptions made in the paper. In real-world scenarios we often do Sys-ID on the robot to find good initial set of hyperparams and then make it robust around that nominal set. But the proposed approach (in the main paper) uses a very broad initial set which is a bit more complex and clearly a non-learning based approach would not work in this scenario.\n\nWe agree with the assessment of how sim-to-real issues are typically addressed, and the limitations of this process are some of the driving motivations for this paper. First, this manual Sys-ID to find a nominal set of hyperparameters requires some human effort on a per-task basis, which we are able to eliminate by selecting broad enough ranges. Second, the nominal parameter values and resulting distributions may not be optimal ones for task performance and robustness. Lastly, the integrated planning system is much more efficient and effective with large parameter ranges as opposed to narrower distributions around nominal values. This is because less information gathering is required by policies that are robust to a larger space of parameters.\n\n> Basically, I am curious to hear how would the experiments (mujoco ones) change if the feasible domain regions are not that complex?\n\nIf the feasible domain regions are centered and regularly shaped, we see the performance of other learning-based methods increase. The experiments in Appendix A.6 demonstrate this property.\n\n> Why was the choice of spline based normalizing flow used to learn the sampling distribution? \n\nWe did not have a specific reason to use spline flows over other models outside of its superior performance compared to other architectures in the Zuko library.", "claims_and_evidence": "Yes, the claims are well supported. The paper claims that their proposed domain randomization appraoch is more robust which is what the experiments correctly validate and show.", "methods_and_evaluation_criteria": "Yes, although more complex tasks/benchmarks could be created (e.g. more complex control tasks). But even the current set of tasks seem adequate.", "theoretical_claims": "No, there is minimal theory and the overall algorithm is sound.", "experimental_designs_or_analyses": "Yes. The toy experiment is a bit contrived in my opinion since there is assumed to be this complex relation in the space. Clearly, a more complex model would be able to fit it much better than other naive approaches (most baselines). \nFor the other mujoco experiments it was not immediately clear how the base sampling distributions were chosen, but it seems they were chosen to higlight larger difference than baseline. Figure 12. in the Appendix provides a more clear picture. I guess one big concern would be how realistic are some of these assumptions made in the paper. In real-world scenarios we often do Sys-ID on the robot to find good initial set of hyperparams and then make it robust around that nominal set. But the proposed approach (in the main paper) uses a very broad initial set which is a bit more complex and clearly a non-learning based approach would not work in this scenario.", "relation_to_broader_scientific_literature": "Domain randomization (DR) is very important for making robots work in the real world especially sim2real. Most legged locomotion stuff relies on DR. However, in those scenarios we often do sys-ID and then use careful engineering/expertise to iterate on things. The proposed approach is more automated, however, it is unclear if it can be applied to complex real-world problems, i.e, does it obviate the need of engineering (most likely not). Overall, the problem statement is important,", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The paper focuses on an important problem. It is quite well written and clearly explains the problem and the solution. The proposed approach is not extremely novel but is well executed. \n\nI do have some concerns about the experiments. Basically, I am curious to hear how would the experiments (mujoco ones) change if the feasible domain regions are not that complex? Also, is the assumption made in the paper realistic?  Most real world robotics works such as OpenAI hand rubik's cube and recent legged robots work often use good old-fashioned engineering and robotics to solve similar problems. How challenging or complex would be to apply the proposed approach for such scenarios, since the base assumption is that a flow based model could learn complex sampling distributions.\n\nWhy was the choice of spline based normalizing flow used to learn the sampling distribution? That part does not seem to be motivated in any way (unless I missed it).", "other_comments_or_suggestions": "none", "questions_for_authors": "please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XAi2h5ZcCs", "reviewer": "Reviewer_95Yv", "summary": "Domain randomization is a known and useful technique to transfer models trained in simulation to the richness of the real-world. There are several known methods in the past. The usual workflow of the algorithm is to learn or estimate the distribution of valid parameters that is solvable by the policy optimization simultaneously with the policy. It requires smart choices of loss function and training algorithms to achieve this goal. In this paper, the authors use a normalizing flow to fit the parameter distribution and use an entropy regularized loss function for training. The conjecture is that neural parameter distributions are more expressive and will be conducive to generalization. Results are demonstrated on some benchmarks to demonstrate more robust generalization as compared to previous baselines. Further, real-world demonstrations are shown on a manipulation task and extensions are made to multi-step planning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and suggestions on expanding our related work and improving figure clarity, which have enhanced the overall presentation of our paper. Below we address each of your comments and questions.\n\n> The citations can be more comprehensive with more mention of domain adaptation, transfer learning and sim2real transfer. Some examples are mentioned below:\n\nWe agree that these are relevant papers, and we have updated our related work to include them.\n\n> Also, the authors are requested to expand further on these lines: \"Some previous works have combined domain randomization with information gathering via system identification (Ramos et al., 2019; Sagawa & Hino, 2024)\"\n\nWe have expanded on this discussion further in the related work section. That section of the related work now reads as follows:\n\"Beyond training robust policies in simulation, learned sampling distributions can be tied to the real-world environmental conditions under which policies are likely to succeed. Previous works have integrated domain randomization with real-world interactions for more informed training distributions [citations] or to find the maximally effective real-world strategy [citations]. However, these methods often necessitate expensive policy retraining or data-intensive evolutionary search based on real-world feedback, posing challenges for real-time applications. Instead, we utilize our learned sampling distribution as an out-of-distribution detector within a multi-step planning framework, enabling fast and data-efficient information gathering in the real world.\"\n\n> The use of domain randomization is to result in domain generalization and sim2real. It is difficult to assume that domain randomization alone will achieve this. There could be some more discussion about the other approaches.\n\nWe agree that domain randomization alone is not always sufficient for transfer. Hopefully our updated related work discussion helps highlight that real-world feedback is an important component of sim2real transfer.\n\n> In terms of the choice of neural sampling distribution, LSDR also uses a learned distribution. Is the main difference that LSDR uses a neural network to fit the GMM parameters? Or do they learn the GMM parameters directly?\n\nThe LSDR baseline directly learns multivariate gaussian parameters. They don’t learn a GMM or use neural networks, but rely a simple Gaussian representation.\n\n> The writing is unclear in certain places. See below.\n\nThank you for pointing these out. We have addressed all of these in the updated manuscript. Specifically, we made the following updates:\n1. We clarified our statement regarding the usefulness of RL in the introduction. It now reads as follows: \"Reinforcement learning (RL) is a powerful tool in robotics because it can be used to learn effective control policies for systems with highly complex dynamics that are difficult to model analytically. Unlike traditional control methods, which rely on precise mathematical models, RL learns directly from simulated or real-world experience [citations].\"\n2. We updated the caption of figure 5 to much more clearly state what each subfigure is. Please see our rebuttal to Reviewer aKt5 for details.\n3. We elaborated on the yaw invariance property of the gear-insertion problem in the main text of the paper. It now reads as follows: \"Despite an unknown yaw dimension, the robot is confident in the insertion because the flow $p_\\phi$ indicates that success is invariant to the yaw dimension. This is due to the fact that success in the insertion task is defined by the distance between the bottom center of the gear and the base of the gear shaft, which is independent of gear rotation.\"", "claims_and_evidence": "The authors claim that using a neural normalizing flow for learning the distribution of domain parameters is an improvement on the state of the art for domain randomization. This is supported by adequate experimental evidence. Also, this same method is illustrated to be useful for learning the precondition distribution in a belief space planner.", "methods_and_evaluation_criteria": "The methods are sound and the evaluation criteria is acceptable. The proposed method achieves a bigger coverage range as compared to the baselines. The previous solution of fitting beta distributions has certain advantages. That is amenable to using a trust region constraint rather than a trust region regularizer which is adhoc.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "There is a toy example which is useful for the reader to understand the essence of the paper. Then, there are results illustrated on MuJoCo benchmarks. The coverage metric is an indicator of how much of the parameter space results in a successful policy. Further experiments on a Bayesian multi-step planner illustrate a novel use case for the algorithm.", "relation_to_broader_scientific_literature": "The topic of domain randomization is under discussion for some time and gained visibility by OpenAI's work on solving the sim2real problem for rubik's cube manipulation. Even though the methods are simple, the effectiveness is usually quite good. This paper is an useful extension. Apart from incremental advances on existing benchmarks and sim2real application, it introduces sampling distribution modeling in the context of belief state preconditions. Further, there is a lot of work in the use of domain randomization for out-of-distribution generalization in the context of images. The dynamics randomization is relatively less explored.", "essential_references_not_discussed": "The citations can be more comprehensive with more mention of domain adaptation, transfer learning and sim2real transfer. Some examples are mentioned below:\n\n[A1] Yu, Wenhao, C. Karen Liu, and Greg Turk. \"Policy transfer with strategy optimization.\" arXiv preprint arXiv:1810.05751 (2018).\n\n[A2] Sagawa, Shogo, and Hideitsu Hino. \"Gradual domain adaptation via normalizing flows.\" arXiv preprint arXiv:2206.11492 (2022).\n\nAlso, the authors are requested to expand further on these lines: \"Some previous works have combined domain randomization with information gathering via system identification (Ramos et al., 2019; Sagawa & Hino, 2024)\"", "other_strengths_and_weaknesses": "Strengths:\n\nThe flow of the paper with the motivation, method and experimental results are good. I find the illustrations on robot assembly tasks and belief-space planning a new addition. Overall, there is a knowledge gain for the reader. \n\nThe strength of the proposed method lies in the effectiveness despite its simplicity. More theoretical study on why this particular loss function and choice of normalizing flow is good will be interesting for future research.\n\nWeaknesses:\n\nThe use of domain randomization is to result in domain generalization and sim2real. It is difficult to assume that domain randomization alone will achieve this. There could be some more discussion about the other approaches.\n\nOn the flipside, the methods by themselves are not very novel though effective. The entropy regularization objective, the normalizing flow and neural probablistic modeling are all well-known. They are used in an interesting and effective way.\n\nThe writing is unclear in certain places. See below.", "other_comments_or_suggestions": "“Reinforcement learning (RL) has proven to be a useful tool in robotics for learning control or action policies for tasks and systems which are highly variable and/or analytically intractable” - unclear statement\n\nFig 5 caption: “The thresholded sampling distribution is further thresholded by the value function to get the belief-space precondition” - unclear statement\n\nSome additional information in the supplementary material could be discussed further in the main section of the paper. For example, there is a passing mention of the policy being independent to the yaw. This could be discussed in more detail.", "questions_for_authors": "In terms of the choice of neural sampling distribution, LSDR also uses a learned distribution. Is the main difference that LSDR uses a neural network to fit the GMM parameters? Or do they learn the GMM parameters directly?\n\nWhat happens if we increase the number of mixtures or the expressivity of the distribution without a neural network?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swRmGLTI0s", "reviewer": "Reviewer_ejdi", "summary": "This work studies the domain randomization (DR) in reinforcement learning and the focus is on the design of task distribution. With the help of normalizing flows, the author proposes an entropy-regularized policy optimization methods for DR. The experiments are conducted in sim simulated and one real-world robotic domain.\n\n---\n\nI've updated the review based on the rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and practical recommendations regarding our claims and experimental metrics, which have significantly strengthened our manuscript. Below we address each of your comments and questions.\n\n> … The claim does not hold all the time, for example, in curriculum learning, we tend to pay more time on solving unsolvable regions of the domain in the later learning process. The reference should be attached in the statement or writing should restrict the scope.\n\nWe agree that this claim was imprecise, and there are other methods that train adversarially on difficult problems. While this would not be a good strategy in our class of problems due to the infeasibility of some parts of the parameter space. That statement has been updated to say \"An alternative strategy is to learn an environment distribution during training with the aim of finding the broadest possible training distribution that can feasibly be solved in order to maximize the chances of transferring to an unknown target environment.\" Additionally, we added a section to related work discussing adversarial training. \n\n> There miss specific metrics or performance indicators about the “outperform” term.\n\nWe have modified the statement to a more detailed claim about coverage.\n\n> Too broad a sampling distribution and the training focuses on unsolvable environments and falls into local minima. On page3. What is the theoretical support or evidence support for this claim? My understanding about the broad sampling distribution is to cover scenarios as many as possible to achieve DR purpose.\n\nThe support for this claim comes from our experimental results and from the results of many other papers on learning for domain randomization. Figure 3 shows that full domain randomization (the broadest possible distribution) is insufficient for reaching the target success threshold. For example, in the “ant” domain where the goal is for the ant to run forward, full domain randomization learns a uniformly applied strategy of bracing the weight of the body so that the negative reward is not received from floor contact. The result is that none of the ants (even the lighter ones) can be considered to have a successful policy because none are running forward.\n\n> However, there are some related works [2-4] that deserve discussion or comparison in experiments.\n\nWe agree that these are relevant papers, and we have updated our related work to include them.\n\n> Throughout the manuscript, I find the coverage ratio works as the primary metric in terms of learning curves and table results. However, its concept is not well introduced at the beginning of evaluation. My understanding about the coverage is related to the reward threshold. Hence, some general metrics can be involved in strengthen this work.\n\nApologies for any confusion, we moved our discussion of the coverage metric to earlier in the evaluation section (section 5.2) and added detail to how it is calculated in our experiments.\n\nRegarding the other suggested metrics\n- Our coverage metric is the same as the success rate metric from Tiboni et al. More specifically, it is the proportion of environments from the entire parameter range that the policy is expected to succeed under. We prefer the name “coverage” because this quantity does not reflect the actual success rate on a robot.\n- We agree that this is a good metric to report. In our updated manuscript, we have added a new table with CVaR scores.  CVaR was computed as the mean of the final rewards falling below the 10\\% percentile (VaR).", "claims_and_evidence": "It is worth noting the line number disappears in the complied pdf, which makes it difficult to point out the content. After going through the manuscript, I find some claims can be further polished and examples are attached as:\n\n(1) **An ideal sampling distribution enables the policy to focus training on areas of the distribution that can feasibly be solved in order to maximize the overall success rate of the\ntrained policy while not wasting time on unsolvable regions of the domain.** on Page1. The claim does not hold all the time, for example, in curriculum learning, we tend to pay more time on solving unsolvable regions of the domain in the later learning process. The reference should be attached in the statement or writing should restrict the scope.\n\n(2) **We show that GoFlow outperforms fixed and other learning-based solutions to domain randomization on a suite of simulated environments.** on Page2. There miss specific metrics or performance indicators about the “outperform” term.\n\n(3) **Too broad a sampling distribution and the training focuses on unsolvable environments and falls into local minima.** On page3. What is the theoretical support or evidence support for this claim? My understanding about the broad sampling distribution is to cover scenarios as many as possible to achieve DR purpose.\n\nSimilar to the above cases, there are other claims, which can be polished in the future version.", "methods_and_evaluation_criteria": "**I. Method:** Learning the task distribution for decision-making is a promising research direction and robust reinforcement learning is also a crucial consideration to bridge Sim2Real gap. The method of this work is the combination of the normalizing flow and DORAEMON [1]. \n\nHowever, there are some related works [2-4] that deserve discussion or comparison in experiments. For example, [2] considers the distribution shifts from a parameterized task distribution to increase RL robustness. [3] also uses normalizing flows to parameterize MDP distributions and learn and avoids the handcrafted task distribution design for robustness improvement. Though [2-3] are into meta RL, the optimization objectives also apply to DR cases. For [4], it considers the active domain randomization and also relates to robustness.\n\n \n**II. Evaluation:** Throughout the manuscript, I find the coverage ratio works as the primary metric in terms of learning curves and table results. However, its concept is not well introduced at the beginning of evaluation. My understanding about the coverage is related to the reward threshold. Hence, some general metrics can be involved in strengthen this work.\n\n(1) In terms of **generalization**, I recommend this work follows [1] and includes the learning curves about the success rate.\n\n(2) In terms of **robustness**, I suggest some conditional value-at-risk (CVaR) of success rates for evaluation and OOD tasks performance.\n\nBesides, I appreciate the author’s effort in real-world scenarios and this is plus of this work.\n\n\n**Reference**\n\n[1] Tiboni, Gabriele, et al. \"Domain randomization via entropy maximization.\" arXiv preprint arXiv:2311.01885 (2023).\n\n[2] Ajay, Anurag, et al. \"Distributionally adaptive meta reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 25856-25869.\n\n[3] Wang, Cheems, et al. \"Robust fast adaptation from adversarially explicit task distribution generation.\" arXiv preprint arXiv:2407.19523 (2024).\n\n[4] Mehta, Bhairav, et al. \"Active domain randomization.\" Conference on Robot Learning. PMLR, 2020.", "theoretical_claims": "Not applicable. There is no theoretical claim in this work.", "experimental_designs_or_analyses": "Overall, this work considers several scenarios on experimental design, however, more indicators and comparisons will strengthen this work.", "relation_to_broader_scientific_literature": "Not applicable.", "essential_references_not_discussed": "[1] Tiboni, Gabriele, et al. \"Domain randomization via entropy maximization.\" arXiv preprint arXiv:2311.01885 (2023).\n\n[2] Ajay, Anurag, et al. \"Distributionally adaptive meta reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 25856-25869.\n\n[3] Wang, Cheems, et al. \"Robust fast adaptation from adversarially explicit task distribution generation.\" arXiv preprint arXiv:2407.19523 (2024).\n\n[4] Mehta, Bhairav, et al. \"Active domain randomization.\" Conference on Robot Learning. PMLR, 2020.", "other_strengths_and_weaknesses": "See the above", "other_comments_or_suggestions": "It would be great if the above suggestions can be incorporated in revision. Meanwhile, the method part needs to exclude some literature work and focuses more on the contributed points. After revision, this will be a strong paper.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces GoFlow, a domain randomization in reinforcement learning that utilizes normalizing flows to learn the sampling distribution of environment parameters aiming to improve the robustness of learned policies by automatically discovering an effective sampling distribution through entropy-regularized reward maximization. The effectiveness of GoFlow is demonstrated on simulated tasks and real-world manipulation planning context.

Reviewers generally found the idea of using normalizing flows for domain randomization promising and the experimental results on the presented benchmarks convincing. The application to a real-world task and belief space planning was also seen as a positive contribution. However, reviewers raised concerns regarding missing experimental details, the contrived nature of the toy problem, the justification for certain design choices (like spline flows), and the limited scope of comparison with related works. Some reviewers also pointed out unclear writing and the need for more detailed explanations of metrics.

In response, the authors clarified experimental setups, hyperparameter selection, and the motivation behind their approach. They agreed to include missing related works and improve the clarity of the manuscript, including figure captions and explanations of the coverage metric. While reviewers acknowledged the rebuttals and some updated their scores, some minor reservations persisted regarding the novelty.

The potential of contribution of automatically learning domain randomization distributions using normalizing flows is exciting, and the results, including real world robot manipulation, are positive. The remaining concerns are relatively minor.
Decision: Accept (poster)