Neural Posterior Domain Randomization
Authors: Fabio Muratore
Theo Gruner
Florian Wiese
Boris Belousov
Michael Gienger
Jan Peters
Venue: CoRL2021 Poster (2021)
Keywords: sim-to-real
domain randomization
likelihood-free inference

Abstract: Combining domain randomization and reinforcement learning is a widely used approach to obtain control policies that can bridge the gap between simulation and reality. However, existing methods make limiting assumptions on the form of the domain parameter distribution which prevents them from utilizing the full power of domain randomization. Typically, a restricted family of probability distributions (e.g., normal or uniform) is chosen a priori for every parameter. Furthermore, straightforward approaches based on deep learning require differentiable simulators, which are either not available or can only simulate a limited class of systems. Such rigid assumptions diminish the applicability of domain randomization in robotics. Building upon recently proposed neural likelihood-free inference methods, we introduce Neural Posterior Domain Randomization (NPDR), an algorithm that alternates between learning a policy from a randomized simulator and adapting the posterior distribution over the simulator’s parameters in a Bayesian fashion. Our approach only requires a parameterized simulator, coarse prior ranges, a policy (optionally with optimization routine), and a small set of real-world observations. Most importantly, the domain parameter distribution is not restricted to a specific family, parameters can be correlated, and the simulator does not have to be differentiable. We show that the presented method is able to efficiently adapt the posterior over the domain parameters to closer match the observed dynamics. Moreover, we demonstrate that NPDR can learn transferable policies using fewer real-world rollouts than comparable algorithms.

Neural Posterior Domain Randomization
Fabio Muratore Theo Gruner', Florian Wiese', Boris Belousov", Michael Gienger', Jan Peters
Intelligent Autonomous Systems Group, Technical University Darmstadt, Germany 2 Honda Research Institute Europe, Offenbach am Main, Germany Correspondence to www.oooroooo-eaaannnnn.
Combining domain randomization and reinforcement learning is a widely used approach to obtain control policies that can bridge the gap between simulation and reality. However, existing methods make limiting assumptions on the form of the domain parameter distribution which prevents them from utilizing the full power of domain randomization. Typically, restricted family of probabil- ity distributions (e.g., normal uniform) is chosen a priori for every parameter. Furthermore, straightforward approaches based on deep learning require differ- entiable simulators, which are either not available or can only simulate a limited class of systems. Such rigid assumptions diminish the applicability of domain ran- domization in robotics. Building upon recently proposed neural likelihood free in- ference methods, we introduce Neural Posterior Domain Randomization (NPDR), algorithm that alternates between learning a policy from randomized simu- lator and adapting the posterior distribution over the simulator's parameters in a Bayesian fashion. Our approach only requires a parameterized simulator, coarse prior ranges, policy (optionally with optimization routine), and small set of real-world observations. Most importantly, the domain parameter distribution is not restricted to a specific family, parameters can correlated, and the simulator does not have to be differentiable. We show that the presented method is able efficiently adapt the posterior the domain parameters to closer match the observed dynamics. Moreover, demonstrate that NPDR can learn transferable policies using fewer real-world rollouts than comparable algorithms.
Abstract:
Keywords: sim-to-real, domain randomization, likelihood- free inference
Introduction
Learning control policies on physical robot is time- and resource-intensive. Crucially, Reinforce- ment Learning (RL) relies on random exploration, which in most cases can not be executed di- rectly on the device. Training in simulation promises to alleviate these problems by generating vast amounts of diverse data faster and cheaper. However, all simulators are only models of reality and therefore guaranteed be flawed. Thus, using data from a single simulation instance is often sufficient to learn control policy which transfers to the real-world counterpart, and might even lead to dangerous overfitting since the optimization process is optimistically biased [1].
The sim-to-real robot learning community suggested numerous approaches to bridge the 'reality gap' within the last few years. Motivated by superior results and flexibility, there is a clear trend towards methods that automatically tune a randomized physics simulator's parameter distribution to closer match the reality [2, 4, 5, 6, 7], also called guided domain randomization. Such approaches generally involve metric to quantify how well the simulated data is matching the observed data, combined with a mechanism to update the domain parameters or their distribution. Likelihood- Free Inference (LFI) methods measure the closeness between two observations by comparing their (logarithmicc probabilities, and provide various ways to update the density estimator which is either connected the posterior [8, 9, 10] or the likelihood [11, 12, 13].
Sth Conference Robot Learning (CoRL. 2021), London, UK.

---

Contributions We contribute to the state-of-the- in robot learning by proposing Neural Posterior Do- main Randomization (NPDR), an algorithm which in- tertwines LFI, domain randomization, and RL to in- fer distribution over simulators which is subsequently used to train transferable policies. Most notably, NPDR does not make any assumptions the simulator apart from requiring the ability to sample from it, works with coarse priors, and can be configured to strictly yield physically plausible domain parameters. Our approach allows seamlessly integrate any (parameterizable) physics engine, hence benefits from the rapid progress in the simulation community. Moreover, we synchronize the simulator segmentwise with the tra- jectories recorded in the target domain. This technique facilitates domain parameter inference for highly dynamical systems. We evaluate NPDR in one sim-to-sim and two sim-to-real experiments, and compare it with (online) BayesSim [14, 15] as well as Bayesian linear regression [16]. Our ex- periments demonstrate the generality of the proposed approach and show the advantages of flexible posterior inference in challenging continuous control problems. Additionally, we release the source code of our implementations a open-source library [17].
Figure 1: Evaluation tasks: robot mini golf and underactuated swing-up
2 Background and Notation
Inferring the dynamics of an environment while solving a task is at the core of model-based RL [18]. We describe this problem using the framework of Markov Decision Processes (MDPs) which the domain parameters are modeled random variables with unknown distribution. This framework is described in Section 2.1. To infer the distribution over the domain parameters without making any additional assumptions on the structure of the model of the MDP, we rely on the LFI methods described in Section 2.2, which only require the ability to sample trajectories from the model.
2.1 Markoy Decision Processes with Randomized Dynamies
Consider discrete time dynamical system
8 P Pe(
81, at),
(8( ~ (T(((())) ((E))))
with the continuous state 8, E Se S R and continuous action a, E Ag S Rd at time step t. The environment, also called domain, characterized by its parameters & R" (e.g., masses, friction coefficients, or time delays) which are assumed an unknown probability distribution p: IR= + g+ The domain parameters determine the transi- tion probability density function Pş: Sş A x Sş R$ that describes the system's stochastic dynamics. The initial state so is drawn from the start state distribution jlg: Se R+. Together with the reward function re: Se A + R. and the temporal discount factor 7 E [o, 1). the system forms an MDP described by the tuple Me = (Se, Ae, Pe: Me, Te, 7
random variables distributed according to
The goal of an RL agent is to maximize the expected (discounted) return, numeric scoring function which measures the policy's performance. The expected discounted return of policy To (ar (x)
with the parameters 0 E θ C R is defined as J(0. g) = E, Ey-)(r) Σ n'rg(s:, a.) 10
While learning from experience, the agent adapts its policy parameters. The resulting acttonaction- reward tuples are collected in trajectories, a.k.a. rollouts, re (e(s), at). When augmenting the RI. setting with domain randomization, the goal becomes to maximize the expected (discounted) return for distribution of domain parameters
{&,, a, r()= T with r.
Σ T-1 1=
J(0)= = [E) E)] = Eg-p(() E ET-)(T)
"rre a:) 0.
The outer expectation with respect the domain parameter distribution p(E) is the key difference compared to the standard MDP formulation. It enables the learning of robust policies that work for whole set of environments instead of overfitting to a particular problem instance.

---

2.2 Sequential Neural Posterior Estimation (SNPE)
A simulator is generative model g E - X' mapping set of domain parameters & E E to observations x E X. Generating data from the simulator can be interpreted as sampling from an intractable likelihood x ~ p(æ((). Given a prior belief p(E) over the domain parameters and an observation 2d LFI provides a way to learn an approximation Po(E(z) of the underlying true domain parameter distribution pE| ==th)),, where & Φ are the parameters of the density esti- mator, c.g., neural network or mixture model. When operating on time series data, LFI requires an embedding to construct features of constant size. This function f : T - X' computes features, also called observations, from the rollouts т E T. may depend on parameters vb E V which state-of-the-ar implementations optimize jointly with the parameters & of the density estimator.
Neural Posterior Domain Randomization (NPDR), as introduced in Section 3, is agnostic to the inference subroutine. In this paper, a neural LFI algorithm SNPE-C [10] is used for its superior per- formance compared to the alternative approaches described in Section 5.3. SNPE-C approximates the true posterior using normalizing flows, generative model which produces tractable distribu- tions where both sampling and density evaluation can efficient and exact [19]. To overcome the intractability of the likelihood, the posterior trained minimize the following objective function
R
N
p(Er.)) Z(x, Φ) plEr.)) 2 (Er.ul Ir,n ~ p(z) Er.n,, Er.n ~ 5(()
[()) = Σ loog
where R is the number of inference rounds N is the number of samples per round, and Z is the normalization constant. Note that the domain parameters are sampled from a proposal prior dis- tribution p(E), which is initialized with the prior p(E), and requires importance reweighting of the posterior except for the first round. Sampling from the proposal prior has the advantage that the do- main parameters could be sampled in more narrow regions where the actual support of the domain parameter distribution lies. In multi-round setup, the proposal prior P(E) is set to be the posterior of the previous round conditioned on the previously observed data P (E|æ x²"))
3 Neural Posterior Domain Randomization (NPDR)
The goal of NPDR is to learn a control policy in simulation such that transfers to the real device, making as few assumptions as possible about the properties of the simulator, e.g., not assuming dif- ferentiability or use of rigid-body dynamics. To achieve this, NPDR augments the nominal RL task with domain randomization and leverages LFI methods powered by normalizing flows to approxi- mate the posterior over the domain parameters. The complete procedure is described in Algorithm 1.
NPDR does not impose any restrictions on the inference subroutine (Line 11) or on the policy optimization subroutine (Lines 2 and 13). key feature of NPDR the ability to use the current belief over the simulation parameters at every iteration. This belief inferred such that it explains the data observed from the real system best. Furthermore, the algorithm that it is possible to jointly condition on multiple real- world rollouts (Lines 6 and 10). To execute the inference (Line 11), we integrated the shi toolbox [20], which enables us to easily switch between LFI methods. The shi package expects all observations used to condition the posterior to have the same size. Satisfying this constrain for arbitrary time series embeddings, requires the rollouts to be of equal length. Thus, if a policy causes an emergency stop, e.g., by exceeding the state boundaries, we pad the remainder of the rollout with zeros. During our experiments, we did not observe any negative effect of this measure on the inference routine.
implemented in way
Complementarily, the rollouts are simulated segmentwise, i.c., the states are synchronized with the real trajectory every K time steps. This repeated synchronization is especially beneficial for systems with a fast dynamics like the Furuta pendulum (Figure 1), since the trajectories diverge quickly even for well-fit parameters. Finally, the policy has an internal model, e.g., true for MPC or the energy- based control law used in Section 4.3, NPDR allows to set the parameters of the internal model to the most likely sample from the current posterior (Line 13). Updating the controller's model was also done in [15], however not in combination with updating the policy parameters.
3

---

Algorithm 1: Neural Posterior Domain Randomization (NPDR)
input :
initial policy "o "o(ass) generative model (E)) physical device gmm., prior p(E), initial density estimator Ug (É(a), summary statistics (T), LFI subroutine INFER, policy optimization subroutine POLOPT,
output: approximate (conditional) posterior over the domain parameters P& (E|æ x"ta)).
policy zg- (als) trained using the approximate (conditional) posterior
Initialize the parameters 0, Φ. ab randomly
Train initial policy 0* < POL POLOPT(O,, p(E) with samples from the prior p(E)
3 Initialize the approximate posterior with the prior P
(E(æ) + P(E)
for each iteration 1 do
Sim-to-real loop
Execute H target domain rollouts rcal +- (paaa(()))
Compute the observations from the rollouts using learned summ. stats. x +- fw (T for each round r = 1: R do
multi-rount LFI, first round is amortized
Sample N domain parameters ~~ & x
Sample N associated rollouts - < gbim (0*,
10
Compute the observations from the rollouts using learned summ. stats. æ
f (Trim)
Optimize the density estimator and embedding Φ², , <- INFER cal x: Φ u/5
11
12
Obtain the cond. posterior from the density estimator P (glæ = (fl==x=maa =aaa)) + 96 (E) 10100=00 xx
Train the policy using the latest conditional posterior 0 + POLOPT 0 Po (E x - relll)]
13
4
Experiments and Evaluations
We conduct three experiments to analyze the properties of the proposed NPDR algorithm. First, the conrecnnss of the algorithm is evaluated on a simulated environment for which ground truth solution is available. Namely, a sim-to-sim experiment on the pendulum is conducted to allow for visual inspection of the complete posterior distribution over the domain parameters (Section 4.1). Next, we conduct an experiment in contact-rich mini golf environment, inferring a 10-dimensional posterior using a state-of-the-ar physics engine (Section 4.2). Finally, we integrate policy optimiza- tion and solve an underactuated swing-up and balancing task on the Furuta pendulum (Section 4.3). The configurations of all experiments provided in the appendix.
Baselines Depending the experiment, chose BayesSim [14] or its online variant [15] as baseline, because they are the only other approaches that work on the same assumptions and yield flexible estimator of the domain parameter distribution as well as a trained policy (see Section 5. for a detailed description of BayesSim). To obtain comparable results, was necessary to adapt online BayesSim such that the policy optimization done until convergence (lines 13-14 of Algorithm in [15)), i.e., we made this part identical to NPDR. For the Furuta pendulum experiment (Section 4.3), additionally validated the system identification module against Bayesian linear regression.
Metrics In order to assess the quality of the domain parameter posteriors found by the LFI sub- routines in NPDR and BayesSim, we compare the state trajectories generated by domain instances sampled from the approximate posterior against the ground truth data that was recorded on the phys- ical system. Before computing the performance metrics, all trajectories are normalized by the state bounds to equalize the importance of the different state dimensions. We decided to use two metrics: the Root-Mean-Squeee -quare Error (RMSE) and the Dynamic Time Warping (DTW) similarity computed by the python-dtw package [21]. Both metrics are desired to be small. DTW a dynamic program- ming algorithm that calculates an optimal match between two temporal sequences [22]. Different DTW methods vary in the cost function and in the assignment of points to each other. We configured DTW to be symmetric, with the beginning and end of the assignment being chosen by the optimizer.
Code The implementations of NPDR well as the baselines are building the SimuRLacra framework [17], the sbi toolbox [20], and on the scikit-leam package [23], all open source.

---

length 1
1.5
length 1a
1.5
1.0
1.0
0.5
0.5
0.5 1.0 1.5
0.5 1.0 1.5
0.5 1.0 1.5
0.5 1.0 1.5
0.5 1.0 1.5
0.5 1.0 1.5 mass mp
mass mp
mass mp
mass 7pp
mass 7p
mass m
(a) rounds of BayesSim [14]
(b) rounds of NPDR
Figure 2: Learned (full) posterior distributions the two domain parameters of the pendulum. The LFI rounds are chronologically sorted from left to right. Warmer colors represent higher prob- abilities. Both algorithms managed to find the manifold of domain parameter configurations which yields observations highly similar ones of the ground truth environment (white circle). Comparing the sub-figures, we a MoG (a) to fit the domain parameter posterior.
benefits of using SNPE-C with MAF (b) instead of SNPE-A with
4.1 Sim-to-Sim Validation with Fully Observable Posterior
The first experiment is set up such that there is manifold of domain parameters = C E producing almost identical trajectories ا' which are highly similar to the nominal ones. Thus, the observations ad = fb (7"d) are close to indistinguishable independent of the summary statistic f. The rationale that the mass mp and the length I, always appear coupled in the pendulum's Equations of Mo- tion (EoMs), and are dominated by the term mylp Thus, we only randomize these two domain parameters, then configurations where the product my?? has value similar to the one of the ground truth environment should be inferred more likely than others. Figure 2 confirms this hypothesis empirically, by clearly capturing the manifold Eid. To generate the rollouts, the pendulum has been excited by applying feed-forward sinusoidal action signal. Both algorithms used the summary statistic function proposed in [14] and the same prior.
4.2 Rim-to-Real Bayesian System Identification in Contact-Rich Environment
One of the main benefits that NPDR and BayesSim inherit from their LFI subroutines is the ability to incorporate of physics engine which expose their parameters. To showcase this advantage, we chose mini golf (Figure and 3) since it requires correct contact modeling to align simulation and reality. We employed the Bullet physics engine randomize the ball's radius rb, mass my, restitution coefficient eb, rolling friction coefficient μb, the two rails' Cartesian offsets Ar, Ar2, Ay,, A12, well their angular offsets around the vertical axis Ay1, Arg. The goal location was a circular piece of double-sided tape, modeled as material with very high rolling
Figure 3: Robot mini golf friction, that stops the ball quickly. To hit the ball reliably, the robot was driven by a hand-tuned, time-dependent policy. For the inference, we used the ball positions, recorded by a Vicon system, and the robot's joint angles.
Figure 4 displays slice through the 10-dimensional domain parameter posteriors learned with NPDR and BayesSim on the same data. As expected, the mass of an ideally rolling object has no influence on the dynamics, thus the marginal of m, should be uniform. The prior for the an- gular offsets were chosen informatively, i.e., the object can be rotated by tT at the beginning of the simulation. Therefore, the posterior should have two modes for %. Both algorithms were able recognize this multi-modality well the coupling between e, and rb. However, the correla- tion between Ar; and Ay; only inferred by NPDR. A quantitative assessment of the modeling accuracy is given in Table 1, which reports the best experiment for every method.
4.3 Ria-to-Real Transfer with Policy Optimization in the Loop
The final experiment is aimed at evaluating NPDR on the challenging continuous control task of swinging up and stabilizing a rotary inverted pendulum (Figure 1). This system, known the Furuta pendulum [24], is particularly hard to control due its underactuated nature and fast oscillatory
5

---

Figure 4: A 7-dimensional slice from the 10-dimensional posteri- ors learned with NPDR (blue) and BayesSim (orange) in the mini golf experiment. The nominal values (green), were either deter- mined by prior measurements or by coarse estimates. Every do- main off-diagonal plot shows the same 200 samples for 2 dimen- sions, whereas the diagonal plots show the marginal distributions. The omitted domain parameters of the first obstacle Ar,, Ay,, 71 are similar Ar2, Ayy. 72. As de- scribed in Section 4.2, the mini golf experiment is set up such that mul- tiple parameter configurations can result the same trajectory. This arises naturally from broad non- minimal choices. Details on the domain parameters ranges can be found in the appendix.
7
770b
6b
Ar2
1/2
Tb
m7
Cb
146
Ar2
Ay2
An2
movements [25]. To capture wide range of dynamics, we chose to randomize the link masses mys Mps the link lengths, l,, lps the motor constant and resistance km Rms the gravity constant g. as well as the viscous friction parameters dp, d, of both joints. As derived the appendix, it is not possible to linearly separate all of these domain parameters and obtain unique solution. Therefore, excluded g. km. and Rm for the Bayesian linear regression, which makes the results baseline comparable to the others. The policy is given parameterized hybrid controller consisting of two components: an energy- mnergy-hhaping swing-up controller that brings the pendulum close to the upright position and a stabilizing PD controller that switches on in the vicinity of the upright position [26]. We used Policy learning by Weighting Exploration with Returns (PoWER) [27] to optimize the policy parameters for all experiments reported Table 2. During policy execution on the physical device, the parameters of the energy-shaping controller's 's internal model were set to the most likely domain parameter set, sampled from the posterior during training.
The results show that our LFI-based domain randomization approach outperforms Bayesian linear regression and the nominal baseline, both in terms of the real-world return and the RMSE error. We believe that the advantage NPDR over BayesSim is due to its superior neural LFI subroutine as well as the fact that our method, in contrast online BayesSim, trains the policy until convergence before executing the next target domain rollout. This hypothesis is supported by the better metries for the trajectory differences (Table 2). See Section 5.1 for more in-depth discussion of the differences between the two algorithms. Comparing the trajectories resulting from the nominal simulation with the ones resulting from the learned distribution (Figure 5), we observe that all approaches improved over the nominal simulator. Moreover, NPDR and BayesSim learned a predictive distribution that includes the ground truth trajectory, except for the segment between 3 and 4s. Since all simulator configurations fail reproduce the recorded trajectories, we conclude that the modeled EoMs are missing effects which are decisive of the last part of the swing-up.
Table 1: Performances of the Bayesian system identification on the real mini golf environment. The metrics quantify how well each approach fits a common ground truth data set.
Metric
NPDR
BayesSim
Nominal
DTW dist.
[3.30 0.16]e+1
[9.21 + 0.64]e+1
1.51e+2
RMSE
[7.19 + 0.65]e-3
[2.19 = 0.51ee-2
8.60e-2

---

1
6
a [rad]
0 [rad]
0
0
0
2
3
2
4
(a) NPDR (blue), nominal (green dashed), ground truth (black)
6
1
a [rad]
B [rad]
0
0
-1
0
2
3
2
4
(b) BayesSim (orange), nominal (green dashed), ground truth (black)
2
6
[rad]
[rad]
0
0
-2
0
2
3
4
2
4
time [s]
time [s]
(c) Bayesian linear regression (purple), nominal (green dashed), ground truth (black)
Figure 5: Distributions over trajectories generated from 200 domain parameter sets sampled by the learned posteriors. The simulations were synchronized with the ground truth (black) data every 200 time steps, hence show the feed-forward predictions 200 time steps. The nominal system (dashed green) uses singe domain parameter set, taken from the manufacturer's data sheet.
Related Work
The topic of sim-to-real transfer in robot learning has grown in prominence in the last few years. For broad overview of the existing approaches, we refer the reader to the survey paper [28]. In the following, we focus the methods directly related to the proposed NPDR algorithm.
5.1 Adapting the Source Domain Distribution
The domain parameter distribution can updated in multiple ways [29, 3, 5, 6, 30). Most ap- proaches assume independent domain parameters, i.c., they cannot represent the correlations be- tween them. In contrast, NPDR, BayesSim [14], and DISCO [31] model the whole posterior, and thus capture the correlations between the parameters. As observed in the experiments in Section 4, these kinds of correlations are abundant in models of robotic systems. BayesSim [14], especially its online variant [15], share the concepts such as the combination of LFI and domain randomization with NPDR. However, the approaches differ in three important ways: (i) NPDR approximates the posterior using a normalizing flow instead of a mixture model, which coupled with using SNPE-C instead of SNPE-A inference algorithm. The results in Section highlight the importance of the
Table 2: Performances of the Bayesian system identification and of the final policies on the real Furuta pendulum. The first two metrics quantify how well each approach fits a common ground truth data set. To quantify the return of methods, we evaluated 5 (final) policies, generating 10 rollouts per policy. A policy can be considered successful if its return exceeds approximately 1400.
Metric
NPDR
BayesSim
Bayes. Lin. Reg.
Nominal
DTW dist.
[1.07 + 0.03]e+3
[1.24 + 0.06]e+3
[2.24 + 0.04]e+3
[1.43 + 0.03 Je33
RMSE
[2.63 + 0.04]e-1
(295 + 0.02]e-1
[4.46 0.07]e-1
[3.84 0.03]e-1
Return sim
[9.34 9.35)e+2
[1.31 131+330 3.57]e+2
[2.05 + 0.30]e+3
[2.13 + 0.05]e+3
Return real
[2.02 + 0.18]e+3
[1.11 + 0.76]e+3
[3.28 + 1.44]e+2
[1.10 + 0.33]e+2

---

LFI subroutine and clearly favour the neural density estimator. This finding is consistent with [32]. (ii) The policy optimization and domain parameter inference in NPDR are not alternated at every (time) step, but trained until convergence, which stabilizes the training. (iii) We demonstrated that NPDR is able to not only update the policy's internal model, as done with MPC in [15], but also to train the policy parameters directly based on the latest posterior.
5.2 Bayesian System Identification for Robotics
A crucial step in NPDR is the approximate inference of the domain parameter distribution. While NPDR makes no assumptions
the simulator properties, other methods provide more domain- specific solutions by assuming more structure in the model. For example, identification of rigid- -body dynamics can be framed as (Bayesian) linear regression problem (33,34) However, the identified parameters commonly turn out to be physically implausible [16]. Nonlinear system identification of rigid-bods dynamics with neural networks [35, 36] allows for greater flexibility and physical plausibility, but it is not straightforward to obtain confidence estimates based on these methods, For more general dynamical systems, approaches utilizing the classification loss between simulated and real samples have been considered [37, 30). If the simulator model is non-differentiable, episodic RL has been proposed optimize the posterior using trajectory similarity the reward signal [3]. Our NPDR method is the closest to the latter family of algorithms, but instead of episodic RL, it utilizes neural LFI methods, which enable the use of more expressive posteriors, such as normalizing flows, and are more sample efficient, requiring only a few rollouts to be trained.
5.3 Likelihood-Free Inference
LFI is a collective term for Bayesian inference methods which are able to learn a posterior distri- bution in scenarios where the likelihood either not available or too expensive to evaluate. Ap- proximate Bayesian Computation (ABC) applies Monte-Carke sampling to infer the parameters by comparing summary statistics of synthetically generated and observed data [38]. Although MCMC- ABC [39] and SMC-ABC [40] enhance the sample efficiency of the classical rejection-ABC by employing more sophisticated sampling schemes, the ABC approach does not scale well because of the inherent problems of sampling in high- -igeeniimnsion spaces. Amortized methods have been more successful in scaling IFI to more challenging problems by utilizing flexible neural density models [.1]. Such models can be used both to generate samples from the posterior and to evaluate the likelihood of parameter configurations. SNPE approaches [8, 9. 10] approximate the conditional posterior, allowing for direct sampling from the posterior, foregoing MCMC. Learning the likeli- hood [11] can be useful in the context for hypothesis testing. Alternatively, posterior samples be generated from likelihood-ratios [12] which can be trained using contrastive learning [13]. A tabu- lar summary of state-of-the-ard I.I approaches can be found in the appendix. For comprehensive survey on LFI from simulations, see [42].
6 Conclusion
In this paper, introduced and empirically studied new adaptive domain randomization algorithm Neural Posterior Domain Randomization (NPDR) that employs normalizing flows for representing the posterior distribution over the domain parameters. Our approach makes no assumptions on the simulator except the ability to sample from it, and it is guaranteed to identify physically plausible parameters. NPDR interleaves approximate posterior inference with policy optimization, leading to an iterative refining of the posterior distribution from which more transferable policies are trained. Our results showed that NPDR improves the flexibility and precision of the existing methods and re- quires only a few real-world rollouts train robust policies in a randomized simulation environment. We believe that the proposed method, powered by likelihood -free inference with normalizing flows as density estimators, is a valuable addition to the toolbox of domain randomization algorithms, enabling rapid domain adaptation and training of effective control policies.
Future work will investigate alternative likelihood-free inference approaches, such as sequential neu- ral ratio estimation. Additionally, we plan to apply NPDR to problems involving more challenging dynamical systems such as soft body simulation and fluid dynamics. It is furthermore of interest to consider other modalities of domain parameters, such parameters of a vision system.
8

---

Acknowledgments
Fabio Muratore gratefully acknowledges the financial support from Honda Research Institute Eu- rope. Boris Belousov and Jan Peters have received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 640554. Calculations for this research were conducted on the Lichtenberg high performance computer of the TU Darmstadt.
References
[1] Muratore, M. Gienger, and Peters. Assessing transferability from simulation to reality for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:1172-1183, 11 2021. doi:10. 11111.1199/PPMMM22099999999 2992229.... URL https://///....... org/
1109/TPAMI 2011/ 2952353.
[2] N. Ruiz, S. Schulter, and M. Chandraker. Learning to simulate. ICLR, New Orleans, LA, USA, May 6-9. OpenReview.net, 2019, URL HJgkx2Aqt7.
[3] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. D. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In ICRA, Montreal, QC, Canada, May 20-24, pages 8973-8979, 2019. 10990...../...A.
2019.8793789 URL pestps://dots org/10....0o 1109/ICRA
[4] B. Mehta, A. Handa, D. Fox, and F. Ramos. user's guide calibrating robotics simulators. In CoRL, Virtual Event, November 16 18, Proceedings of Machine Learning Research. PMLR, 2020. URL https: htt.ttps::/ //axxiv. org/ abs/ /011.. 2001..0991........ 08985.
[5] X. B. Peng, E. Coumans, T. Zhang. T. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. RSS, Virtual Event Corvalis, Oregon, USA, July 12-16, 2020, do5:70..5 Tttp://S..000.XX00 2020.XVI.064. URL https::tp//// :ttp:://ddo... http:///ooo.oo/.....oo 15601 /RSS.
2020 XVI. 064.
[6] F. Muratore, C. Eilers, M. Gienger, and J. Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics Automation Letters, 6(2):911-918, 2021. doi: 109/ LRA.2021.3052391. URL https: / http://ii.oogggg/// wttg//...////.o/oooo 1109/LRA. 2021. 3052391.
[7] J. Truong, S. Chernova, and D. Batra. Bi-directional domain adaptation for sim2real transfer of embodied navigation agents. IEEE Robotics Automation Letters, (2):2634-26411 2021. 10::10.119/LRA.2000 000622333 URL https://doi. org/10. 1109/LRA 2021 2020003222000000 3062303.
[8] G. Papamakarios and Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. In NIPS, Barcelona, Spain, December 5-10, pages 1028-10366 2016. URL
[9] J. Lueckmann, P. J. Gonçalves, G. Bassetto, K. Ócal, M. Nonnenmacher, and J. H. Macke. Flexible statistical inference for mechanistic models of neural dynamics. In NIPS, Long Beach, CA, USA, December 4-9, pages 1289-1299, 2017. URL httpp::/ neurips.
Awstract.
[10] D. S. Greenberg, M. Nonnenmacher, and J. H. Macke. Automatic posterior transformation for likelihood-free inference. In ICML, Long Beach, California. USA, 9-15 June, volume 97 of Proceedings of Machine Learning Research, pages 2404-2414. PMLR, 2019. URL http: wmw..rr..........mmm.m.......mm. //proceedings.. mlr www.s//97/sppeeeeeeeeeeee gwwennerggee..e...... / htmm..
[11] G. Papamakarios, D. C. Sterratt, and Murray. Sequential neural likelihood: Fast likelihood- free inference with autoregressive flows. In AISTATS, Naha. Okinawa, Japan, April 16-18, volume 89 of Proceedings of Machine Learning Research, pages 837-848. PMLR, 2019, URI. http://proceedin
mlr. www.epreessveee /ttpe///////pppaaaaaaiee11..... html.
[12] Hermans, Begy, and G. Louppe. Likelihood-free MCMC with amortized approximate ratio estimators. In ICML, Virtual Event, 13-18 July, volume 119 of Proceedings of Ma- chine Learning Research, pages 4239-4248 PMLR, 2020. URL http://proceeding: mlr. www.ppeeviiiiii 1ttss//1111//hmmm22///0 www.eamm200..aaee.m.mmea html.

---

[13] C. Durkan, I. Murray, and G. Papamakarios. On contrastive learning for likelihood-free ffre infer- ence. In ICMI, Virtual Event, July 13-18, volume 119 of Proceedings of Machine Learning Research, pages 2771 PMLR, 2020. URL http:://rroceedings mlr, patss///ss//1/////// durkan20a.htar html.
[14] F. Ramos, R. Possas, and D. Fox. Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators. In RSS, Freiburg im Breisgau, Germany, June 22-26, 2019. 100.10.1567//SS..22 211.S..211..V.002. URL https: /tt////oo.... org/ 10. 15607/RSS, 2019 .V....VV...... .29.
[15] R. Possas, L. Barcelos, R. Oliveira, D. Fox, and F. Ramos. Online bayessim for combined simulator parameter inference and policy improvement. In IROS Las Vegas, NV, USA, October 24 Jamuary 24, pages 5445-5452. IEEE, 2020. doi: 100//R0054444..220995444 attps://dot org/10. 10. 1119/110000//// 2020. 2020.9341401.
URL
[16] J. Ting, A. D'Sza, and S. Schaal. Bayesian robot system identification with input and output noise. Neural Networks, 24(1):99-108, 2011. doi: 0.100166..eeeuue22000..0m
URL https:
//doi. wwtoo///...11111 neunet. 2010. 08..0.8.000 01101.
[17] F. Muratore. SimuRl framework for reinforcement learning from randomized simula- tions. https://githu/ wtt.c/o///aaa/ http://aaaaaiiimmuma
2020.
[18] T. M. Moerland, J. Broekens, and C. M. Jonker. Model-based reinforcement learning: A
survey. arXiv, 2006.16712, 2020. URL attpp://arxxiie http//axiv..orgbb/////aaa 16712.
[19]
Kobyzev, S. Prince, and M. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. doi: N00:100.100//PAAM1.00000000
[20] Á. Tejero-Cantero, J. Boelts, M. Deistler, J. Lueckmann, C. Durkan, P. J. Gonçalves, D. S. Greenberg, and H. Macke. sbi: A toolkit for simulation- inference. J. Open Source Softw., 5(52):2505, 2020. doi: 10.21 105/100s 02505.. URL attps://don htw.doi..oog////////oooo org/ 10/ 21105/ jos8.02505.
[21] T. Giorgino. Computing and visualizing dynamic time warping alignments in r: The dtw package. Journal of Statistical Software, 31(7):1-24 2009.
[22] H. Sakoe and S. Chiba. Dynamic programming algorithm optimization for spoken word recog- nition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1):43-49, 1978.
[23] L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Mueller, O. Grisel, V. Niculae, P. Pretten- hofer, A. Gramfort, J. Grobler, R. Layton, J. VanderPlas, A. Joly, B. Holt, and G. Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pages 108-122, 2013.
[24] K. Furuta, M. Yamakita, and Kobayashi. Swing-up control of inverted pendulum using pseudo-state feedback. Proc. Inst. Mech. Eng. Part 1, 206(4):26332)))) 1992.
[25] C. Eilers, J. Eschmann, R. Menzenbach, B. Belousov, F. Muratore, and Peters. Underactuated waypoint trajectory optimization for light painting photography. In ICRA Paris, France, May 100.////A4444420000.......
31 - August 31, pages 1505-1510. IEEE, 2020. https://doi. org/10. A0099IIA449945 2020. 9196516.
URL
[26] J. A. Acosta, J. Aracil, and F. Gordillo. Nonlinear control strategies for the furuta pendulum. Control and intelligent systems, 101 2001.
[27] J. Kober and J. Peters. Policy search for motor primitives in robotics. Machine Learning, 84(1-2):171-2033 2011. 100100000077/1009400000222333333 URL http://dxxdoi..rrg
1077/81094-010-5228-
[28] W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learn- ing for robotics: a survey. In SSCI Canberra, Australia, December 1-4, pages 737-744 IEEE, 2020.
11010,1100//SC1188822200000800 SSCI47803. 2020. 9308468.
URL attps://dde org/o0../// /ttg//0..1000 1109/
[29] J. Harrison, A. Garg, B. Ivanovic, Y. Zhu, S. Savarese, L. Fei-Fei, and M. Pavone. Adapt: Zero- shot adaptive policy transfer for stochastic dynamical systems. In ISRR, Puerto Varas, Chile, December 11-14, volume 10 of Springer Proceedings in Advanced Robotics, pages 437-453.
10

---

Springer, 2017.
db::10010077998---300---2800 34. URL ptttps:/ddoi.rr//// 11--3-00--22619-4-34
[30] Y. Du, O. Watkins, T. Darrell, Abbeel, and D. Pathak. Auto-tuned sim-to-real transfer. arXiv, 2104.07662, 2021. URL httpp/////aarii..g/abab/////////
07662.
[31] L.. Barcelos, R. Oliveira, R. Possas, L. Ott, and F. Ramos. DISCO: double likelihood-free
inference stochastic control. In ICRA, Paris, France, May 31 August 31, pages 10969-10975. IEEE, 2020. doi: I09/CCA449945.00000000 2020.919911. URL
1109/
ICRA40945. 2020. 9196931.
[32] J. Lueckmann, J. Boelts, D. S. Greenberg, P. J. Gonçalves, and J. H. Macke. Benchmarking simulation-based inference. In AISTATS, Viriual Event, April 13-15, volume 130, pages 343- html. [33] C. G. Atkeson, C. A. H, and M. Hollerbach. Estimation of inertial parameters of manipulator loads and links. The International Journal of Robotics Research, 501)1011111, 1986.
351. PMLR, 2021. URL http:/ uttttp:/pprocceeding wwtpeeediinngmm....eee.e halp:pprres/.113//uuccmmmm22
[34] Ting, M. N. Mistry, J. Peters, S. Schaal, and Nakanishi. A bayesian approach to nonlinear parameter identification for rigid body dynamies. In RSS, Philadelphia, Pennsylvania, USA, August 16-19. The MIT Press, 2006.
URL http://wwn
roboticaproceddings. wttpa//g...ggrra///2...... html.
[35] G. Sutanto, A. S. Wang, Y. Lin, M. Mukadam, G. S. Sukhatme, A. Rai, and F. Meier. Encoding physical constraints in differentiable newton-culer algorithm. In L4DC, Virtual Event, Berkeley, CA, USA, 11-12 June, volume 120 of Proceedings of Machine Learning Research, pages 804 103. PMLR, 2020. URL http://proceeddnng..ni pat..ppres//v1/// sutanto20a. html.
[36] M. Lutter, J. Silberbauer, J. Watson, and J. Peters. Differentiable physics models for real- world offiine basel-based reinforcement learning. arXiv, 2011.01734, 2021. URL https:
//arxiv..r wttprii..rrg/bb/2211111 01734.
[37] Y. Jiang, T. Zhang, D. Ho. Y. Bai, C. K. Liu, S. Levine, and J. Tan. Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. arXiv, 2101 10600, 2021. URL https: / /axxiv.... org a2100 .60055.
[38] M. Sunnåker, A. G. Busetto, Numminen, J. Corander, M. Foll, and Dessimoz. Ap- proximate bayesian computation. PLoS Comput. Biol., 9(1,, 2013. 103:10..37l/jouraalll 1002803. URL https: / /oi.. org/10. 1371/journal pcbi. pcb.. 1002803.
[39] Marjoram, J. Molitor, V. Plagnol, and S. Tavaré. Markov chain monte carlo without likeli- hoods. Proceedings of the National Academy of Sciences, 100266;:5324-15322, 2003. ISSN 0027-8424. doi:10. 26/15324.
URL https: / http://wwwwppaaas.. http:://www.pass.rrr//atttt///// org/content/100/
[40] M. A. Beaumont, J.-M. Cornuet, J.-M. Marin, and C. P. Robert. Adaptive approximate
Bayesian computation. Biometrika, 96(4):983-99) 990, 10 2009. ISSN 0006-3444. doi:
10. 10999///oomee/spp52.. URL https://dde org/10. 1093/biomet/asp022.
[41] G. Papamakarios, Murray, and T. Pavlakou. Masked autoregressive flow for density estimation.
In NIPS, Long Beach, CA, USA, December 4-9, pages 2338-2347, 2017. URL https: /ppec/////cceenigs www.nnu..n... httpe/ccppppee//1/////// 6c cc1a26662267222061770088888 Abstract.
[42] K. Cranmer, Brehmer, and G. Louppe. The frontier of simulation inference. Pro- ceedings of the National Academy of Sciences, 117(48):30055 30062, 2020. URL http:
30066,
//arxiv.oo arg/abs/191// 014299.


Reviews:
[{"review_id": "wUTkLZjG_o6", "reviewer": "Reviewer_wTxL", "summary": "This work proposes a new way to leverage domain randomization to estimate and adapt the posterior distribution over the simulator’s parameters, and consequently allowing optimization of a policy suitable for the target environment in the real world. Specifically, it combines likelihood-free inference methods based on normalizing flows with domain randomization to estimate the posterior. This makes fewer assumptions about the domain parameter distribution, and can model correlations between parameters. This approach is evaluated in a sim-to-real experiment on a swing-up pendulum task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- NPDR combines a strong LFI method with domain randomization to flexibly model the posterior distribution over parameters given a small set of observations. The experiments demonstrate the wide applicability of the method, and also include results on a real pendulum system.\n\n- The paper is overall well written and easy to read. The visualizations help illustrate and characterize the differences between NPDR and BayesSim. \n\nWeaknesses:\n- Perhaps, a weakness is that the method is a somewhat straightforward combination of a LFI method based on normalizing flows with domain randomization. However, this work highlights settings that necessitate a stronger neural density estimator.\n\n- It would be useful to include the sample mean and standard deviation of the results across multiple experiments in Table 1, in addition to reporting the max over experiments.\n\n- The studied pendulum control task is quite simple. The returns achieved by NPDR and BayesSim are thus quite close with overlapping intervals. It would be good to apply the method on a wider range of control tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending a weak accept. Overall, the paper clearly communicates the method and its strengths. It also experimentally evaluates several aspects such as its ability to capture correlations between parameters. However, the studied tasks and quantitative results are a bit limited.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- In terms of writing organization, having the related work earlier in the paper such as after the introduction would help a reader who has less familiarity with domain randomization better understand the contributions and differences of this work.\n\n- Describing what the Dynamic Time Warping (DTW) metric is in more detail would make the paper more self-contained.\n\n- As mentioned earlier, it would be useful to include the sample mean and standard deviation of the results across multiple experiments in Table 1, in addition to reporting the max over experiments.\n"}}, {"review_id": "wHogbLOO3uN", "reviewer": "Reviewer_KMoW", "summary": "This paper presents a method for training policies using Domain Randomization, where the randomization distribution is fit with some real world data. The algorithm, named Neural Posterior Domain Randomization (NPDR) in the paper, consists of using a likelihood-free inference method, Sequential Neural Posterior Estimation with Normalizing Flows (SNPE-C or APT) , to determine the distribution used to train policies in simulation with Domain Randomization. This is similar to BayesSim, which this paper uses as a baseline for comparison.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper demonstrates the power of likelihood-free inference methods for the tasks of parameter identification in robotics tasks. Furthermore, it shows how likelihood-free inference can be integrated naturally into a domain randomization pipeline for training robot controllers in simulations. This work also demonstrates the proposed Domain Randomization algorithm on a real physical system, the Furuta pendulum. The paper is clearly written and touches upon a very relevant topic for robotics research. The experimental results on the Furuta Pendulum task show an improvement over the online BayesSim baseline.\n\nThe strengths of this paper are in the simplicity of the proposed method and the experimentation with real robot systems.\n\nThe weaknesses of this paper are in the originality and impact of its contributions and in the experimental evaluation. \n\nIn terms of originality and impact, the proposed algorithm is not very different to BayesSim at a high level. In a detailed view, the proposed algorithm  can be viewed as the same as BayesSim, but replacing the posterior inference algorithm and density model (both of which aren't contributions from this paper). \n\nAn important shortcoming of the experimental results is that the differences in experimental results can be attributed to multiple factors (the ones listed in Section 5.2), but the paper does not provide isolated ablations or improvements to the baselines to determine the influence of the components of NPDR in the final performance. The comparison doesn't seem to account for model capacity. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the main technique used in this paper (likelihood free inference with normalizing flows) is of great interest to roboticists, the contribution is thin. I would still recommend this for acceptance given that it would be the first documented application of SNPE with normalizing flows to learning control policies. But I would condition my acceptance to a proper ablation study (to ensure that the method was compared to the best possible baseline)", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "The paper compares NPDR to online BayesSim, but not to vanilla BayesSim.\n\nThe paper attributes the difference in performance to 1) the choice of inference algorithm 2) the choice of density model 3) that NDPR trains the policy until convergence. These three factors can be removed from NPDR (or added to vanilla BayesSim) independently to determine their importance."}}, {"review_id": "q9MQz9RgVpK", "reviewer": "Reviewer_tnJY", "summary": "This paper use SNPE-C (a Likelihood-Free Inference method) to learn a density estimator to compute the posterior conditioned on real observation, and train the RL agent in simulation environment based on the parameters sampled from posterior (domain randomization) to minimize the dynamic sim to real gap, the advantage of the method was shown by one sim2sim and two sim2real experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe method applied in this paper can be applied to simulators that can only sample parameters which is an easy-to-use method. Requirement of real rollout is also reduced by the method, according to experiment setting, only one iteration of training can get good performance, which outperforms than the baseline. Both insight and real experiments are conducted to show the advantage of the method.\n\nWeaknesses:\nThe method applied in this paper do have a good performance, but this seems mainly because of the strength of the SNPE-C itself, and in a previous paper [1], SNPE-C has already been proposed to compute posterior for simulator. It looks like the improvement for the method is not enough when apply this method to sim2real.\n\n[1] Greenberg, David, Marcel Nonnenmacher, and Jakob Macke. \"Automatic posterior transformation for likelihood-free inference.\" International Conference on Machine Learning. PMLR, 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the SNPE-C seems just applied from a previous paper, but the previous paper is just use SNPE-C for one simulator, not involve sim2real, and the result in this paper also show good performance. The proposed method has improved a lot compare to the baseline, also considering the low requirement for real data, this maybe a useful method  for dynamic sim2real.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. Clarify the contribution of improvement when apply the SNPE-C to the sim to real compare with the previous paper [1].\n2. Show some experiment about how the number of real rollouts influence the performance of the method.\n3. It seems the trajectory of each rollout was not provided in paper, it’s hard to evaluate the total number need for real data.\n"}}, {"review_id": "PtquRMU9xHD", "reviewer": "Reviewer_FjeG", "summary": "This paper applies neural likelihood-free inference (LFI) to domain randomization, aimed to tackle the sim-to-real transfer problem. LFI brings the benefit of relaxing the restrictive assumptions on the parameter distribution of the simulators and removing the requirement for a differentiable simulator.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method is a natural step towards more general and better-performing domain randomization for better sim-to-real transfer.\n- The paper is well written.\n\nWeaknesses:\n- My most major concern is regarding whether the improvement of the current paper over BayesSim is significant enough. Firstly, in lines 230-231, because of this difference (ii), does your proposed method also incur more computational cost? I think an empirical comparison with [14,15] (or a discussion on it if comparison is not feasible) in terms of the computational costs is needed. Secondly, in lines 231-233, why isn't BayesSim [14,15] able to do this? That is, why can't their method also train the policy parameters directly based on the latest posterior? Moreover, I think the performances of BayesSim and the proposed NPDR are actually very similar in Figure 5.\n- lines 140-142: I think what is described here are the advantages of the proposed method (i.e., free of potentially restrictive assumptions on the domain parameter distribution, doesn't require a differentiable simulator, etc.), however, it doesn't exclude the comparison with other methods that don't have these advantages (e.g., those methods that assume a parametric distribution on domain parameters). Therefore, I think comparisons with more domain randomization methods should be performed.\n- Table 2: why is \"Return sim\" so high for \"Bayes. Lin. Reg.\"? Do they overfit to the simulations?\n- lines 36-39: what's the motivation for using LFI for inferring the domain parameter distribution, instead of other inference methods such as those based on variational inference or MCMC?\n- Algorithm 1, line 6 and line 10: I think giving more details on this function $f_{\\boldsymbol{\\psi}}$ will be helpful", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper improves over the BayesSim method for domain randomization for sim-to-real transfer. My most major concern is the significance of the improvement over BayesSim which determines the technical novelty of the paper, please see details below.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "My most major concerns is whether the improvement of the current paper over the method of BayesSim is significant enough, and I have listed the relevant questions to be addressed in Section \"Strengths and Weaknesses\" above under \"Weaknesses\" (the first point there). Also, I have some less major concerns/questions which I have also listed under \"Weaknesses\"."}}]

Author rebuttal:

Meta-review: The paper proposes the use of sequential neural posterior estimation with normalising flows as a likelihood-free inference engine for domain randomisation in robotics tasks. Experiments are provided for both sim-to-sim and sim-to-real with a Furuta Pendulum task.

Reviewers were mostly positive about the paper. The authors are encouraged to reply to all comments from the reviewers, in particular:

1. Better elaborate on the contributions and the differences with respect to the baseline (BayesSim)
2. If possible, provide ablation studies to indicate which factors are affecting performance the most.
3. Short description for DTW and why this metric is more appropriate compared to other divergences such as KL, Wasserstein or MMD.
4. Comparison of computational cost between the proposed and baseline
5. As SNPE is using conditional NFs, the incorporation of the statistical prior leads to a non-analytical posterior (the division of the two priors multiplied by q). How is the new posterior used as a prior in the next iteration?

========= Post discussion update ===========

I thank you the authors for the effort in addressing the reviewers comments and improving the paper. The reviewers agree to accept the paper and this is also my recommendation. 

Decision: Accept (Poster)