Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for training policies using Domain Randomization, where the randomization distribution is fit with some real world data. The algorithm, named Neural Posterior Domain Randomization (NPDR) in the paper, consists of using a likelihood-free inference method, Sequential Neural Posterior Estimation with Normalizing Flows (SNPE-C or APT) , to determine the distribution used to train policies in simulation with Domain Randomization. This is similar to BayesSim, which this paper uses as a baseline for comparison.
strengths_and_weaknesses: This paper demonstrates the power of likelihood-free inference methods for the tasks of parameter identification in robotics tasks. Furthermore, it shows how likelihood-free inference can be integrated naturally into a domain randomization pipeline for training robot controllers in simulations. This work also demonstrates the proposed Domain Randomization algorithm on a real physical system, the Furuta pendulum. The paper is clearly written and touches upon a very relevant topic for robotics research. The experimental results on the Furuta Pendulum task show an improvement over the online BayesSim baseline.

The strengths of this paper are in the simplicity of the proposed method and the experimentation with real robot systems.

The weaknesses of this paper are in the originality and impact of its contributions and in the experimental evaluation. 

In terms of originality and impact, the proposed algorithm is not very different to BayesSim at a high level. In a detailed view, the proposed algorithm  can be viewed as the same as BayesSim, but replacing the posterior inference algorithm and density model (both of which aren't contributions from this paper). 

An important shortcoming of the experimental results is that the differences in experimental results can be attributed to multiple factors (the ones listed in Section 5.2), but the paper does not provide isolated ablations or improvements to the baselines to determine the influence of the components of NPDR in the final performance. The comparison doesn't seem to account for model capacity. 
summary_of_recommendation: While the main technique used in this paper (likelihood free inference with normalizing flows) is of great interest to roboticists, the contribution is thin. I would still recommend this for acceptance given that it would be the first documented application of SNPE with normalizing flows to learning control policies. But I would condition my acceptance to a proper ablation study (to ensure that the method was compared to the best possible baseline)
issues: The paper compares NPDR to online BayesSim, but not to vanilla BayesSim.

The paper attributes the difference in performance to 1) the choice of inference algorithm 2) the choice of density model 3) that NDPR trains the policy until convergence. These three factors can be removed from NPDR (or added to vanilla BayesSim) independently to determine their importance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes a new way to leverage domain randomization to estimate and adapt the posterior distribution over the simulator’s parameters, and consequently allowing optimization of a policy suitable for the target environment in the real world. Specifically, it combines likelihood-free inference methods based on normalizing flows with domain randomization to estimate the posterior. This makes fewer assumptions about the domain parameter distribution, and can model correlations between parameters. This approach is evaluated in a sim-to-real experiment on a swing-up pendulum task.
strengths_and_weaknesses: Strengths:
- NPDR combines a strong LFI method with domain randomization to flexibly model the posterior distribution over parameters given a small set of observations. The experiments demonstrate the wide applicability of the method, and also include results on a real pendulum system.

- The paper is overall well written and easy to read. The visualizations help illustrate and characterize the differences between NPDR and BayesSim. 

Weaknesses:
- Perhaps, a weakness is that the method is a somewhat straightforward combination of a LFI method based on normalizing flows with domain randomization. However, this work highlights settings that necessitate a stronger neural density estimator.

- It would be useful to include the sample mean and standard deviation of the results across multiple experiments in Table 1, in addition to reporting the max over experiments.

- The studied pendulum control task is quite simple. The returns achieved by NPDR and BayesSim are thus quite close with overlapping intervals. It would be good to apply the method on a wider range of control tasks.
summary_of_recommendation: I am recommending a weak accept. Overall, the paper clearly communicates the method and its strengths. It also experimentally evaluates several aspects such as its ability to capture correlations between parameters. However, the studied tasks and quantitative results are a bit limited.
issues: - In terms of writing organization, having the related work earlier in the paper such as after the introduction would help a reader who has less familiarity with domain randomization better understand the contributions and differences of this work.

- Describing what the Dynamic Time Warping (DTW) metric is in more detail would make the paper more self-contained.

- As mentioned earlier, it would be useful to include the sample mean and standard deviation of the results across multiple experiments in Table 1, in addition to reporting the max over experiments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper applies neural likelihood-free inference (LFI) to domain randomization, aimed to tackle the sim-to-real transfer problem. LFI brings the benefit of relaxing the restrictive assumptions on the parameter distribution of the simulators and removing the requirement for a differentiable simulator.
strengths_and_weaknesses: Strengths:
- The proposed method is a natural step towards more general and better-performing domain randomization for better sim-to-real transfer.
- The paper is well written.

Weaknesses:
- My most major concern is regarding whether the improvement of the current paper over BayesSim is significant enough. Firstly, in lines 230-231, because of this difference (ii), does your proposed method also incur more computational cost? I think an empirical comparison with [14,15] (or a discussion on it if comparison is not feasible) in terms of the computational costs is needed. Secondly, in lines 231-233, why isn't BayesSim [14,15] able to do this? That is, why can't their method also train the policy parameters directly based on the latest posterior? Moreover, I think the performances of BayesSim and the proposed NPDR are actually very similar in Figure 5.
- lines 140-142: I think what is described here are the advantages of the proposed method (i.e., free of potentially restrictive assumptions on the domain parameter distribution, doesn't require a differentiable simulator, etc.), however, it doesn't exclude the comparison with other methods that don't have these advantages (e.g., those methods that assume a parametric distribution on domain parameters). Therefore, I think comparisons with more domain randomization methods should be performed.
- Table 2: why is "Return sim" so high for "Bayes. Lin. Reg."? Do they overfit to the simulations?
- lines 36-39: what's the motivation for using LFI for inferring the domain parameter distribution, instead of other inference methods such as those based on variational inference or MCMC?
- Algorithm 1, line 6 and line 10: I think giving more details on this function $f_{\boldsymbol{\psi}}$ will be helpful
summary_of_recommendation: This paper improves over the BayesSim method for domain randomization for sim-to-real transfer. My most major concern is the significance of the improvement over BayesSim which determines the technical novelty of the paper, please see details below.
issues: My most major concerns is whether the improvement of the current paper over the method of BayesSim is significant enough, and I have listed the relevant questions to be addressed in Section "Strengths and Weaknesses" above under "Weaknesses" (the first point there). Also, I have some less major concerns/questions which I have also listed under "Weaknesses".
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper use SNPE-C (a Likelihood-Free Inference method) to learn a density estimator to compute the posterior conditioned on real observation, and train the RL agent in simulation environment based on the parameters sampled from posterior (domain randomization) to minimize the dynamic sim to real gap, the advantage of the method was shown by one sim2sim and two sim2real experiments.
strengths_and_weaknesses: Strengths:
The method applied in this paper can be applied to simulators that can only sample parameters which is an easy-to-use method. Requirement of real rollout is also reduced by the method, according to experiment setting, only one iteration of training can get good performance, which outperforms than the baseline. Both insight and real experiments are conducted to show the advantage of the method.

Weaknesses:
The method applied in this paper do have a good performance, but this seems mainly because of the strength of the SNPE-C itself, and in a previous paper [1], SNPE-C has already been proposed to compute posterior for simulator. It looks like the improvement for the method is not enough when apply this method to sim2real.

[1] Greenberg, David, Marcel Nonnenmacher, and Jakob Macke. "Automatic posterior transformation for likelihood-free inference." International Conference on Machine Learning. PMLR, 2019.

summary_of_recommendation: Although the SNPE-C seems just applied from a previous paper, but the previous paper is just use SNPE-C for one simulator, not involve sim2real, and the result in this paper also show good performance. The proposed method has improved a lot compare to the baseline, also considering the low requirement for real data, this maybe a useful method  for dynamic sim2real.
issues: 1. Clarify the contribution of improvement when apply the SNPE-C to the sim to real compare with the previous paper [1].
2. Show some experiment about how the number of real rollouts influence the performance of the method.
3. It seems the trajectory of each rollout was not provided in paper, it’s hard to evaluate the total number need for real data.
review_title: null

Meta-review: The paper proposes the use of sequential neural posterior estimation with normalising flows as a likelihood-free inference engine for domain randomisation in robotics tasks. Experiments are provided for both sim-to-sim and sim-to-real with a Furuta Pendulum task.

Reviewers were mostly positive about the paper. The authors are encouraged to reply to all comments from the reviewers, in particular:

1. Better elaborate on the contributions and the differences with respect to the baseline (BayesSim)
2. If possible, provide ablation studies to indicate which factors are affecting performance the most.
3. Short description for DTW and why this metric is more appropriate compared to other divergences such as KL, Wasserstein or MMD.
4. Comparison of computational cost between the proposed and baseline
5. As SNPE is using conditional NFs, the incorporation of the statistical prior leads to a non-analytical posterior (the division of the two priors multiplied by q). How is the new posterior used as a prior in the next iteration?

========= Post discussion update ===========

I thank you the authors for the effort in addressing the reviewers comments and improving the paper. The reviewers agree to accept the paper and this is also my recommendation. 

Decision: Accept (Poster)