Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a regularization scheme for training vision-based control policies that are robust to variations of the visual input, which the paper classifies as "visual randomization". Visual randomization, as the paper notes, is one of the state-of-the-art techniques proposed from simulation to real robot transfer of vision-based policies trained on synthetic images. The regularization proposed in the paper aims to produce policies that rely on features that are invariant to the visual randomization, so that the resulting behaviour of the agent is consistent across different variations of its visual input for the same states. 

The paper proposes that forcing a policy trained under randomization to have a Lipschitz constant of K, over the randomization parameters, causes the optimal policy to be similar across randomization parameters, with the difference in expected returns of two randomized environments being bounded by K times the maximum difference in their randomization parameters.

I recommend this paper to not be accepted until the following issues are addressed. 

* There are missing details from the experimental setup, which makes the results hard to interpret (see, below). 

* There are  missing details on how vanilla domain randomization was implemented. Domain randomization aims to maximize the expected performance over the environment distribution. This can be implemented properly by computing the expected gradients with data from more than one environment. From the algorithm descriptions in the appendix, it is not clear that this is how vanilla domain randomization was implemented. 

* The title, introduction and conclusions do not reflect the scope of the paper. The paper only addresses situations where the same behaviour is achievable on all environments, an assumption (Mehta et al, 2019) also makes, and its proposed regularization is based on the assumption that the optimal behaviour is achievable with the same policy on all environments. But this is not true in general: for dynamics randomization, different environments may require different policies (e.g. driving a car on a road vs driving off-road). The regularization method may result in overly conservative policies i such situations.  

Questions about experimental details: 

What are the maximum returns  for Cartpole when trained until convergence without randomization? (175? 200? 1000?) If the maximum returns are higher than 175, how does Figure 4 look with more data? This is crucial to understand, for example, the results in Figure 11. That figure shows the proposed regularization slightly hinders the performance for the environments near l=1, g=50 (that region is a darker shade of green on the left subfigure). How do we know if the task has been successfully solved in the green vs purple regions? In all experiments, are the training curves showing the performance of the policies over the same environments (same seeds)? If not, how are the training curves comparable?

Other things to improve:

The conclusions of this paper can be made stronger by adding a comparison with EpOpt-PPO (i.e. optimizing the worst case performance over a set of trajectories sampled from multiple environments)
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:

To improve the generalization ability of deep RL agents across the tasks with different visual patterns, this paper proposed a simple regularization technique for domain randomization. By regularizing the outputs from normal and randomized states, the trained agents are forced to learn invariant representations. The authors showed that the proposed method can be useful to improve the generalization ability using CartPole and Car Racing environments. 

Detailed comments:

I'd like to recommend "weak reject" due to the following reasons:

1. Lack of novelty: The main idea of this paper (i.e. regularizing the outputs from normal and randomized states) is not really new because it has been explored before [Aractingi' 19]. Even though this paper provides more justification and analysis for this part (Proposition 1 in the draft), the contributions are not enough as the ICLR publications.

2. As shown in [Cobbe' 19], various regularization and data augmentation techniques have been studied for improving the generalization ability of deep RL agents. Therefore, the comparisons with such baselines are required to verify the effectiveness of the proposed methods.

3. For domain randomization, it has been observed that finding a good distribution of simulation parameters is a key component [Ramos' 19, Mozifian' 19, Chebotar' 19], but the authors did not consider training the distribution of simulation parameters in the paper. 

[Ramos' 19] Ramos, F., Possas, R.C. and Fox, D., BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators. In RSS, 2019.

[Cobbe' 19] Cobbe, K., Klimov, O., Hesse, C., Kim, T. and Schulman, J., Quantifying generalization in reinforcement learning. In ICML, 2019.

[Mozifian' 19] Mozifian, M., Higuera, J.C.G., Meger, D. and Dudek, G., Learning Domain Randomization Distributions for Transfer of Locomotion Policies. arXiv preprint arXiv:1906.00410, 2019. 

[Chebotar' 19] Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N. and Fox, D., May. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 8973-8979). 2019

[Aractingi' 19] Michel Aractingi, Christopher Dance,  Julien Perez, Tomi Silander,  Improving the Generalization of Visual Navigation Policies using Invariance Regularization, ICML workshop 2019.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper introduces the high variance policies challenge in domain randomization for reinforcement learning. The paper gives a new bound for the expected return of the policy when the policy is Lipschitz continuous. Then the paper proposes a new method to minimize the Lipschitz constant for policies of all randomization. Experiment results prove the efficacy of the proposed domain randomization method for various reinforcement learning approaches.

The paper mainly focuses on the problem of visual randomization, where the different randomized domains differ only in state space and the underlying rewards and dynamics are the same. The paper also assumes that there is a mapping from the states in one domain to another domain. Are there any constraints on the mapping? Will some randomization introduces a larger state space than others?

The paper demonstrates that the expected return of the policy is bounded by the largest difference in state space and the Lipschitz constant of the policies, which is a new perspective of domain randomization for reinforcement learning. 

The proposed method minimizes the expected variations between states of two randomizations but the Lipschitz constant is by the largest difference of policy outputs of a state pair between domains. Should minimizing the maximum difference be more proper?

The center part of Figure 2 is confusing, could the authors clarify it?

In the Grid World environment, how does the random parameter influence the states?

The baselines are a little weak. The paper only compares the proposed with training reinforcement learning algorithm on randomized environments. Could the authors compare with other domain randomization methods in reinforcement learning or naively adapt domain randomization methods from other areas to reinforcement learning?

Overall, the paper is well-written and the ideas are novel. However, some parts are not clearly clarified and the experiments are a little weak with too weak baselines. I will consider raising my score according to the rebuttal.

Post-feedback:
I have read the rebuttal. The authors have addressed some of my concerns but why minimizing the expected difference is not convincing. I think the paper should receive a borderline score between 3 and 6. 
review_title: Official Blind Review #3

Meta-review: The paper presents a technique for learning RL agents to generalize well to unseen environments.

All reviewers and AC think that the paper has some potential but is a bit below the bar to be accepted due to the following facts:

(a) Limited experiments, i.e., consider more appealing baselines/scenarios and provide more experimental details.
(b) The proposed method/idea is simple/reasonable, but not super novel, i.e., not enough considering the ICLR high standard (potentially enough for a workshop paper).

Hence, I think this is a borderline paper toward rejection.


Decision: Reject