Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper aims to achieve generalization under appearance change by explicitly optimizing for invariance under the visual transformations present in the training data, by leveraging privileged information about which states should be treated identically. Results demonstrate improvements over both a separated RL+supervised version and the naive domain randomization approach.

I have settled on a weak rejection, because although the results are reasonable, I feel that the approach is not particularly novel, requires significant access to privileged information, and is not sufficiently situated in the literature.

This is essentially a generalized visual place recognition problem, in which representations are sought that are invariant to visual changes in the environment such as texture and lighting. This is an extremely well-studied problem, and many important references are missing in the discussion of related work [1]. This problem has also been tackled with deep learning approaches [2,3,4] and so it is somewhat surprising that none of this work is referenced in a discussion of visual invariance.

The approach of using privileged information (image pairs that are the same place with different appearance) to train representations to be similar [3,4] is a very straightforward and not particularly novel auxiliary loss, and it is not surprising that it would improve generalization here. A more interesting and important direction is how to identify these points of invariance without supervised labels.

The results in Fig. 2 look very noisy and don't have error bars. Are these results statistically significant?
This paper could be improved with clearer experimental results showing some kind of monotonic improvement with the number of environments in Fig. 2, perhaps this could be achieved by running more experiments.

The paper would also be improved by considering unsupervised ways to provide the labeled pairs required for the method. For example, what about generating random transformations of the observations instead of using privileged information? The authors mention an adversarial approach; results from a technique like this would significantly improve my rating.

[1] Lowry, Stephanie, et al. "Visual place recognition: A survey." IEEE Transactions on Robotics 32.1 (2015): 1-19.
[2] Chen, Zetao, et al. "Deep learning features at scale for visual place recognition." 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.
[3] Arandjelovic, Relja, et al. "NetVLAD: CNN architecture for weakly supervised place recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[4] Carlevaris-Bianco, Nicholas, and Ryan M. Eustice. "Learning visual feature descriptors for dynamic lighting conditions." 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
The goal of the paper is to improve generalization of RL agents to a set of known transformations of the observation. 
The authors propose to explicitly include a term into the PPO loss function that incentivizes invariance to transformations of the environment which should not change the policy, in their case changing textures of walls. 

The idea itself is straightforward but worth exploring, although it does make the fairly strong assumption that one has access to or knowledge of the transformation function of the environment that can be applied to the observations. 
Overall, the paper is clearly written and easy to read. 

I'm currently recommending rejection based on the experimental evaluation as I don't believe that, in their current form, they sufficiently show the utility of the method (see detailed comments below).
However, I'm happy to change my rating if some or all of my comments and questions are addressed. 

My main concerns are with the experimental evaluation:
- I would encourage evaluation on a second environment. In particular, the CoinRun environment which is cited in the paper would in my opinion make an excellent second evaluation setting as it also allows randomization of the texture, so the setup is fairly similar to the setup here and, importantly, it allows comparison to published results. 
-I am surprised that training on _more_ environments reduces the performance for the baseline agent. My suspicion is that training on 100 or 500 task is, at first, much harder to learn for the agent as it sees individual levels much more rarely. With the number of training steps fixed, I think it is possible that those agents haven't finished training yet. It would be good to include the training curves for the agents or at the very least their final performance on the training set (for example in the appendix).
- Another question that was not clear to me from the text: When training the IR objective, is the transformation function restricted to produce observations from the set of limited environments? I.e. when training on 10 envs, does the transformation function produce only observations from those 10 or potentially from all 500 in the 'maximum' training set? Having access to all 500 would explain why the success rate for IR is constant across all number of training set sizes.
- I think figure 2 needs more random seeds and needs to show the standard deviation across them, as it might be fairly large. 

Edit:
Thank you for your response and your comments. 

Overall I think this is interesting and very promising work. Consequently, I am raising my score to "weak reject". Below, I discuss several points how I think the paper could be improved. 

Additional environment: I understand that Coinrun requires a significant amount of compute and that's not easily affordable for everyone. However, I do strongly believe that this work would profit a lot from additional experiments. For future versions of the paper, I do think Coinrun would be very interesting, but alternatively, maybe the Multiroom environment used in [1] might be an easier alternative? (With transformations being symmetry transformations?). Just an idea. 

More environments leading to a deterioration of performance: Unfortunately, I still don't understand how more data can lead to more overfitting. I believe it would be useful to investigate this further, either to avoid an unfair comparison to PPO if there's some training instability that can be easily avoided. Or alternatively, if this result holds, this could be a very useful insight as well if the authors can explain why this is the case. 

Good and constant performance of IR: I still find it surprising how constant the IR performance is independent of how many levels are being used. Providing more data/ablation studies to better support this result and provide further insight into IR would strengthen the paper a lot. Additionally/alternatively: Submitting the code for reviews to inspect would help here as well. 

>> "We hypothesize that adding this penalty encourages the model to find the invariance in the environment and the model is able to deduce the shared invariance within these environments and reach a certain performance with this added information only by observing a few .  

In particular, if you can show some further support for this claim, I think this would strengthen the paper a lot. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Claims: They formalize generalization in a reinforcement learning setting by defining a distribution over POMDPs from which they can sample multiple POMDPs for training. The authors posit that using domain randomization causes overfitting to those domains, and propose an invariance regularization to add to the training objective that prevents this overfitting.

Decision: Weak reject. While I believe the problem setting is important and the framework interesting to evaluate RL in a multi-POMDP or MDP setting and assuming a distribution over those multiple MDPs, the assumptions made in order to perform this invariance regularization are too strong. In Equation 1, in order to add the regularization penalty term, it requires that they have access to the transformation \mathcal{T} or to observations that correspond across the MDPs. If the authors can explain why this is a reasonable assumption to make or explain how this assumption can be relaxed, I will consider changing my score. It is also unclear from the writing how this assumption is enforced. Is the initial state always the same across environments, so the agent is always performing the same actions to stay sync-ed across environments?

Can you give some intuition as to why the agents trained on 100 and 500 environments are generalizing worse compared to ones that are trained on fewer environments? It seems one would expect the opposite to be true. 
review_title: Official Blind Review #2

Meta-review: All the reviewers recommend rejecting the submission. There is no basis for acceptance.
Decision: Reject