Improving the Generalization of Visual Navigation Policies using Invariance Regularization

Abstract: Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment. However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order to be useful. This has not been the case for agents trained with reinforcement learning (RL) algorithms. In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks. Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. 
      We propose a regularization method which combines RL with supervised learning methods by adding a term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken. The results of this method, called invariance regularization, show an improvement in the generalization of policies to environments not seen during training.
      

Reviews and author responses:

Summary:
The goal of the paper is to improve generalization of RL agents to a set of known transformations of the observation. 
The authors propose to explicitly include a term into the PPO loss function that incentivizes invariance to transformations of the environment which should not change the policy, in their case changing textures of walls. 

The idea itself is straightforward but worth exploring, although it does make the fairly strong assumption that one has access to or knowledge of the transformation function of the environment that can be applied to the observations. 
Overall, the paper is clearly written and easy to read. 

I'm currently recommending rejection based on the experimental evaluation as I don't believe that, in their current form, they sufficiently show the utility of the method (see detailed comments below).
However, I'm happy to change my rating if some or all of my comments and questions are addressed. 

My main concerns are with the experimental evaluation:
- I would encourage evaluation on a second environment. In particular, the CoinRun environment which is cited in the paper would in my opinion make an excellent second evaluation setting as it also allows randomization of the texture, so the setup is fairly similar to the setup here and, importantly, it allows comparison to published results. 
-I am surprised that training on _more_ environments reduces the performance for the baseline agent. My suspicion is that training on 100 or 500 task is, at first, much harder to learn for the agent as it sees individual levels much more rarely. With the number of training steps fixed, I think it is possible that those agents haven't finished training yet. It would be good to include the training curves for the agents or at the very least their final performance on the training set (for example in the appendix).
- Another question that was not clear to me from the text: When training the IR objective, is the transformation function restricted to produce observations from the set of limited environments? I.e. when training on 10 envs, does the transformation function produce only observations from those 10 or potentially from all 500 in the 'maximum' training set? Having access to all 500 would explain why the success rate for IR is constant across all number of training set sizes.
- I think figure 2 needs more random seeds and needs to show the standard deviation across them, as it might be fairly large. 

Edit:
Thank you for your response and your comments. 

Overall I think this is interesting and very promising work. Consequently, I am raising my score to "weak reject". Below, I discuss several points how I think the paper could be improved. 

Additional environment: I understand that Coinrun requires a significant amount of compute and that's not easily affordable for everyone. However, I do strongly believe that this work would profit a lot from additional experiments. For future versions of the paper, I do think Coinrun would be very interesting, but alternatively, maybe the Multiroom environment used in [1] might be an easier alternative? (With transformations being symmetry transformations?). Just an idea. 

More environments leading to a deterioration of performance: Unfortunately, I still don't understand how more data can lead to more overfitting. I believe it would be useful to investigate this further, either to avoid an unfair comparison to PPO if there's some training instability that can be easily avoided. Or alternatively, if this result holds, this could be a very useful insight as well if the authors can explain why this is the case. 

Good and constant performance of IR: I still find it surprising how constant the IR performance is independent of how many levels are being used. Providing more data/ablation studies to better support this result and provide further insight into IR would strengthen the paper a lot. Additionally/alternatively: Submitting the code for reviews to inspect would help here as well. 

>> "We hypothesize that adding this penalty encourages the model to find the invariance in the environment and the model is able to deduce the shared invariance within these environments and reach a certain performance with this added information only by observing a few .  

In particular, if you can show some further support for this claim, I think this would strengthen the paper a lot. 


We would like to thank the reviewer for spending the time and effort reviewing this paper. We find his notes to be very useful for  future projects based on this work, especially the provided references. Future version of this work will contain more experiments and seeds per experiment. While we agree about the fact that presenting an unsupervised method for providing labeled pairs is a more interesting route, this paper was only a proof of concept about the idea that policies learned with RL should not be expected to generalize and in order to ensure generalization one can provide an auxiliary loss for that, e.g., the IR penalty we propose. After that, we find it interesting to pursue methods of how to provide the labeled pairs or the transformation function without access to privileged information, as mentioned in the final section. 

We would like to thank the reviewer for taking the time to review and comment on our work. We will attempt to answer his concerns: 

Comment 1: "It requires that they have access to the transformation $\mathcal{T}$ or to observations that correspond across the MDPs. If the authors can explain why this is a reasonable assumption to make or explain how this assumption can be relaxed." 

We experimented on VizDoom because of the easy access to these transformations and  to have a proof of concept for our idea. However, we believe that these transformations could be replaced with simple image processing manipulations. There is a wealth of transformation functions (equalize, colorize, posterize, invert, brighten, sharpen ….) that can be combined and applied at different magnitudes in order to get a transformed version of the observation. One can use these operations as the transformation function or use learning methods to find the most suitable data augmentation for training as done in Cubuk et. Al. (2019).  

Comment 2: "It is also unclear from the writing how this assumption is enforced. Is the initial state always the same across environments, so the agent is always performing the same actions to stay sync-ed across environments?" 

There is no syncing between agents. Each agent has his own random seed. At the start of an episode the agent and the goal objects are initialized randomly. Each agent operates in the original training environments and a randomly textured environment, the log probabilities of the actions are recorded in both environments to calculate the penalty and then the agent executes the same action on both environments. 

Comment3: "Can you give some intuition as to why the agents trained on 100 and 500 environments are generalizing worse compared to ones that are trained on fewer environments?" 

We hypothesize the reason in the drop of performance when trained on 100 and 500 environments is that the overfitting is happening. Looking at the training curves for some of the seeds of the 100 and 500 experiments, we see an overfitting behavior. Unfortunately, the training curves are not included in the paper now but they will be added in consecutive versions. For a detailed description of why we believe that overfitting is happening and causing this drop in performance please refer to the stared {*} points in our response to reviewer 3, which we did not add here in order to avoid repeating the same text. The point is that all of the reported experiments achieve a 90%+ success rate on the training set and there is always a gap between the training curves and the test curves. That gap is bigger and more evident when training on 100 or 500 variations of the training environments. This leads us to believe that overfitting is happening when using vanilla-PPO and it is more severe when training on a large number of environments.

 

[1] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. CoRR, abs/1805.09501, 2018 

We thank the reviewer for taking the time to read and comment on our paper. We will try to the best of our abilities to respond to some of the reviewer’s concerns. 

Comment 1: "I would encourage evaluation on a second environment. In particular, the CoinRun environment which is cited in the paper would in my opinion make an excellent second evaluation setting as it also allows randomization of the texture, so the setup is fairly similar to the setup here and, importantly, it allows comparison to published results."  

We see the relevance of running experiments on CoinRun  as it is very suitable for testing the generalization of learned policies. However, in order to be comparable to the related literature [1], we have to run huge experiments for a large number of training iterations. Due to hardware and time constraints we couldn't achieve that now but it is a goal for future work. 

 Comment 2: "I am surprised that training on _more_ environments reduces the performance for the baseline agent. My suspicion is that training on 100 or 500 tasks is, at first, much harder to learn for the agent as it sees individual levels much more rarely. With the number of training steps fixed, I think it is possible that those agents haven't finished training yet. It would be good to include the training curves for the agents or at the very least their final performance on the training set." 

We hypothesize that the reasoning behind the declining performance for the baseline vanilla-PPO agents as more environments are added is due to the fact that the learning model is overfitting. 

We agree with the reviewer about the necessity of adding the curves of the train/test loss vs training iteration in order to show that overfitting is happening. We will be including the appendix of this paper. However, here is a summary of the behavior to justify our overfitting explanation:  

* For RGB/Grayscale input. Throughout training there is a clear gap between the training and test curves. On average across the seeds, over the first 1 to 2.5 million training iterations the mean training success rate rises to 90% while the mean test success rate rises to 15%-50%. Thereafter, the mean test success rate continues to oscillate in the 15%-50% interval until training ends.

* For RGB-D input, in the 10-50 training environment cases. The training success rate reaches 90% at around 1 to 1.5 million iterations, while the test success rate follows the training success rate with a 5%-10% gap throughout training. 
 
* For RGB-D input, in the 100 training-environment case, the behavior of the test success rate was far less stable. In the reported experiments, for three of the seeds, the training and test success rates follow very similar trajectories, with only a 2%-5% gap between them, indicating that very little overfitting is going on. However, for the two other seeds, the test success rate peaks at around 50% and then collapses from this peak soon after the beginning of training, decaying to around 20% after 1 to 1.5 million iterations. 
 
* For RGB-D input, in the 500 training-environment case, the behavior is similar to that with 100 environments. 

Generally, all the reported experiments are able to reach 90%+ success rate on the training set, this is one of the reasons that we did not include the training curves in the paper because the experiments reached almost perfect performance on the training set.

Comment 3: "Another question that was not clear to me from the text: When training the IR objective, is the transformation function restricted to produce observations from the set of limited environments? "

The transformation function only has access to a subset of the possible transformations, e.g., when training on 10 environments a transformation can be sampled from a fixed set of 10 environments and not the entire superset of available transformations. As for the constant success rate of PPO-IR. We hypothesize that adding this penalty encourages the model to find the invariance in the environment and the model is able to deduce the shared invariance within these environments and reach a certain performance with this added information only by observing a few .  

Comment 4: "I think figure 2 needs more random seeds and needs to show the standard deviation across them, as it might be fairly large."

We thank the reviewer for pointing this out and will proceed to add more seeds to the consecutive versions of this work in order for the results to have statistical validity.

[1] Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. CoRR, abs/1812.02341, 2018 

Claims: They formalize generalization in a reinforcement learning setting by defining a distribution over POMDPs from which they can sample multiple POMDPs for training. The authors posit that using domain randomization causes overfitting to those domains, and propose an invariance regularization to add to the training objective that prevents this overfitting.

Decision: Weak reject. While I believe the problem setting is important and the framework interesting to evaluate RL in a multi-POMDP or MDP setting and assuming a distribution over those multiple MDPs, the assumptions made in order to perform this invariance regularization are too strong. In Equation 1, in order to add the regularization penalty term, it requires that they have access to the transformation \mathcal{T} or to observations that correspond across the MDPs. If the authors can explain why this is a reasonable assumption to make or explain how this assumption can be relaxed, I will consider changing my score. It is also unclear from the writing how this assumption is enforced. Is the initial state always the same across environments, so the agent is always performing the same actions to stay sync-ed across environments?

Can you give some intuition as to why the agents trained on 100 and 500 environments are generalizing worse compared to ones that are trained on fewer environments? It seems one would expect the opposite to be true. 

This paper aims to achieve generalization under appearance change by explicitly optimizing for invariance under the visual transformations present in the training data, by leveraging privileged information about which states should be treated identically. Results demonstrate improvements over both a separated RL+supervised version and the naive domain randomization approach.

I have settled on a weak rejection, because although the results are reasonable, I feel that the approach is not particularly novel, requires significant access to privileged information, and is not sufficiently situated in the literature.

This is essentially a generalized visual place recognition problem, in which representations are sought that are invariant to visual changes in the environment such as texture and lighting. This is an extremely well-studied problem, and many important references are missing in the discussion of related work [1]. This problem has also been tackled with deep learning approaches [2,3,4] and so it is somewhat surprising that none of this work is referenced in a discussion of visual invariance.

The approach of using privileged information (image pairs that are the same place with different appearance) to train representations to be similar [3,4] is a very straightforward and not particularly novel auxiliary loss, and it is not surprising that it would improve generalization here. A more interesting and important direction is how to identify these points of invariance without supervised labels.

The results in Fig. 2 look very noisy and don't have error bars. Are these results statistically significant?
This paper could be improved with clearer experimental results showing some kind of monotonic improvement with the number of environments in Fig. 2, perhaps this could be achieved by running more experiments.

The paper would also be improved by considering unsupervised ways to provide the labeled pairs required for the method. For example, what about generating random transformations of the observations instead of using privileged information? The authors mention an adversarial approach; results from a technique like this would significantly improve my rating.

[1] Lowry, Stephanie, et al. "Visual place recognition: A survey." IEEE Transactions on Robotics 32.1 (2015): 1-19.
[2] Chen, Zetao, et al. "Deep learning features at scale for visual place recognition." 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.
[3] Arandjelovic, Relja, et al. "NetVLAD: CNN architecture for weakly supervised place recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[4] Carlevaris-Bianco, Nicholas, and Ryan M. Eustice. "Learning visual feature descriptors for dynamic lighting conditions." 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014.

Meta-review: All the reviewers recommend rejecting the submission. There is no basis for acceptance.

Decision: reject