Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes applying random convolutions to the observation space to improve the ability of deep RL agents to generalize to unseen environments. To encourage the learning of invariant features, the authors further include a loss term to align features of perturbed and unperturbed observations. Thorough experiments on multiple generalization benchmarks show that this method outperforms many previously used regularization and data augmentation techniques.

Although the proposed method is simple, it represents a useful contribution. The need to generalize across low level transformations in the observation space features prominently in several environments, including DeepMind Lab and CoinRun. The clear need for agents to be invariant to these low level transformations well motivates the proposed approach, as does the failure of many existing methods to provide this invariance.

The authors could more explicitly discuss the main drawbacks of this approach. As with any data augmentation, there is an assumption that the applied transformation generally won’t destroy information pertinent to the task. While this is true for the MDPs investigated here, it is easy to imagine slight variants of these MDPs for which this approach would fail. If an optimal policy must condition on color or texture information from observations, then using these random convolutions would render training impossible. Encountering such MDPs is not farfetched, so this weakness seems worth acknowledging.

In Figure 5 it would be useful to visualize performance of an agent trained directly on these unseen environments, as this presumably serves as an upper bound for the zero-shot performance of “PPO + ours”. How close does “PPO + ours” come to closing this gap? Without any context on the reward scale, it’s hard to infer how well this method is generalizing, beyond seeing that it beats some (possibly weak) baselines. Admittedly some closely related curves can be found in Appendix Figures 9 and 14, though they’re a bit out of the way.

Section 3.1 mentions that using alpha = 0 complicates training. It is somewhat surprising that using alpha > 0 is necessary or significant and yet the value used (alpha = .1) is relatively small. Any further comments on this choice?

I appreciate the discussion in Appendix F. It’s natural to wonder about alternative injection sites for the random network, and it’s good to see how the proposed method compares to these alternatives.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This work proposes using a randomly parameterized convolutional layer as additional processing of the input observation to provide data augmentation to make policies more robust to environments with different observation spaces. The empirical results are thorough, comparing with other regularization techniques, including dropout, L2 regularization, and batch normalization with the same policy gradient method, PPO on a variety of generalization in RL benchmarks. There are additional experiments of this method to check that it actually removes visual bias in a computer vision problem better than other methods. 

They also incorporate a feature matching loss that explicitly forces the learned representations of equivalent states to be close in L2 distance. While the empirical results are impressive, it is hard to feel excited about this work, which relies on the inductive bias of a randomly parameterized convolution layer to modify the texture of the observation and show it works in certain settings. I'd like more discussion and showcasing of failure modes, it seems that this wouldn't work for settings where the train and test environments are different in ways beyond texture and changes in small objects, and additional analysis in terms of the dogs and cats database about why it performs so much better than other methods. What exactly is the desired and meaningful information in images that a random convolution layer can keep while removing something that is able to generalize to different shades of cats and dogs? Why would this perform better than grayscaling?  What about grayscaling and additive Gaussian noise?

The comparison of PPO is also unfair in that the author's method uses an ensemble of policies to act, which other methods do not. A more fair comparison would use ensembles in all other baselines as well or results showing how their method performs without this ensemble.

Overall, the presentation, analysis, and writing can all be improved to match the strong empirical results produced. 
review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: This paper proposes methods to improve generalization in deep reinforcement learning with an emphasis on unseen environments. The main contribution is essentially a data augmentation technique that perturbs the input observations using a noise generated from the range space of a random convolutional network. The empirical results look impressive and demonstrate the effectiveness of the method. The experiments are thorough (includes even adversarial attack) and the core method is novel as far as I am aware.

That said, I have a couple of concerns regarding this paper and I would be willing to change my score if authors can address these.

1) Feature matching loss (Eq 2) is presented as a novel contribution without referring to related work in semisupervised learning literature. This is essentially consistency training. See:
a) Miyato, Takeru, et al. "Virtual adversarial training: a regularization method for supervised and semi-supervised learning." IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.
b) Xie, Qizhe, et al. "Unsupervised data augmentation." arXiv preprint arXiv:1904.12848 (2019).

2) The main contribution appears to be a data augmentation technique where we add a random neural net based perturbation to the state. My question is:

*Why don't you first evaluate this on computer vision tasks given that the core idea is data augmentation for images?*

If this technique is so powerful, shouldn't this do a great job in CIFAR10, Imagenet etc? Instead authors only provide a niche example (bright vs dark cat/dogs).

If this can compete with top augmentation techniques on Imagenet (e.g. autoagument), then it can explain the RL performance. Otherwise, please provide some intuition on why this works so well on RL but not as well on computer vision tasks. Is it the unseen environment diversity of RL challenges?

3) While proposed method performs well on the benchmarks, it is not clear whether authors compare to the state-of-the-art algorithms. For each task (CoinRun, DeepMind Lab, etc), please explicitly state the best prior result (e.g. Espeholt et al, Tobin et al, Cobbe et al etc) so that proposed method's performance can be better assessed.

-------------------------

After rebuttal: Authors addressed most of my comments. I also found the new experimental results (Fig 5 and 7) very insightful. I increase my score to Weak Accept.

For future improvement: More realistic experiments on computer vision tasks (besides cats and dogs) would be welcome. Otherwise, please justify why proposed strategy is particularly good for RL (rather than traditional computer vision benchmarks) in boosting robustness to new domains.
review_title: Official Blind Review #3

Meta-review: This submission proposes an RL method for learning policies that generalize better in novel visual environments. The authors propose to introduce some noise in the feature space rather than in the input space as is typically done for visual inputs. They also propose an alignment loss term to enforce invariance to the random perturbation.

Reviewers agreed that the experimental results were extensive and that the proposed method is novel and works well.

One reviewer felt that the experiments didn’t sufficiently demonstrate invariance to additional potential domain shifts. AC believes that additional experiments to probe this would indeed be interesting but that the demonstrated improvements when compared to existing image perturbation methods and existing regularization methods is sufficient experimental justification of the usefulness of the approach.

Two reviewers felt that the method should be more extensively compared to “data augmentation” methods for computer vision tasks. AC believes that the proposed method is not only a data augmentation method given that the added loss tries to enforce representation invariance to perturbations as well. As such comparisons to feature adaptation techniques to tackle domain shift would be appropriate but it is reasonable to consider this line of comparison beyond the scope of this particular work.

Ac agrees with the majority opinion that the submission should be accepted.
Decision: Accept (Poster)