Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper extends the DeepMind Control Suite benchmark by adding a series of visual variations across different tasks (e.g. lighting, color, background, textures, camera angles etc.). They also contrast a few recent self-supervised learning and data augmentation RL methods and measure these agents’ ability to generalize and transfer across a variety of visual variations provided by their benchmark. They also investigate different aspects of the variation in the scene that these agents seem to be most affected by.

Overall, this paper is clearly written and the analysis is very thorough and well motivated. However the results in my view are not that surprising: the observation that agents that have been trained only on a 'single' visual variation of an environment overfit to the visual features and fail to generalize outside of this distribution is not particularly novel. While I appreciate the amount of work that has been done here and the helpful analysis, contrasting different state of the art methods on self supervised learning, I have some questions/concerns:

1. There are a variety of existing benchmarks that allow for variation of the environment visual properties as well as variation in task distribution (e.g. [RLBench](https://arxiv.org/abs/1909.12271) & [OpenAI Fetch environments](https://arxiv.org/abs/1802.09464)). While Control Suite is one of the most widely used benchmarks for contrasting continuous control methods, I’m not convinced that it is on its own the best setting for studying visual generalization of agents as the majority of the tasks involve only a single agent/object and the appearance of agent/object in the environment is often not relevant to the task (making it a rather extreme example). I believe in order to better understand which methods allow for better generalization, one needs to combine tasks with different properties.
2. To my understanding the agents are trained only on a single instance of visual properties of the environment while tested on a large range of variations in the visual properties of the environment at test time. Maybe a more realistic test would be to allow a small variation of the environment during training time and then assess the generalisation gap?  
3. Related to my first point, I would like to perhaps better understand what is the desired outcome for this benchmark. I am concerned that in order for some method to avoid overfitting to spurious correlations in such tasks, this might result in some trivial task-specific heuristics for augmenting the data, which would perhaps solve this benchmark while still failing to generalize to more interesting/complex settings involving many objects where some visual properties of the environment are indeed task relevant. Could you comment on that? 

In summary, I enjoyed reading this paper and I believe the authors have done a nice job of contrasting different self-supervised learning algorithms in a challenging benchmark. I’m not fully convinced of the impact of the benchmark on future research but I think there are still interesting takeaways that would be valuable for the community.

review_title: Official Blind Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a "remastered" version of the Deep Mind Control suite that allows for testing of visual generalization to a number of environmental changes (e.g., pattern/texture of the floor, background, and lighting).  

I will begin my review by stating that I have somewhat limited knowledge of soft actor critic, Deep Mind Control and some of the most recent related work in this space and my review should be taken with that caveat.  However, the proposed forms of data/simulation augmentation that are proposed seem quite useful as a way of moving the Deep Mind Control suite from towards a more realistic and richer set of visual data. While this paper seems positioned as a more systematic baselining and data set paper and the novelty from an methodological perspective is somewhat low.  I think these are helpful in increasing how challenging the baseline is and providing more sight about how well the current methods perform.  

Having said that I think there are some ways the paper could be improved.  The link to the anonymous page wasn't working for me, I tried several PDF viewers but couldn't access it.  Also the image assets did not seem to be included in the supplementary material.  The main reason I was looking for these is that the details in section 3.3. are quite limited.  If possible I would like to be able to gain more insight about the diversity of the types of augmentation.  I appreciate that spaces is limited in the paper but showing an image with small thumbnails seems like it could be an option.   I link to the assets would be very helpful.

Another element that is a little unclear to me is how the backgrounds change as the characters move.  Are these full 3D environments or 2D backgrounds?  I am referring mainly to the "Background" elements. By the description it seems that they are photo graphs and therefore they don't capture 3D elements. 

The literature review around the use of simulation and/or graphics to created augmented data for interrogating models is a little bit limited.  I understand that these augmentations might be novel in the domain of RL, however, I think it is worth noting how adding conceptually this is something that has been done in past in several computer vision domains/tasks.  Not that that necessarily diminishes the utility of the proposed dataset.

Tables 1 and 2 don't have clearly labeled units.  This might seem minor but is a constant frustration for readers.  
For Figure 2. The it isn't clearly indicated what the error regions represent or the units of the axes.

To summarize, I think that this work is a helpful contribution, albeit with somewhat limited innovation.  I think more information about the data augmentation and positioning in the space of other uses of graphics/simulation for testing and augmenting vision data might be helpful. I would argue the paper as it stands is borderline and needs some improvements, but those could be achieved with revisions making it a good contribution.

review_title: This paper presents a "remastered" version of the Deep Mind Control suite that allows for testing of visual generalization. The paper presents an interesting augmentation to the DMC environment.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a new dataset for continuous control tasks based on the DeemMind control suite, but with varying backgrounds, colors and lighting/camera conditions. Instead of the fixed appearance in the original suite the frames from this dataset contain a lot of varying structure. The purpose of the dataset is to measure the generalization and transfer abilities of pixel-based continuous control agents across different visual variations.
The paper presents a comparison between several different representation learning method for CC models and concludes that augmentation plays a big role in making such models generalize well.

All in all I think this is an interesting contribution and the dataset may be useful for the purpose it was created, but I have several reservations.

First, I would say that this is still a rather simplistic notion of visual variability - especially the choice for not changing the camera positions in a substantial manner - this makes the task of the visual representation much easier and by "ignoring" color (as would be the result of color augmentation) most of the generalization problem really does go away. Demonstrating the ability of methods to be robust to more severe changes in the viewpoint would be a harder and more convincing test case (as it affects, directly, some task-relevant properties which can be extracted from the scene). Also - maybe changing the actual *appearance* of the limbed model (say, by skinning it) would have been more interesting - the underlying dynamics would stay the same but the appearance would change greatly while still be a deterministic function of the underlying state.
Finally, in terms of the baselines used and the results shown - I think it would have been good to show another RL method other than SAC - just to make sure results (in terms of augmentation and generalizablity) to not depend greatly on the specific choice of underlying RL algorithm.

To conclude - I think this is a nice contribution with a potential to be even nicer.

*Post Rebuttal*

The authors have mostly addressed my concerns - I still think SAC should not only be the only RL method in the paper, and though I'm still not convinced more drastic camera variations wouldn't make this more interesting I think all in all this is a decent contribution and probably should be accepted.
review_title: A nice dataset and idea but maybe a bit too simplistic?

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### **Summary and Contributions of Paper** 
This paper proposes a new RL Generalization benchmark based on the DM Control Suite, where there are multiple changing backgrounds but fixed dynamics. Multiple baselines, most of which involve data augmentation for RL, are tested and shown to possess poor performance on this benchmark, which suggests that it is challenging and meaningful enough to be used for future research.

### **Strengths**
- This benchmark appears to be easy to use, and highly accessible once released.
- The baselines used are comprehensive, and contain multiple recent and competitive methods, especially in data augmentation.
- This benchmark is highly motivated, as it is important to cleanly separate the visual and dynamics factors of RL generalization, and provides good background/related work.
- Paper is written well and is easily understandable.

### **Weaknesses**
- Unfortunately, there have already been multiple benchmarks proposed with the "changing background" setting, such as in (Zhang et al 2018b, Sonar 2020). Nearly identical DeepMind Control/background benchmarks have been proposed in (Zhang et al 2020). Therefore, this paper will need to emphasize its novelty component. Some ways to provide novelty are described in the next points.
- One unfortunate trap that many RL generalization benchmark papers fall into, is the pattern of "Propose benchmark, test algorithms/hyperparameters", without providing revelations/ablations as to why such methods work conceptually, or why certain phenomena exist. Many times it is difficult to tell if problems in generalization occur specifically with the certain benchmark, or actually occur at a broad level from some deep reason. Differentiating from the "N+1-th changing background benchmark" would allow this paper to provide much more impact.
- It would be helpful if the authors could provide more insights (other than only showing reward numbers) into why overfitting occurs in this benchmark, such as e.g. 1. Saliency plots on what the RL agent focuses on, 2. Properties of the policy/SAC algorithm, such as how different data augmentations affect its optimization process.

Overall, I currently give a score of 6 (marginally above acceptance) because the usefulness/assumed accessibility of this benchmark outweighs the weaknesses, but I would be happy to increase my score if the authors address some of my concerns.

### **Clarity Questions**
- One of the most important details I was looking for was how/if the background moves compared to the agent, i.e. what is the precise definition of \phi? This is very important, as from experience in running these types of benchmarks, overfitting does not usually occur when the background moves independently to the agent (e.g. when the agent is stopped, the background can still keep moving). I assume from Fig. 8 that the background moves along with the agent, and thus forward progress (via reward function) can be spuriously correlated with the background, but could you please clarify this point?


### **References**
- https://arxiv.org/pdf/1811.06032.pdf (Zhang et al 2018b)
- https://arxiv.org/abs/2006.10742 (Zhang et al 2020)
- https://arxiv.org/pdf/2006.01096.pdf (Sonar 2020)

### **Rebuttal Update**
I thank the authors for updating their draft. I think this paper is worth accepting due to its ease of use and many different features, but I will keep my current score at a 6.

**Here's why:**

My experience in these types of benchmarks have usually shown that **completely static** backgrounds (when cleanly used - there can be very subtle things that can still correlate with progress) do not actually affect generalization, but rather, anything correlated with progress will cause overfitting. The difference between this benchmark's background and e.g. a ProcGen/CoinRun's background is that CoinRun's background still moves when the character moves (which implies a slight correlation with forward progress).

From the authors' responses to Q 3.2 and Q 3.3, I believe that the main visual overfitting is occurring with the floor tile, as it is the only spurious object that is correlated with progress. This is because, as the authors have stated, the actual background image does not actually change when the agent is moving as the background is too far away. The authors also reinforce this aspect when they added the saliency/spatial attention maps in Appendix A.1, which shows that most of the attention is focused on the floor tiles, especially on test environments.

This means that I suspect that the randomized static background portion of the benchmark does not affect generalization, and I urge the authors to rethink this portion of the benchmark. 
review_title: Solid and useful benchmark. Some questions about clarity and novelty.

Meta-review: The paper proposes a modification to the DeepMind Control Suite to measure generalization with respect to visual variation. The authors run baseline experiments against their new benchmark and discover, unsurprisingly, that RL agents learning from visual observations overfit to spurious details of the observations.

Reviewers generally found the work to be clearly written, and the experimental analysis to be thorough and well done, though concerns about the rather simple nature of the visual augmentations persisted even after updates and author rebuttals. There were also concerns that by focusing only on Soft Actor Critic in the experiments.

3 of 4 reviewers felt the work met the acceptance bar, albeit only marginally. The dissenting reviewer's concerns centered on clarity (many specific issues appear to have been remedied), the relatively limited nature of the augmentations, and the fact that reviewers were not given access to the code. 

While the submission has potential, improvements needed are not minor, and given the short process, we can only accept papers as is, rather than expecting certain changes. Please take the reviewers' comments into consideration as you revise and resubmit to a future venue.
Decision: Reject