Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an empirical study of different mechanisms often set in place to prevent overfitting in Deep RL. It draws conclusions at what the challenges are and suggests future work directions.
main_review: This paper proposes an empirical study of different mechanisms often set in place to prevent overfitting in Deep RL.
Although the empirical findings are interesting and well-conducted, they don't fully cover the span of regularization methods. More importantly, the authors point out the challenges and weaknesses but do not address them, limiting the impact of the contribution.

## Writing and clarity

This article appears to have been written in a hurry, without any proof-reading or work on the structure.
There are lots of typos, many articles or words missing, acronyms are undefined, one bibliographical reference is replaced with a question mark, the ordering of the bibliography is shady, etc.
Seeing the related work section after the conclusion is an unorthodox practice at best, and looks more as if it was written and just dropped there. Additionally, it is mostly redundant with the introduction section and most citations already appear in the paper.
Many ideas are repeated several times (e.g. the last paragraph of the introduction just repeats the previous ones, Section 8 is a repetition of the main take-away messages from Sections 3 to 7).
Despite these aspects and the fact that it lacks content (see next comments), the paper is rather clear, well-organized and pleasant to read.

## Lack of perspective on regularization for RL

The idea of preventing overfitting and fostering generalization is often tackled as regularization. As the authors point out, many recent work have concentrated on this aspect, including the three papers by Raileanu et al, which serve as a basis for this work. iDAAC being somewhat state-of-the-art, it is justified, but other approaches such as IBAC-SNI, random convolutions, or the "historical" RAD algorithm would be legitimate candidates to verify the empirical study conducted here. Additionally, as noted in (Song et al, 2019) regularization can be separated as explicit regularization (explicit penalization of the empirical risk) or implicit regularization on the architecture and optimizer (dropout, batchnorm, momentum, early stopping) or on the data (data augmentation). This categorization applies both to supervised and to reinforcement learning. The only place where regularization is covered as a whole is the very last paragraph of the paper. And a natural question is then to assess whether the techniques evaluated in the paper actually cover a significant span of the possible regularization techniques overall.

## Insufficient discussion

Most take-away messages take the form of a challenge and some perspectives for future work. Besides the raw results, the reader does not get much insight from the reading.

In Section 4, I think the discussion is very incomplete. It seems to me that it is necessary for the policy to rely on level-specific features and that, ideally, an algorithm that generalizes well should disentangle (I didn't see any reference to the topic of disentanglement in the paper) the theme information from the layout information and only base the policy on the latter. Therefore, I think the problem should not be to evaluate whether "policies have level-specific features" but rather to assess whether these features are theme-independent. the discussion on this question is too shallow (almost inexistent).
More anecdotically, the claim that "policies have level-specific features regardless of whether it generalizes or not" is not backed up by Table 1 since the policies that have been used in this table "have good generalization capabilities".

In Section 5, the discussion on auxiliary tasks deserves a lot more details. The presented results display a lot of variability and the actual benefit of the proposed auxiliary task is very dependent on the game tackled. This section would benefit a lot from discussing the reasons of this variability. For example, auxiliary tasks could be expected to bring in game-dependent benefits since it depends on whether the auxiliary tasks features help disambiguate between equivalent sets of features for the policy. Unfortunately, the discussion stops after the plain presentation of results.
Also, data augmentation was mixed up with domain randomization. I assume it is a moment of inattention coupled with a lack of proof-reading. 

Section 6 (unfortunately) is nothing new. Hyperparameter sensitivity is (unfortunately) a recurrent, known problem in RL (even without the problem of generalization). And (unfortunately), no new insights are provided in this section.

Figure 6 (Section 7) deserves a much longer discussion. Why does training on more levels hinder the overall performance? Is the training done sequentially or in parallel (in the latter case, is it only a matter of number of samples to process before moving on to a next step?)? Even the raw results are not sufficient here to inform the reader on the mechanisms at hand.
summary_of_the_review: This paper proposes an empirical study of different mechanisms often set in place to prevent overfitting in Deep RL.
Although the empirical findings are interesting and well-conducted, they don't fully cover the span of regularization methods. More importantly, the authors point out the challenges and weaknesses but the discussion is very shallow and they do not address them, limiting the impact of the contribution.
The paper itself needs an important effort of editing (it is still a very preliminary version, assembled in a rush), and a significant contribution to improve the generalization capability of RL algorithms, to have a meaningful impact for the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper looks into why some of the recent methods for improving generalization in deep RL still do not perfectly generalize to new levels of a game. For data augmentation methods, they show that better augmentations which make use of the underlying process that generates the data / ground truth task variation. For approaches that learn domain invariant features, the authors claim that you actually need domain-dependent features to generalize well and that these methods may work well for different reasons (i.e. the additional signal and regularization from the auxiliary losses). They investigate approaches that use data augmentation and domain invariant features and provide some new insights. The authors also show that simple auxiliary tasks can improve generalization policies and that other approaches. 
main_review: ### Strengths 

The paper's overarching goal is to understand why current RL algorithms (aimed at improving generalization) still fall short in certain settings. This is an important problem and answers to this question would be valuable for the community. The paper contains some interesting insights regarding some recently proposed algorithms.

I also liked the findings related to the fact that simple models e.g. using a simple inverse model as auxiliary task can be quite effective at improving generalization in RL and even comparable with other SOTA (aiming to learn domain invariant features) methods like DrAC or DAAC. This suggests a fruitful direction for future work might be understanding which types of auxiliary losses or self-supervised training may lead to good generalization. I think these insights would be valuable for the community and those experiments seemed reasonably well designed for the conclusions to hold. 

### Weaknesses

However, the paper does not seem to provide a clear and definite answer to the question. It seems to contain a number of disconnected experiments that answer slightly different questions than the main question.

In addition, some of the claims do not seem to be supported by the experiments. Some of the experiments are lacking important details and are not very well designed to answer the questions they are supposed to. 

The paper seems to look at both the setting of zero-shot generalization and the setting of few-shot / fast adaptation to new tasks. However, the authors should clarify which setting they are referring to and use more specific nomenclature throughout the paper since the writing can be confusing at the moment (especially at the beginning of the paper when readers may not know you are investigating multiple settings). In addition, referring to a level as a separate task can be misleading since that typically refers to a different reward and / or transition function, which is not typically the case in Procgen where the levels differ in layout / theme but the overall goal is the same. 

In many places throughout the paper, including the figures, it is not clear whether you are referring to the training or test set performance, which makes it difficult to assess the results and conclusions. 

1. I am not convinced that the best way to test whether a method is theme invariant is to see if its test performance is the same when training it on levels with the same theme because this may confound better optimization with better generalization (train and test performance)? A better way to test whether a method is theme invariant is to test it on levels that have the same layout but different themes and see if the performance differs from the training levels. However, this does not seem to be the case here, but I found the description to be quite unclear. For DrAC-FT, do you use the exact same layout for the training levels as those used to train DrAC, with the only difference being that the theme is the same for all levels?

2. Does DrAC-TR use observations with same layout but different themes as augmentations or does it simply add more levels to the training set and uses the same types of augmentations as DrAC? Again, I found this section to be missing important details and clarifications. The comparison between DrAC-TR and DrAC isn’t fair since DrAC-TR essentially gets to train on many more levels of a game and because it sees the exact same levels in different themes it has the opportunity to learn theme invariance (perhaps even without the need to explicitly regularize the policy and value function). 

3. It would be interesting to compare with PPO-TR to see how it compares to DrAC. You can also apply DrAC but with augmentations given by different themes of the game. An interesting question is whether you need to explicitly regularize the policy and value when you have this type of data and perhaps how many themes do you need to capture this invariance (to any other theme)?

4. In Figure 2, the aggregated results don't seem to differ very much. However, there are games where the differences are significant, and others where they are not. I don't think the text reflects this very well and some of the claims can be misleading. 

5. Can you provide some intuition on why DrAC-FT would be better than DrAC? Are they being trained and tested on the exact same dataset? 

6. While your experiments seem to suggest that the methods still capture level specific features, while having good generalization, and thus those features are needed to learn generalizable policies, I am not sure the claim is correct. There are level specific features which you need in order to take reasonable actions and others which you do not (so you want to be invariant to these). Perhaps a better test is whether the theme can be predicted (instead of the level), which is typically not useful for solving the task. In contrast, the layout typically is needed to both solve the task and predict the level of the game. It also depends how different the test set is from the train set. If you collected this using a random policy you might end up with essentially identical observations in the train and test set and thus it will be very easy to predict the level (due to memorization). I suggest collected the dataset using a pretrained policy or checking that the test set is completely different from the train set. 

The problem might still be that the visual features, while containing the necessary elements to make good decisions, they also contain elements that are superfluous, level-specific, and thus making the policy overfit and fail on certain levels at test time (similar to spurious correlations in SL). I encourage the authors to design experiments aimed at teasing out the two to better understand whether those features are beneficial or not (and in what settings). 

7. I think Procgen may not be the best benchmark for this study since it can be difficult to control all the elements of variation across the different levels of a task. I would suggest designing simpler experiments in domains where you can change one variable at a time (e.g. the theme or layout, or agent-goal distance etc.). Perhaps you can then use Procgen to see if the conclusions hold in more complex / different domains. 

### Minor 
1. Sometimes you refer to UCB-DrAC as UCB-DrAC while other times as DrAC. I think it's better to decide on one of these and mention the convention at the beginning of the paper and then use consistent notation throughout the draft.
2. The related work section should be moved earlier in the paper and definitely before the discussion and conclusion 

summary_of_the_review: Overall, I think this paper is the beginning of some interesting investigations. However, I am not convinced the experiments described in this paper support the claims made by the authors. In addition, I think it would be better to focus on a one or two questions, clearly state them, and try to answer them as clearly as possible and provide as much insight and depth as possible. In its current form, it is not clear what questions the authors are trying to answer and what the conclusions are. 

Given all the above, I do not think the paper is ready for publication. However, I do think the subject of the paper is important and would be of interest for the community. I encourage the authors to take the feedback into account in order to improve the paper, and resubmit at a future conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies generalization of several previously published visual RL SOTA algorithms with PPO as a baseline in ProcGen environment (2d platformed style, procedurally generated maps). Generalization is studied in terms of theme (colors and styles of objects and background) and tasks (levels with different layouts and specifics things to do). DrAC is shown to not be theme-invariant by comparing result from vanilla version, a version where themes are fixed and version with randomized themes (vanilla version performs worse). For PPO, DAAC and iDAAC, linear classifiers on top of visual feature extractor layers (pretrained on the rl task) are trained to classify states (images from level) based on the level they belong to. Authors claim that accuracy around 90% hints that tested policies contain level specific information regardless of any generalization capabilities. Moreover authors propose a simple auxiliary task "inverse model" for ppo and show that it's comparable with UCB-DRAC. Finally it is mentioned that all methods need careful on per-game basis hyperparameter tuning which requires further research
main_review: The paper asks important questions about generalization theme and task wise and answers them but I am not sure (not saying that they are not, just cannot wrap my head around it) if generalization of the conclusions is justified. Especially considering results showing that hyperparameters must be finetuned for every game separately. 

- In abstract starting from "Contrary to..." it's not clear what you meant. Introduction part which states the same info is clear.
- Introduction is sturdy, clean and clear.
- Probably would drop "investigating" in headers and separate motivation/setup/results/discussion parts  of each subsection in more clear way
- Some figures could use better comments (e.g. Figure 2 is ok while Figure 3 lacks similar details like what is the dotted line).
- Figure 2: isn't it an overstatement to say that TR matched FT? on 8-12 out of 15 it look like significantly or slightly  worse. 
- The paper is not fully explicit in terms of experimental part but it's usually "like in the <link provided>" so I assume that the sources are explicit enough. Similar approach is taken to descriptions of algos and such - no elaboration and just reference (usually I feel that there is too much but here is too little perhaps). It surely benefits the brevity and cleanness but requires more jumps in other papers than usually if not acquainted with previous work. However I'm not sure whether to count it as positive or negative.


summary_of_the_review: The whole paper is not groundbreaking but makes its point in a clear and understandable way and seems to be natural consequence of previous research.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies generalization of policies trained for video games. More specifically, it analyzes transfer to new game situations via changing the theme (appearance) or the level (layout) of the game. The Procgen framework is used to procedurally generate new levels or themes to evaluate a set of hypotheses. The paper investigates (1) data augmentation, (2) policy generalization via domain confusion (3) generalization via auxiliary tasks (4) effects of hyperparameter selection and (5) adaptation to new levels.
main_review: **Strengths**

The paper attempts to address why policies learned for video games have generalization issues, which is an important problem and helps us design better models and learning algorithms in the future.  

**Weakness**

The main issue is that there is almost no novel finding in the paper. Most of the conclusions are already known. See details below:

- Section 4: I am wondering why the policy should not have level-specific features when it is trained for it. Also, the experiment is not flawless. Two states in train and test might be very close to each other. This doesn't mean that the policy has level-specific features.

- Section 5: The paper shows an inverse model (inferring the action between two states) achieves good performance. That is also already known, and several others papers have shown that. Some of these papers have been cited in the paper as well.

- Section 6: The conclusion of this section is that the performance drops if a single set of parameters is used for all games, which is expected.

- Section 7:  It is mentioned that "we see that end-to-end training of policy on levels 201 − 400 is less sample-efficient than finetuning the policy head as well as finetuning the entire policy". This statement is not correct. The blue and black curves in Figure 5 are almost identical or have a large overlap for most games. 

- Some of the conclusions are not aligned with the observations of other works. For example, the paper recommends that "This suggests a fruitful avenue for future work: instead of learning better visual features, focus on how to leverage the learned features to solve a new task". However, works such as [A] show better visual encoders are important for generalization to unseen environments.

**Comments about the details:**

- It is mentioned that "Figure 5 shows that the average returns of the policy on levels 201−400 become similar to the average
returns of the policy on levels 1 − 200". I am wondering if average returns are comparable. These are two different sets of layouts.

- It is mentioned that  "Furthermore, we implemented a variant of the ProcGen suite that uses a separate randomization
process for the theme versus the game layouts". I am not sure what this means. It is unclear how the same randomization process was used for both theme and layout in the other scenario.

[A] Wijmans et al., How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget, 2020.
summary_of_the_review: First, I do not see any novel conclusion in the paper:

(1) It is already shown by several papers that inverse models are helpful.

(2) It is already known that data augmentation techniques are required for better transfer.

(3) It is expected that using the same parameters for all games results in inferior performance compared to using game-specific parameters.

Second, the conclusion that improving visual encoders is not that important contradicts other works.
review_title: null

Meta-review: This paper presents an empirical study of generalization in visual reinforcement learning. This study is carried out in the domain of video games and it addresses the benefits of techniques such as regularization, augmentation and training with auxiliary tasks. The reviewers for this submission were positive about the goal and setups in this paper. They agreed that understanding why present day methods that attempt to improve generalization continue to fall short, is an important problem. However, most reviewers were underwhelmed by the findings presented in the submission. As examples: Reviewer 185P mentions that "the paper does not seem to provide a clear and definite answer to the question" and " I am not convinced the experiments described in this paper support the claims made by the authors." and Reviewer SFef mentions that "Most of the conclusions are already known". Some reviewers also found a lack of clarity and several typos in the initial submission. The authors have provided detailed responses to the reviewers. In particular they have fixed most writing issues. They also detailed why certain algorithms and techniques were benchmarked in this submission and others were left out. I think this is reasonable. One cannot expect a paper to benchmark every algorithm out there, and choosing promising and representative ones is sufficient. My takeaway from detailed discussions about this paper are that: The paper is much improved from a writing point of view and the rebuttal addresses some concerns well. However, I do agree with the reviewers that the findings presented in the paper are for the most part expected. This reduces the value of the paper to readers. When this is the case, it may be beneficial to dig deeper into these findings and present a narrow but deep analysis. Please see Reviewer 185P's suggestions in this regard. Given the above, I am recommending rejection for this conference, but I encourage the authors to take into the reviewers suggestions and resubmit.
Decision: Reject