Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors investigate the question of how and when to use data augmentation (DA) in the deep reinforcement learning (DRL) training pipeline since it can interfere with the training and its effect is going to be time-sensitive for non-stationary RL. In this context, they propose two DA schemes depending on the use case, whether for sample efficiency (InDA) or for generalization (ExDA). They also propose an adaptive DA scheme, UCB-ExDA, with all the augmentations, including the option to not augment as arms of the MAB problem, followed by ExDA. The idea behind these DA schemes is to have a standalone distillation method at any point of the training-- during it as in the case of InDA, or after it as in the case of ExDA. DRL training and DA distillation are kept separate, and the authors also who the usefulness of postponing DA to the end of DRL training.
strengths_and_weaknesses: *Strengths*

The proposed DA strategies are well-motivated and well-discussed and the authors also find significant contrasting time-sensitivities of DA methods, which is interesting and demands further investigation. Distillation losses are well-illustrated and well-explained. Experiments and ablations are helpful-- the timing, performance, and different types of transformations are extensively studied.

*Weaknesses*

- The experimental design is sound, but it would be good to see some experiments on less computationally intensive benchmarks that do not require training for 25mn timesteps-- are there any, why not? Or at least, this should be addressed in the paper.
- Minor/writing: Possible typo on line 189 on Page 5-- N_k(s) is the number of *times*?





questions: - It would be interesting to see more discussion on DrAC+PAGrad, especially looking at its performance when considering separating out distillation from RL training. The authors say that they devise it as another mechanism to relieve the interference between RL training and augmentation, but why?
- It's interesting that removing the value inconsistency measure from the distillation loss doesn't degrade RL performance-- why would this be so?
limitations: The authors note that their proposed framework and findings are limited to the unique setting of non-stationary RL. The authors also state that there is potentially no negative societal impact of their work, and to the best of my understanding and imagination, I can also not think of one, since the paper is more investigative in nature.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Data augmentation is a critical regularization tool in supervised learning, used to induce prior domain knowledge into the model. In reinforcement learning however, data augmentation hurts and causes interference with training. In this work, the authors propose simple methods to incorporate data augmentation during learning, improving model generalization while alleviating the interference with training. Additionally, the authors propose a method that adaptively select the appropriate set of augmentations for an RL task.
strengths_and_weaknesses: Strengths:
1. The authors present a simple and novel framework for using data augmentation in reinforcement learning. The motivation for the method is grounded in observations of the effects of data augmentation in supervised learning. The proposed method addresses the problem of RL training interference in a simple way. The results on the OpenAI Procgen benchmark are convincing and the authors both validate their methods and propose different insights into how augmentation works in the context of RL. 
2. This work is relevant and I believe it can of interest to the community at large. Data augmentation has long been a critical component of supervised learning (and self-supervised learning), and it is important to have work such as this that utilized this framework in RL.
3. The authors clearly discuss the differences between the two proposed methods InDA and ExDA, and provide sufficient details to enable the reader to choose between the two. ExDA, in particular, is promising due to its low computational overhead.
4. UCB variants of the two methods are also proposed. They enable the model to find, in a set of augmentations, the most beneficial ones. While this approach is not novel, the addition of the "identity" augmentation appears to be a critical and relevant component to avoid interference with training in particular cases.

Weaknesses:
1. PPO is used as the baseline RL algorithm, no other algorithm is tested, so it is uncertain how general the method and the results are. The same thing goes for the benchmark RL tasks that were considered: all video games, with 16x16 images.
2. In section 5.2, ExDA is applied after 20M timesteps of RL training. After the 20M timesteps, no further RL training is performed, which is not the case for InDA. InDA continues to switch between RL optimization steps and distillation steps. For fair comparison between ExDA and InDA, it might be necessary to have the same number of total RL training steps, i.e. training for 25M steps before introducing ExDA

questions: 1. While InDA and ExDA individually make sense, what would happen if both methods are applying (performing distillation during and after training)? 
2. When performing adaptive selection, how do the UCB-ExDA and UCB-InDA methods differ in their choice of the most beneficial augmentations? Do they always converge to the same augmentation?
limitations: The authors address and discuss the main limitations of the work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes two data-augmentation techniques to improve sample efficiency and generalization capability based on timing. (1) intra distillation separates proximal policy optimization and distillation optimization, and is applied during the whole training period; (2) extra distillation is applied after training. 
strengths_and_weaknesses: ## Strengths

- It is well-written and easy to understand. The figures describe the proposed paper in an easy-to-understand graphical format.
- the proposed method is simple and effective.

## Weaknesses

### The formulation is not general enough

In the background section, the authors fix the observation representation into two-dimensional square image. It seems to be the implementation detail instead of the formulation. For example, the observation could not be limited to images, the image size could not be limited to be square (i.e., $\mathbb{R}^{m\times n}$).

### Ambiguous statement and lack proof. 

In line 146-147, the authours state that the proposed method is more stable than the alternatives. I wonder stability competes in terms of learning theory, sample efficiency, or optimization? Besides the intuitive illustration in Figure 2, can you provide the proof to support your statement?

### Data distribution modelling vs. proposed separation training between $\mathcal{L}_{PPO}$ and $\mathcal{L}_{DA}$]

The authors propose to alternate PPO and DA optimization, what is the benefits compared with directly modeling the data distribution, or can you provide some insights for distribution modeling?

### Lack of ellobration for ExDA

The proposed ExDA is one of the main contributions in this paper, but the authors leave some details of computational cost reduction into the supplementary part. It would be better if you can give some definitions/remarks/proofs to show why removing the value inconsistency measurement helps? If due to the space issue, I suggest combining Figure 3 and Figure 4 to fill the blank area.

### Lack of SOTA baselines  
The comparisons between the proposed method and baselines are not thorough enough. For example, CURL[1] and DrQ[2] are missed.

Some minor typos are listed below:   
* Line 72, learn with -> to learn with;
* Line  74-75,  a limited applicability -> limited applicability;  
* Line 78, a curriculum learning -> curriculum learning;  
* Line 203, clear advantage -> a clear advantage.  

*Reference*  
[1] Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. "Curl: Contrastive unsupervised representations for reinforcement learning." International Conference on Machine Learning. PMLR, 2020.  
[2] Yarats, Denis, Ilya Kostrikov, and Rob Fergus. "Image augmentation is all you need: Regularizing deep reinforcement learning from pixels." International Conference on Learning Representations. 2020.
questions: - From Table 1, it seems that the improvement is not significant.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores several modifications to UCB-DrAC [25]. It decouples the optimization of the RL loss and the data-augmentation regularization losses in two separate stages, and then empirically studies what is the best strategy for scheduling those two stages: alternating between them at fixed intervals during training (InDA), or training with RL only followed by a few epochs of regularization (ExDA). It also augments the UCB bandit in UCB-DrAC with an arm that does not perform any type of data augmentation. These methods are evaluated on a variant of the Procgen benchmark, where they compare favorably to the considered baselines (PPO, RAD, DrAC and DrAC+PAGrad). 
strengths_and_weaknesses: **Strengths**

- Generalization in RL is an important topic and the paper provides interesting insights about how data augmentation interacts with RL training.
- The proposed methods are simple to implement on top of DrAC and could be easily adopted by practitioners.


**Weaknesses**

- The evaluation protocol differs from [25] despite the proposed methods are a direct extension of DrAC. If I understood it correctly, authors consider a more challenging case where the agents are only trained on a single background per task (*easybg*) instead of Procgen's *easy* setup; this is motivated in L203 as a way to *see clear advantage from visual augmentations*. Given that [25] already showed an advantage of using augmentations in the standard Procgen benchmark, this choice is difficult to understand. I would strongly encourage including results on both the original setting as well.
- While the difference in evaluation protocol makes is difficult to compare results in this paper with those in the literature, it is surprising to see that DrAC underperforms PPO in most experiments (e.g. Figure 3, Table 2). This contradicts the results of [25]. Why is this the case?
- Given that the methods are an extension of DrAC (splitting the optimization of the RL and consistency losses in two stages, and adding an identity arm to the bandit), presenting DA/InDA/ExDA as entirely novel contributions (e.g. Sections 1 and 4) is somewhat of an overclaim. Note that properly positioning itself as an analysis or extension of DrAC would not reduce the merits of the paper but would make it easier for readers to understand the problems being studied.
questions: **Major**

I am particularly concerned about the following points described under *weaknesses* above which I believe should be resolved before this paper is ready for publication at NeurIPS:
- Results under standard Procgen evaluation protocol
- Ensuring that baselines replicate published results
- Better positioning of the proposed method with respect to the literature

**Minor**

- InDA and ExDA require additional learner steps compared to methods that optimize both the RL and the consistency loss in parallel (e.g. DrAC). What's the impact on wall-clock time?
- It would be interesting to understand how the performance of InDA and ExDA changes as a function of the size of the replay buffer used for distillation. In particular, I would expect a collapse if this buffer is too small. Have authors experimented with this?
- Figure 9 (Appendix D) shows train scores for different number of backgrounds. It is somewhat expected that increasing the number of backgrounds will make training slower, given that the agent is exposed to a more difficult task distribution. It would be interesting to compare those same experiments in the test set, where agents trained on fewer background might underperform due to overfitting.
limitations: Section 6 briefly mentions one limitation of the approach (i.e. the framework is specialized, and thus restricted, to RL). The paper does not describe potential negative societal impacts, although there aren't any specific to the proposed approach.
review_title: null

Meta-review: metareview: Although data-augmentation is  now a common practice in Deep RL  this submission provides a new RL-friendly approach to how and when use augmentation in Deep RL, supported by extensive experimental study, which is a non-trivial contribution to the field.
confidence: Certain
Decision: Accept