Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for improving generalization in reinforcement learning using data augmentation. It identifies a key theoretical limitation of previous work in the naive application of data augmentation to actor critic methods and proposes a solution that avoids biased estimation of the policy gradient objective while still allowing for regularization of the policy and value function. The resulting loss is paired with three distinct approaches for identifying an optimal data augmentation strategy. The best of these approaches, a bandit algorithm, obtains SOTA performance (relative to published works) on the Procgen environment. 

main_review: Strengths:

- The paper provides a well-motivated and justified means of applying data augmentation to improve generalization in reinforcement learning.

- The use of data augmentation to promote invariance in the value function and policy directly, rather than indirectly via applying the data augmentation to the inputs used to compute the policy gradient, is a nice way of avoiding the issue of biased gradients in policy gradient learners.

- The UCB method to select data augmentation transformations is a nice addition to promote more effective augmentation policies. 

- The paper is well-written and clear, with the experimental settings easy to follow and the theoretical motivation compelling.

- The empirical analysis of different approaches to automatically learning data augmentation is reasonably complete and provides some insight into the robustness of the proposed regularization strategy to possibly-suboptimal augmentation sets.

Weaknesses:

- The proposed method isn’t particularly novel as data augmentation approaches have been widely used in both supervised and unsupervised learning to regularize the representation. Applying data augmentation only to enforce invariance in a representation, as opposed to using it to increase the size of the training dataset, has been deployed previously in papers such as BYOL.

- The meta-learning baseline is unclear: does the meta-learner only have the capacity to apply a single augmentation to the inputs? If so, this seems like a fairly severe limitation. A more fair comparison might have been against a learned Spatial Transformer Network, which has the capacity to apply more complex sets of augmentations. 

- The empirical performance indeed outperforms the baselines, but in many instances the performance improvements are relatively modest. 
UCB assumes a fixed reward distribution, but as mentioned by the authors the policy improvements of the reinforcement learner mean that the rewards are likely to be highly nonstationary. Additionally, the use of the “training” environment to select the data augmentation, rather than a “validation” environment, seems like it might encourage overfitting.

Questions and Points of Improvement

- There are additional approaches to RL that seek to throw away irrelevant information to improve generalization, such as DBC and PBL. I would be interested in seeing how the method compares to these other approaches that use methods other than data augmentation to improve robustness and generalization.

- The theoretical setup should clarify whether the goal is truly to maximize the expected return over all environments or to maximize some sort of normalized return. For example, if one environment provides significantly more rewards than another, it might make sense to focus policy learning on this environment. This presumably isn’t an issue for the environments under consideration in the empirical analysis because of the shared structure, but may be worth noting for correctness. 

- I would have liked to see more justification for the use of training-set performance improvement in selecting a transformation rather than a validation-environment performance improvement as the reward signal for UCB, as it seems the principal goal of this method is not to increase the performance on the training environments, but to improve test-set performance.

- The best distribution of augmentations to maximize generalization performance may not be to greedily select the augmentation that improves the policy improvement step the most. I would be interested in seeing if a similar approach that can identify stochastic optimal policies over augmentations might yield better performance.

limitations_and_societal_impact: While I don't believe this work is likely to lead to any direct negative societal impacts, the authors should include a statement saying as much in the text.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper claims that the current data augmentation methods in RL relies on expert knowledge and large number of transformation methods, and proposes three methods that can automatically find a useful augmentation for a given RL task.
The paper introduce a way of using data augmentation with actor-critic algorithms under settings where the input observation has images.

main_review: Originality: This paper deals with the data augmentation problem in RL. While the problem setting is not new, the method proposed in this paper do have some merits in automatically learning the augmentation method to reduce the computational requirements. This paper also provides different perspectives on data augmentation, and provided three methods to augment data correspondingly. In the related work, it also discussed adequate paper and differs itself from them. 

Quality: This paper technically sound under the scope that the input observations are images.  This paper conducted a bunch of experiments under different environments to validate the effectiveness of their proposed method. However, it seems like in the experiments the input have to be images so that the transformations (candidate data augmentation methods) are applicable. It would be more interesting to see how this method could generate to scenarios where the inputs are vectors (as in many real world cases).  However, I did not find any discussions on this issue in this paper.
Clarity: This paper is clearly written and well organized.

Significance: The results seems to be less important considering the input have to be images.  But I do appreciate that in the experiment part, this paper has investigated the robustness, which is corresponding to their motivations mentioned in the introduction.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary: Reinforcement learning has been shown to overfit to training environments. Some recent works (laskin et al, kostrikov et al) have shown data-augmentation to be an effective technique in curbing the overfitting issue. However, different games have different biases and hence require different data augmentation. This paper proposes to use UCB algorithm to automatically select a good data augmentation (from a list of data augmentations) for a game. However, naively applying data augmentation to RL algorithms like PPO isn’t theoretically sound. To address this problem, the paper introduces novel policy and value regularization. Overall, the paper contains extensive experiments and ablation studies.
main_review: Originality: The paper is a mix of original ideas as well as novel combinations of well known techniques. The paper introduces novel policy and value regularization to combine data augmentation with PPO in a theoretically sound manner. In addition, it uses UCB algorithm to select appropriate data augmentation for a game to be used during the policy training.  

Quality: The claims are empirically well supported, the experiments and the analysis are extensive and proper ablations are considered. 

Clarity:  The paper is clearly written and well organized.

Significance: The results are important in a way as the paper suggests a theoretically sound and a stable way to use data augmentation with RL. 
 
Minor point: I wouldn’t call the method SOTA as DAAC/iDAAC gets better results (Raileanu et al). But that didn’t affect my final score as I feel this method is complementary to DAAC/iDAAC and it fixes (or atleast makes a good attempt at fixing) the issue of stability wrt data-augmentation in RL.
 
References:
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas. Reinforcement Learning with Augmented Data. NeurIPS 2020.
Ilya Kostrikov, Denis Yarats, Rob Fergus. Image Augmentation is all you need. ICLR 2021.
Roberta Raileanu, Rob Fergus. Decoupling Value and Policy for Generalization in Reinforcement Learning. ICML 2021.
limitations_and_societal_impact: It's not discussed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a framework for automatic data augmentation in deep RL. It proposes three algorithms based on 1) UCB 2) RL2 3) meta-learning, combining PPO and regularization of both the actor and critic to make sure that the policy and value function are invariant to the transformation corresponding to the augmentation. Experiments on ProcGen and the DMC with distractors show that the UCB-based algorithm (UCB-DrAC) outperforms vanilla deep RL and previous data augmentation baselines and is competitive with the best augmentation type chosen post hoc.
main_review: # Originality #

The paper combines existing ideas to create a novel framework for automatic data augmentation in RL: 1) UCB, RL2, MAML 2) regularizing the value function. Thus, in my opinion, this work has sufficient novelty.

# Quality #

The paper does not contain theory, aside from some intuition for why we should include regularization. I do not consider this a major weakness since the paper is primarily an applied work, but I do think there may be some theoretical support behind regularization, as TRPO does something very similar.

With respect to the empirical results, I found the results to be fairly convincing. The authors use two benchmarks, ProcGen and DCM,  and compare to recent work for generalization in deep RL, both based on data augmentation and other techniques. Ablations are done on the proposed algorithm showing that both the regularization and UCB are necessary for improvements over baselines. The performance of UCB-DrAC is comparable to that of choosing the best augmentation post hoc, with lower sample efficiency. 

I would've liked to see more discussion comparing the three proposed algorithms, in particular in relation to the assumption that there is a single best augmentation. If there is a single best augmentation (that happens to be in the set of eight transformations), then it is not surprising that UCB-DrAC works best, as it has the fewest parameters. Do the authors think that that is the case, or could it be that the higher expected sample complexity of RL2-DrAC and Meta-DrAC require training for more timesteps than was done in the paper?

# Clarity #

Overall, the paper is clear, and the results were presented well. However, there are a few places where edits are recommended:
- The description of RL2-DrAC is a little incomplete, it is not clear what the input to the meta-learner is from the main text. After looking at the appendix, I see that it is the sequence of augmentations, not the sequence of states as in the original RL2. I found this to be a bit confusing, so I would suggest the authors to clarify in the final draft.
- The second paragraph of section 4.5 seems a little out of place. Should it have been earlier?

# Significance #

The significance of this work is that it is the first attempt at an automatic data augmentation framework for deep RL. It shows competitive performance to choosing the best augmentation after running all of them, thus leading to improved sample efficiency. As UCB-DrAC seems fairly simple to implement, I think it would be useful for practitioners. 

# Updated review #

After reading the responses and reviews, I have decided to keep my score, with the understanding that the authors include more discussion of the assumptions of each algorithm in the final draft. 
limitations_and_societal_impact: # Limitations #

The authors mention one limitation of the work in the conclusion, that only one augmentation is chosen. I agree that this is a good direction for future work, as in practice many augmentations are often used. Another possible limitation is that the choice of the augmentation is currently based on only the reward signal, but intuitively the state (image) would also be helpful for picking the augmentation.

# Societal Impact #

The authors do not discuss any potential societal impact. I think this is fine since the paper is methodological, and I'd like to mention the following paper on the societal impacts of deep RL.
- Whittlestone, Jess and Arulkumaran, Kai and Crosby, Matthew. The Societal Implications of Deep Reinforcement Learning. JAIR 70, 2021.
review_title: null

Meta-review: The paper presents a solid contribution to improve generalization in reinforcement learning through data augmentation. Reviewers appreciated the theoretical motivations provided for the method and the addition of UCB to select transformations. I encourage the authors to make revisions promised in their response and address the issues raised by the reviewers. 
Decision: Accept (Poster)