Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method that utilizes data augmentation for image-based reinforcement learning. The data augmentation is used to regularize the policy and function approximation in the proposed method. In addition, a method for automatically identifying effective ways of data augmentation is proposed. The experimental results show that the proposed method outperforms the baseline methods.

The study shows that the regularization of policy and function approximation using the transformed images is more effective than training a policy by using the transformed image as a state. Regarding identification of effective ways of data augmentation,  the proposed method seems improve the performance of image-based RL methods, although the proposed approach is simple.

I have some suggestions for improving the paper.
- The term for regularizing the policy is given in Eq. (3). I think that the first \pi(a|s) in the KL divergence should also have the subscript \theta because it seems that the two policies are based on the same model. Likewise, the first term in Eq. (4) should have the same subscript as the second term. 

- It would be good to show the Jensen-Shannon divergence and the cycle consistency of RAD in Table 2. 


== comments after discussions and the paper update ==

I appreciate the authors' efforts to improve the clarity and provide additional results. I believe that the proposed method is now clearly presented and the claims are properly supported by experiments. I raise the score to "accept". 

review_title: Interesting results

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary after Discussion Period:
-----------------------------------------------
After reading the other reviewer's comments and corresponding with the authors, I have become convinced that the author's proposed regularization method is novel and effective, and would recommend this avenue of research be further explored. Yet it has also become clear to me that the author's claims on why their method works are not yet supported by evidence. Further, I don't believe the author's proposed further ablation studies would fix the theory, since such experiments don't address whether their method works by fixing problems with Laskin’s work (as the author's claim) or because it provides a more direct way of enforcing invariance to transformation (as I claim).

So we're left with a difficult situation, the method and the experiments are good while the theory is lacking. In such a situation both acceptance or rejection seem reasonable. Yet, as per ICLR reviewer guidelines, one should answer three questions for oneself:

 - What is the specific question and/or problem tackled by the paper?
 - is the approach well motivated, including being well-placed in the literature?
 - Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.

Since the theory is lacking and the approach is not well motivated, and since the theoretical claims haven't been rigorously supported, I feel as per ICLR guidelines the paper is not yet ready for acceptance.

Initial Review:
-------------------

Summary In this paper, the authors introduce three approaches for automatically finding an augmentations for any RL task, and a novel regularization scheme to make such augmentations work effectively.

Positive aspects:
-----------------------
The paper’s language is clear and the authors provide a good overview of the problem of data augmentation for reinforcement learning. Furthermore, they nicely explain why data augmentation for RL isn’t as straightforward as augmenting data for supervised learning learning. I believe that data augmentation could be a nice tool in the Reinforcement Learner’s toolbox, and I’m glad to see a paper advancing the idea.


Major Concerns:
-----------------------

This paper has not provided sufficient evidence that the author’s proposed way of doing data augmentation is effective. In this paper, there are two main novel methods for doing data augmentation / insights in RL. I will discuss my concerns with both methods separately.

Policy and Value function Regularization.
-----------------------------------------------------

The authors criticize the naive application of transformers in the PPO’s buffer, as done in Laskin et al. (2020), saying that this changes the PPO objective. While I agree that such a naive transformation as in Eq. 2 is problematic, I fail to understand why application of transformation to states in the buffer would result in the Eq. 2, as the transformation would happen to the states being fed into both $\pi_\theta$ and $\pi_{\theta_\text{old}}$, resulting in an equation different from Eq. 2. One just needs to save the old policies (and old Advantage function), so that $\pi_{\theta_\text{old}}$ can be applied to transformed states, and not just use the actions from the buffer. This would seem like a straightforward fix.

Yet the authors have proposed a different regularization fix which judging the experiments does seem to work, as shown in Figure 2. I suspect it works for another reason: since the regularization forces $V(s) = V(f(s))$ and $\pi(\cdot | s) = \pi(\cdot | f(s))$ I wonder if this isn’t a method to allow prior knowledge to flow into the policy and value estimation. If the transformation(s) $f_i$ have been chosen such that one can be reasonably sure that true value and policy functions should be invariant to said transformations, then by enforcing $V(s) = V(f(s))$ and $\pi(\cdot | s) = \pi(\cdot | f(s))$  one is constraining V and \pi to fit the prior knowledge contained in $f_i$.

So now we’re comparing apples and oranges, since your method (DrAC) gets to incorporate this prior knowledge, while PPO and RAD don’t, and DrAC’s good performance isn’t surprising.

Automatic Data Augmentation
----------------------------------------

Here, given some candidates for data augmentation, the authors propose three methods to discover which candidate work well. Unfortunately, I don’t fully understand the approach. 

The authors are examining a Meta-Reinforcement Learning setting, where one wishes to find a policy which performs well not just on one MDP, but on a whole distribution of MDPs. This leads to an inner-and-outer for-loop like setting, in the inner for loop, the agent does multiple episodes with a single environment, in the outer loop the agent gets new environments.

I had expected this inner-outer for-loop structure to pe present in the meta learning of augmentations, and for the authors to clearly describe how knowledge about the effectiveness of augmentations is transferred from past episodes on one environment to future episodes in the same environment, and how it’s transferred from past environments to future environments.

The only support given for these methods is the experimental results. Yet we see that the performance of DrAC and UCB-DrAC lie within a standard deviation of one another. So the evidence of the effectiveness of UCB-DrAC over the simpler DrAC is weak at best.

Minor Comments:
-------------------------
$T_m(s’|s,a)$ is the transition function, $R_m(s,a)$ is the reward function: Here, $T$ and $R$ are generally distributions and not functions

Eq. 6 is confusing, since $f_*$ can refer to both $f_t$ (function at timestep $t$) and $f_i$ (the $i$-th transformation function). 

Do the update with something like $N_t(f) \leftarrow N_{t-1}(f) + 1$ is a bit clearer, since then it’s the number of times $f$ has been pulled before timestep $t$

review_title: Data sugmentation is a nice tool for RL, the authors propose a simple and effective solution but fail to motivate their method or support theoretical claims

final_score: rating: 7: Good paper, accept
review_content: #######################################

Summary:
This paper tackles the problem of generalization in deep RL via data augmentation. It provides a framework for automatic data augmentation based on UCB, RL^2, or MAML. When UCB is combined with regularization of the policy and value function so that their outputs are invariant to transformations (such as rotation, crop, etc.), it shows improvements over similar algorithms on the ProcGen benchmark. 

#######################################

Pros:
1. The paper is well written and the proposed algorithms are straightforward to understand.
2. UCB-DrAC is shown to be statistically superior to the tested baselines. Surprisingly, when all ProcGen games are taken into account, some existing methods such as Rand-FM are actually worse than PPO.
3. Ablation studies are able to show that both ingredients of UCB-DrAC are important. UCB is shown to find the best augmentation asymptotically, and the DrAC is shown to be better than PPO and RAD. 

Cons:
1. The experiments do not compare the proposed algorithms to DrQ, the algorithm proposed in Kostrikov et al. (2020). Since it also tackles generalization in deep RL through data augmentation, it seems that it should be included as a baseline. Can the authors explain why it was not included?
2. The proposed algorithms are only combined with PPO. It would be good to have some results for other actor-critic algorithms, such as SAC, to verify if the SOTA behavior holds.

#######################################

Overall:
I would vote for accepting this paper, due to the strength of its experimental results. The proposed approaches are novel in the data augmentation for generalization in deep RL subfield, although AutoAugment (cited in the paper) also uses RL for choosing data augmentations in the supervised learning case.

#######################################

Further comments and questions:
1. It may be helpful to rewrite some of the algorithms in Section 3.3 in pseudocode, as text is harder to read.
2. Is there intuition for why only UCB-DrAC leads to statistically significant improvements over PPO, and not RL2-DrAC and Meta-DrAC? Can it be shown that the former has lower sample complexity than the latter two?
3. Why is the cycle consistency percentage in Table 2 always low? Is it some idiosyncrasy of the metric or due to the inherent variance in the trajectories?

#######################################

Update after reading other reviews and author responses:

I am happy to keep my score and support accepting this paper. I agree with Reviewer 4 that conducting a more detailed ablation study of the impact of regularizing both the policy and value function (i.e. a comparison with an algorithm like that of Kostrikov et al. (2020)) would improve the paper and hope that it will be included in the final paper. 
review_title: An effective framework for automatic data augmentation in deep RL

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 

This paper proposes an automatic data augmentation approach for RL tasks. Specifically, it takes UCB for data selection and introduces two regularization terms for actor-critic algorithms' policy and value function. Then this paper evaluated the approach based on the Procgen benchmark and demonstrated that it outperforms existing methods. It is also shown that the learned policies and representations are robust to irrelevant factors.


Reasons for score:

On the one hand, I feel that the proposed approach is incremental (UCB for data selection with the actor-critic algorithm as the RL algorithm plus additional regularization terms). It would also be ideal if more experiments can be included to prove that the proposed approach is effective in most RL tasks. However, on the other hand, based on the current experimentation results, the proposed method seems promising and can be useful for generalization in reinforcement learning.


Pros

1. The proposal of the automatic data augmentation/selection approach is useful, given that the existing primary methods either rely on expert knowledge or separately evaluate a large number of transformations to find the best one, both of which are expensive. 

2. The proposed approach achieves SOTA performance in the Procgen benchmark.

3. It shows that the proposed method is robust to irrelevant factors. 
	

Cons

1. I think the primary difference between the proposed approach and existing approaches is the additional regularization losses. This would make the new approach seem incremental. 

2. There is a lack of performance comparison between the automatic data augmentation method and non-automatic approaches.

3. The data augmentation approach is only evaluated in the Procgen benchmark. It would be great if this paper includes more experiments to demonstrate the performance, especially since the proposed method is intended for any RL task.

review_title: Tend to accept 

Meta-review: As of now, automatic data augmentation methods have mostly been proposed for supervised learning tasks, especially classification. This paper introduces automatic data augmentation to deep (image-based) reinforcement learning agents, aiming to make the agents generalize better to new environments. A new algorithm called data-regularized actor-critic (DrAC) is proposed, with three variants that correspond to different methods for automatically finding a useful augmentation: UCB-DrAC, RL2-DrAC, and Meta-DrAC. Promising results are reported on OpenAI’s Procgen generalization benchmark which consists of 16 procedurally generated environments (games) with visual observations. Further experiments have been added in the revised version.

**Strengths:**
  * This work is among the first attempts that propose an automatic data augmentation scheme for reinforcement learning.
  * The paper articulates well the problem of data augmentation for reinforcement learning.
  * The experiment results are generally promising.

**Weaknesses:**
  * Although the experiment results reported seem promising, there are missing pieces in order to help the readers gain a deeper understanding to justify more thoroughly why the proposed regularization-based scheme works.
  * Theoretical justification is lacking.

This is a borderline paper. While it presents some interesting ideas supported empirically by experiment results, the paper in its current form is premature for acceptance since a more thorough, scientific treatment is lacking before drawing conclusions. Moreover, considering that there are many competitive submissions to ICLR, I do not recommend that the paper be accepted. Nevertheless, the authors are encouraged to address the concerns raised to fill the gaps when revising their paper for resubmission in the future.

Decision: Reject