Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: (Auxiliary Review only -- the authors need not respond to this).

This work proposes issues with data augmentation in RL and proposes a couple of techniques to address those issues. It applies augmentation only on the current state and not future state to address erroneous bootstrapping. It also proposes a Q objective that uses both the augmented and un-augmented data. Also, for actor critic experiments, the actor is optimised only with un-augmented data and learn a  policy that is general via sharing parameters between the networks. These are simple techniques that help the paper demonstrate good results in a range of tasks.

This is a clear paper with good set of experiments. The problem has been described clearly and the work proposes simple refinements to the data augmentation strategies used in RL. While I am convinced of the quality of the method itself and the experiments, this work could have benefitted from more rigour with the baselines to make a stronger case for the method. Hence, I am offering a score of 6 noting that the paper is marginally above acceptance threshold.
main_review: Note : * This is an AUXILIARY REVIEW only. The authors are not expected to respond to this review or any of the concerns presented here. * *

The paper is well-written, clear and provides a good description of the experiments. It clearly sets up the problem that standard data augmentation methods suffer from. It also provides empirical solutions to these problems and backs up these claims with experimental results. Even though the paper justifies the claims only empirically based on experimentation, the ideas proposed here have made simple and easily extendable modifications, which can be very useful in complex state action spaces when RL+Data augmentation is employed. The ideas proposed here are pretty simple and straightforward, but I am not convinced these methods alone provide us a full-fledged framework for data augmentation as claimed. Also, any theoretical justification provided with this work is not rigorous enough to demonstrate that these methods are universal for all data augmentation problems.  On the novelty front, the paper's key contribution seems to be the idea of using un-augmented inputs to the targets. There are two ways to look at this -- just as a simple innovation being not novel enough on its own or trying to understand how the work arrives at this simple innovation and whether the innovation helps improve the algorithm performance over a wide variety of tasks. I am more inclined towards the latter. I believe this is still a useful enough contribution to practitioners and researchers alike, where a few simple modifications help greatly stabilize performance.

My major concern with the paper was the hyper-parameters and architectures used for the baselines. It looks like the adapted baselines were handicapped a bit by architectural changes and these baselines were not tuned to optimality, or at least it was not very evident that this was done thoroughly. This takes a bit of the shine off the impressive results presented by the paper and I am not sure if tuning the parameters would change any relative comparison between those baselines and this method.

The paper's proposed methods are good and the results are useful on their own to the community. Having said that, because of the other minor issues mentioned earlier, I am not able to give a higher score than 6 for this paper.
limitations_and_societal_impact: Yes, the authors have addressed the limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of high-variance Q-targets under data augmentation, and proposes a method to reduce the variance of the targets called SVEA. The key idea contains three parts: (1) Apply augmentation to the Q-value estimation of the current state and not the next state Q-targets for bootstrapping, and (2) modify the objective to include both the targets using augmented and original observations and (3) optimize the actor using only the unaugmented data for actor-critic algorithms, with parameter sharing. 

The paper performed experiments in DeepMind control suite and its variants (with raw pixel inputs). Compared to other data augmentation methods (e.g. DrQ) their proposed SVEA method is robust to the type of data augmentation and is more sample efficient with a better final performance. Some ablations were also presented among the above components to indicate the usefulness of all components. Generalization performance was demonstrated in the DistractingCS and robotic manipulation environments. The method also appears to help when using a Vision Transformers architecture. 

main_review: **Originality**: The proposed framework can be seen as a generalization of existing data augmentation approaches, by considering combinations of both augmented and unaugmented data in the loss function. The related work section is adequately touching upon the several areas which this framework can be viewed from. 

**Quality**: There is no theoretical analysis for the proposed framework, but the paper presents several empirical results to back up the claims, including ablations on the various components. However, I would suggest backing up the claim on Pitfall 1: Non-deterministic Q-target with some empirical data as well. How bad is the variance with non-deterministic Q-targets in practice? With SVEA, even though we have deterministic Q-targets, we still have noisy predictions due to the data augmentation. So there can still be comparison of variance in the case of combination of augmented and unaugmented current state and next state. 

For the alpha and beta hyperparameter: it was mentioned that 0.5/0.5 works well for the experiments. Have the authors tried varying this parameter throughout the training (i.e. some sort of annealing schedule), and whether that hinders or helps the performance even more?

**Clarity**: The paper is overall fairly clear to read. 

**Significance**: Given the recent popularity of applying data augmentation in vision-based reinforcement learning agents, I believe that the proposed technique, which is simple to implement, can be easily incorporated as a toolbox of RL practitioners/researchers. Demonstrating that this framework still scales to ViT architectures is also promising. It would be nice to have some more theoretical justification for the framework by the original authors, but perhaps that could be a follow up work (potentially by others). 

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a simple technique to improve the robustness and sample-efficiency of reinforcement learning. The technique can be applied to any algorithm via a simple modification of the input pipeline to said algorithm's Q-learning subroutine. The claim is that this modification reduces the variance of Q-targets and therefore stabilizes learning.
main_review: Disclaimer: Emergency review with all the short-comings that comes with.

# Originality

While the idea of data-augmentation has been extensively used in both supervised and reinforcement learning, this paper proposes an approach that in some ways generalizes the DrQ approach, and in others simplifies it. In my view the modifications, though subtle, are novel.

# Quality

The work produces an extensive array of empirical results showing reobust generalization and impressive sample-efficiency. The comparisons to DrQ lack specification of the hyperparamters K and M, which can easily be added. If, as I suspect, the comparison is with DrQ[K=1, M=1], it would be nice to see a comparison to the default DrQ[K=2, M=2] and/or at the very least DrQ[K=2, M=1], which matches the number of forward passes through the Q-network.

Similarly, in the empirical demonstration of Pitfall 2 (Figure 1) it would be nice to see the equivalent plot for various values of K.

Finally, I would slightly quibble with the claim of "theoretically grounded" on line 30, as this paper does not feature any rigorous theoretical statements.

# Clarity

Overall the paper is very clearly and well written. The only major point of contention is that DrQ is parameterized by a K and M which control how many samples the target Q-values and TD loss, respectively, are averaged over. It would be good to specify which is being used here. It sounds like [K=1, M=1] (?) but I would think the fairest comparison would be [K=2, M=1].


## Minor points

- The targets on line 215 and line 9 of Algorithm 1 presumably look different in continuous control cases, which is the main focus of the experimental section.
- Very minor point: would the expressions not be equivalent but simpler if the trade-off was simply alpha and (1 - alpha), instead of beta?
- The experimental setup is slightly unclear in the text, copying the caption of Figure 3 into the main body would help. In particular, the sentence "all methods are trained for 500k frames and evaluated on the full set of tasks".
- Figure 6 (left) needs error-bars.

# Significance

Data-augmentation seems to be a very simple technique for improving the sample-efficiency and generalizability of RL agents. This is particularly important for real-world applications where interactions with the environment can be expensive and the sim-to-real gap can be prohibitive.
limitations_and_societal_impact: I did not see any mention of limitations. The authors mentioned the growing footprint of large models and how this work could help negotiate that.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the question of zero-shot or fast generalization of value-based deep reinforcement learning agents using visual inputs to image transformation which do not change the (conceptual) state of the environment. The long term goal is therefore training agents which learn efficient, high return control policies both which are robust to irrelevant variations in high dimensional input observations. The ultimate goal seems to be generalization of learned skills to novel environments, but as far as I can tell this particular evaluation philosophy is implemented in experiments.

The paper reviews previous attempts to learn robust state representations to visual variations and formulates clear hypotheses for explaining the causes of difficulties which prior methods faced and were hindered by. The said hypotheses suggest a simplified approach (SVEA) which is empirically developed and evaluated in the remainder of the paper. Control tasks from widely used benchmarks are used to evaluate the method, somewhat less usual is the focus on vision-based high-dimensional versions of such tasks, instead of standard domains such as Atari games and benchmarks for generalization such as Natural Atari [1], CoinRun/ProcGen [2].


main_review: I am generally happy with the paper, I have a few suggestions:

* The goal of reinforcement learning is discovering an optimal control policy, not necessarily an action-value function. Strictly speaking, value estimation is just a crutch. Hence, why bother with robust value estimation? Why not learn value functions without augmentations, use them to discover an optimal policy, and then use supervised learning [3, 4] to encode a robust policy? Please clarify in the text! The proposed method is attempting to solve a problem (RL under observation augmentation) which may not be relevant if all that is needed is good control under diverse observation augmentations. How am I getting this wrong?
* Indeed, for all these environments it is not clear that value prediction from vision is needed at all, since a policy could be derived much faster from full state representations in terms of sample efficiency and computational speed, and then have that policy supervise a vision to action mapping, circumventing the robust value estimation problem altogether.
* There are many possible definitions of generalization in reinforcement learning; one is accurate value prediction in unseen states, as realistic problems have enormous state spaces; another is robust value prediction from high-dimensional observations, where states may have been seem before but there is extraneous variance in the observations which needs to be dealt with. Finally, there are progressively stronger definitions of generalization, namely across domain gaps which may or may not introduce variability relevant to the control policy itself. Which definition have you tested for in your experiments?
* Could you please clarify in the main text which experiments use some form of state information, e.g. "proprioceptive" data common in control tasks, as well as target object information, and which do not? If such information is used, how does it reflect on the ability to extrapolate the presented results to problems where such information is not available.
* Could you please specify for each set of results whether the set of initial states was randomized and by how much? One surefire way to get robustness to high variability in observations is open-loop control, which can also be implemented by non-recurrent architectures, e.g. by relying heavily on proprioceptive inputs or general statistics of observations. 


References:
1. (Natural Atari) Amy Zhang et al. 2018. Natural Environment Benchmarks for Reinforcement Learning. https://arxiv.org/abs/1811.06032
2. (CoinRun/ProcGen) Karl Cobbe et al. 2019. Leveraging Procedural Generation to Benchmark Reinforcement Learning. https://arxiv.org/abs/1912.01588
3. Andrei Rusu et al. 2015. Policy Distillation. https://arxiv.org/abs/1511.06295
4. Wojciech Czarnecki et al. 2019. Distilling Policy Distillation. https://arxiv.org/abs/1902.02186

limitations_and_societal_impact: I understand why every paper claims the maximal general applicability possible which cannot be refuted by reviewers without experiments; please note that proving negatives is hard, and much rests on your honesty and expertise alone. I am not comfortable with text which talks about aspirational goals as if they were achieved in this paper. The abstract talks about generalization to novel environments and great improvements in stability and sample efficiency, it seems to me that not enough attention to detail has been expended in quantifying the limitations of this approach and the wider family of algorithms to which it belongs. For example, a claim of "greatly improved sample efficiency" sounds outstanding, but then it is not clear to a non-RL specialist how many order of magnitude we are talking about, and how augmentations themselves make the problem worse. While the paper deserves credit for exploring this subject in detail, such claims are vague at best and may not age well; sample efficiency may mean very different amounts of experience in 5 years time.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a way of improving training stability of DrQ (an image-based RL algorithm that uses data augmentation) by applying image augmentation more carefully. The authors propose to input a mix of augmented and un-augmented observations into the critic, and stop augmenting the actor's inputs. On the considered benchmarks the proposed method demonstrates better generalization results. 
main_review: The paper identifies an issue with the increased variance of the target estimation in DrQ (image-based RL with data augmentation). The authors conjecture two root causes for the increased variance: non-deterministic Q-targets and over-regularization.
To remedy these issues they propose to apply data augmentation more carefully. Specifically, the authors suggest to mix augmented and un-augmented inputs to the critic and stop augmenting of the actor's inputs. This method is then empirically justified in a series of experiment on tasks from the DeepMind Control Suite.

Pros:
* The author investigate an interesting question of an effect of data augmentation on generalization in image-based RL. 
* The method is thoroughly verified in different settings. 
* The paper is clearly written and easy to follow.
* The exploration of alternative architecture, such as ViT.

Cons:
* The paper proposes a small trick that lacks in novelty. The impact of data-augmentation on the estimation variance was also studied in the original DrQ paper (Kostrikov et. al), where a very similar technique was proposed -- the target estimation was done over 2 different augmentations. This work introduces a tiny modification of the DrQ's variance reduction approach, that just combines one augmented and one un-augmented observation instead of combining two augmented observations. Furthermore, the authors fail to mention this prior result in their paper and compare to it. 
* From the paper it is not clear what hyper-parameters are used for the baselines, specifically for DrQ. If the authors use the original DrQ's parameters, then I see several major issues with the empirical results: 1) SVEA uses encoder arch that consists of 11 conv layers, while DrQ only uses 4. 2) The size of replay buffer of SVEA is 500K, while DrQ's 100K. 3) SVEA virtually increases the batch size by factor of 2, which usually leads to better performance on the DeepMind Control Suite tasks, it is not clear if the authors increase batch size for DrQ or not. These discrepancies alone could result in a drastic difference in performance and can render the empirical study to be invalid. 
* There is no experiments that compare individual components of SVEA to verify is that the improved performance comes from the proposed method or better/different hyper-parameters. 
* The authors claim in line 30 that they "theoretically ground" their findings, but this is not the case as there is no theoretical results presented. In general, the authors don't do a good job justifying that the two pitfalls are real. Their argument is hand-wavy and is based on a series of indirect comparisons to the baseline (DrQ). Unfortunately, there are too many moving parts in those comparisons that make it virtually impossible to validate the claim (see also the point above about the hyper-parameter discrepancies).

Given the very limited scope of novelty and technical contribution, the failure to mention the prior work that is almost identical, and the inconclusive empirical study I recommend to reject this paper from a high-profile venue of NeurIPS. 

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper investigates the problems related to the high variance in the targets that arise in TD-learning when the network is trained with data augmentation. Thee paper proposes a simple and effective way to address this problem by applying the data augmentation only on the online Q-network not on the target Qs and a modified objective function. The paper presents results on a range of tasks.

The paper is well-written and it is well-motivated. The authors investigate the proposed method on a range of interesting tasks and provide a very simple but seemingly effective solution.

The authors have provided an extensive rebuttal during the discussion period. However, some of the reviewers complaints are not completely addressed and I would appreciate if they can be addressed in the camera-ready version of this paper:

1) The theoretical justification provided in this paper is not rigorous enough. There is no guarantee that the proposed approach would work on any type of data augmentations. It would be nice if the authors are more careful about this and highlight that in the paper in order to avoid over-claiming.
2) The Reviewer MseA, xDk5 and UP5u asked several questions about the hyperparameters of the model. It would be nice if the authors can provide more analysis on the impact of the different hyperparameters used for different baselines. For example Reviewer UP5u pointed the different hyperparameters of DrQ and suggested more experiments to compare against.
3) More clear descriptions of the experimental setup along with a table of all hyperparameters used for all models (potentially in the appendix).
4) Adding error bars to the figures where applicable.
 
In addition to those points, I would recommend the authors to address all the other concerns by the reviewers as much as possible.
Decision: Accept (Poster)