Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors tackle the problem of generalization to novel visual environments in Reinforcement Learning.  The authors differentiate between weak augmentations (those that help with learning to solve the task in the same environment) and strong augmentations (those that hurt the training performance on the same environment, but may help generalization across domains).  The authors propose a curriculum where after M steps of only training on the weak augmentations, additional optimization on strong augmentations is included as well.  The authors also propose a data augmentation called Splice, where task-relevant information is cut out from the environment and pasted over a distracting background.
strength_and_weaknesses: A strength of this paper is in its simple, easy to grasp idea - this would definitely help it be applied out-of-the box to a variety of RL methods.

A main weakness of this paper is in its novelty, which will be elucidated upon in the section below.  Furthermore, I sense that a major weakness of this paper is how restricted it is to the DMC task suite.  Firstly, the selection of weak and strong augmentations were predefined specifically for the DMC tasks (‘Known weak augmentations for all DMC tasks are: crop, translate, and shift’).  This limits the application of the proposed model to other domains for which there is no prior knowledge about the weakness or strength of augmentations.  It would be interesting if the authors could automatically uncover such properties of elements in a proposed augmentation set automatically - however, as such, it looks like this approach was specifically designed for the DMC task suite rather than being general.

Also, the numerical results of the proposed AugCL do not appear to outperform the existing art (in particular, SVEA) on the listed tasks, with the exception of Walker, Walk.  Given how design decisions were specifically selected for this domain, I would have expected more substantial performance improvements.

The claim that AugCL enables generalization to unseen environments would also be greatly reinforced if it were demonstrated across true unseen visual domains and tasks, rather than remaining in the DeepMind Control suite.  I believe that given the claims of the paper, a more thorough set of experiments would be useful in demonstrating the power of their proposed curriculum.
clarity,_quality,_novelty_and_reproducibility: I suggest the authors double check that citations are performed correctly according to the ICLR style guidelines - reading the citations throughout the paper was jarring and disorienting due to the lack of differentiation within the sentences.

I can identify no further issue with the clarity, quality, or reproducibility of this work.  In terms of the novelty, I believe the work is slightly novel; I believe that exploring a mix of strong and weak augmentations exists in prior art and the novelty of this work mostly revolves around this two-stage curriculum schedule.  In terms of the Splice data augmentation, I believe similar works have been done before [1, 2], where the agent and task-relevant features are extracted out and placed over a different distracting background.  There were no model or architectural improvements - the authors apply their curriculum on top of an existing SAC model.

[1] Stone et al. “The Distracting Control Suite – A Challenging Benchmark for Reinforcement Learning from Pixels”, 2021.

[2] Hansen et al. “Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation”, 2021.
summary_of_the_review: The proposed approach in this work is intuitive, and simple to apply to many current RL solutions.  However, I believe it suffers from explicit domain-specific information (distinguishing weak from strong augmentations specifically for DMC), as well as a lack of convincing experiments that fully showcase the strength of the model with respect to the claims.  I also believe the only substantial novel contribution in this work is the curriculum schedule (as the differentiation of weak vs strong augmentations is assumed, not learned/identified ad hoc per environment), which is limited in terms of its intellectual impact.  I therefore recommend that this paper undergo continued revisions before being considered for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of generalization in reinforcement learning. The authors seek to find algorithms for training a policy such that the policy will perform well on novel test-time environments. The authors propose a two-phase approach. In phase 1, the policy and value function are both trained with "weak" data augmentation. In phase 2, a second value function is trained in parallel with "strong" data augmentation and is used to update the policy. The method achieves SOTA performance on the Deep Mind Control Generalization Benchmark (DMCGB).
strength_and_weaknesses: The main strength of this paper is the empirical performance. Achieving the top performance on the DMCGB is certainly important. The method proposed in the paper beats baselines in all but 1 of the DMCGB tasks.

However, I do think there are a number of ways the paper needs to improve. Currently, I think the paper feels a bit rushed and a few edits and ablations would increase the impact of the paper substantially. I've included a list of comments and edits below.

- the authors are using the ICLR 2022 template

- Section 3.1 should just be $R$? not $R_t$

- Algorithm 1 - in lines 13 and 16, $L_\pi$ should take as input a $\phi$ as well to specify which Q function weights are being used in the equations in section 4.1 (I think in line 13, it should be $\phi^S$ and in line 16 it should be $\phi^W$?). I think this detail is important for understanding the algorithm.

- add equation labels for the equations in section 4.1

- at the top of page 5, I think the authors should reference a line different from "line 1"?

- I don't think Figure 1 is particularly informative since the arrows don't communicate which parts of the loss each model is being used for. I would personally prefer if the authors wrote out explicitly the loss for the policy and critic(s) in phase 1 and phase 2. For instance, the weak critic and target critic are updated the same way in phase 1 and 2, the only thing that changes in phase 2 is we begin training the strong critic and the policy uses the strong critic in its update (I believe, although currently ambiguous in line 13 of Algorithm 1). Currently, it's not obvious to me looking at Figure 1 that the updates for two of the networks are unchanged in phase 2, but that information would help a lot for internalizing the author's method.

- Table 2 and Table 3 - did the authors evaluate SODA and SVEA with splice? The authors should demonstrate that SODA and SVEA don't get the same performance boost as AugCL when switching from overlay to splice. Otherwise my sense is the main contribution of the paper is the use of splice augmentation, not the particular choice of training weak and strong critics in parallel which is what the authors emphasize in the method section.

- page 9 - ideally the authors would sweep across 3-5 values of M to get a sense of how sensitive the algorithm is to choice of M
clarity,_quality,_novelty_and_reproducibility: I think given that the method in this paper beats baselines handily on the DMCGB, it's definitely an acceptable quality method. Tables 2 and 3 suggest to me that splice augmentation is the main reason that the method won the benchmark, not the particular choice of losses that the authors use for incorporating splice augmentation. Including SODA and SVEA with splice augmentation in the tables would go a long way in clarifying the importance of each of the authors' contributions. With respect to originality, the method seems very similar to SVEA to me.
summary_of_the_review: Currently, I feel this paper is somewhat difficult to read and the method has limited impact outside of the DMCGB. However, if the authors address my comments in the "Strengths and Weaknesses" section, I will consider increasing my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
This paper proposes to decouple critics in an actor-critic scenario, where one critic is trained on so-called weak pixel-based augmentations and the other on strong augmentations. The argument is that while weak augmentations help stabilize training within an MDP, strong augmentations help generalize to a larger set of variations on that MDP but destabilize training.

- This induces four parameterized functions, $\pi_\psi$, $Q_{\phi^W}$, $Q_{\phi^{target}}$, and $Q_{\phi^S}$ ($W$ and $S$ standing for weak and strong).
- A new strong augmentation is introduced, called Splice, which pastes in natural image distractors forcing the model to pay attention to the relevant parts of the image.
- Training is separated into two phases. The method introduces a hyperparameter $M$, the number of steps for which training is in the first phase.
  - For $M$ steps, $\pi_\psi$ is trained on $Q_{\phi^W}$, $Q_{\phi^W}$ is bootstrapped to the EMA (of $\phi^W$) $Q_{\phi^{target}}$ with weak augmentations for both $\pi$ and $Q$
  - Then, $\pi_\psi$ is trained on $Q_{\phi^S}$ with strong augmentations, $Q_{\phi^W}$ and $Q_{\phi^S}$ are bootstrapped to the EMA (of $\phi^W$) $Q_{\phi^{target}}$ with weak and strong augmentations respectively.
- This method shows improvements on an array of environments from the DeepMind control benchmarks.

strength_and_weaknesses: The paper is relatively clear and well motivated. It introduces methods that make sense and motivates them well.

I have some issues with the empirical evaluation of the proposed methods:
- It seems that one of the the main drivers of performance wrt baselines is the augmentation used. Were any baselines trained with Splice?
- As the authors point out, the role of $M$ is still unclear, but its role is a central part of the method. 
    - Here's a suggestion for a plot: run experiments with varying $M$s and display results with $M$ on the $x$ axis and the AUC or average reward after training on the $y$ axis for two settings, in-distribution and out-of-distribution MDPs.
    - If what the authors posit is correct, with $M=0$ training should be too hard and not perform well on either MDPs, while with $M\to \infty$ (or simply $M>$total number of steps trained on) training should not generalize well to OOD MDPs but work well on in-distribution MDPs; and there should be a sweet spot in the middle where performance is ideal. Not only would this be a nice ablation, it would provide evidence that the _hypotheses_ that underlie this paper are correct.

Other than that, I'm also a bit unsure that invoking curriculum learning is correct here. While you could argue this is _technically_ CL, with two phases we typically only say the the first phase is pretraining. A curriculum on the other hand usually implies a continuously changing "difficulty level" and data distribution adjustment, and one that is _adaptive_ to the learner's performance--here $M$ is a hyperparameter.
clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear, but its novelty is somewhat limited. Perhaps I am lacking context here but if I understand correctly training two separate critics for different augmentations is not new, although this particular setup may be an improvement over past work. In-pasting natural image distractors is also not new, although the specifics here may matter for performance.

I believe I could reproduce this paper or at least its broad strokes relatively easily.

Note on formatting: it seems all the non-inline citations are missing parentheses. Are you using the `\citep{...}` command?

summary_of_the_review: While improvements in performance are always valuable, the scientific contribution here appears relatively minor to me. The proposed methods build on existing ones, but without deeply investigating them. This leaves us with higher numbers but not much new knowledge. 

As is I'm on the fence about accepting this paper, but I think it could be a much stronger contribution with a proper empirical validation _of the hypotheses_.

Update: the new results provide more clarity to this paper. In particular they test the hypothesis that $M$ matters, this is interesting in itself, but the other interesting aspect is that in terms of performance, it is the Splice augmentation that seems to offer most of the boost. I'm still leaning towards accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents AugCL, a method for learning to generalize to unseen environments in pixel- based RL using strong visual augmentations. Training is split into two phases of using weak and strong augmentations respectively to leverage a curriculum. The authors use a separate critic network for each phase and demonstrate how this disentangelment induces superior generalization. They further introduce an image augmentation method called Splice, and show how the combination of this with their method achieves state-of-the-art performance on the DMC-GB benchmark.
strength_and_weaknesses: > Strengths

The paper is well-written, well-structured and easy to follow.
The paper achieves SOTA performance on 13/15 environments of a popular benchmark compared to previous methods. The authors prove the relevance of the key aspects of their proposed methods by running an ablation study.

> Weaknesses

The authors conclude the value of M (timesteps for the1st augemntation phase) is an open question. I believe that the determining of this value is rather crucial, as the whole notion of curriculum (and hence the core method) relies on this parameter. The authors do report experimenting with 0 and 100,000 as values for M, which yielded poor results. I would like to see more analysis on how to select M and what effect this value beholds in other environments.
The paper talks about addressing how to effectively incorporate strong augmentations such as random convolutions and overlays, but then mainly focuses on their own proposed augmentation (Splice) in their experiments. Results using the Overlay augmentation are indeed presented, but 1) originate from a different dataset than which was used in SVEA and SODA, and 2) are not superior in all environments compared to previous works. It can thus be argued, that the proposed method AugCL does not necessarily surpass previous methods, but the new augmentation method (Splice) is the reason for improved performance. It would be interesting to see how much the proposed method outperforms previous methods using the same augmentations.
The contribution of the new augmentation method Splice is marginal, as it is not applicable in most other RL benchmarks. Since Splice is tailored for the DMC suite, it cannot be considered as a general augmentation method, which is why I would consider it a contribution of low significancy.
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel, but its impact is questionable, which reduces the overall quality of the paper.
The authors have provided their code with instructions, to be used in combination with the DMC benchmark repository, which significantly contributes to reproducibility.
The paper is written in a very clear and concise manner.
summary_of_the_review: The paper utilises a very trivial method to marginally improve generalization of pixel-based RL on one benchmark. The contribution is novel, however, it is not empirically proved to be beneficial in other benchmarks, due to which I don't find it to be particularly significant. Further, it is dubious to label the simplistic two-phased learning as a curriculum.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of generalization to novel visual environments in reinforcement learning. The authors propose a curriculum that incorporates both weak and strong augmentations to improve performance on tasks in the DeepMind Control (DMC) suite. The reviewers agree that the idea of using a curriculum to achieve better results in continuous learning (CL) settings is interesting, they felt like the impact of the curriculum was not established conclusively through the empirical evidence presented in the paper. In particular, the reviewers felt that the effect of the curriculum was confounded by other factors investigated in the paper, which made the overall story less clear. Additionally, the reviewers had concerns about the fact that the results of the proposed AugCL curriculum did not outperform existing methods by a significant margin on the tasks in the DMC suite, and the claim of generalization to unseen environments was not well supported. Although the reviewers appreciated the work the authors put into addressing their concerns during the rebuttal period, they did not feel like the paper was strong enough in its current state to publish at ICLR.
justification_for_why_not_higher_score: Although the idea of using a curriculum to achieve better CL performance is interesting, the authors did not provide enough clear empirical evidence for it. 
justification_for_why_not_lower_score: NA
Decision: Reject