Augmentation Curriculum Learning For Generalization in RL
Authors: Dylan Yung
Andrew Szot
Prithvijit Chattopadhyay
Judy Hoffman
Zsolt Kira
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
generalization
pixel-based RL
embodied learning

Abstract: Many Reinforcement Learning tasks rely solely on pixel-based observations of
the environment. During deployment, these observations can fall victim to visual
perturbations and distortions, causing the agent’s policy to significantly degrade
in performance. This motivates the need for robust agents that can generalize in
the face of visual distribution shift. One common technique for doing this is to ap-
ply augmentations during training; however, it comes at the cost of performance.
We propose Augmentation Curriculum Learning a novel curriculum learning ap-
proach that schedules augmentation into training into a weak augmentation phase
and strong augmentation phase. We also introduce a novel visual augmentation
strategy that proves to aid in the benchmarks we evaluate on. Our method achieves
state-of-the-art performance on Deep Mind Control Generalization Benchmark.

Under review as a conference paper at ICLR 2023
AUGMENTATION CURRICULUM LEARNING FOR GEN- ERALIZATION IN REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Many Reinforcement Learning tasks rely solely on pixel-based observations of the environment. During deployment, these observations can fall victim to visual perturbations and distortions, causing the agent's policy significantly degrade in performance. This motivates the need for robust agents that can generalize in the face of visual distribution shift. One common technique for doing this to ap- ply augmentations during training; however, it comes at the cost of performance. We propose Augmentation Curriculum Learning novel curriculum learning ap- proach that schedules augmentation into training into a weak augmentation phase and strong augmentation phase. We also introduce a novel visual augmentation strategy that proves to aid in the benchmarks we evaluate on. Our method achieves state-of-the tfef-hh-tttttt art performance on Deep Mind Control Generalization Benchmark.
INTRODUCTION
Reinforcement Learning (RL) has shown great success in large variety of problems from video- games Mnih et al.] (2013), navigation Wijmans et al.] (2019), and manipulation Levine et al. (2016); Kalashnikov et (0118) even while operating from high-dimensional pixel inputs. Despite this success, the policies produced by RL are only well suited for the same environment they were trained for and fail to generalize to new environments. Instead, agents overfit task-irrelevant visual features, resulting in even simple visual distortions degrading policy performance. A key objective of image-based RL is building robust agents that can generalize beyond the training environment.
Several existing approaches to training more robust agents include domain randomization Pinto et al. (2017); Tobin al. (2017) and data augmentation Hansen Wang (02T); Hansen et al. (202 b);; Fan et al. (2021). Domain randomization modifies the training environment simulator to create more varied training data, whereas data augmentation deals with augmenting the image observations rep- resenting states without modifying the simulator itself. Prior work shows pixel-based augmentation improves sample efficiency and helps agents achieve performance matching state- based RL Laskin et al. Yarats et al. (202I). Therefore, in this work, we focus data augmented generalization to visual distribution shift while the semantics remain unchanged. We specifically aim to do this in a zero-shot manner, i.e., where shifted data is unavailable during training.
Unlike for supervised and sulf-super image classification tasks, augmentation for pixel-based RL has demonstrated mixed levels of success, Prior work categorized augmentations into weak and strong ones based on downstream training performance Fan et al. (2021). Specifically, works define those allowing the agent learn policy with higher episodic rewards in the training environment than training without augmentation. Strong augmentations refers to augmentations that lead empirically worse performance than training with no augmentations. Classifying augmentations according to this definition is dependent on the task. For example, cut- out color has empirically been shown to be detrimental ("strong augmentation") for all tasks in Deep Mind Control Suite (DMC) Tassa et al. (2018), but is effective ("weak augmentation") for Star Pilot in Procgen Cobbe et al. (2019) as shown in Laskin et al.) Methods exist that attempt to automate finding the optimal weak augmentation
weak augmentations
per-task basis Raileanu et al. (020). but these still do not expand the effectiveness of many augmentations.
Many RL generalization methods leverage weak augmentation for better policy learning training and add strong augmentations in training for generalization to visual distribution shift [Hansen & Wang (2021) Hansen et al. (2021b): Fan et al. (2021). However, these methods suffer from strong

---

Under review as a conference paper at ICLR 2023
augmentation making training harder due the difficulty of learning from such diverse visual ob- servations, destabilizing training. This results in strong augmentations causing the agent to not learn policy with strong performance using weak augmentations alone. In this work, introduce a new training method that avoids the training instabilities caused by strong augmentations through curriculum that separates augmented training into weak and strong training phases. Once the net- work has been sufficiently regularized in the weak augmentation phase, it is cloned to create a policy network that is trained strong augmentations. This disentangles the responsibilities of the net- works into accurately approximation the Q-value (network trained on weak augmentations) of the agent and generalization (doing well on shifted test distributions). Crucially we separate the two net- works to avoid the destabilizing effect of strong augmentations. We also demonstrate the power of method under even more severe augmentation, namely new splicing augmentation that pastes relevant visual features into an irrelevant background. We show that our curriculum learning ap- proach can effectively leverage strong augmentations, and the combination of our method with this new augmentation technique achieves tfte-of-thheart generalization performance.
Our main contributions are summarized as follows:
We introduce Augmentation Curriculum Learning (AugCL), a new method for learning with strong visual augmentations for generalization to unseen environments in pixel- based RL.
A new visual augmentation named Splice, which by simulating distracting backgrounds helps prevent overfitting to task irrelevant features.
We demonstrate AugCL achieves state-of-the-art results across suite of pixel-based RL general- ization benchmarks.
2
RELATED WORKS
2.1 CURRICULUM LEARNING
Inspired by how humans learn, (Elman 1993) proposed training networks curriculum style by starting with easier training examples and then gradually increasing complexity as training ensues. (Bengio et al.) 2009) showed that this training style yielded better generalization results faster, and by introducing more difficult examples gradually, online training could be sped up. This ideology has shown to be transferable to RL across varying types of generalization (Cobbe al., 2019). (Wang et al. 2019), (Florensa et al. 2018). (Sukhbaatar al., (017), but to our knowledge has never been explored for generalization to visual perturbations pixel-based RL.
2.2 RL GENERALIZATION BENCHMARKS
There are many benchmarks designed for evaluating agents under different distribution shifts CChat-
topadhyay et al.
2021 Dosovitskiy et al.) 2017 Stone et al., 2021 Zhu et al., 2020 Li et al. 2021
Szot et al. [022]] We chose Deep Mind Control Generalization Benchmark (DMC- GB) (Hansen &
Wang 202 1) as the current SOTA methods have been benchmarked on this, allowing us to compare
directly to the results shown in previous works. DMC-GB offers different generalization modes: easor color hard, video easy, and video hard. These modes can be applied to all DMC tasks, and visual examples can be seen in Figure Color casy is not benchmarked on as it is considered solved (Hansen & Wang 2021). Color hard dynamically changes the color of the agent, background, and flooring. Video easy changes the background to another random image, and video hard will change both the background and floor to random image. Under these extreme perturbations, the agent must learn to identify the relevant visual features to the task in order to maximize reward.
2.3 GENERALIZATION IN VISUAL RL
There have been many advances in visual RL generalization. In this section, we will briefly sum- marize each method. SODA (Hansen & Wang 2021) leverages an approximate contrastive loss focused on minimizing the distance of embedded vectors of the same state augmented with crop and strongly augmented copy closer together in dimee-dimeniionall space. SECANT (Fan et al. 202T) trains an agent using crop and then leverages that agent's 's policy for training a new agent under strong augmentation in an imitation learning fashion. The prior SOTA approach DMC-GB was
2

---

Under review as a conference paper at ICLR 2023
SVEA (Hansen et al. 2021b). SVEA leverages data-mixing, which modifies the critic loss Equa- tion (T) to a weighted weak augmented and strong augmented temporal difference error (Sutton & Barto 2018). We take inspiration from the SVEA in method by bootstrapping Q target network that has gradient updates under weak augmentation for critic trained under strong augmentation.
3 BACKGROUND
3.1 PROBLEM: GENERALIZATION IN PIXEL CONTROL TASKS
We formulate problem as interacting with Markov Decision Process (MDP) defined as the tuple M = (S, A. P, R, ?) for state-space S, action space A. transition distribution P(s"|s, a), reward function R(s, a), and discount factor 7. In our setting, images observability. We therefore represent the state with A stacked observations to maintain the Markov property. The goal is to learn policy ((ae(s)) which maps states to a distribution over actions to T (', az)
O offer only partial
maximize R = == EYNT
In this work we focus on learning a policy that generalizes new MDPs M. M has new observa- tion space, but other elements of the MDP are unchanged. Policy is then evaluted on the return on new MDPs, without any additional samples from the new MDPs for updating x.
3.2 SOFT ACTOR-CRIIE CRITIC
SAC (Haarnoja al. (017) is considered the state-of-the-ee art (SOTA) off-policy RL algorithm for most continuous control tasks. SAC trains an actor network (a)()))) to take actions and critic network Qo(St, ae) predict state-action values. SAC follows the maximum entropy RL prin- ciple which is to maximize actor entropy while maximizing expected reward au [E. r2 + {H((s(-(s))]] The critic is trained by minimizing the temporal difference error using samples (s,, S1+1, Te) ~ D, where D is a replay buffer storing the agent's previous interactions with the environment during training. The critic parameters can be updated using an approximation to the expected rewards. Let & be the critic parameters and grerget be another set of critic parameters for the bootstrapped value target.
fo( B.n a.) - (r. VV o (target)))
The value of the next state is estimated through the single-step boostrap target based on the target critic network parameters:
V($+41: (5441:0farget) = Egtwa (Q (St+1, a') - a log Ty. (a'(s(44)
(2)
Target critic parameters (target are typically updated exponential moving average (EMA) of the regular critic parameters 6. The actor network learns a policy by minimizing the following.
(x -- (Q a) log T. (a [c]s
Typically, Th is represented by Gaussian distribution for continuous control and updates are made via the reparameterization trick (Kingma & Welling, 2013) and a is learned relation to entropy. All methods in the experiments section use the same base neural architecture and SAC with EMA updates of Q-target network from the critic network. key thing to note that the actor and critic share the same CNN encoder. This is crucial to the success of our's and previous methods. More details in appendix
METHOD
4.1
AGENT ARCHITECTURE
The key differences between AugCL and previous works are: 1) We train weak and strong aug- mented network in parallel. 2) We train only on weak augmentation for the early phases training. 3) We bootstrap the strong augmented network from weakly augmented Q target network. The driving intuitions behind this is strong augmentations incur non-zero degradation to policy learning but help with generalization. Hence mitigate this issue, have separate network trained only on
3

---

Under review as a conference paper at ICLR 2023
Weak Augmentation Phase (Train Step
Train Step M
Strong Augmentation Phase
(Train Step >= M)
π V
Q GW
Q @
Loss
Q li
Copy
Qtorget
Week
Q pss
Strong Critic Loss
EMA
Figure I: Neural architecture and tensor flow across different phases for AugCL.
strong augmentation and update the target network using EMA from the weak augmented network. This duo circumvents the policy degradation by allowing the weak critic and Q target network to learn state and action value approximation without being impeded by strong augmentation. Then by bootstrapping this Q target network with an accurate approximation strong augmented network learns generalize under strong augmentation. The weak augmented pre-training is required to regularize the CNN from biasing itself to high-frequency features. We show both necessary achieve SOTA performance Section 5.3
the value function, the
AugCL builds off of SAC, learning a critic and policy network as shown in Figure | The critic network (s ax) takes input a stacked sequence of image observations, and an action produces the expected return for taking action d₂ in state st. We also learn parameterized policy (s;) that outputs normal distribution parameterized by learned mean and variance which then samples an action for the current state. Q6 and मe share visual encoder (s:) which takes as input the high- dimensional image observations and produces a oow-dimessional state encoding. The parameters of d and & both include the encoder parameters μ4, which are updated part of both the policy and critic losses.
AugCL trains Q6 and TT through the standard SAC losses with random image augmentations. We sample augmentations f from a distribution over augmentations F. then use ʃ to augment the states s, in the SAC losses from Section 3.2 We augment only the current state sı in the SAC loss similar to (Hansen et al.) 2021b) since we found this improves stability as shown see The resulting losses for the critic and policy with the augmentations
respectively:
Lo (0: starget F) = R. (f))) - (r, + 7V (arget) ))}
(4)
(+b d, -Ea- (f(s), a log o. (a))(((s)))
(5)
Notice that the losses are defined relative to a distribution over augmentations F. The choice of this augmentation distribution is an important consideration in the algorithm's 's stability and robustness to new MDPs. Prior work breaks up augmentations for pixel-based control into two classes: weak augmentations and strong augmentations.
Weak Augmentations (denoted as FW) are augmentations that help stabilize and improve training performance in the source MDP M, but are insufficient for generalization to new MDPs M.. (Cetin et al. 2022) shows that training with weak augmentations is important to prevent overfitting to high- frequunqqccc features in the image space when learning from bootstrapped targets in actor-critic methods. The consequences of this overfitting have been shown to cause the critic network to overfit to its own predictions, and decrease in correlation to Monte Carlo returns as training ensues. Train- ing with weak augmentations is therefore important part of any actor-critic control-from-poe

---

Under review as a conference paper at ICLR 2023
method, such AugCL. Known weak augmentatiins for all DMC tasks are: crop, translate (Laskin et al.) and shift Yarats et al. 2021).
Strong Augmentations (denoted as FF) are augmentations that help improve policy performance in new MDPs M. Visual perturbations such as random color changes of the environment can be simulated with strong augmentations such as: random convolution and random color jitter. While distracting backgrounds can be simulated with mix-up (Zhang et al. 2017). these augmentations difficult train with, as they increase the difficulty of the learning problem due the duo of stochastic parameter sampling for JS and high visual variance between samples. AugCL addresses how to effectively incorporate strong augmentations such as: random convolution, overlay (variation of mix-up), and our novel augmentation into training.
4.2 AUGCL: CURRICULUM LEARNING WITH STRONG AUGMENTATIONS
As mentioned, strong augmentations are detrimental to learning but important to train with for gen- eralization performance. The key idea of our method therefore to leverage curriculum learn- ing (Bengio et al.) 2009) to avoid this destabilization. Specifically, AugCI. defines curriculum over augmentations to enable better training and generalization. It well known that CNNs are inher- ently biased to high-frequency features (Geirhos al.) 2018 Jo & Bengio 2017). the consequences of this in RL is a lower average episodic reward in the train environment. AugCL uses weak aug- mentations early in training to regularize the CNN. Then later in training AugCL introduces strong augmentations to improve the robustness of the policy. We train two separate networks parallel we believe that strong augmentation incurs non-zero degradation to policy learning, as shown in Section 5.3
Algorithm AugCI.
1: ΦW, .b. ofaagge critic parameters, policy parameters and Q-target parameters
2: a, C. actor learning rate, critic learning weight and momentum encoder weight
3: M: Update step to switch to strong augmentation
4: for timestep /. -= 1,..,, do
5: ag ~ Th. (-(s,)
> Sample action
6:
81+1 (l. at)
>> Step environment
7:
B - ((s,, at, (Stt oe), 81+1)
Add transition to replay buffer
8:
ift = M then
9:
Φc =W"
Clone critic
10:
{s; Og,
1, N} ~ B
> Sample transition
11:
@W & @W
BV VoLQ( FFW (gargett)
Weak Critic loss
12:
if 2 M then
13:
s + v - a Cr (vi; FS)
A Strong Actor loss
14:
S t 65 - 3V L (05 JJS (argget))
1 Strong Critic loss
15:
else
16:
t v- - an (v; FW)
Weak Actor loss
17:
darget + (1 - S)otarge + C
Q-target EMA update
AugCL is described in Algorithm AugCL begins by acting in the environment with Ty. and then adding the observed transition to the replay buffer (lines 5-7). We then sample data batches from the replay buffer for updating the policy and critic. Our curriculum learning schedule breaks the updates into two phases. For the first M policy updates, AugCL in the weak augmentation phase and updates the critic and policy from weak augmentations alone (lines 11,16). Target critic parameters gfarget are updated as exponential moving averages of the learned critic parameters (line 17), and used in the bootstrap term of the critic loss. The purpose of the weak augmentation phase to stabilize policy learning. Prior work shows that it is easy for the critic to overfit in image- based RL and weak augmentations 2022]. However, the weak augmentations do not make the policy robust to new visuals. Improving generalization performance is the purpose of the next strong augmentation phase.
important achieve strong training performance (Cetin et al.)
Then, after M policy updates, AugCL switches to the strong augmentation phase and incorporates strong augmentations into training (lines 12-14). A new strong critic network de is copied from
5

---

Under review as a conference paper at ICLR 2023
the weak critic network Φ (line 9). Now, the weak critic network is updated like in the previous phase by training with weak augmentations. However, the separate strong augmentation network with parameters 6 is now trained with strong augmentations (line 14). The policy is also updated with the strong augmentations (line 1)- Separating the strong and weak augmentations into two networks is important for stability. The weak critic helps stabilize bootstrap targets by leveraging weak augmentation to better approximate the state, action value function while the strong critic focuses on generalization performance. Previous methods have attempted this parallel training of the weak and strong augmented network, but with little success (Fan et al. 2021). We show that this due to not regularizing the CNN encoder first in see sec [..3 A figure of the architecture and the flow of tensors representing input and output of each neural layer can be seen in Figure
The advantage of AugCI. separating strong augmentations into later phase of learning it does not require a delicate balance between potentially conflicting losses from strong and weak augmen- tations. SODA and SVEA incorporate strong augmentations as an auxiliary learning signal that is always applied in conjunction with learning an accurate approximation of the state, action value from weakly augmented data. By learning from both data at the same time, the networks must contend with the trade off between stronger augmentations improving generalization yet harming training performance. The auxiliary objective in SODA may suffer from gradient interference from the confficting losses as the critic network is optimized learn accurate state, action value ap- proximation, and a contrastive loss in parallel. SVEA suffers from similar issue in that it requires a hyperparameter balance the combination of losses from strong and weak augmented data. On the other hand, AugCL disentangles the responsibilities of state, action value approximation, and gener- alization between the weak augmented critic and the strong augmented critic respectively. We em- pirically demonstrate this by showing AugCL, performs better than SVEA and SODA on variety of benchmarks and has minimal train environment performance degradation, as shown in Appendix [D] Also note SODA and SVEA only pass weakly augmented observations to the policy network during training, the issue with this is that no ( (s+) # To TFF (st)). AugCL does not suffer from this issue as we pass JS (s:) through the policy network at train time, and interestingly we find train performance still improves and converges to marginally worse performance the train environ- ment than training with weak augmentation alone as shown in appendix We summarize the key differences between AugCL and prior work in Appendix Table 5
4.3 SPLICE AUGMENTATION
Since AugCL is well suited to train with challenging strong augmentations, we introduce a novel augmentation called "Splice" to improve generalization in visual RL. RL generalization benchmarks that incorporate background distractions (Hansen & Wang 2021 Stone et al. (022)) are challenging for state-of-the- visual RL approaches. The standard solution to introduce variation of mix- up augmentation (Zhang et al., 2017) to RL training (Hansen & Wang 2021 Fan et al.] 2021 Hansen al., (022bb) (Hansen & Wang 202T) theorized that previous strategies failed to adapt to severe background distractions because task-relevant visual features such as the agent's shadow were removed.
Our new augmentation Splice solves this issue by pasting relevant visual features into an irrelevant background. This explicit separation of tesk-relevant versus task-irrelevant features helps general- ization. Specifically, we mask out all non-relevant parts of the visual observation through a seg- mentation mask which is available in the simulation. We then replace all the non-task parts of the image with a random background image. We use COCO (Lin et | (204)) for our experiments as the background replacement images. Further discussion and PyTorch (Paszke et al.] 2019) style code for Splice and a visual example is provided in Appendix F
EXPERIMENTS
We now evaluate AugCL and baselines on how well they can generalize to visual distribution shifts in the DMControl Generalization Benchmark (DMC-GB). In Section 3.1 we describe the experi- mental setup for how our method and baselines are configured. Next, in Section 5.2) we show that AugCL achieves state-of the-tr performance in the majority of settings in DMC-GB. Finally, in Section 5.3
analyze what hyperparameters are necessary for the benefits of AugCI..

---

Under review as a conference paper at ICLR 2023
SODA (conv)
SVEA (conv)
AugCT. (conv)
Domain, Task
CURL
RAD
DrQ
PAD
Walker,
445 = 99
400 + 61
520 91
468 - 47
697 + 66
760 - 145
890 + 36
Walk
Walker,
662 + 54
614 + 88
770 + 71
797 + 16
930 + 12
942 + 26
956 + 17
Stand
Cartpole,
454 + 110
590 53
586 52
630 63
831 + 21
837 + 23
852 +
Swingup
Ball In Cup,
231 + 92
541 +: 29
365 + 210
563 + 50
892 1 37
961 1
957 1: 18
Catch
Finger,
691 12
667 + 154
776
134
803
72 901 + 51 977
980 +
Spin
Table I: Results from DMC-GB benchmark color hard. All methods are evaluated on 5 seeds over 30 episodes. The mean and standard deviation provided. AugCL outperforms baseline in out the 5 tasks.
5.1 EXPERIMENTAL SETUP
Environments and Evaluation: The purpose of our experiments is to evaluate how well policies trained with various methods can generalize new visual disturbances. All methods are first trained in source environment without any visual disturbances. We then evaluate the trained policy in the same environment but with random visual disturbances. DMC-GB tests how methods to random colors, backgrounds. and camera poses. All methods are trained for 500,000 frames and evaluated 5 tasks from DMC-GB in three different evaluation settings from DMC-GB (color- hard, video-easy. and video-hard). The 5 tasks from DMC-GB used in this paper are described in Appendix Table [ We report the mean and standard deviation across 5 seeds per method, where each seed is evaluated by taking the average episode return across 30 episodes. For table and table 3 we added training with SVEA during the weak augmentation phase.
generalize
We compare AugCL against other recent bixed-based RL methods, some of which were explicitly designed for learning robust policies that can generalize unseen environments. Specifi- cally, we compare against CURL,, RAD, SVEA, SODA, DrQ as well as PAD (Hansen et al. 2021a), which adapts the test environment using self-supervision. Hyperparameters and baseline results are taken from (Hansen et al.) 2021b). We don't 't compare to SECANT as it requires double the train- ing frames to all other baselines and requires training models sequentially. Also note that SVEA and SODA augment each batch twice, thus doubling the data the agent trains whereas AugCL only uses a single batch.
Baselines:
Data Augmentation Setup: We apply random shift Yarats al. 1021 for AugCL. For all experiments, we selected M = 200, 000 for AugCL, meaning we first perform 200k updates in the weak augmentation phase before switching the strong augmentation phase. All hyperparameters shared between AugCL. and baselines are kept the same. Random convolution produced the best results prior works on color hard and overlay for video DMC-GB benchmarks (Hansen et al. 2021b), and we therefore use those augmentations for their respective benchmarks. Note that DrQ AugCL and SVEA all use shift as their weak augmentation and CURL, PAD, RAD and SODA use crop. This is important to note as shift has been shown to give stronger empirical results than crop in DMC tasks (Yarats al. 202T). "Overlay" in tables 2 3 refers to (Hansen & Wang 2021 version of mix-up. The original SVEA and SODA paper use the Places dataset (Zhou al. 8a) for Overlay, but during the time this paper was written Places was unavailable due to maintenance, instead, we used COCO for AugCL. We felt this
weak augmentation
fair comparison as long as both datasets were different from RealEstate 10k (Zhou et al. 2018b), which is used by DMC-GB. A full list of hyperparameters can be found in table 4 We apply random shift (Yarats et al. 2021) as our weak augmentation for AugCL and set M = 200, 000, which we determined empirical ly. All overlapping ceper-parameters between methods are kept the same.
We also include results using Splice on the DMC-GB video easy and video hard benchmarks. In the DMControl tasks, we segment out the agent by filtering, converting the RGB image to HSV, and

---

Under review as a conference paper at ICLR 2023
Domain, Task
CURL
RAD
DrQ
PAD
SODA (overlay)
SVEA (owerlay)
AugCL (overtay)
SODA (splice)
SVEA (splice)
AugCI (splice)
AugCI SVEA (splice)
Walk
556 1. 133
600 63
682 100:1:0
717 1. 79
708 38
819 1.
839 625 1: 29
882 1163
679 1:3
904 1: 29
Walker,
Sunnd
8G2 75
745 + 146
873 83
1336 + 20
903 13
961
956 875 + 71
969 +-4
968 +
972
Cunpole,
Swingup
404
738
795 12
764
Ball Cup.
Catch
316 1: 119
681 1:
318 1: 157
436 1: 55
873 36
871 1: 106
910 3:
907 -00
963 11
969 1
967 1.3
Finger.
602
400 633 691 605 97
752 +-40
888 975
983 975 +
Table Results from DMC-GB benchmark video easy generalization benchmark, AugCL, with the new splice augmentation outperforms prior work in 4 out of 5 of the tasks.
Domain, Task
CURL
RAD
DrQ
PAD
SODA (overlay)
SVEA (overlay)
AugCL (overlay)
SODA (splice)
(splice)
AugCL (splice)
SVEA (splice)
Walker,
Walk
58 18
56 9
104 + 22
93 +
381 72
377 +33
364 65
619 + 25
861 + 50
864 + 34 888 + 30
Walker.
Stand
5
872
Carpole,
Swingup
114 15
110 16
188 t.
123 1: 24
429 64
393 1t 45
390 2:
394 1:90
776 1: 28
7/22 :: 35
784 1: 16
Ball Cup.
115 33
100
174
034 125
200
806
916 21
905 35
Finger.
Spin
27 27+
322 +
71 +
56 18
302 +
335 58
312++99
873 + 163
948 + 20
962 + 24
960 + 11
Table 3: Results from DMC-GB benchmark video hard. AugCL with the splice augmentation out- performs baselines in all of the tasks.
then taking pixels with HSV values only greater than threshold. We set a consistent value threshold of 0.6 for all tasks to remove all aspects of the image, including the shadow, leaving only the agent. The hue threshold was 0 for all tasks except "Cartpole, Swingup" which required the hue threshold to be set to 3.5 due to the background being mix of lighter and darker blues in these environments. The saturation threshold was Appendix Table
for all tasks. The full list of hyperparameters can be found in
5.2 DMC-GB RESULTS
Firstly, Table 1 shows that AugCL outperforms all baselines in 4 out of 5 tasks in DMC-GB color- hard environments. This further closes the gap between the performance of the policy from training and its generalization performance under color hard are close to matching the current SOTA the train environment thanks to SVEA and AugCL shown in Table 7 Despite SODA and SVEA using the same augmentations as AugCL. and also being designed for generalization in pixel-based RL, AugCL outperforms them in evaluation return.
the test environment. Ball In Cup, Catch and Finger. Spin
Next, in the DMC-GB video easy and video hard environments AugCL again outperforms baselines in almost all of the settings. AugCL outperforms baselines in out of tasks in video-easy (Table and in 5 out of 5 tasks in video hard (Table 3). A combination of the splice augmentation and AugCL performs best. Splice combined with AugCL does well on "Finger, Spin" under video easy as it's only 1 average episodic reward off from the train environment SOTA seen in Table 7 We theorize that Splice performs better than Overlay because Overlay a weighted sum of pixels from the state image and an irrelevant image. (Laskin et al.) Cetin et al. (002) showed the utility of weak augmentation was that regularized CNN improved spatial attention mapping to task relevant features. Overlay may impede this process by making task relevant features less visible.
5.3 ABLATIONS
We analyze two design choices of AugCL: the curriculum and using separate critic networks for weak and strong augmentations. We use Random Convolution as the strong augmentation for all variations of AugCL. in this section. No Pretrain in Figure represents AugCL., but without the
8

---

Under review as a conference paper at ICLR 2023
Walker Walk
Walker Walk
800
Episode Reward
Episode Reward
600
400
400
No Pretrain
No Pretrain
Single Critic
200
Single Critic
AugCL
AugCL
200000 250000 100000
150000 400000
450000
500000
200000 250000 300000
350000
400000
650000 500000
Step
Step
(a) Train
(b) DMC-GB color hard
Figure 2: No Pretrain: No weak augmented pre-training on the strong network. Single Critic: A single critic is used for training under weak and strong augmentation. The line represents the mean over seeds, and the shadow represents variance. Figure 2a shows performance on the train environment, and performance during training DMC-GB color hard. The lines represent averages and the shaded regions the standard deviation of the results across seeds.
copying of weights to the strongly augmented network at step M (omitting line 9 in Alg. []. Single Critic: represents having a single critic network for both strong and weak augmentation (omitting line 11 and line 17 becomes gargget + (1 O CØ in Alg.
As we can see without the pre-training even while bootstrapping from Q-target network updated using a weakly augmented network. No Pretrain is much higher variance across the seeds and not able to match AugCL's performance on the walker walk task, thus showing the importance of first regularizing the CNN on weak augmentation, which motivates the curriculum. While Single Critic performs much better than No Pretrain, to a lower solution than AugCI. on both the train environment and the test environment. We can that while Single Critic is improving the policy it is learning it is does not perform well as distributing learning the state, action value to the weakly augmented network, thus showing the non-zero destabilization strong augmentation incurs.
see it's much less sample efficient and converges
We also experimented with setting M = O and found it was unable to learn as the strong critic couldn't learn useful representation and could not bootstrap the target network's predictions to improve learning. We include further exploration of selecting M in appendix AA| This was indicated to by the episodic reward not improving as training continued. We believe this also points to- wards the importance of weak augmentation regularization early in training. We helieve that these experiments are clear evidence that strong augmentations do indeed incur a non-zero degradation to policy learning and that the key to getting good generalization performance is to disentangle the strong and weak augmentation as AugCL does, which is not possible without the curriculum as an unregularized CNN has difficulty learning task relevant representations.
CONCLUSION
AugCL. shows tremendous improvements to generalized environments by disentangling strong and weak augmentations into their respective networks. The combination of AugCI. and Splice have substantially improved performance on DMC-GB, giving new SOTA. We also effectively show the importance of weak augmented pre-training for parallel weak and strong augmented network training, highlighting the missing ingredients to previous attempts. An issue with our method is selecting the optimal M is still open question. M 100, 000 yielded much worse results and we theorized that the CNN was not regularized enough. This tricky balance can lead to significant changes in results and hope find more developed method for selecting M.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning. ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 1553330 URL hattp:://ci.orq// 1145/ 1553374 1533380
Edoardo Cetin, Philip J Ball, Stephen Roberts, and Oya Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pp. 2784- 2810. PMLR, 2022.
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Ani Kembhavi. Robustnav: To- wards benchmarking robustness in embodied navigation. International Conference in Computer Vision (ICCV), 2021.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. arXiv preprint arXiv: 190:::992.01555 2019.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings the 1st Annual Conference on Robot Learning, 1-16, 2017.
Jeffrey L. Elman. Learning and development in neural networks: the importance of start- ing small. Cogmition, 48(1):71-99 1993. ISSN 0010-0277. doi: https://doi.org/10 1016000100227703990055-4 URL https://www. http:///www.cc.eeccerrecccco htwp://ccirett..cooccceeeeeeee
atticle/p110000027790800
Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 3098. PMLR, httssss/vvv/////
18-24 Jul 2021. URL
html
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume Proceedings of Machine Learning Research. pp. 1515-1528. PMLR, 10-15 Jul 2018. URL httpp:://pooceeeaaiii
mmr.
#reas//80//lozensall. html
Robert Geirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenct-trained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness, 2018. URL httphs:////axii http:///aw.....r/a/////111111111
112111111//111.112111
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. 2017.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. In International Conference on Robotics and Automation, 2021.
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Picter Abbeel, Alexci A. Efros, Lerrel Pinto, and Xiaolong Wang. Seff-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021a.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Conference Neural Information Processing Systems, b021b.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities, 2017. URL httpt:ttt//attt fitts:///ariiv www..orgggboo... /tt.bb///1111111
10

---

Under review as a conference paper at ICLR 2023
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. In 2nd Annual Conference Robot Learning, CoRL. 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 651-673. PMLR, 2018. URL http://
proceedings mlr. wwwpppwwwss...e https////88/aaaaaaaoooolaaaa.a.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes, 2013,
URL https:
/ //arxiv.org abaaaaas/1/ 6114
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learming with augmented data. arXiv:200.149900
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep vi- suomotor policies. Journal of Machine Learning Research, 17(39)11-40, 2016. URL http:
7/jmlr. wttp::ppppppppp.////oooooooap///
522 html
Chengshu Li, Fei Xia, Roberto Martín Marttn, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu, Hy- owon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In 5th Anmual Conference on Robot Learning, 2021. URLhttpss: Dectps://openreviieee http://wewettoooodddeee/eeeeeaee
JNJRRR
Tsung- Yi Lin, Michael Maire, Serge Belongie. Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zimicck and Piotr Dollár. Microsoft coco: Common objects in context, 2014. Utttssss://aaxx wwt.a//iiiiir/...aa....ba.a
1405. 0312
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL
httpps://axxt wttg/as////////e/e 5602
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Newral Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http: http::/se.ppppee...errpppppoopep pdf
library..
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning, 2017. URL https://arxiv. owtpiis/rrg/////o/oo/o/
1710.06542
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus, Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. 12862, 2020.
Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3674-3683, 2020.
Austin Stone, Oscar Ramirez, Kurt Konolige. and Rico Jonschkowski. The distracting con- trol suite challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play, 2017. URL
https::/aaaa http///gg//////ooo///tt///.///// 7703.44 05407
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduciinn. The MIT Press, 2nd.
second edition, 2018. URL
html

---

Under review as a conference paper at ICLR 2023
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Von- drus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira. Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 00690, 2018
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world, 2017.
URL https:t/axxii http:/iivvrrggaaee hblgg/bb/1773... 06907
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions, 2019.
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.000 2019.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representa- tions, 2021. URL
risk minimization. arXiv preprint arXiv: http://wwwweeee/oommmm/e////// TTGaa
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 40(6)::452-14466 1464, 2018a. doi: 10. 1001109/TAMM2221777220000
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018b.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martnn robosuite: A modular simu- lation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. /2293, 2020.
A CHOICE OF M ON PERFORMANCE
We explored how different M selections effect AugCL in figure between M and average episodic reward. We see on the test environment the relationship between M and test environment performance is parabolic, with performance peaking at M=100k or 200k on the test environment. While it seemed that 100k and 200k for M gave the same performance higher and lower values of M had much lower average performance. We theorize this is due to striking a good balance between regularizing the CNN with weak augmentation and then training to adapt to the strongly augmented version of the environment, which requires lot of frames to approximate.
3 We found a parabolic relationship
B DRQ VS RAD VS NON-NAIVE AUGMENTATION
In this paper we also explored the RAD style of augmentation Laskin et al. versus DrQ Yarats et al. (202)) and Non-Naive Augmentation Hansen al. (2021b). In order to get the best generalization results possible we required achieving best results possible on the train environment as that acts as an upperbound to what can be achieved during generalization. We theorized the main strengths of DrQ stem from the shift augmentation introduced and the multiple sampling are for mitigating the issues introduced by RAD style augmentation. We show in Figure 5 that Non-Naive Augmentation does indeed perform better equal to DrQ and RAD style augmentation in terms of average episodic reward on the train environment at the end of 500,000 train steps and is more sample efficient on out of 5 tasks. RAD is equivalent to "DrQ 1k1m" in Figure This is exciting development as
12

---

Under review as a conference paper at ICLR 2023
600
650
01
01
0
100000 200000
300000
#C0000
SC0000
0
310000 200000
300000
#300C0
300000
(a) Train
(h) DMC-GB color hard
We show the effects of M selection on AugCL by choosing [0, 100k, 200k, 300k, 400k, 500k]. AugCL. is trained with random convolution as the strong augmentation on Walker, Walk for total of 500k frames. Blue circles represent performance of different seeds using the average episodic reward over 50 runs and the red x represents the mean across all 3 seeds.
Figure 3:
sampling multiple augmentations of images can be costly especially with off-policy methods which known take multiple days train. Though more exploration is required as we only ran this assessment on 5 tasks in DMC, we encourage practitioners of mixel-based RL to leverage Non-Naive Augmentation over the current alternatives. Results shown in Figure
C NEURAL ARCHITECTURE
In this section go into more detail about neural architecture. Neural architecture adopted from [Hansen et al. (2021b)]. A Convolutional Neural Network (CNN) encoder is shared between the and critic network. The encoder is only updated with respects to the Equation (I) and is detached during back propagation through Equation 3 The encoder is implemented with an 11- layer CNN encoder that takes a stack of RGB frames rendered 84 84 x and output features of size 32 21 X 21 where 32 is number of channels and 21 x 21 spatial feature mapping dimension. convolutional layers use 32 filters and
kernels. The first convolutional layer has stride of 2, while all others have stride of 1. The CNN encoder is shared between the actor and critic and followed by linear projection layers for both the actor and critic. The linear projection layers consist of three fully connected layers with hidden dimension of 1024. All hyperparameters shown in table
D AUGCL TRAIN ENVIRONMENT PERFORMANCE
We believe key aspect generalization includes maintaining the best policy possible on the train environment as well. We include in Table O train environment performance of AugCL across dif- ferent strong augmentations we benchmarked on. We also include results for non-naive shift on the train environment as well upper bound to what all the generalized methods can achieve in Table
E TASKS DESCRIPTIONS
In this section we include table with brief description, dimensions of the action vector and whether the task is dense or sparse reward task in Table g
F MORE DETAILS ON SPLICE AUGMENTATION
The inspiration for Splice came when we noticed that in many robotics task the relevant visual fea- tures had higher brightness. We noticed that DMC fit this criteria well the ground and background
13

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Frame Rendering
84 x 84
Frames Stacked
3
Random Shift
pixels
M
200,000
Action Repeat
(finger), (cartpole), (otherwise)
Discount Factor
0.99
Episode Length
1000
Learning Algorithm
SAC
Number Of Frames
500,000
Replay Buffer Size
500,000
Optimizer (3)
Adam(3 (2=0.999)
Optimizer (a)
AdammBB==0.. 8y=0.999)
Learning Rate (0)
1e-3
Learning Rate (a of SAC)
le-4
Batch Size
128
S Update Frequency
2
Momentum Coefficient Seeds
0.05(encoder), 0.01(critic)
[0.4]
Table 4: Hyperparameters used for all experiments
Auxiliary
Method
Weak
Neural
Auxiliary
Modified Critic Loss
Uses Strong Augmentations
Augmentation
Architecture
loss
AugCI.
Shift
Additional critic network
Additional Critic loss
NIA Data-mixing
Yes
SVEA
Shift
N/A
N/A Contrastive loss N/A
Yes
SODA
Crop
Projection network
N/A
DrQ
Shift
NIA
Averaged Q & Q-target
No
CURL
Crop
NIA
InfoNCE
N/A
No
PAD
Crop
Self supervied network
Adaptive self-supervision
N/A
No
Table 5: This table summarizes what we assessed as key differences and similarities between meth- ods. "N/A" stands for "Not Applicable."
tended to be dark blue, while the agent is combination of bright colors (typically yellow and bright blue). Splice converts an RGB image to HSV color space then sets a threshold for hue, sat- uration and value. We use kornia [Riba et al. [2020]] for color space conversion. Hue represents color, saturation represents chromatic intensity and value represents brightness. If all values in cell in the HSV converted image exceed the preset thresholds then they are imparted on a new im- age. In our
splice out the agent and paste it onto randomized background. Laskin al.) Cetin et al. (2022) show weak augmentation leads to a better spatial attention mapping of features the agent control like the robot over high frequency features like the background and flooring. Splice has the ability to impart human prior knowledge about the tasks through tuning the thresh- olds. By tuning the thresholds accordingly the user can parse out only the relevant visual features
Strong Aug
Walker, Walk
Walker, Stand
Ball in Cup, Catch
Cartpole Swingup
Finger, Spin
Conv
894 36
958 13
965 + 6
853 +
979 + 9
Overlay
903 + 26
969 + 7
964 + 5
869 + 10
967 + 9
Splice
878 + 38
958 + 7
962 + 6
865 + 13
976 + 15
Table 6: Train environment performance at the end of 500,000 train steps of AugCL with varying strong augmentations. The mean and standard deviation over seeds where each seed is evaluated using the mean of 30 episodes done for each seed.
14

---

Under review as a conference paper at ICLR 2023
(a) Training environment (walker).
(b) Training environment (cartpole).
(c) color hard environment (walker).
(d) color hard environment (cartpole).
(e) video easy environment (walker).
(f) video easy environment (cartpole).
(g) video hard environment (walker).
(h) video hard environment (cartpolc).
Figure 4: Image taken from Hansen et al. [2021b) with examples from cartpole and walker of DMC-GB.
Ball Cup Catch
Finger Spin
810
800
Upiood Bround
620-
430
DrQ 1k1m
DrQ 1k1m
DrQ 2k2m
2k2m
Non naive aug
Non naive oug
Ball Cup, Catch
(b) Finger, Spin
Cartpole Swingup
Walker Walk
Walker Stand
1002
D02
803
Eppoodk Taorand
Ejoodk Bosaad
800
602
Spoiods
602
DrO 1k1m
Dro 1k11
DrQ 1k1m
2k22m
202
2k2m
Non naive aug
Non naive aug
203
Non nalve aug
100000
520038
100003
230000
302030
100000
230000
Step
(c) Cartpole, Swing Up
(d) Walker, Walk
(c) Walker, Stand
Figure 5: We compare DrQ and Non-Naive Augmentation the train environment. Dr with k, m=l and k, m=2 where represents number augmented samples for s, in Equation (I) and m represents number of augmented samples for 81+1, all methods shift. We find that non-naive augmentation with shift is more sample efficient and lower performance variance between seeds on out of 5 tasks. The mean and standard deviation of the average of 30 episodes across seeds: [0, 2] are used to generate plots. The lines represent the means and the shadow represents the variance.
in the task. This allows us to circumvent the high frequency feature bias that CNNs inherently have by pasting the relevant features random backgrounds, thus the high frequency features between
15

---

Under review as a conference paper at ICLR 2023
Walker, Walk
Walker, Stand
Ball in Cup, Catch
Cartpole Swingup
Finger, Spin
916 + 25
975 =
971 =
869 + 11
984 =
Table 7: Shift augmentation evaluation results the train environment across 5 seeds with the mean and standard deviation across 30 episodes for each seed. This table serves as an upper bound to what can be achieved in generalized benchmarks.
Domain, Task
Description
Action Vector Size
Dense Rewards
Walker,
planae walker rewarded walking
target velocity.
Yes
Walker,
Sland
planar walker that rowanded for standing with on upright torso at constant minimumn height.
Yes
Swingup Cup,
Curpole,
Swing and balance an unaciosted pole applying forces to cart at its threshold angle.
The rewanted for balancing pole within
Yes
Catch
An actuated planzr receptacle to swing and catch a atlached by a string bottom
No
Finger,
Spin
manigulition problem with plenar 3 DoF finger. The tack continually spin free body
No
Table 8: Table containing: action space dimension, brief description of task and if rewards dense. Descriptions are taken from Hansen et al. (20216).
frames becomes the task relevant features. Example code given below an comparative example is shown in Figure
import torch import kornia
def splice (x, hue_t, saturation value_t):
b. h, w = shape
x.SV kornia. wowor..colo.... rgb hxv(x)
overlay sample.background batch_size=b)
thresholds torch FloatTensor h[uut,, saturation.t, value_t])
thresholds thresholds (1, -l, 1, re.ectt 1, h, w) mask x HSV > thresholds
mask torch all (mask, dim=1)
overlay [mask] x x[mask]
return overlay
16

---

Under review as a conference paper at ICLR 2023
H
(a) Vidco Hard
(b) Splice
Figure 6: Value threshold 0.6, with threshold of 0 set for hue and saturation for splice augmenta-
tion shown in Figure 6b Figure 6a taken from DMC-GB.
17


Reviews:
[{"review_id": "aH5TEfgFlRg", "reviewer": "Reviewer_iZtt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper proposes to decouple critics in an actor-critic scenario, where one critic is trained on so-called weak pixel-based augmentations and the other on strong augmentations. The argument is that while weak augmentations help stabilize training within an MDP, strong augmentations help generalize to a larger set of variations on that MDP but destabilize training.\n\n- This induces four parameterized functions, $\\pi_\\psi$, $Q_{\\phi^W}$, $Q_{\\phi^{target}}$, and $Q_{\\phi^S}$ ($W$ and $S$ standing for weak and strong).\n- A new strong augmentation is introduced, called Splice, which pastes in natural image distractors forcing the model to pay attention to the relevant parts of the image.\n- Training is separated into two phases. The method introduces a hyperparameter $M$, the number of steps for which training is in the first phase.\n  - For $M$ steps, $\\pi_\\psi$ is trained on $Q_{\\phi^W}$, $Q_{\\phi^W}$ is bootstrapped to the EMA (of $\\phi^W$) $Q_{\\phi^{target}}$ with weak augmentations for both $\\pi$ and $Q$\n  - Then, $\\pi_\\psi$ is trained on $Q_{\\phi^S}$ with strong augmentations, $Q_{\\phi^W}$ and $Q_{\\phi^S}$ are bootstrapped to the EMA (of $\\phi^W$) $Q_{\\phi^{target}}$ with weak and strong augmentations respectively.\n- This method shows improvements on an array of environments from the DeepMind control benchmarks.\n", "summary_of_the_review": "While improvements in performance are always valuable, the scientific contribution here appears relatively minor to me. The proposed methods build on existing ones, but without deeply investigating them. This leaves us with higher numbers but not much new knowledge. \n\nAs is I'm on the fence about accepting this paper, but I think it could be a much stronger contribution with a proper empirical validation _of the hypotheses_.\n\nUpdate: the new results provide more clarity to this paper. In particular they test the hypothesis that $M$ matters, this is interesting in itself, but the other interesting aspect is that in terms of performance, it is the Splice augmentation that seems to offer most of the boost. I'm still leaning towards accept.", "strength_and_weaknesses": "The paper is relatively clear and well motivated. It introduces methods that make sense and motivates them well.\n\nI have some issues with the empirical evaluation of the proposed methods:\n- It seems that one of the the main drivers of performance wrt baselines is the augmentation used. Were any baselines trained with Splice?\n- As the authors point out, the role of $M$ is still unclear, but its role is a central part of the method. \n    - Here's a suggestion for a plot: run experiments with varying $M$s and display results with $M$ on the $x$ axis and the AUC or average reward after training on the $y$ axis for two settings, in-distribution and out-of-distribution MDPs.\n    - If what the authors posit is correct, with $M=0$ training should be too hard and not perform well on either MDPs, while with $M\\to \\infty$ (or simply $M>$total number of steps trained on) training should not generalize well to OOD MDPs but work well on in-distribution MDPs; and there should be a sweet spot in the middle where performance is ideal. Not only would this be a nice ablation, it would provide evidence that the _hypotheses_ that underlie this paper are correct.\n\nOther than that, I'm also a bit unsure that invoking curriculum learning is correct here. While you could argue this is _technically_ CL, with two phases we typically only say the the first phase is pretraining. A curriculum on the other hand usually implies a continuously changing \"difficulty level\" and data distribution adjustment, and one that is _adaptive_ to the learner's performance--here $M$ is a hyperparameter.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TI_ff43m4fS", "reviewer": "Reviewer_QnFd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents AugCL, a method for learning to generalize to unseen environments in pixel- based RL using strong visual augmentations. Training is split into two phases of using weak and strong augmentations respectively to leverage a curriculum. The authors use a separate critic network for each phase and demonstrate how this disentangelment induces superior generalization. They further introduce an image augmentation method called Splice, and show how the combination of this with their method achieves state-of-the-art performance on the DMC-GB benchmark.", "summary_of_the_review": "The paper utilises a very trivial method to marginally improve generalization of pixel-based RL on one benchmark. The contribution is novel, however, it is not empirically proved to be beneficial in other benchmarks, due to which I don't find it to be particularly significant. Further, it is dubious to label the simplistic two-phased learning as a curriculum.", "strength_and_weaknesses": "> Strengths\n\nThe paper is well-written, well-structured and easy to follow.\nThe paper achieves SOTA performance on 13/15 environments of a popular benchmark compared to previous methods. The authors prove the relevance of the key aspects of their proposed methods by running an ablation study.\n\n> Weaknesses\n\nThe authors conclude the value of M (timesteps for the1st augemntation phase) is an open question. I believe that the determining of this value is rather crucial, as the whole notion of curriculum (and hence the core method) relies on this parameter. The authors do report experimenting with 0 and 100,000 as values for M, which yielded poor results. I would like to see more analysis on how to select M and what effect this value beholds in other environments.\nThe paper talks about addressing how to effectively incorporate strong augmentations such as random convolutions and overlays, but then mainly focuses on their own proposed augmentation (Splice) in their experiments. Results using the Overlay augmentation are indeed presented, but 1) originate from a different dataset than which was used in SVEA and SODA, and 2) are not superior in all environments compared to previous works. It can thus be argued, that the proposed method AugCL does not necessarily surpass previous methods, but the new augmentation method (Splice) is the reason for improved performance. It would be interesting to see how much the proposed method outperforms previous methods using the same augmentations.\nThe contribution of the new augmentation method Splice is marginal, as it is not applicable in most other RL benchmarks. Since Splice is tailored for the DMC suite, it cannot be considered as a general augmentation method, which is why I would consider it a contribution of low significancy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FYSKJZ_jN5", "reviewer": "Reviewer_3K31", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of generalization in reinforcement learning. The authors seek to find algorithms for training a policy such that the policy will perform well on novel test-time environments. The authors propose a two-phase approach. In phase 1, the policy and value function are both trained with \"weak\" data augmentation. In phase 2, a second value function is trained in parallel with \"strong\" data augmentation and is used to update the policy. The method achieves SOTA performance on the Deep Mind Control Generalization Benchmark (DMCGB).", "summary_of_the_review": "Currently, I feel this paper is somewhat difficult to read and the method has limited impact outside of the DMCGB. However, if the authors address my comments in the \"Strengths and Weaknesses\" section, I will consider increasing my score.", "strength_and_weaknesses": "The main strength of this paper is the empirical performance. Achieving the top performance on the DMCGB is certainly important. The method proposed in the paper beats baselines in all but 1 of the DMCGB tasks.\n\nHowever, I do think there are a number of ways the paper needs to improve. Currently, I think the paper feels a bit rushed and a few edits and ablations would increase the impact of the paper substantially. I've included a list of comments and edits below.\n\n- the authors are using the ICLR 2022 template\n\n- Section 3.1 should just be $R$? not $R_t$\n\n- Algorithm 1 - in lines 13 and 16, $L_\\pi$ should take as input a $\\phi$ as well to specify which Q function weights are being used in the equations in section 4.1 (I think in line 13, it should be $\\phi^S$ and in line 16 it should be $\\phi^W$?). I think this detail is important for understanding the algorithm.\n\n- add equation labels for the equations in section 4.1\n\n- at the top of page 5, I think the authors should reference a line different from \"line 1\"?\n\n- I don't think Figure 1 is particularly informative since the arrows don't communicate which parts of the loss each model is being used for. I would personally prefer if the authors wrote out explicitly the loss for the policy and critic(s) in phase 1 and phase 2. For instance, the weak critic and target critic are updated the same way in phase 1 and 2, the only thing that changes in phase 2 is we begin training the strong critic and the policy uses the strong critic in its update (I believe, although currently ambiguous in line 13 of Algorithm 1). Currently, it's not obvious to me looking at Figure 1 that the updates for two of the networks are unchanged in phase 2, but that information would help a lot for internalizing the author's method.\n\n- Table 2 and Table 3 - did the authors evaluate SODA and SVEA with splice? The authors should demonstrate that SODA and SVEA don't get the same performance boost as AugCL when switching from overlay to splice. Otherwise my sense is the main contribution of the paper is the use of splice augmentation, not the particular choice of training weak and strong critics in parallel which is what the authors emphasize in the method section.\n\n- page 9 - ideally the authors would sweep across 3-5 values of M to get a sense of how sensitive the algorithm is to choice of M", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2fxDrpCyikW", "reviewer": "Reviewer_wrjJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors tackle the problem of generalization to novel visual environments in Reinforcement Learning.  The authors differentiate between weak augmentations (those that help with learning to solve the task in the same environment) and strong augmentations (those that hurt the training performance on the same environment, but may help generalization across domains).  The authors propose a curriculum where after M steps of only training on the weak augmentations, additional optimization on strong augmentations is included as well.  The authors also propose a data augmentation called Splice, where task-relevant information is cut out from the environment and pasted over a distracting background.", "summary_of_the_review": "The proposed approach in this work is intuitive, and simple to apply to many current RL solutions.  However, I believe it suffers from explicit domain-specific information (distinguishing weak from strong augmentations specifically for DMC), as well as a lack of convincing experiments that fully showcase the strength of the model with respect to the claims.  I also believe the only substantial novel contribution in this work is the curriculum schedule (as the differentiation of weak vs strong augmentations is assumed, not learned/identified ad hoc per environment), which is limited in terms of its intellectual impact.  I therefore recommend that this paper undergo continued revisions before being considered for acceptance.", "strength_and_weaknesses": "A strength of this paper is in its simple, easy to grasp idea - this would definitely help it be applied out-of-the box to a variety of RL methods.\n\nA main weakness of this paper is in its novelty, which will be elucidated upon in the section below.  Furthermore, I sense that a major weakness of this paper is how restricted it is to the DMC task suite.  Firstly, the selection of weak and strong augmentations were predefined specifically for the DMC tasks (‘Known weak augmentations for all DMC tasks are: crop, translate, and shift’).  This limits the application of the proposed model to other domains for which there is no prior knowledge about the weakness or strength of augmentations.  It would be interesting if the authors could automatically uncover such properties of elements in a proposed augmentation set automatically - however, as such, it looks like this approach was specifically designed for the DMC task suite rather than being general.\n\nAlso, the numerical results of the proposed AugCL do not appear to outperform the existing art (in particular, SVEA) on the listed tasks, with the exception of Walker, Walk.  Given how design decisions were specifically selected for this domain, I would have expected more substantial performance improvements.\n\nThe claim that AugCL enables generalization to unseen environments would also be greatly reinforced if it were demonstrated across true unseen visual domains and tasks, rather than remaining in the DeepMind Control suite.  I believe that given the claims of the paper, a more thorough set of experiments would be useful in demonstrating the power of their proposed curriculum.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject