Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
Authors: Byungchan Ko
Jungseul Ok
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Reinforcement Learning
Data Augmentation
Scheduling
Curriculum Learning

Abstract: In deep reinforcement learning (RL), data augmentation is widely considered as a tool to induce a set of useful priors about semantic consistency and improve sample efficiency and generalization performance. However, even when the prior is useful for generalization, distilling it to RL agent often interferes with RL training and degenerates sample efficiency. Meanwhile, the agent is forgetful of the prior due to the non-stationary nature of RL. These observations suggest two extreme schedules of distillation: (i) over the entire training; or (ii) only at the end. Hence, we devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL), and a simple yet efficient framework to automatically schedule the distillation. Specifically, the proposed framework first focuses on mastering train environments regardless of generalization by adaptively deciding which {\it or no} augmentation to be used for the training. After this, we add the distillation to extract the remaining benefits for generalization from all the augmentations, which requires no additional new samples. In our experiments, we demonstrate the utility of the proposed framework, in particular, that considers postponing the augmentation to the end of RL training.

Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
Byungchan Ko"
Jungseul Ok GSAI, POSTECH jungseul postech..ss.t..t.kkkkkk
NALBI kbc nalliaaa
Abstract
In deep reinforcement learning (RL), data augmentation is widely considered a tool to induce a set of useful priors about semantic consistency and to improve sample efficiency and generalization performance. However, when the prior is useful for generalization, distilling to RL agent often interferes with RL training and degenerates sample efficiency. Meanwhile, the agent is forgetful of the prior due to the non-stationary nature of RL. These observations suggest two extreme schedules of distillation: (i) the entire training; or (ii) only the end. Hence, we devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL), and simple yet efficient framework to automatically schedule the distillation. Specifically, the proposed framework first focuses on mastering train environments regardless of generalization by adaptively deciding which or no augmentation to be used for the training. After this, we add the distillation to extract the remaining benefits for generalization from all the augmentations, which requires no additional new samples. In our experiments, we demonstrate the utility of the proposed framework, in particular, that considers postponing the augmentation to the end of RL training. https://gitub..con/// 6723/es-da
Introduction
Deep reinforcement learning (RL) aims finding a neural network to represent policy or value functions taking raw observation as input, of which the most common form in practice is visual data or images of digeniimensionlltty, e-g., video games [23] board games 29 28 and robot controls 32 18] RL handling high- digeedimensional input often suffers from poor sample efficiency and 15). To overcome these issues,
generalization capability, mainly due to the curse of dimensionality it has been widely considered to augment data based on prior knowledge that set of transformations preserve the meaning or context of input observations, e.g., cropping out unimportant parts of images, and changing colors (20 19 21 [4]. On one hand, RI. agent can be directly fed with the original and augmented data so that it implicitly learns a representation with the prior and improves the sample efficiency and generalization (20). On the other hand, the prior knowledge in data augmentation can be explicitly distilled via self-supervised learning, which introduces additional regularization to ensure consistency between responses original and augmented inputs 25 14.
However, data augmentation shows highly task-dependent effect in RL, and it is prone to generate severe interference with the training even when it truly conveys useful prior to train and test environ- We address the problem of alleviating the interference between data augmentation and RI training to improve sample efficiency for acquainting train tasks, and generalization capability for unseen test environments. This problem in (online) RL. is more critical and challenging than that in supervised learning (SL) since the objective function and data distribution are time-varying in RL,
ments 20 25 14
This work was done while Byungchan Ko studied in GSAI, POSTECH.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

while they are not in SL. Indeed, according to [0, is sufficient to partly apply data augmentation just for a short period of SL at any time. Meanwhile, we empirically observe that the prior from data augmentation can be easily forgotten in RL of the non-stationary nature (see Section 5.3 ), i.e., the effect of augmentation is time-sensitive.
Based on our observation about the interference and the time-sensitivity, we propose two simple yet effective methods according to timings of data augmentation Intra Distillation with Augmented observations (InDA) and Extra Distillation with Augmented observations (ExDA). Data augmentation beneficial for the sample efficiency needs to be applied over the entire RL training, i.e., InDA. Conversely, data augmentation useful only for the generalization should be postponed to the end of RL training, i.e., ExDA, so that we can minimize the interference in the training, while enjoying the benefit in the testing. InDA and ExDA are equipped with Distillation with Augmented observation (DA). DA a stand-alone self-supervised learning which enables us to induce the prior after RI. training, and shows relatively small interference with RI. training by explicitly preserving the response of RI. agent to the original input.
The best timing (InDA or ExDA) depends on traits of train task and augmentation. We hence suggest framework of adaptive scheduling, named UCB-ExDA, that (i) first aims at maximizing the sample efficiency by adaptively selecting which or no augmentation to be used during RL training; and then, (ii) distill the priors from all the augmentations after RL training. Specifically, inspired by [25). we devise UCB-InDA for the adaptive selection in the first part by modifying the upper confidence bound (UCB) algorithm 3 for multi-armed bandit, where differently from [25), the set of arms includes all the augmentations and the option to not augment. In summary, UCB-ExDA is nothing but executing
UCB-InDA followed by ExDA. Our experiment demonstrates the utility of the proposed framework.
Our contributions are summarized as follows:
We devise DA (Section 4.1 which explicitly preserves the knowledge of RL agent so that enables distilling the consistency prior of augmentation into RL agent not only during but also after RL training, while other methods 22 25 14 need be applied concurrently with RL training and thus show relatively strong interference in our experiments (Section 5.2).
(ii) We identify the simple yet effective timings of data augmentation: either InDA or ExDA (Section 4.2 Section 4.3) based upon the discovery of the time-sensitivity (Section 5.3) that has not been observed in SL [10], i.e., the proposed timings are effective particularly for RL.
(iii) We finally establish UCB-EXDA which automatically decides the best timing of augmenta- tion, and demonstrate its superiority compared to others (Section 5.4). The advantage of UCB-ExDA particularly substantial when the best strategy is ExDA postponing augmen- tation to the end of RL training.
2 Related Works
Augmented experience in RL. To solve the problem of poor generalization and sparse data, popular approach is to fabricate virtual experiences and train the RL agent to learn with them. Domain randomization is technique to produce such experiences from a simulator of targeted system, 32, 24 26]. Accurate simulators of practical systems are difficult to obtain, and it has limited applicability. However, visual augmentation has no such limit because the method uses simple image transformations such as cropping, tilting, and color jitter, although applications require careful understanding of the targeted system to guide the design of an appropriate image transformer. A method of curriculum learning for domain randomization, in which the difficulty is gradually increasing (26) provided insights that coincide with some of our findings. However, we provide a further understanding of the types of visual augmentation that should be early or late during training.
Regularization from augmented data in based RL has been implemented in various learning frameworks, including but not limited to representation [4 31], self supervsue [25]. and contrast
30. One proposed algorithm (25] applies the UCB algorithm 3 to automatically select the most
effective augmentations over RL training, where each augmentation is considered as an arm and then
evaluate the effectiveness of augmentation by using a sliding window average. The idea of adapting augmentation concurs with our main message regarding the timing of augmentation. In [25]. 'not
augmenting" is not an option, whereas our findings indicate that it should be. In addition, 255 does not consider post augmentation followed by RL training, in ExDA.
2

---

Other time-sensitivity in deep learning, During deep learning, the early stage of training often has significant effect I Therefore, we devised time-sensitive methods that adapt the progress of training, such as learning rate decay [5 and curriculum learning 33 Golatkar et al. [10] studied such a time-sensitivity of regularization techniques for SI., where the effect of data augmentation in different time does not change much. We find that the time-sensitivin of augmentation can be significant in RL. This contrast may occur because of the non-stationary nature RL, which SL does not have. Although a set of techniques originally developed for SL such convolutional neural network, weight decay, batch normalization, dropout and self-supervised learning improve deep RL [16 6 22 30 34 [B,, thorough study should be conducted before introducing method from different learning frameworks, because we find the contrasting senettenstiviites of data augmentation. This spirit is also shared with application [17] of implicit bias in SL 122 to RL.
3 Background
Notation. We consider a standard agent-environment interface of vision-based reinforcement learning in discrete Markov decision process of state space S, action space A, and kernel P P(st+1,71| Se/se at) which determines the state transition and reward distribution. The goal of the RL agent to find a policy that maximizes the expectation of cumulative reward Σi'-1 7'r,, where I' is terminating time and E [0, 1] is discount factor. At each timestep 4, the agent selects an action a, E A and receives reward Tr and an image 0++11 O(s++1) & as observation of the next state S1y1- Data augmentation can be described by an image transformation function Φb IR kxk' + k of which output is presumed to have the same or similar semantics of the input.
Baseline RL algorithm. Throughout this paper, use Proximal Policy Optimization (PPO) [27 as baseline, although we believe our findings and methods can be easily adjusted to others. PPO is a representative on-policy RL algorithm to learn policy (a o) and value function Vo of neural agent parameterized by B. Storing a set of recent transitions тę == (ot, ae. Tt, 0t+1) in experience buffer D, the agent is updated to minimize the following loss function:
(Pro = - L (0) + aLy
(1)
where a is a hyperparameter and canonical regularization terms are omitted. The clipped policy objective function L, and value loss function Lv are defined as:
I (0) E min (L)) 1- c,1+))A))
(2)
(0) i8 (Vo(o.) V. y.uuB)) 27
where the expectation T. is taken with respect to T. ~ D, we denote by Oold the parameter before the update, (f) is the importance ratio xp(axlo)) New (Arlo,)
and A, is the generalized advantage estimation [27
Method
In what follows, we present methods: Distillation with Augmented observations (DA; Section 4.11 Intra DA (InDA; Section 4.2). Extra DA (ExDA; Section 4.3), and then the adaptive scheduling frameworks based on UCB (UCB-InDA and UCB-EXDA; Section 4.4) DA is stand-alone knowl- edge distillation method, which can used at any time to instill the underlying prior of augmentation into a given RL agent. InDA and ExDA conduct either DA PPO in each epoch but have different schedules (Figure [), where InDA interlcaves PPO and DA, whereas ExDA performs PPO first then DA. UCB-InDA adaptively decides which no augmentation to be used in each DA epoch of InDA based on the UCB of estimated gain from each option. UCB-EXDA performs ExDA preceded by UCB-InDA.
4.1 Distillation with augmented observations (DA)
The underlying prior of augmentation can be infused by minimizing a measure of inconsistency between the agent's responses to original and augmented inputs (resp. o, and Φ(o;)). For instance,
3

---

DA (Augmenation)
(o
PPO (RL training)
(A)
Vooia (o,)
Epoch
(c)
Distillation with Augmented Observations
After training
(B)
Extra Distillation with Augmantad Obsorvations
Epoch
Ve (o)))
Figure 1: An illustration of InDA and ExDA
Figure 2: An illustration of distillation losses.
with PPO agent learning policy Ty and value Vo, Raileanu et al. 22 proposes the following measure:
(0, Φ) == EOLND
=e(-lo(o))) + BO [(Vo(o.) - Vo(o(o))))
(4)
which uses Kullback- Leibler divergence (first term) and mean squared deviation (second term) for policy and value inconsistencies, respectively, Noting that Lais (0; Φ) can be minimized to be zero by constant response to all inputs. the distillation with Eq 44) can distort the RL agent, in particular, when applying it outside of RL training.
We hence devise a network distillation technique (DA) which explicitly preserves the RL agent's s response to original input and thus is applicable even after RL training. DA distills the knowledge of RL agent Oota into / using not only original but also augmented observations. More formally, the loss of DA is written as:
LoA (0, Φ; Dota) = Lais (0, I1; Dota) + (0, di; Ooia)
(5)
We here denote the identity transformation by II such that I(o) = o for o, and extend the definition of Las in Eq 4 as follows:
Lais (0, Φ; Oaaa) = BoyND
(:ooe), no( (=((())))]] + Ro, [(Vo (0+) Vo(o(o:))) (6)
In Eq (5) the first term ensures that 0 and Oold behave identically for the original inputs, and the second one infuses the consistency prior. In Figure 2 (e, I; fosa), Lais (0, Φ; foua) and Lax d; 0) graphically correspond to (A), (B), and (C), respectively. From this, it follows that minimizing LDA in Eq (5) eventually reduces Lais [6, Φ: 0) in Eq (4) In addition, the minimization of LDa secures the responses of 0 to the original inputs (which can be pre-computed to reduce computation cost) the those of Oala, while the alternatives (e.g., (A)+(C): (ais (0, II; Oaaa) + Lais (0, d; 0)) does not and thus may generate interference with RL training. Our experiments (Section 5.2 Figure 4 justifies the
design of DA by showing substantial advantage compared to the other alternatives such as DrAC
25 using Lais (0, o; ()) in Eq (4). We note that this advantage becomes much clearer when wrong augmentation is given, c.f., the supplementary material.
4.2 Intra distillation with augmented observations
InDA (Algorithm D alternates between minimizing Lero and LDA, i.e., PPO and DA are explicitly separated, whereas they are often executed simultaneously in other methods (25 Such clear separation reduces the interference [14). We can control the frequency and timing of applying distillation with hyperparameters I, S and T, where we perform DA after each I rounds of RL training only if the current epoch n is in the interval of [S, T), i.e., S and are the epochs to begin and terminate DA, respectively. We vary S and T to study the time-dependency of data augmentation. We denote InDA (S, T) indicate the period apply DA, while we omit the indication when DA applied over the entire period. We provide further details on InDA in the supplementary material.

---

Algorithm InDA
Algorithm 2 ExDA
Require: N, S.T
Require: N, N.M. e
Initialize I close to origin.
1: Initialize 0 close to origin.
2: for n = 1....... N do
2: //Pre-training phase with RL algorithm
3: /L training
3: for 1, ..2....,A N do
4: Store sampled transitions to D;
4: Store sampled transitions to D;
5: Optimize RL objective (0) with D;
5: Optimize RL objective (0) with D;
6: I/ Distillation
6: end for
7: if n E [] and mod(n 1, f) 0
7:
then
8: Store Oold 8;
8: Store 8old t- 8:
9: Distillation at the end of RL training
9: Minimize (D) for D, Oold and Φ;
10: for m 1.2.....AA M do
10: end if
11: Minimize (0) for D, 8ota and o;
11:
end for
12: end for
4.3 Extra distillation with augmented observations
ExDA (Algorithm{) 2 performs the distillation after the end of RL training, where the lengths of DA and RL training are parameterized by M and N, respectively. We note that computational cost can be reduced by removing the value inconsistency measure Eonn (o.) - Vo(o(o:))) from Las Eq (6) because the value consistency is not necessary for RL. training nor distillation in the actor-critic framework and including it has potential risk of generating additional interference. In the supplementary material, is empirically verified that this reduction does not degrade RI. perffrrmrm.cc. We leave more interesting details in the supplementary. For instance, one can consider re-initializing 0 before starting DA as part of exploiting the implicit bias [22 [7] to improve generalization. However, test performance is often dropped. This is mainly because the dataset to distill Tea. has much less diversity than that observed along the trajectory. Thus, we use no re-initialization for the experiments in the main paper.
4.4 UCB-based adaptive scheduling frameworks
The training benefit by augmentation differs depending on the task. This dependency complicates the choice of whether use InDA or ExDA for augmentation. Hence, we devise an auto-augmentation method, called UCB-InDA, inspired by UCB-DrAC (25), where each augmentation is corresponded to an arm multi armed bandit (MAB) problem and assessed its potential gain in training with upper confidence bound (UCB) 3. More formally, in UCB-InDA, the set of arms is the set of augmentations. = (Ø1..... ΦK) ). which must include the identity function I, i.e., the option not to augment. The inclusion of identity function is small but makes crucial difference than UCB-DrAC [25] since we observe that using augmentation sometimes needs to be postponed after RI. training for the sake of better sampling complexity and test performance.
round of MAB corresponds to every epoch of InDA, where we let (b(s) E & be the augmentation selected at the 8-th round of DA. Let Gss) be the average return, the sum of estimated advantage Á and predicted value Vj. over (I 1) epochs of PPO followed by the s-th DA. UCB algorithm assumes that each arm generates reward independently drawn from fixed distribution, and estimates the empirical mean over the entire sampling process. However, in RL, the return G(s) is non-stationary, so InBA computes moving average gain Gs((s), instead, taken over certain number (chosen be in our experiment) of most recent rounds selecting di as Raileanu et al. [25 proposed. Then, inspired by UCB1 algorithm
3 JCBB-IDA selects action k(s)
at round s as follows:
log(s)
A(s)
arg max Gs(s) +1 RE(L.....K)
(x(s)
where is the UCB exploration coefficient, and Nx is the number times selecting d's up to round We refer to the supplementary material for the hyperparameter choice. We remark that compared to UCB-DrAC 255. the proposed UCB-InDA has subtle but important differences, summarized two folds: (i) the inclusion of identity transformation (i.c.. augmentation) and (ii) the distillation with
5

---

PAGrad
PAGrad
Baux
Jaux
Imain
Imain
(a) non-conflicting: (gaxx; Gmain) 2 0
(b) conflicting: (gaux Pmnin] O
Figure 3: An illustration of gradient conflicting and PAGrad. We here let Imain and Jaux denote the gradients of main (shown in red; e.g., Lppo) and auxiliary (shown in blue; e.g., Lddss) losses, respectively. In the computation of PAGrad (shown in black), the component of 9axx conflicting to Imain only when it exists (i.e., (Jaux. 9main) < 0).
augmentation via InDA. The gain of each component is numerically studied in Section [ Finally, we note that UCB-ExDA is nothing but UCB-InDA followed by ExDA.
4.5 PAGrad
Inspired by [36]. we devise an alternative approach to reduce the interference by interpreting RL multi-task learning, where LPrO and Lais correspond to the main and auxiliary task losses, respectively. In [36]. the degree of conffict from task A to task B estimated by the inner product of the gradient of task A and the negative gradient of task B, e.f., Figure 3 From this, we propose PAGrad (Projecting Auxiliary Gradient) to compute a modified gradient of Lppo excluding the conflict from the auxiliary gradient Ldis to the main one Lppo. Formally, PAGrad computes the gradient given as follows:
training with data augmentation
Lppo +
(VLats
(Oi Lomo V Lyool 2
V LPPO)
Lpro is the components of Ldis opposite to Lpro which may disturb optimizing the main objective LpPO- Based this, we devise DrAC+PAGrad that updates the model parameter toward the negative direction of (8). This is an alternative of InDA, while it concurrently optimizes Lero and Ldis differently from InDA adopting the time-separation of optimizing Lrro and Lais We also note that it differs from the original method proposed in [36] alternating the main and auxiliary tasks to accomplish every task at equal priority, while we have clear priority on RL task.
where min((U Lu. Lreoll
| Leroll
5 Experiment
5.1 Setups
Train and test tasks. We use the OpenAl Procgen benchmark of 16 video games (5). where main character tries achieve specific goal, e.g., finding exit (Maze) collecting coins (Coinrun), while avoiding enemies given 2D map. At each time t, visual observation or is given as an image of size 64 64. train or test task achieve high score on of environments configured by game and mode, where a mode describes predefined sets of levels (e.g., complexity of map) and backgrounds. Cobbe et al. provide easy mode for each game, consisting of 200 levels and certain set of backgrounds.
We simplify easy mode and train agents in easybg mode, of which the only difference from easy mode (5] is showing only a single background. This is useful to investigate the case that using visual augmentation is helpful for testing but not for training. We denote the task configuration by game name(mode). e.g., Coinrun(easybg). Then, we evaluate generalization capabilities using two modes: test-bg and test-lv, which contain unseen backgrounds and levels, respectively, addition to easybg mode that we use for training.
Types augmentation. For clarity, we mainly focus on two types of visual augmentation, each of which conveys distinguishing inductive bias:

---

(a) Random color transforms an image by passing through either color jitter layer or random convolutional layer proposed in [2]]. From this, we can impose the consistency color changes, which may provide strong generalization to diverse backgrounds of test-bg mode.
(b) Random crop leaves randomly-selected rectangle and pads zeros to the rest of the image [25]. This augmentation is particularly useful in fully-observaale games (e.g.. Chaser and Heist), because imposes an efficient attention mechanism.
We also report the result with other augmentations including color jitter, random convolution, gray, and cutout color in the supplementary material, from which the same messages can be interpreted.
Baseline methods for RL with data augmentation. We mainly compare the proposed methods (InDA and ExDA) to the following basclines:
(a) RAD 20
simply feeds PPO with experiences of original and augmented observations.
(b) DrAC 25
is method to simultaneously minimizing LePO in Eq ① and Lais in Eg
(c) DrAC+PAGrad is variant of DrAC, which we devise investigate another mechanism to relieve the interference between RL training and augmentation in Section 4.5
The supplementary material presents further details and experiments, which omitted for simplicity. All results in the main paper are averaged over five runs.
Train
Test-lv
0.8
Mean score
0.8
0.6
0.4
20
20 -ExDA -ExDrAC
PPO DDAC
InDA
PAGraddDrrdd Timesteps (M)
Figure 4: Benefit of separating distillation from RL training. We compare InDA, ExDA, DrAC, DrAC+PAGrad and ExDrAC when we start to apply each of them to distill the prior of random crop after 20M timesteps of PPO. ExDrAC is variant of DrAC without RL training, i.e., minimizing only Las in Eq 4
5.2 Benefit of separating distillation from RL training.
In Figure 4 after training PPO agent up to 20M timesteps, we start to suddenly apply one of the different distillation methods with random crop. We report averaged scores over 6 environments (Bigfish, Dodgeball, Plunder, Chaser, Heist, Maze) after normalized by the highest train score of PPO on each environment. After 20M timesteps, ExDA and ExDrAC have no RL training but minimize LDA in Eq (5) and Lais in Eq 4. respectively. The substantial gap between ExDA and ExDrAC justifics the design DA explicitly preserving the knowledge from RL when distilling the prior. More importantly, it is remarkable that ExDA promptly learns the generalization ability once it starts to distill the prior. This validates the idea postponing the distillation after RL training.
We now compare the distillation methods (InDA, DrAC, and DrAC+PAGrad) concurrently optimizing the RL objective and distilling the prior in Figure Each method has performance degeneration due to the interference generated by distillation. However, InDA and DrAC+PAGrad have clearly smaller degeneration than DrAC which the only one without any separation of optimizing the RL objective and distillation loss. We note that DrAC+PAGrad has more interference than InDA, and it seems to fail to impose the prior since there not much difference to PPO testing. Hence, this verifies the superiority of InDA which enables distilling the prior while alleviating the interference.
7

---

Chaser
Chaser
Chaser
10.25;
(0,5)
(0.0)
Staree
Timestops (M)
(a) train (easybg)
(b) test-Iv
(c) Policy Distance
Heist
Heist
Soom
Timesteus (M)
Timestape (MI
Timestops (M)
(d) train (easybg)
test-lv
Policy Distance
Figure 5: Time- Sime-sensitivity of applying augmentation. We compare train and test performance of InDA[S, I with random crop, where the timing of applying DA is governed by starting time S and terminating time T, and evaluate four different pairs of [S, [0. 0]. 00, 5]. [20, 25j. [0, 25] up to 25M timesteps. Furthermore, we show the change of distance between two policies on an original observation and augmented observation. Note that InDA with [S, T [0, 0] means RL training with no augmentation, i.e., vanilla PPO. We focus on Chaser and Heist since they exemplify two representative deneddepeneecces. Each train task easybg mode. We present further details and results with other games in the supplementary material.
5.3 Effective timings of distillation
In what follows, we aim to identifying effective timings of distillation. this end, in Figure 5 we test several schedules of applying DA two representative environments of distinguishing traits. The supplementary material presents the result on more environments, which similar one of the representatives.
effective timing: InDA. Chaser with random crop (Figure and 5(b) represents the case when augmentation improves the sample efficiency of training and thus the generalization ability in training. To compare the generalization ability, we measure policy distance between original and augmented observations using Jenson-Se -hannnon divergence (Figure 5(c)). InDA[O, 5J's policy distance is increased after it stops using DA. Thus, the generalization ability is degraded, if we do not continue to DA. In this case, it is clear that InDA should be applied during the entire RL training, as InDA(O, 25] shows the best performance also important apply DA as soon as possible since the effect of InDA[0.5] applying DA in the beginning is relatively prompt and significant comparing to that of InDA[[0,25]. This suggests that the automatic framework needs explore more in the early stage.
both training and testing. In addition, is
An effective timing: ExDA. Heist with random crop (Figure 5(d) and 5(c) shows the opposite of data augmentation to what Chaser case suggests, i.e., postponing data augmentation as much as possible. Random crop generates slight interference, although it immediately improves the generalization ability. We remark that the inductive bias from the random crop easily forgotten, as the test performance of InDA[O, 5] is saturated right after stopping the distillation. This can be explained by time-varying nature of sample distribution and objective in RL. Interestingly, is in contrast to the data augmentation in SL, where an early distillation is sufficient to impose the prior [10] On the other hand, the test performance curve of InDA[20. 25] soars right after DA. Furthermore, Figure 5(f) shows that InDA[20, 25] narrows the gap between two policies the

---

original and augmented observation. Recalling the interference between RL training and distillation, this suggests postponing the distillation the end of RL training, and motivates our ExDA.
Performance benchmark InDA and ExDA. In Table we summarize the train and test performances of InDA and ExDA on a set of games with different augmentations and modes. ExDA outperforms other baselines with random color test-bg. Moreover, we note that ExDA consumes only 0.5M timesteps to inject the prior at the end of RL training. whereas the others use all the training data. ExDA in both sample efficiency and generalization ability with random crop on test-Iv. It is worth noting that DrAC+PAGrad is slightly better than DrAC, while there is a substantial gap between InDA and each DrAe- algorithm. This again confirms the benefit from the separated distillation of InDA observed in Figure 4 These results demonstrate that each combination of environment and augmentation has suitable time at which apply augmentation, and the gain from using the right distillation timing, i.e, online (InDA, DrAC, or DrAC+PAGrad) or offline (ExDA), is rigid regardless of the choice of algorithms.
Augmentation
Task
PPO
RAD
DrAC
DrAC+PAGral
InDA
ExDA
Train
1.00
0.98
0.88
0.89
0.88
0.98
Rand conv
Test-bg
1.00
1.08
1.86
1.88
1.92
2.11
Rand color
Test-lv
1.00
0.81
0.84
0.84
0.7
0.87
Train
1.00
0.94
0.95
0.95
0.96
0.98
Color jitter
Test-bg
1.00
1.37
1.44
1.44
1.43
1.48
Test-lv
1.00
0.83
0.86
0.86
0.84
0.88
Train
1.00
0.28
1.08
1.25
Rand crop
Test-bg
1.00
0.64
0.87
0.94
0.94
0.95
Test-Iv
1.00
0.46
1.52
1,53
1.80
Table 1: Benchmark InDA and ExDA. We report normalized train and test scores of InDA, ExDA
and DrAC with PAGrad on Open AI Procgen, compared to baselines PPO, DrAC [25], RAD [20] Boldface indicates the best performance. We average the score among several environments [(Rand coinrun, ninja, climber, fruitbot, jumper, heist, mazc). (Rand crop: Bigfish, Dodgeball, Plunder, Heist, Chaser, Maze)] after normalization considering PPO score be 1. Every method is trained
color:
on 200 levels, using easybg mode. We evaluate test performance on both test-bg and rest-lv. The results be interpreted as upper bound of potential gain from using data augmentation perfect timing.
5.4 Adaptive scheduling methods
Adaptive selecting of timings: UCB-InDA and UCB-ExDA. It is hard to know in advance whether certain augmentation helps the training or not 25). We hence employ UCB-InDA which estimates the gain or damage from each augmentation from trial and error and identifies the one
Chaser
Heist
color
identity
Numberd thilose
3001
crop
Numberd of choicse
300
Timesteps (V))
Timesteps (M)
(a) Chaser
(b) Heist
Figure 6: Exploration & exploitation to find the most beneficial augmentation. We show that selected augmentation during training with UCB-InDA for each Chaser and Heist. UCB-InDA automatically selects the augmentation among three options, random color, identity and random crop.

---

Eov
Method
PPO
DrAC
UCB-DrAC
InDA
ExDA
UCB-IaDA
UCB-ExDA
0.46
10.3
7.41.
4.9 1
9.14 0.56
9.67.1
0.29
Hoist
Test-bg
5.18 + 1.53
3:58 + 0.31
3,76: 0.54
4.9 + 0.87
7.05 = 1.29
6,23 1.29
7.86::: 0.83
Tost-lv
4.13 = 1.39
3.07 0.33
3.49+0488
1,47 0.77
0.98 5.05
4.8 1.24
5.74+ 0.67
Train
5.63.: 1.12
0.16::0:22
4.6 1,22
5.691: 1.42
5.582 1,33
1.49
7.23:: 1.15
Chaser
Test-bg
0.87# 0.06
0.1= 0.02
0,57+0.12
3,5 1.38
1,02
0.04
1.08 0.08
3. 18-2 0.79
Tost-lv
4.83+ 0.88
0.14+00.11
4.14+11001
4.96+ 1.21
5.11+ 1.05
6.45+ 0.8
6.43+ 1.28
Table 2: Full exploitation of augmentation to improve generalization on both test-bg and test-Iv. We compare InDA, ExDA, DrAC, UCB-DrAC, UCB-InDA, UCB-ExDA and PPO about train, test-bg and test-Iv. Boldface indicates the best method. InDA and DrAC use both random color and random crop during RL training. ExDA use both augmentation after RL training. UCB-InDA and UCB-DrAC are trained as automatically selecting the augmentation during training. UCB-ExDA trains ExDA after UCB-InDA with both augmentation.
with most help. Recall Table 2 where PPO (without augmentation) shows much better training performance than InDA in Heist. As shown in Figure/6(b) UCB-InDA is able to identify that no augmentation is best for training in Heist. This implies that ExDA is more appropriate than InDA. automatically
Conversely, random crop is selected Chaser (Figure 6(a) As the result, select InDA or ExDA appropriately for each task.
Fully exploitation of augmentation In Table 2 when both random color and random crop are used to improve generalization on both test-bg and test-Iv, report numerical evaluation of UCB-InDA and UCB-ExDA with other baselines on train and test tasks. Decreased train performance of DrAC and InDA compare to PPO show the difficulty of simultaneous training with several augmentations. Train performance of UCB-InDA and UCB-DrAC are improved by adaptive selecting, especially, UCB-InDA is better than UCB-DrAC. The gap is made due to the robustness about the change of augmentation during training. In terms of generalization, UCB-Ext BCEDA clearly surpasses UCB-InDA thanks ExDA to extract all the priors from the complete set of data augmentations at the end RL training.
6 Discussion
We have identified two most effective yet simple timings (InDA and ExDA) of data augmentation for RL, and proposed -A framework to adaptively select the best scheduling augmentations. We note that the effectiveness of this framework is restricted but specialized for RL with the unique non- stationary nature. Indeed, in SL without shift of data distribution and objective, is sufficient to apply data augmentation at the beginning algorithm with a finite set of data augmentation. It is interesting to investigate a room to improve by further optimizing continuous parameters of data augmentation for RL, c.f., auto-augmentatiion technique to optimize continuous augmentation parameter per sample for SL [1] Another promising direction accelerate the distillation process of DA by data condensation with augmentation [37]. This is possible with our framework clearly separating between RL training and distillation, and may be particularly useful train distributed RL agents since condensed data for agent's distillation is usable for the other.
[10]. Our framework employs the most basic multi-armed bandit
Acknowledgments and Disclosure of Funding
We thank Kimin Lee for helpful discussions. This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-00) NN..201--0 Artificial Intelligence Graduate School Program (POSTECH) and Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021 10102110-0266- Artificial Intelligence Innovation Hub) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) Byungchan Ko was supported by the Institute of Information &
(No. 202552222200 Communications Technology Planning Evaluation (IITP) grant funded by Korea (MSIT) (2020-0- 01594, PSAI academic joint research and education program).
10

---

References
[1] Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference Learning Representations, 2018.
[2] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, pages 7411-7422, 2019.
[3] Peter Auer. Using confidence bounds exploitation-exploraiio trade-offs, Journal of Machine Learning Research, 3(Nov):397-427 2002.
[4] Richard Bellman. Adaptive control processes. Princeton university press, 2015.
[5] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. In International conference machine learning, pages 2048-2056. PMLR, 2020.
[6] Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization reinforcement learning. 2019.
[7] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201-208. JMLR Workshop and Conference Proceedings, 2010.
[8] Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in dqn, 2020.
[9] Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamies in linear neural networks. In Advances in Neural Information Processing Systems, pages 3196-3206, 2019.
[10] Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems, pages (0678-10688, 2019.
[11] Denis Gudovskiy, Luca Rigazio, Shun Ishizaka, Kazuki Kozuka, and Sotaro Tsukizawa. Autodo: Robust autoaugment for biased data with label noise via scalable probabilistic implicit dif- ferentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16601-16610, 2021.
[12] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, pages 6151-61555 2017.
[13] Nicklas Hansen, Yu Sun, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. arXiv preprint arXi::2007.04300 2020.
[14] Nicklas Hansen and Xiaolong Wang- Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13611-1361 IEEE, 2021.
[15] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters, 2019.
[16] Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning, 2018
[17] Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. The impact of non-stationarity on generalisation deep reinforcement learning. arXiv preprint arXiv:2006.053 058266 2020.

---

[18] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et Qt-opt: Scal- able deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018.
[19] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforeement learning from pixels. arXiv preprint arXiv:2004.13649 2020.
[20] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004.14999, 2020.
[21] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: simple technique for generalization in deep reinforcement learning. arXiv, pages arXiv-1910, 2019.
[22] Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization an empirical study on continuous control, 2020.
[23] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
[24] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.0654., 2017.
[25] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. 12862, 2020.
[26] Sharath Chandra Raparthy, Bhairav Mchta, Florian Golemo, and Liam Paull. Generating automatic curricula via self-supervised active domain randomization, 2020.
[27] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprins arXiv:1506.02438, 2015.
[28] David Silver, Thomas Hubert, Julian Schritwieserr, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140-11141111-1144 2018
[29] David Silver, Julian Schritwiisser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
[30] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning, 2020.
[31] Adam Stooke, Kimin Lee, Pieter Abbecl, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv:2009.083/99 2020.
[32] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23-30. IEEE, 2017.
[33] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work?, 2021.
[34] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2020.
[35] Kaichao You, Mingsheng Long, Jianmin Wang, and Michael Jordan. How does learning rate decay help modern neural networks?, 2019.
12

---

[36] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782 2020.
[37] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685. PMLR, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We claim our three main contributions in and target
on DeepRI.
(b) Did you describe the limitations of your work? [Yes] We describe the limitations of our work about limited set of data augmentation for UCB methods Section 6
(c) Did you discuss any potential negative societal impacts your work? [N/A] We believe that our research will not have a negative impact on society.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read the ethics review guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our works are based on experimental results.
(b) Did you include complete proofs of all theoretical results? [N/A] Our works are based on experimental results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We describe our training environments and include code the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We describe training details in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We indicate standard deviation with score in Figure [ and Table 2 except normalized score with several environments such as Figure 4 and Table II
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We explain about training time in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our experiment environments are baed on OpenAl Procgen (5).
(b) Did you mention the license of the assets? [Yes] We include the license of the asssets
in the code.
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "qLeBF3nFT8t", "reviewer": "Reviewer_SwS3", "summary": "This paper explores several modifications to UCB-DrAC [25]. It decouples the optimization of the RL loss and the data-augmentation regularization losses in two separate stages, and then empirically studies what is the best strategy for scheduling those two stages: alternating between them at fixed intervals during training (InDA), or training with RL only followed by a few epochs of regularization (ExDA). It also augments the UCB bandit in UCB-DrAC with an arm that does not perform any type of data augmentation. These methods are evaluated on a variant of the Procgen benchmark, where they compare favorably to the considered baselines (PPO, RAD, DrAC and DrAC+PAGrad). ", "questions": "**Major**\n\nI am particularly concerned about the following points described under *weaknesses* above which I believe should be resolved before this paper is ready for publication at NeurIPS:\n- Results under standard Procgen evaluation protocol\n- Ensuring that baselines replicate published results\n- Better positioning of the proposed method with respect to the literature\n\n**Minor**\n\n- InDA and ExDA require additional learner steps compared to methods that optimize both the RL and the consistency loss in parallel (e.g. DrAC). What's the impact on wall-clock time?\n- It would be interesting to understand how the performance of InDA and ExDA changes as a function of the size of the replay buffer used for distillation. In particular, I would expect a collapse if this buffer is too small. Have authors experimented with this?\n- Figure 9 (Appendix D) shows train scores for different number of backgrounds. It is somewhat expected that increasing the number of backgrounds will make training slower, given that the agent is exposed to a more difficult task distribution. It would be interesting to compare those same experiments in the test set, where agents trained on fewer background might underperform due to overfitting.", "limitations": "Section 6 briefly mentions one limitation of the approach (i.e. the framework is specialized, and thus restricted, to RL). The paper does not describe potential negative societal impacts, although there aren't any specific to the proposed approach.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Generalization in RL is an important topic and the paper provides interesting insights about how data augmentation interacts with RL training.\n- The proposed methods are simple to implement on top of DrAC and could be easily adopted by practitioners.\n\n\n**Weaknesses**\n\n- The evaluation protocol differs from [25] despite the proposed methods are a direct extension of DrAC. If I understood it correctly, authors consider a more challenging case where the agents are only trained on a single background per task (*easybg*) instead of Procgen's *easy* setup; this is motivated in L203 as a way to *see clear advantage from visual augmentations*. Given that [25] already showed an advantage of using augmentations in the standard Procgen benchmark, this choice is difficult to understand. I would strongly encourage including results on both the original setting as well.\n- While the difference in evaluation protocol makes is difficult to compare results in this paper with those in the literature, it is surprising to see that DrAC underperforms PPO in most experiments (e.g. Figure 3, Table 2). This contradicts the results of [25]. Why is this the case?\n- Given that the methods are an extension of DrAC (splitting the optimization of the RL and consistency losses in two stages, and adding an identity arm to the bandit), presenting DA/InDA/ExDA as entirely novel contributions (e.g. Sections 1 and 4) is somewhat of an overclaim. Note that properly positioning itself as an analysis or extension of DrAC would not reduce the merits of the paper but would make it easier for readers to understand the problems being studied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o_pV3A9RrEh", "reviewer": "Reviewer_fohF", "summary": "This paper proposes two data-augmentation techniques to improve sample efficiency and generalization capability based on timing. (1) intra distillation separates proximal policy optimization and distillation optimization, and is applied during the whole training period; (2) extra distillation is applied after training. ", "questions": "- From Table 1, it seems that the improvement is not significant.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- It is well-written and easy to understand. The figures describe the proposed paper in an easy-to-understand graphical format.\n- the proposed method is simple and effective.\n\n## Weaknesses\n\n### The formulation is not general enough\n\nIn the background section, the authors fix the observation representation into two-dimensional square image. It seems to be the implementation detail instead of the formulation. For example, the observation could not be limited to images, the image size could not be limited to be square (i.e., $\\mathbb{R}^{m\\times n}$).\n\n### Ambiguous statement and lack proof. \n\nIn line 146-147, the authours state that the proposed method is more stable than the alternatives. I wonder stability competes in terms of learning theory, sample efficiency, or optimization? Besides the intuitive illustration in Figure 2, can you provide the proof to support your statement?\n\n### Data distribution modelling vs. proposed separation training between $\\mathcal{L}_{PPO}$ and $\\mathcal{L}_{DA}$]\n\nThe authors propose to alternate PPO and DA optimization, what is the benefits compared with directly modeling the data distribution, or can you provide some insights for distribution modeling?\n\n### Lack of ellobration for ExDA\n\nThe proposed ExDA is one of the main contributions in this paper, but the authors leave some details of computational cost reduction into the supplementary part. It would be better if you can give some definitions/remarks/proofs to show why removing the value inconsistency measurement helps? If due to the space issue, I suggest combining Figure 3 and Figure 4 to fill the blank area.\n\n### Lack of SOTA baselines  \nThe comparisons between the proposed method and baselines are not thorough enough. For example, CURL[1] and DrQ[2] are missed.\n\nSome minor typos are listed below:   \n* Line 72, learn with -> to learn with;\n* Line  74-75,  a limited applicability -> limited applicability;  \n* Line 78, a curriculum learning -> curriculum learning;  \n* Line 203, clear advantage -> a clear advantage.  \n\n*Reference*  \n[1] Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. \"Curl: Contrastive unsupervised representations for reinforcement learning.\" International Conference on Machine Learning. PMLR, 2020.  \n[2] Yarats, Denis, Ilya Kostrikov, and Rob Fergus. \"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.\" International Conference on Learning Representations. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2lUXPE3k8J7", "reviewer": "Reviewer_tBmt", "summary": "In this paper, the authors investigate the question of how and when to use data augmentation (DA) in the deep reinforcement learning (DRL) training pipeline since it can interfere with the training and its effect is going to be time-sensitive for non-stationary RL. In this context, they propose two DA schemes depending on the use case, whether for sample efficiency (InDA) or for generalization (ExDA). They also propose an adaptive DA scheme, UCB-ExDA, with all the augmentations, including the option to not augment as arms of the MAB problem, followed by ExDA. The idea behind these DA schemes is to have a standalone distillation method at any point of the training-- during it as in the case of InDA, or after it as in the case of ExDA. DRL training and DA distillation are kept separate, and the authors also who the usefulness of postponing DA to the end of DRL training.", "questions": "- It would be interesting to see more discussion on DrAC+PAGrad, especially looking at its performance when considering separating out distillation from RL training. The authors say that they devise it as another mechanism to relieve the interference between RL training and augmentation, but why?\n- It's interesting that removing the value inconsistency measure from the distillation loss doesn't degrade RL performance-- why would this be so?", "limitations": "The authors note that their proposed framework and findings are limited to the unique setting of non-stationary RL. The authors also state that there is potentially no negative societal impact of their work, and to the best of my understanding and imagination, I can also not think of one, since the paper is more investigative in nature.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nThe proposed DA strategies are well-motivated and well-discussed and the authors also find significant contrasting time-sensitivities of DA methods, which is interesting and demands further investigation. Distillation losses are well-illustrated and well-explained. Experiments and ablations are helpful-- the timing, performance, and different types of transformations are extensively studied.\n\n*Weaknesses*\n\n- The experimental design is sound, but it would be good to see some experiments on less computationally intensive benchmarks that do not require training for 25mn timesteps-- are there any, why not? Or at least, this should be addressed in the paper.\n- Minor/writing: Possible typo on line 189 on Page 5-- N_k(s) is the number of *times*?\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-lMoQoE9mhl", "reviewer": "Reviewer_ZudK", "summary": "Data augmentation is a critical regularization tool in supervised learning, used to induce prior domain knowledge into the model. In reinforcement learning however, data augmentation hurts and causes interference with training. In this work, the authors propose simple methods to incorporate data augmentation during learning, improving model generalization while alleviating the interference with training. Additionally, the authors propose a method that adaptively select the appropriate set of augmentations for an RL task.", "questions": "1. While InDA and ExDA individually make sense, what would happen if both methods are applying (performing distillation during and after training)? \n2. When performing adaptive selection, how do the UCB-ExDA and UCB-InDA methods differ in their choice of the most beneficial augmentations? Do they always converge to the same augmentation?", "limitations": "The authors address and discuss the main limitations of the work", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors present a simple and novel framework for using data augmentation in reinforcement learning. The motivation for the method is grounded in observations of the effects of data augmentation in supervised learning. The proposed method addresses the problem of RL training interference in a simple way. The results on the OpenAI Procgen benchmark are convincing and the authors both validate their methods and propose different insights into how augmentation works in the context of RL. \n2. This work is relevant and I believe it can of interest to the community at large. Data augmentation has long been a critical component of supervised learning (and self-supervised learning), and it is important to have work such as this that utilized this framework in RL.\n3. The authors clearly discuss the differences between the two proposed methods InDA and ExDA, and provide sufficient details to enable the reader to choose between the two. ExDA, in particular, is promising due to its low computational overhead.\n4. UCB variants of the two methods are also proposed. They enable the model to find, in a set of augmentations, the most beneficial ones. While this approach is not novel, the addition of the \"identity\" augmentation appears to be a critical and relevant component to avoid interference with training in particular cases.\n\nWeaknesses:\n1. PPO is used as the baseline RL algorithm, no other algorithm is tested, so it is uncertain how general the method and the results are. The same thing goes for the benchmark RL tasks that were considered: all video games, with 16x16 images.\n2. In section 5.2, ExDA is applied after 20M timesteps of RL training. After the 20M timesteps, no further RL training is performed, which is not the case for InDA. InDA continues to switch between RL optimization steps and distillation steps. For fair comparison between ExDA and InDA, it might be necessary to have the same number of total RL training steps, i.e. training for 25M steps before introducing ExDA\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept