Revisiting Data Augmentation in Deep Reinforcement Learning
Authors: Jianshu Hu
Yunpeng Jiang
Paul Weng
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Data Augmentation
Primary area: reinforcement learning

Abstract: Various data augmentation techniques have been recently proposed in image-based deep reinforcement learning (DRL).
Although they empirically demonstrate the effectiveness of data augmentation for improving sample efficiency or generalization, which technique should be preferred is not always clear. 
To tackle this question, we analyze existing methods to better understand them and to uncover how they are connected.
Notably, by expressing the variance of the Q-targets and that of the empirical actor/critic losses of these methods, we can analyze the effects of their different components and compare them.
We furthermore formulate an explanation about how these methods may be affected by choosing different data augmentation transformations in calculating the target Q-values.
This analysis suggests recommendations on how to exploit data augmentation in a more principled way.
In addition, we include a regularization term called tangent prop, previously proposed in computer vision, but whose adaptation to DRL is novel to the best of our knowledge.
We evaluate our proposition and validate our analysis in several domains. 
Compared to different relevant baselines,  we demonstrate that it achieves state-of-the-art performance in most environments and shows higher sample efficiency and better generalization ability in some complex environments.

Published as a conference paper at ICLR 2024
REVISITING DATA AUGMENTATION IN DEEP REINFORCEMENT LEARNING
Jianshu Hu, Yumpeng Jiang
Paul Weng
UM-SJTU Joint Institute
Data Science Research Center
Shanghai Jiao Tong University
Duke Kunshan University
Shanghai, China
Kunshan, Jiangsu, China
(his1998, jesss6leesssss.eeeeee
paul . weng@duke, edu
ABSTRACT
Various data augmentation techniques have been recently proposed in image- based deep reinforcement learning (DRL). Although they empirically demonstrate the effectiveness of data augmentation for improving sample efficiency or gener- alization, which technique should be preferred is always clear. To tackle this question, we analyze existing methods to better understand them and to uncover how they are connected. Notably, by expressing the variance of the Q-targets and that of the empirical actor/critic losses of these methods, we can analyze the ef- fects of their different components and compare them. We furthermore formulate explanation about how these methods may be affected by choosing different data augmentation transformations in calculating the target Q-values. This anal- ysis suggests recommendations on how to exploit data augmentation in more principled way. In addition, we include a regularization term called tangent prop, previously proposed in computer vision, but whose adaptation to DRL is novel the best of our knowledge. We evaluate our proposition and validate our analysis in several domains. Compared to different relevant baselines, we demonstrate that it achieves state-of-ther performance in most environments and shows higher sample efficiency and better generalization ability in some complex environments.
INTRODUCTION
Although Deep Reinforcement Learning (DRL) has shown its effectiveness in various tasks, like playing video games (Mnih et al.] 2013) and solving control tasks (Li al., 2019), it still suffers from low sample efficiency and poor generalization ability. To tackle those two issues, data aug- mentation, a proven simple and efficient technique in computer vision (Kumar et al. 2023) starts to be actively studied in image-based based DRL where it has been implemented in various ways (Laskin cct al. 2020 Kostrikov et al. 2020 Raileanu et al. 2021). However, most such work has been mainly experimental and more principled comparison between these state-of-the-as the-art methods is lacking.
In this paper, we study data augmentation techniques in image-based based online DRL. We formulate a general actor-critic scheme integrating data augmentation. We then show that current data augmen- tation methods, categorized as explicit/implici regularization, are instances of this general scheme. In explicit regularization, image transformations augmented samples to explicitly enforce invariance in the actor and critic. By contrast, image trans- formations are directly applied on the observations during training in implicit regularization.
used in regularization terms calculated with
Following the analysis about implicit and explicit regularization, we propose a principled data aug- mentation method in DRL and further justify its design. We start the justification with discussion about applying different image transformations in calculating the target Q-values. Hansen et al. (2021) propose to avoid using complex image transformations in calculating the targets stabilize the training. We provide further analysis on why some image transformations are complex and not suitable for calculating the target and how to judge whether an image transformation is complex or not.
TThe code of method: httpp:://iituub.ccm//aaaaaa

---

Published as a conference paper at ICLR 2024
In addition, we justify other components of our method (e.g., KL regularization in policy) by an- alyzing the variance of the actor/critic losses and the variance of the Q-estimation under image transformation, which is important to control since applying random image transformations neces- sarily increase the variance of those statistics. The analysis also reveals the importance of learning the invariance in eritic for stabilizing the training. This latter observation motivates to include an adaption of tangent prop (Simard et al. 1991) regularization in the training of the critic.
Contributions: (1) We analyze existing state-of-the-art data augmentation methods in image- based DRL and show how they are related. (2) We provide an empirical and theoretical analysis for the different components in these methods to justify their effectiveness. (3) Based on our anal- ysis, we propose a principled data augmentation actor-critic scheme, which also includes tangent prop, which is novel in DRL. (4) We empirically validate our analysis and evaluate our method.
RELATED WORK
Many data augmentation techniques for DRL have been proposed, mostly for image-based based DRL
(Ma et al. 2022). although the fully-observable setting has also been considered Lin et al. (2020). Data augmentation can be used generate artificial observations, transitions, or trajectories for an existing DRI. algorithm or for improving representation learning. Most propositions investigate the usual online DRL training, but recently data augmentation with self-supervised learning DRL (Srinivas et al. 2020; Schwarzer et al., [021) has become more active following its success in com- puter vision (He al.) 2020, Grill et al. 2020p. For space reasons, we focus our discussion on the most relevant methods for our work: data augmentation for image-based online DRL.
The first methods to leverage data augmentation in DRL apply transformations directly on obser- vations to generate artificial ones to train the RL agent, which can lead to better generalization (Cobbe et al. 2018). particular, Reinforcement learning with Augmented Data (RAD) (Laskin et al. 2020) extends Soft Actor-Critic (SAC) (Haarnoja et al.) 2018) and Proximal Policy Opti- mization (PPO) (Schulman et al., 2017) to directly train with augmented observations. Instead, Data-regularized Q (DrQ) (Kostrikov et al. 2020) introduces in SAC the idea of using aver- aged Q-target by leveraging more augmented samples. Using Deep Deterministic Policy Gradient (DDPG) (Lillicrap al. 2015), DrQ-v2 (Yarats et al., (221)) provides various implementation and hyperparameter optimizations and gives up the idea of averaged Q-target. SVEA (Hansen et al.) 2 avoids using complex image transformation when calculating the target values because doing 80 increases the variance of the target value. Raileanu et al. (202T) argue that simply applying image transformations on the observations in PPO may lead to a wrong estimation of the actor loss and instead propose adding regularization terms in the actor and critic losses. They also propose some methods of automatically choosing image transformation. recent work like (Liuu et al. 2023 Yuan et al.) 2022), they investigate problems of using a trainable image transformation which is orthogonal to ours.
3 BACKGROUND
In this section, we define notations, recall invariant transformations in DRL, formulate a generic actor-critic scheme exploiting data augmentation, and explain how existing methods fit this scheme according to how data augmentation is applied in the actor and critic losses.
For any set X' A(X') denotes the set of probability distributions over X. For a random variable X, [[ (resp. V[X) denotes its expectation (resp. variance). For any function and i.i.d. Notations
samples Z.. EN of X, B [X)] = 1/4 Σ o(x;) is an empirical mean estimating E[o(X)).
A Markov Decision Process (MDP) M = (S, A,r, T, pa) is composed of set of state S, set of action A. a reward function r S x A + R, a transition function T : S x A + ASS), and a probability distribution over initial states Po A(S). In reinforcement learning (RL), the agent is trained by interacting with the environment to learn a policy πT. s) E ((A) such that the expected return E,, Eee 7're 80 pa] is maximized.
2

---

Published as a conference paper at ICLR 2024
Algorithm Data-Augmented Off-policy Actor-Critic Scheme
Hyperparameters: total number of training steps T, mini-batch size N. policy update frequency K, image transformation set Fi = {fr E T}, distributions P, B E A(T), random variables transform states.
p.
1: Initialize critic Qs(s, a). actor (s), and empty replay buffer D.
2: Start with initial state so-
3: for 0... do
4: Interact with the environment using action from current policy ar ~ T(. st).
5: Save transition (s,, a,, ft, St-1) in replay buffer D.
6: if episode ends then reset 814-1 an initial state end if
7:
// Actor-critic update
8: Sample mini-batch {(s;, oz. st)) l i= I..... I......N N3 from D.
9:
L
Ei (8, 0z, Mi, 8/, U1
10:
Update ø (critic parameters) with gradient of L then Lg N E (si.
if t mod
12:
Update (actor parameters) with gradient of Lg.
14:
end if
end for
In image-based RL, the underlying model is actually partially observable MDP (POMDP): the agent observes images instead of states. However, following common practice, we approximate this POMDP with MDP by assuming that state is stack of several consecutive images. With data augmentation, the same transformation applied on all the stacked images of that state.
Invariant Transformations Data augmentation in image-based control tasks assumes that a set Fi {Sr RAx + k _ T E T of parameterized (h x w-sized) image transformations fr that leave optimal policies invariant is given for some parameter set 'T. Exploiting this property in online RL is difficult, since an optimal policy is unknown. However, it suggests to directly focus learning policies that are invariant with respect to this set FT. Recall:
Definition (r-innariancee) A policy æ is invariant with respect to an image transformation fr if:
x(a (a f+()) for all S, A.
(1)
Interestingly, the Q-functions of those policies satisfy the following invariance property: Definition (Q-invariance). @-function is invariant with respect to image transformation fr if: Q(s,a) Q(f((s),) a) for all s E S,a E
The definitions imply that Fy can be assumed be closed under composition without loss of gener- ality. We assume that set T contains To such that fr, is the identity function, which would allow the possibility of performing any image transformation. Examples of image transformation instance: (small) random shift, which pads and then randomly crops the images; random overlay, which combines original images with extra images. Empirically, random shift has been shown to be one of the best transformations to accelcrate DRL training across a diversity of RL tasks, while random overlay helpful for generalization.
for
We formulate simple actor-critic scheme (Algorithm 1). which boils down to standard off ofo-cyoicy actor-critic if the original losses are applied. Existing lutaaauumentttioon based DRL methods fit this scheme by enforcing Q-invariance and/or r-invariance via explicit or implicit regularization, as explained next. We discuss them next and explain how they fit Algorithm [ Due to the page limit, recall all the related DRL algorithms in Appendix A and consider SAC (Haarnoja et al. 2018 as the base algorithm the main text and discuss the variant with DDPG (Lillicrap et al, 2015 in the corresponding appendices.
3.1
EXPLICIT REGULARIZZTION
The invariant transformations F. can be directly used to promote the invariance of the learned O-function and learn more invariant policies with respect them. Formally, this can be achieved
3

---

Published as a conference paper at ICLR 2024
by formulating the following (empirical) critic loss and actor loss with two explicit regularization terms: for a transition (s,a, r, s') and random variables v. μ over T
as,
(Qo a) a's + aqEu((Qo(f((ss)) Qp a)) and (3) Qa(s, a) + a (x.,,, (xi. [1) TT( (+()))
(4) where 0, 8 are the parameter of the critic Qa and actor kp, target y(s', a²)) is defined as r + (Q a') ) - a log (a(asss)) with a' TT(- s') and target network parameter o, action â is sampled from T 8). coefficients a, aq, and Or respectively correspond to the entropy term and the regularization terms to promote invariance in the critic and actor, and subscript sg represents "stop gradient" for the corresponding term.
(s, a [a 8)
In practice, Equations 3 and 4 could be used in an actor-critic algorithm: for instance, they could replace the losses in lines 9 and 12in in Algorithm[] [ Note that DrAC (Raileanu et al. [202] uses them by setting the distribution of and ş to be uniform (although DrAC is based on the PPO algorithm).
3.2 IMPLICIT REGULARIZATION
Another commonly used data augmentation method in DRL is directly applying the image trans- formation on states during training. Formally, for a transition (s, a, r, r²)) and random variables v, J over T, the (empirical) critic and actor losses with implicit regularization are respectively:
- -Ell((f(()) E, S', p, E., (
and
(5)
(0, v) = B log (ā (a(()) - (s) a log no(a') (8)) (s), a' nT(: fx(s')), and anda- no(. fu(s))-
(6)
with (d)
The empirical means can be calculated with different numbers of samples: M and K samples for E, and E,, in the critic loss, and J samples for E, in the actor loss. Note that with K >>> 1, the corresponding empirical expectation corresponds to the average target used in DrQ. If v and μ have a uniform distribution, Eguations [5] and 6 are the losses used in DrQ (Kostrikov et al. (020) with
1 and RAD (Laskin et al. 2020) with M 1.K 1, and =11. Moreover, different image transformations for A/ and j can be used. In SVEA (Hansen et al. 202I), complex image transformations are only applied on states s and included in random variable in the critic loss.
THEORETICAL DISCUSSION
In this section, we theoretically compare explicit regularization with implicit regularization.
4.1
CRITIC LOss
Before comparing the critic losses in these two regularizations, we first compare an alternative ex- plicit regularization one may think of to enforce Q --invariance, which uses:
(s,,,,,,, ,V) (Qo(s,a) (Qo(s.a)-((s'ccll (()))'') + E ((oolt())).) (
u(s', ('),
(7) The only difference compared with Equation iss to replace the target in the regularization term with yss, a'). Intuitively, using y(s', a²) is better target because it not only promotes the invariance in the critic, but also serves as target improve the critic. In Appendix C.T we show that using v(s', a') leads to smaller bias which may be preferred in explicit regul arization.
Now the critic losses in explicit regularization and implicit regularization can be connected by setting the distributions of the image transformation parameters, as shown in the following simple lemma:
Lemma 1. C.2 There exist distributions for è and ju such that we have for any sample v, (aQ + 1) (s. a, .2,62,,0 (i) = 1 (s, ,,s',v)
Note that the extra factor aq + 1 can be controlled by adjusting the learning rate of gradient descent. Therefore, explicit regularization amounts to assigning a higher probability to To-
All detailed derivations/prrofs are in the appendix. The appendix number is provided for ease of reference.
4

---

Published as a conference paper at ICLR 2024
4.2 ACTOR LOSS
Obviously, using the actor loss in explicit regularization (Equation 4 promotes the invariance in the policy. More interestingly, the actor loss in implicit regularization (Equation 6) also implicitly enforces policy invariance, but under some conditions, as we show next. First, recall that like SAC, the target policy used in the actor loss for state s can be defined as g(a s) exp( Qo (s, a) log (s)), in which Z is the partition function for normalizing the distribution. We can prove that the actor loss implicit regularization can be rewritten with a KL regularization term:
B. Assume that the critic is invariant with respect transformations in Fr, i.e. Qo(f+(s), = Q(s, a) for all T E T, a E A. For any random variable μu, the actor loss in implicit regularization (Equation 0 can be rewritten:
Proposition 4.1.
(gs pb) - =Eu To(a fu(s)) log
To.sg (a 1 8) + Ep DKL T fu(()) To ( (8)
g(a s)
Intuitively, with the actor loss in implicit regularization, the policy To fu)(s) for each augmented state is trained to get close to its target policy g(- (s). which is completely defined by the critic. When the invariance of the critic has been learned, the To(- (s)) for different augmented states are actually trained to get close to the same target policy, which then induces policy invariance.
Interestingly, the direction of the KI divergences in Equation 4 and Equation 8 is reversed with respect to the stop gradient. The KL divergence in Equation 4 should be preferred because the other one also implicitly maximizes the policy entropy (see Appendix D). which leads to less stable training. as we have observed empirically. Since policy invariance is only really enforced when the critic is already invariant and that the implicitly enforced KL divergence has opposite direction, seems preferable to directly enforce policy invariance like in explicit regularization (Equation (
Following the idea of using averaged target in the critic loss, another interesting question about applying KL regularization is whether to use policy No. [s)) of transformed state as the target:
E (s) log (ā s) Qs(s, a) E. (D.L(TT.. (s)) |I To( (
(9)
where 77 and μ follow the same distribution. Intuitively, the regularization term in this equation enforces the invariance across policies of different transformed states and somehow is equivalent to
using an averaged policy as the target. As shown in Appendix C.3] the loss in Equation is an upper
bound of the loss with an averaged policy Tang(: (: 8) = [ [m
[n]())] (1
as the target:
E (8, μ1) = a log (ā 8) - Qo(s, aa By D (Tang |s)) ne (-Ifx(())))
(10)
So using Equation 9 is somehow equivalent to using averaged policy as the target in the KL regularization term and thus has an effect of smoothing, which may be preferred.
PRINCIPLED DATA-AUGEETTED OFF-POLICY ACTOR-CRITIE ALGORITHM
Following these analyses, we finally propose our generic algorithm and provide justification for it.
5.1 GENERIC ALGORITHM
Formally, our generic algorithm follows the structure in Algorithm but with the following (empir- ical) critic and actor losses, ar, ,, M) and Co p)), for a single transition (s, a, r,s")
lo EarEn, (Q)(L..(o)),)-BBLlll.
+ Fa.pB.lVy
and
(11)
6 Enp aa log To (a)) +a, ( (.,,
To fa (+)(
(12)
"oo simplify notations, drop the parameters of the losses when there is risk of confusion.
5

---

Published as a conference paper at ICLR 2024
where Por is the number of types of image transformations used in the training. fu, (resp. fas fu)
corresponds to the transformation parameterized by random variable vs (resp. 1177) and a (resp. atp. ax) is the coefficient of the mean-squared errors for different image transformations (resp.
tangent prop term, KI. regularization term). Note that 1) and j follow the same distribution.
5.2 JUSTIFICATION FOR THE GENERIC ALGORITHM
In this section, we justify further the formulation of our generic algorithm by analyzing first the effects of applying different image transformations in calculating the target Q-values and then the estimation of the empirical actor/critic losses under data augmentation. While the variances of these estimations are unavoidably increased when random image transformations are applied, they can be controlled by several techniques, as we discuss below. Recall that a lower variance of an empirical loss implies a lower variance of its corresponding stochastic gradient, which leads to less noisy updates and may therefore stabilize training. Moreover, a lower variance of target Q-values entails a lower variance of the empirical critic loss, which then similarly leads more stable training.
Applying image transtormations in calculating the target A simple technique to reduce the variance of an estimator is to use more samples. Recall that in contrast to RAD, DrQ leverages more augmented samples and introduces the average target, which can be interpreted as using more augmented samples for estimating the target Q-values. In Appendix [..]] we show that DrQ enjoys smaller variance of the empirical critic loss and the target Q-values than RAD, which may explain the empirical better performance of DrQ (see e.g.. (Kostrikov et al. 2020
However, when facing complex image transformations, |Hansen et al. (2021) propose avoid using them in calculating the target Q-values to reduce the variance of the target and thus stabilize the training. Interestingly, we observe experimentally that even using complex image transformation such as random conv in the target does not induce a large variance in the target while much larger bias is observed for the trained critic, shown in Appendix F. Compared to using random shift which is a finite set of image transformations, using random conv actually applies an infinite set of transformations. To maintain the invariance among augmented states, more updates are required to train the agent sufficiently. To validate the idea that the invariance with respect to complex image transformations is harder to learn and more updates are required when using them in calculating the target, evaluate increasing the number of updates in each iteration when using different image transformations in calculating the target, as shown in Appendix F From the experimental results, observe that the cosine similarity between the augmented features at early stage (e.g., 100k training steps) could be used as a criterion for judging if an image transformation is complex or not. Moreover, for those complex image transformations, updating more is helpful when we apply them in calculating the target Q-values.
Adding KL regularization The analysis Section 4.2 suggests that explicitly adding a KL regu- larization term, D, DKI(TTf (-1 (-1l,(()) || ( fu with 17 and μ following the same distribu- tion over Fr, in the actor loss can help better learn the invariance of the actor. Thus, we define the actor loss in the generic algorithm as the of the actor loss in implicit regularization (Equation and this KI. regularization term. Below, we provide two additional justifications for this choice.
Firstly, show in the following proposition that the variance of the actor loss in implicit regu- larization can be controlled by a KL divergence if the invariance is already learned for critic Qo- Hence, adding KL regularization term in the actor loss may both enforce invariance and reduce the variance of the implicit actor loss.
Proposition 5.1. we have:
(E2) Assuming that critic Q is invariant with respect to transformations in FT,
1 ).Ex (E,,) clf c(Fu(s)
Vpled(s, 1J] < n
(13)
where c(fu(s)) > 0 and n is the number of samples to estimate the empirical mean (a(s, (s, u))
In addition, under invariance of the target critic Q5. we can prove that the variance of target Q-values can be controlled by KL divergence. Therefore, training with KL regularization term may further reduce the variance of the target, which would also consequently reduce the variance of the critic loss.

---

Published as a conference paper at ICLR 2024
Proposition 5.2. in Fr, we have
1E.3
Assuming that target critic Q5 is invariant with respect transformations
VulY 1)] T
E (En max(y(f - (Dn.l))
(14)
where target ((). =EE PullBuswaottlll ooteootttttoeeolllueee ( d)) v(f((s'), - ++ ar llooooa' (fa))((()')) and n is the number of samples to estimate the empirical mean (s',11 11).
Although those results are obtained under invariance of the (current or target) critics, we may expect an approximate reduction of the variances by adding KL regularization term, when the critics approximately invariant. Experimentally, we can observe that the variance of target and critic losses can be reduced thanks to this KL regularization term.
Tangent Prop We now introduce an additional regularization to promote the invariance of the critic. This addition motivated by our previous analysis, which indicates that the critic invari- ance is an important factor for stabilizing training. This additional regularization term is based on tangent prop (Simard et al. [991). which was proposed for computer vision tasks, to promote in- variance by enforcing that the gradient of the trained model with respect to the magnitude of image transformation be zero. In DRL, when applied to the critic, it can be formulated as follows.
If the invariance of Q-function over the whole set of the image transformation parameter is required and the Q-function is differentiable with respect to т, tangent prop regularization is actually adding the constraints the derivative of the Q-function with respect to
(s), δT
(Q(f+(s), a) (sr(s) (Q(f (r) fs() 0f-(s)
0.
(15)
0f+(s) Or
8r
Although the computational cost is increased due to the differentiation of Q, this regularization is still beneficial since sample efficiency often one of main concerns when applying DRL. Compared to the original tangent prop, we also extend it to a broader application by applying it not only on the original state but also on the transformed states. Specifically, instead of only calculating the derivative around To. the derivative is estimated at any E T. A theoretical justification for how tangent prop regularization is related to the training of the critic can be found in Appendix B.2
EXPERIMENTAL RESULTS
In order to validate our theoretical analysis and show the effectiveness of our proposed algorithm, we perform series of experiments to (1) experimentally validate our propositions, (2) conduct case study explicitly showing the statistics we analyzed, (3) compare our final proposed algorithm with state-of-the-at baselines (RAD, DrAC, DrQ, DrQv2, SVEA) to verify its sample efficiency, and evaluate its generalization ability against SVEA, which was specifically-desinne for this purpose.
Experimental Set Up We evaluate different methods on environments from Deep.indd Control Suite TTssa et al., 2018) with normal background for evaluating sample efficiency and distracted background for evaluating generalization ability. DeepMind Control Suite is a commonly used benchmark for evaluating methods of applying data augmentation in DRL. Across different envi- ronments, all hyperparameters are listed Appendix [G such as learning rates and batch size for the actor and critic. All experiments are performed with 5 different random seeds and the agent evaluated every 10k environment steps, whose performance measured by cumulative rewards averaged over 10 evaluation episodes.
Ablation study To validate our propositions, we first compare RAD M=1,K=1], RAD+ [M=2.K=1] and DrQ [M= using the official implementation from DrQ where [M, K) are respectively the numbers of samples used for estimating the empirical mean over random variables [v, p1] in the critic loss. We then show the effectiveness of KL regularization and tangent prop reg- ularization by adding them one by one based on DrQ. All these methods are trained with normal
https: tttp//ttttttt.ooe wtw.hhhw.....eeeeececoeeeeeeeeee wwteeenisyaatttttt way.aatt////. drq

---

Published as a conference paper at ICLR 2024
Median
IQM
Mean
Optimality Gap
ours DrQ+KL DrQ RAD
RAD
0.60
0.75
0.900.56
0.64
0.72 0.80
0.56
0.64
0.72 0.80
0.24
0.32
0.40
0.48
Normalized Score
Figure 1: Aggregated results of validating our propositions.
Methodk
Sid critic
target
loss
wallenr _un_ab ation
590
RAD
RAD
0.772
0.321
4.911
0.366
BAB+
#1.335
20.084
#1444
#0.015
DID
D/D
Dipert Ex targer
(a)
+-00043
0.271
+00031
+0.663
5.011
+0.021
120
OUIT
DQQ
10.123
0.358
10.055
0.225
10.961
5.077
10.055
0.291
DeQ+KL(ix target)
10,042
0.384
=0.007
0.221
11225
5.708
#0:000
0.135
300
DrQ+KL
+0.053
0.519
(b)
+0.18
0.183
(b)
+-1.745
4.408
(b)
+0.010
0.099
100050
200080
300000
430000
ours
(c)
10.001
0.280
(
10.001
0.182
10.412
4.15
10.007
0.101
evelusiion 58O55
Figure Performance of different methods in walker run environment.
Table 1: Important statistics of different methods in walker run environment. (a)(b(() are discussed in the main text.
background using random shift as the image transformation and also evaluated with normal back- ground. When applicable, we adopt hyperparameters from (Kostrikov et al.) 2020) except that we reduce the batch size from 512 to 256 (to make the algorithms easier to run on our computing device, equipped with one NVIDIA RTX 3060 GPU and Intel i7-10700 CPU). Considering that only one image transformation (random shift) is applied, the weight a; for it is set as 1 in Equation [T] We tune the weights agL and atp based on quick grid search in {0.1, 0.5, 1.0} and finally choose 0.1 for both L and Ottp- The results of validating our proposition are shown in Figure | We follow the recommendations from /Agarwal et al.) (2021) to plot the aggregated performance over totally environments. Detailed training curves in each environments are included in Appendix
H.I H.I
Case study We conduct a comprehensive evaluation in walker-run using the same setting as above to answer the following questions: 1) Does our proposition help reduce the variance we are con- cerned about? 2) Does our proposition help learn the invariance the feature space? Moreover, we run the experiment of using a fixed target in the KL regularization validate our analysis in Section 4.2
Note that the set of parameters T is finite when using random shift image transformation. So, augmented Q-values (f( a) and augmented target Q-values Qz(fr (s"), a') ) for all T, E T recorded within mini-batch for every 10k environment steps. These are used to caleulate the standard deviation of the empirical critic loss and the target Q-values. The KI. divergence between policies for augmented samples is also recorded with the same frequency. The results for this case study are shown in Figure 2 and Table I From the table, we can see (a) the variance of the critic loss and the variance of the target Q decrease thanks more augmented samples, (b) KL regularization helps enforce the invariance in the actor and reduce the variance of the target and the variance of the actor loss, (c) tangent prop further improves learning the invariance in the critic.
Meanwhile, the feature vectors of augmented samples within a mini batch after the encoder from the actor and critic are recorded with the same frequency. To estimate the invariance the feature space, we first calculate the cosine similarities between the augmented features. We also apply t- SNE (van der Maaten Hinton| 2008) to project the latent features into 2D space and calculate the La-distance between the projected points. With these measures, the learned invariance of the actor and critic features along the training are shown in Appendix H.2 Following our proposition, the agent leam well the invariance in the feature spaces of both the actor and critic. From another view, this achieves the same goal as using the self--suervised learning losses (Srinivas et al. 2020
Grill al. 2020) to explicitly enforce the invariance in the latent space.
8

---

Published as a conference paper at ICLR 2024
Tager AOP J9ah
WAcher PAPE
Sager SUTT SAr male
5000
310m
5000
300
Figure 3: Partial results of evaluating sample efficiency for our methods.
Favirmments
SVEA
ourS
Methods
Std of critic loss
Std of target
KI.
walker walk
64.52260.33
50% 008.28879.82
5.780 =1.029
0.936 +0.097
0.404
wulker stand
788. 89..31
843.27=51.33
SVEA
+0001
cartpole swingup
351.96_62..66
366.77-158806
ball in cup catch
445.44
169.50::14..99
ours
=0.760
3.197
=0.038
0.530
0.033
0.205
finger spin
77..6
447.99 73.52
Table 2: Comparison of SVEA and ours in DMControl with video-hard backgrounds for training 500k env steps.
Table Statistics for the trained model of SVEA and our method using random overlay in walker walk en- vironment.
Main result We evaluate our proposed methods in sample efficiency and generalization ability. For sample efficiency, we evaluate our method (ours) against state-of-the-art baselines: RAD, DrQ, DrAC (with SAC as the base algorithm), DrQv2 (with DDPG as the base algorithm) and SVEA (with random overlay). The experimental results shown in Figure 3 confirm that our method outperforms previous ones. Due to the page limit, the additional evaluations included in Appendix H.3
For generalization, we compare SVEA with our method based on the official implementation of SVEAT Considering that random overlay shows the best generalization ability among the image transformations listed in SVEA, our comparison also includes random overlay. When applicable, adopt the same hyperparameter values in SVEA. Thus, shift and random overlay, reuse a = 0.1 from previous experiments and tune Ctpp based on a quick grid search in (0.1, 0.5) and finally choose 0.5 for 0ttp Both methods (SVEA and ours) trained in normal background for 500k environment steps, and evaluated in vidco-hard backgrounds. The final results comparing SVEA and our method are shown in Table [ We can see the improvement in generalization especially in environments such as ball in cup catch, finger spin, and walker walk. Meanwhile, similar statisties we discussed before are recorded, as shown in Table 3 The complete
the a; 0.5 for random
curves for evaluations and the recorded statistics are listed in Appendices H.4 and H.5
7 CONCLUSION
We revisit ffte-of-the-aite data augmentation methods in DRL. Our theoretical analysis helps under- stand and compare different existing methods. addition, this analysis provides recommendations on how to exploit data augmentation in more theoretically-motivaee way. We included tangent prop, regularization term to promote invariance, which novel in DRL. We validated our proposi- tions and evaluated our method in DeepMind control tasks with normal background for comparing sample efficiency and distracted background for comparing generalization ability with the state-of- the-art methods. Limitations are discussed in Appendix due to the page limit.
https::
ok http:////mm..kkaaaaaaaaaa/anoooo -ennera
general iizttin
benchmark

---

Published as a conference paper at ICLR 2024
Reproducibility For the experimental results, the code with comments on how to reproduce the results is released, the experimental settings are described in the main paper (Section 6 and the hyperparameters required for reproducing the results are recorded in Appendix G For the theoretical results, all the detailed proofs can be found in the appendix.
This work has been supported in part by the program of National Natural Science Foundation of China (No. 62176154).
Acknowledgments
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. CoRR, 0tos/2108.112660 2021. (RLLttts:://rre
org/ abs/ 22000 13264
Randall Balestriero, Ishan Misra, and Yann LeCun. A data-augmentation worth a thou- sand samples: Analytical moments and sampling-free training. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 19631-19644. Curran Associates, Inc.,
hetp.//edde............ooooo./eo Fi F111://000cSSaaa0/0//aaaaa131110
2022. URL
Karl Cobbe, Oleg Klimov, Christopher Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. CoRR, abs/1812.023311, 2018. URLphttp://arxiv
1t5//bb/18111 02341
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. CoRR, abs/2006.07733, 2020. URL
https::/aaxxe w0w.://///../....oo////
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs/1801. ttss/180.012200 001.01200. 2018. http://ppp//axxii.orrr/aaaa/////
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep leaarnnng with convnets and vision transformers under data augmentation. CoRR, abs/2107.00644 2021. URL /tRppsttt://axii org/abs 21 07. 00644
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference Computer Vision doi: 10.109CCYP4420002000000077
and Pattern Recognition (CVPR), pp. 9726-9735 2020.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regulariz-
ing deep reinforcement learning from pixels. CoRR, abs/2004. 13649, 2020. URL https:
//axiv.crg//ab//2000 13649
Teerath Kumar, Alessandra Mileo, Rob Brennan, and Malika Bendechache. Image data augmenta- tion approaches: A comprehensive survey and future directions, 2023.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas.
Reinforcement learning with augmented data. CoRR, abs/2004, 14990, 2020. URL https:
7/arxiv. //wxxiv.org/aas////e.e/e
abap////////ae /200/ 14990
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. A simple randomization technique for generalization in deep reinforcement learning. CoRR, abs/1910.05396. 2019. URL http://
arxiv.org/ abs 910.00.00 05396
Tingguang Li, Weitao Xi, Meng Fang, Jia Xu, and Max Q.-. Meng. Learning to solve rubik's cube with dexterous hand. In 2019 JEEE International Conference on Robotics and Bionimetics (ROBIO), pp. 1387-1393, 2019. doi: 10.1 01099//0000000442220099991111110
10

---

Published as a conference paper at ICLR 2024
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR. abs/1509.02971, 2015. URL
org/
Corpus ID: : 6326763
Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas, and Paul Weng. In- variant transform experience replay: Data augmentation for deep reinforcement learning. IEEE Robotics and Automation Letters and IROS, 2020.
Sicong Liu, Xi Sheryl Zhang, Yushuo Li, Yifan Zhang, and Jian Cheng. On the data-efficiency with contrastive image transformation in reinforcement learning, In The Eleventh International Conference on Learning Representations, 2023. URLhttps: hptp:////e//peeeee..eoeee net forum?
id=- hea.m-rxxic00 5ga
Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A com- prehensive survey of data augmentation in visual reinforcement learning, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URI htttt:://arii..rr//aaaaaee
5602
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http::::///ppeeeei..ee. n forum? f CCS fcJZVW
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs/1707.063477 2017. URL http://arxiv..rgg////oo
1707. 06347
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021.
net/ www.:/net/ffruut forum?
Id=uCQEPZwRaUi
Patrice Simard, Bernard Victorri, Yann Le Cun, and John Denker. Tangent prop: A formalism for specifying selected invariances in an adaptive network. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS'91, pp. 895-903, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1558602224.
Aravind Srinivas, Michael Laskin, and Picter Abbeel. CURL: contrastive unsupervised representa- tions for reinforcement learning. CoRR, abs/2004.04136, 2020. /RRLittts:/aaxxvvcoo
abs 2004. 04136
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Mar- tin A. Riedmiller. Deepmind control suite. CoRR, abs/1800.000690 2018. URL http: 7/arxiv. //Axx/v.orrr/aba//11100000
Manan Tomar, Utkarsh Aashu Mishra, Amy Zhang, and Matthew E. Taylor. Learnnng representa- tions for pixel-based control: What matters and why?, 2022. URL htttps:://ppeeeviiiiii
met//Forun22o--11222004222
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 986):2579-2605, 2008. URL http: jmlr. ort.pppers/////
wardeermaatenDDaahin
Denis Yarats, Rob Fergus, Alessandro Lazaric and Lerrel Pinto. Mastering visual continuous trol: Improved data-augmented reinforcement learning. CoRR, abs/2107.09645, 2021. URL
https::/aaxtie www./bg..s....ooooooeeeo..ooeeee 2210 09645
Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. Don't touch what matters: Task-aware lipschitz data augmentation for visual reinforcement learning, 2022.
11

---

Published as a conference paper at ICLR 2024
A BACKGROUNDS
Deep Deterministic Policy Gradient Incorporating a parameterized actor function pa(s), Deep Deterministic Policy Gradient uses the following actor and critic and loss to train the agent:
(0) B.,= (s a+ (t)lo.=no(a.)l
(16)
(Ø = E. a:) Q(s,, (() (m
where ax) = + 7Q ( ( ())+))), which the target Q-value defined from a target network, & and & represents the parameters of the actor and the critic respectively, 0 and á represents the parameters of the target actor and the target critic respectively. and D represents the replay buffer. The weights of a target network are the exponentially moving average of the online network's s weights.
Soft Actor-Critic Maximum entropy RL tackles an RL problem with an alternative objective func- tion, which favors more random policies: J = Rł,, IE 7'r, + (H( (- &t)]], where "y is the dis- count factor, a is a trainable coefficient of the entropy term and H( st)) is the entropy of action distribution x-- st). The Soft Actor-Critic (SAC) algorithm (Haarnoja et al. 2018) optimizes it by training the actor ny and critic Q with the following respective losses:
(6) D..~D,amn (a log no (a st) Qo(s),a),,
(17)
(o = E., a (Q (s., ar) - ((s).))))
where Q(st, = ((st,aa))r++(((((0 + ac+1) log (((+1 which the target Q-value defined from a target network and a++1 TT (. 11+1). 0, and represents the parameters of the actor, the critic and the target critic respectively, and D represents the replay buffer. The weights of target network are the exponentially moving average of the online network's 's weights.
Reinforcement Learning with Augmented Data Reinforcement Learning with Augmented Data (RAD) (Laskin et al. 2020) applies data augmentation in SAC by replacing the original observation with augmented observations in the training of the actor and critic. Given image transformation fu, the actor and critic losses are
(0) = E., E,,~D, (a To [b(())) - Q((((())))) (Q(0) Fe,, a.. (Qo() Q(). a.))
(18)
Reguaareul Q Data-regularized Q (DrQ) Kostrikov et al. 2020) extends RAD by using data augmentation in the training of the critic two new ways. Given a type of image transformation if parameterized by data augmentation is first applied in the calculation of the target Q-value for every transition (s, a, a,T. &.):'):
K
Σ
where a' ~ T(- (s')).
(19)
K
k=1
Q₂ is the slowly updated target network. Then the critic is updated with different augmented & and this averaged target:
(O(0)
1
Σ Σ
(s) v)²
(20)
NM
Note that DrQ recovers RAD when and K 1.
SVEA
In order to avoid non-deterministic Q-target and over-regularization, Hansen et al. (2021)
propose using state without complex augmentation for calculating the target. Let Ti and T set of random shift and set of random shift plus one of the data augmentation mentioned in the paper such as random convolution (Lee et al. 2019). The critic loss used for training is
N
Lo(0)= = N
O a (Qolf,, - 41)² + Bweea (Qo (f) v1)
(21)
T=1
12

---

Published as a conference paper at ICLR 2024
where Csvea and Seeea constant coefficients for naively and complexly augmented data respec- tively, T1, E Ti and T2,1 E T2, Yi = 7 + (22(( + a log T(a (s)),, where T. TT,((s)).
DrAC Instead of directly replacing the original samples with augmented samples in the training, Raileanu et al. (202T) use two regularization terms in the training of the actor and critic to explicitly enforce the invariance. When upplying
the PPO algorithm (Schulman al. 2017) learn
state-value estimator (s(s) and policy Te (s), the regularization terms are
Gy (Vss)-V((f(ss))
(22)
G = DKL s) na(a fs(s)
where (s) is the sum rewards collected by the agent after state and random variable for parameterizing the image transformation.
Tangent Prop Regularization Tangent prop (Simard et al. 1991) is regularization term used for learning invariance for function G(s) with respect small image transformation parameterized by on 8:
aG(s, da
=0 |
Σ
(23)
B EXPECTED LOSS UNDER DATA AUGMENTATION
B. ACTOR LOSS
SAC as base algorithm For imsee control tasks, data augmentation f parameterized by over T is applied on the observations. The actor loss with implicit regularization for the state in transition is
()( = By log To fs(()) Qo(Fu(s), lam ]
(24)
=Ryy DKL (T(-I5 (T exp( Qo(ss())))) (e log ((nx ((n(())))
Let g(fa(s),)) expl (o(((((s)))))) log (s)())))
(s
Eu |KL To TT( (T((111(66))l(((
Er [KK (T6((1((666)))10(61))))))))) DKL (=((
DEL Too (To((11((6))lo((s.)))
R, I/ To(aa (a(a)s((s)
To(a))p(s)) log {o(aa{((s))
g(alfu(s))
g(als)
+ E.
(25)
E, J. (T)a g(a)fu(s))
g(als)
+ E, (o)
E J. Te(alfu
g(als)
+ [
(s))
DKL To(-If (8) mm + DKL (T
Te(a) g(als)
(als)
- E.
+ Ey (o)a)fu(s)) log g(als)
(s))fu(s))
+ (TT
13

---

Published as a conference paper at ICLR 2024
If the invariance in the critic has been learned:
Q(u(ss) = Q(s,a) allalla aorall A,
(26)
the actor loss with implicit regularization becomes
(z) - E [. nO mo(a)fu(() log
[als) g(als) + Ey DxL ( (To((
(27)
because
g()lfu())) = g(als) for all A.
(28)
If the actor is well learned for state this actor loss become
( (1) = [m[ (To(
(29)
because
g(als) ne.sg (als)(a)s) for all A.
(30)
DDPG as base algorithm For image-based control tasks, data augmentation f parameterized by i over T is applied on the observations. Considering that the Q-invariant transformation is also x #"-invariant, training all policies of the transformed states to get close to the same optimal policy equivalent to training the policy of original state and enforce invariance the policy. Considering actor loss with implicit regularization, we can apply a Taylor expansion with respect to the optimal action x² (Fa(s) = x xs) = arg max (s)
(8,
1ME Qo((f((s, la=mo(9.(0)
-EE [E T (fx(s))
+ J(a -E((
(fx((s))
+ 2 (a (() xz - alla x x (=)
En (ā - ((==+(((<s)) (V,(())))
2
(31)
_E, (a- (5) To. - (f
(a (g) (s) (s) - ((,(6)))aa=m0(
1. (a To, (())))(((aaa))ea
H(a Ts))
2
+ (TT, (8) - n²" (s)( H (8) (fa(s)))
+ 2(a - ()))²
(
The first term above is enforcing the invariance the actor with respect to the transformation.
B.2
CRITIC LOSS
B.2.1
LINEAR MODEL
According the analysis by Balestriero et al. (2022). the expected Mean Squared Error (MSE) under data augmentation for a linear regression model can be expressed by the expectation and variance the transformed images. Now we want derive a similar regularization term from the critic loss.
we use linear model for the critic and actor:
Q(s, a) W, s + W, *
(32)
((s,
W, Wi bn
(33)
14

---

Published as a conference paper at ICLR 2024
(()) W, * cW, * bi, (0.,1)
(34)
which Wg E RI W. E R! Wr E R!418|, Wo E RI and by. b are parameters for the model. Q the exponential moving average of Q.
The critic loss for transition (s, a, r') under data augmentation v P and μ ~ P' for state s and next state s' is
l = E. (Q((((t))) E.lw)
(35)
()) 2E (Q(( 1x
in which
y v=r+aQUa()), log (
(36)
Considering the last term is not used update Q, we only need focus on the first two terms.
Expectation E, Bv(((f((s) = EE( + Waa + bul
=W,lew [fo [s][ Waa + bo (()]
(37)
Variance
E, [e((((((s)
-E, (
E, ('..) Wof (Waa + bo)2 2(Waa (o)(ff(((s))
E,
(Waa bn)² 2(W.a (V).
(38)
=Tr (W. + (Wa t0
=Tr (W.
E, E. (s,))))))
[[] + (Waa bu)² + 2(Wa (W,,..((
= TTr [WW
Whole loss
Ey (((((())) 2R. [I( [ R,
Tr (w(w (wW.w.//o(ss) -f + Enlri² (39)
Σ - - - -
B.2.2 NON-LINEAR MODEL.
According to the analysis by Balestriero et al. (2022). the expected loss of transformed state has an upper bound related to the variance of the transformed state:
( - (f))) s (e ()[(()))) + ((e))
(40)
in which variance of the transformed image can be decomposed into
v[f(z)] = H (z))
(41)
The second term in the RHS of Equation 40 recovers tangent prop regularization.
15

---

Published as a conference paper at ICLR 2024
C EXPLICIT VS IMPLICIT REGULARIZATION
C.I CRITIC LOSS IN EXPLICIT REGULARIZATION
For (s. a, s', sampled from the replay buffer D, given current estimation Qo and true estima- tion Q" without error, the bias of the target (=))(=)))( ,a)) is smaller than the target Qa.eg (8, a):
E (Bar))(te))((dddd) -
=E,wD 1
a mlggss'
(r + (t log
-BrwD -[
(42)
m B (Earosttoo)00oo00.. Q"
<B,uD (Boost/rr)00sssss) -
<Er-Da (Ccesos))(osoooo.ooo Q¹ (s(, $
-E,wD [ [ -
The bias of using a target ÿ in the explicit regularization
(ύ) -E, ((Qo((f(s)) -) b)² - () - -E,
(43)
(s, 0) +) Q0ss -
We only need to consider the first term (Q constant with respect o. So the bias of using different targets the regularization term decided by the bias of the target compared to the true estimation. The bias of using (or= 6 a')] in the explicit regularization term is smaller than using Qo.eg(e (s, a) according to the equations above:
g). considering that other terms is
i = Eg' Ela'=(( ,a))) < <( (s,)),
(44)
In practice, we use the sampled value y(s',a' as the target, which leads to smaller bias and relatively larger variance.
C.2 CRITIC LOSS CONNECTION
Assume given (5(s,, a, r, v),v), by appropriately setting the random variables in Equations recovers the critic loss in explicit regularization (Equation as shown below. If the distributions of i and f are defined as follows:
Mlverlao11 ao+1
if To
P()) = {
1. if 0, if To
PP ( 7)
P(uerJag agti
(45)
if f To
then we have for any sample (s,a,r,s)):
(an + -2
C.3 ACTOR LOSS
Considering that the policy is parameterized normal distribution in SAC, we first define:
T [x())) = ()) Fo [s)) NN
(46)
16

---

Published as a conference paper at ICLR 2024
For simplicity, we consider y and 17 are defined over discrete set T with probability P(u Ti) P(n = T²) = P., T4 E T. The derivation can be easily extended to using a continuous set.
Tang( (.0gg:11.. = [o, [n(())] = VV(aag Pang) = N N
> PiATO E PP
(47)
E, (- ( [o(-
=B, [1og
"
+
o
+
(2n
- 22 1
on
20
20
2-
(48)
Σ, P.o
Ei P.(T. - -xp)²
=1 - Σ
P. or; +
+
20
20
2
(Tun (-1)) mo(()su(s))
Sang
(aagg Xx) 1
=log
Gang
20
20 / 2
(49)
Σ,
P.O C P.Ay. - ^x,²
1 2
log o₂
2 log
PfO +
20
20
Comparing the two equations above, the first term and the last term are the same, and the second term is a constant with respect to the parameter 0 of the actor. For the third term, it is obvious that
>; P.o?
2, of
because P. 2 P? for any For the forth term, we have:
207
207
Σ; Po(Ar, Ax)² C P.Ar. - An
20
20
Σ, PA + +1
22 Xn)
X (E P.,,, -2,x Ee Pidr.
20
20
(50)
X2 + E, P. -2 Ex P.A..
22 +,E, P.AY,) )2 -2 E, P.A
20
20
Σ, P.A. .E, 32
20
20
20 #
So the loss of using the policy of a transformed state as the target is an upper bound of using the average policy as the target:
B, Drr
(T (-(- 2 (1)) Ta (+)))())))
(51)
D KL DIVERGENCE
Given a transformation fs( on state s, considering that the KL divergence is not symmetric, we discuss the differences between two kinds of KL regularization here:
DKL (TT and D
(52)
Detach First
ne. (als) (a|s)]agg log (als) To (a(a)fu(s))
DKL (s)) (e({(())))
Te. (als) log (als)
(oggo((lf((s)) (53)
- -H((TO.sg(s)) - (als) log (o((f()
17

---

Published as a conference paper at ICLR 2024
Detach Second
DKL (xL(Fe(((())]]Tee.. = J. Te(a] log To(a)fu(s))
Tg (a|s)
| To To(aaff log To To -T((aff(ss) log ne, an (a(ss)
-H((mo(f(s))) - [T(alf log To. (als),
(54)
in which H represent the entropy for distribution.
"Detach second" introduces an entropy term for the policy of the transformed state. This regulariza- tion not only makes the policy of the augmented state and the original state elose, but also maximizes the entropy of the policy of the transformed state. However, in "detach first", the entropy term with the sign sg not used to update the policy.
E
VARIANCE UNDER DATA AUGMENTATION
E. MORE AUGMENTED SAMPLES REDUCE THE VARIANCE OF THE CRITIC LOSS
Considering one transition (8, a, r, 8') and M transformations {Srm. m = 1,... M). K transfor- mation {(5r k = 1,... K} respectively and s', the Q-values and target Q-values for the transformed samples are
Qm (s), r (aalt (s),, a log (adlff (aaaas (1)))
(55)
where m E (1,... e (1, (I,...,K) K}.
RAD+ loss becomes
M
CAAD = M
Σ (Qm m=1
yk )2
(56)
DrQ loss becomes
M
K
DDQ =
Σ
Q2m
yk)
M
K
m=1
h=1
M
K
Q2
+
yk
2Qm
yk
M
K
m=1
k=1
k=1
K
(57)
Σ
Q
U1 )2
1
Σ 2Qm
yk
M
K
k=1
k=1
M
K
M
K
Σ
Mi
2
Σ
Qm 3/
M
m
K2
M
K
m=l
k=1
m=
1k=1
18

---

Published as a conference paper at ICLR 2024
If all the combinations of above Q and y values used for estimation, the loss becomes:
M
fall
Σ (0m
yx)²
M K
1==1
M
K
M
K
1 K
O
Σ
M.
Qm
yk)
M m=1
K
(58)
k=
1 1==1
M
K
1 M
Q
2 M
5E Qm
1k
yk
K
The second terms in both Equation 57 and 58 can be ignored because the gradients of target values Yj with respect to critic parameters stopped.
Obviously, latt and lrQ have same gradients with respect to trainable parameters of the critic. The comparison between DDQ and RADD is exactly the comparison between lall and RRAD For one transition in one gradient step. M K pairs of Qm and yk are used to formulate fall while only pairs of Qm and yı are used formulate RAAD+. Therefore, can find out that DrQ outperforms RAD by leveraging more augmented samples and the averaged target. These operations indeed reduce the variance of the estimated critic loss.
E.2 KL REDUCES THE VARIANCE OF ACTOR LOSS
SAC actor loss respect to for all A, the variance of the actor loss V, (2, v)] ] is bounded by term that depends on the KI. divergence DA, = (xL(T(- (n(s))][ (8))) for v, P:
Given data augmentation ful v P on state s, if Q(f((s), a) is invariant with
V. (s, - LE, (E,,DDn, + c(fu(s))
(59)
where c(fu(s)) > O, is the number of samples to estimate the empirical mean (s,
Proof. For image-based based control tasks, a data augmentation f parameterized by P is applied on the observations. The actor loss of SAC becomes
(g(s v) R. DKL (o(- exp( ( Qo(((())))) Let (sf, exp() (o log (s)).
log ((c(())))]
(60)
The variance of empirical mean can be derived as the true variance divided by the number of samples
v[š[=]] ((((()) E[x:)²]]
EE( - (x
[[]]] + 12 B[x] + + I2 [x()))
(61)
n² Viz]
n
I V[p]
The variance of (s, with respect to for given number of samples is
v. Vele V.JDxautdd((Lee)))ldd00000000000
n
E, (DaLTa((asa)))) h.gp:aaae......auuuuttuttullooan
n
(62)
E, (PRAA(((1LL)))))))00) Dorpoaatthilidoooooooooooooaaaaa DPoo.ooooo-DPPoooooooooooooooooa
1 E, n
19

---

Published as a conference paper at ICLR 2024
For the term inside the above equation, can further derive:
1) (o())((())
S. xo(-1f((s)) log ((fn(s),.) To(((f(s)) s(fu(s),.)
nol [y()))
J. no(-1f1 log To (
(e())fu(s) (g0
(o(-lf((s)) log To (s)
(T(-
- (eeeff0)))-.aakkk/00000000001100
1. To(-(f((8)) (s)
+ {(f((s).))
(s))
{(f(()),))
/ (ma((15((()))<(((((((()) (((1(0)))-)((((()()(dd(((((())))
- =o(-lfu()))
+ /. (0(-1f7(8)) log ())
())
(63)
Then plug the above results into the equation of (, v)]
V. vo,
1 Er
1 B {<
(PnDDectttccll666))1111111111114 log)
Σ (
+
+ Σ P(r)) / (8) log o(fu(e):)) 91
(s)
(64)
1 E (E (1)
+ Σ P(n) / 11
(-og))))) logg(f((s)..)
Σ P(n)) /. -
+ P(m) (o(- log 7(L(()))
9 2(F,(s))
20

---

Published as a conference paper at ICLR 2024
For the second term the right hand side of Equation 64 by applying Pinsker's inequality, get
P((n) [ (T0((17((8)) Te(-)fu(s))) (log (logmo(-lff(s)) - logg(f((s)..))
P(r) /. (mo((als(())) mm(a)())) (1)()) max log (()(((()((())) logg(f((s),a))
(65)
$ Σ P(nnl 2 (To((((((((())))00((1111 max -
For the third and fourth terms of Equation 64 given data augmentation fu v ~ P on state 8, (s)((s), a) is invariant with respect for all A. both the third and the fourth terms of V. v)] are zero.
Therefore, if (x)(,, a) is invariant with respect for all E A, the variance of the augmented actor loss V. Vuled (8, (8, w)] bounded by the divergence Dæ,y = D [,(+)) (( [+))))
for v, P:
Vo(e((s, v -Ry
(En +aa.+eofeee +c(cc())) I
(66)
where cs)) max, log To (a)s((s)) log log.o(((s)) 0, is the number of samples estimate the empirical mean.
DDPG actor loss
Based on Equation 31 the DDPG actor loss (a pi) becomes,
()) = - E, (TO(su(s)) - (I - ((((()))) 2
(67)
The variance of the actor loss is reduced if we minimize the mean squared error between two deter- ministic actions [[ The (T(S((s'))ll where 77, P.
Proof. Let My. (s)) (ff(s))). Assuming that the Hessian matrix Hy have a lower bound and upper bound:
113H, 1 L
(68)
have
4 M (b) 1.
(69)
21

---

Published as a conference paper at ICLR 2024
Vol6((s
1 -B,(((((,v)) -))(())))) - -.. Σ P(n)e((s.V) ((mn)a))))
n
-BI ((n)(e((s,v) ((((s,)))){{ - _
1
Eyl P(n)(e((s.v) (g (s.)))))
n
7
1 Eu (s, n)) - n
(max (s min m))
1
$ n
(max L, |M 2 min Ln M, 2/ (Lumanc | 11? Lomn |I Ammin )1
Let Umax = arg max (, v); Mmin arg min (o.
V, [()(
((Lvmex - mmmin My 112 + mmmi (llMvrmn 112 M 11?
71
1 (Ihymen Amnii Ml|Memen 112
n
Iimin (ITTa (s)) T (8)] [| (b)) - x (s)]1222
1
((Lymax - lymmin M| 112
n
+ maiin
E (TralJeama (8)); x (s)e)² ( ((To() (8)); T (s)))))²
'(Lvmax - Ammmn Il.M 11?
n
E (TTe (Fro(fumac (s))s (s),)² (fm (5)))- x"' -s()))))
+ Ammn
1
((Lvmex - bemii IlAMemoxl! 11?
n
(((ro(femma (8)) 2 at" (s)x) (no (8)); (00)a=)(ss)))>>>> =s)))))))
1 ((Lvmex hemin M|AMssas 11?
T2
+ Ammn ((To(fe (s)) + To (fffmm (s)); - - -x)²(s)) )² (m (s)) ne (fomin (f)mm)(()))) (s))s) 2
1
((Lvmex - bemiin Ml|demaa 1|?
n
E ((To(feem (8)) + To(Tmmmn (8)); 2n 08)) (Toma (8)); no (fmmm (8));)²
+ Inmin
1
((Lvmex homan I|AAlymsse 11?
n
+ mmon ((To(Femm (s)) + No (f (s)); - 2n (s), ()))))(()) (Fomoon To (frmm ())))))()))) (s))k)
(70)
22

---

Published as a conference paper at ICLR 2024
Since
(a- () + b- 2c)²
(a b) + (a + b - 2c)4
2
(a (a- 6)$ + ((a
b c)3)²
2
(a 6) (2(a
c)² c)
(71)
(a 6) + 4((a c)² + (b - c)
2
(a b) + 8(a c)* + 8(b c)
have
V.
1
(Lyman Lamin MlAM m 11
n
mymn E(mo(fenm (To(Semma (8)) + TooSmmm (8)); 2xx (s)) (To (rrmam (8)); To (J (o))
1 (Lymen Impuii 21 11
12
+
2 Vmain ( (frmes (8)) Te Te(Sn (on (s))
+ 44 Monin
1| ( (fumes (s)) (s))111
41 Naio 1 Vmnia (s)) nT" (1))
(72)
In Equation 72 the third and fourth terms are minimized by the actor loss. If we minimize the second term of Equation by minimizing the mean squared error between two deterministic actions (fr))()))))) - xo (s I)Il in the case of DDPG, the variance of the actor loss is reduced.
E.3 KL REDUCES THE VARIANCE OF THE TARGET Q- V-VALUE
DDPG target values For DDPG, when we compute target values, we add Ornstein- -hateene-hhleebckk noise to deterministic actions for exploration. Then the policy can be regarded as probability distribution म.
For image-based control tasks, data augmentation f parameterized by P is applied on the observations. Then the target value y for a given transition (s,a,r,8') is
v(fa('),a)) = r + QQ((m(s') a'), where a' ~ (x(-1(((66)). The expectation of ((f (s),,')))))) a') with respect to ~ T ((-If(ss))
(73)
E. (v(a(())))) a')l = 7B, (9 =r Σ ( (l
(74)
The expectation of (s)))s)) a') with respect n P and a' ~ (-((f(()))
Ey (V(T)
= r + BB (a((n(()))
(75)
= Σ P()) Σ ((a'l ((d1
23

---

Published as a conference paper at ICLR 2024
We create two tables to better illustrate the meanings of Eg (s' a')] and Ea (v(ff
μ
Jr.. (s') with P(u = Tm
a'
v(5-on (s').oi) with P(u Tm) To (ai |5 (8'))
a
v(Trm (s)),a)) with P(u = Tm) To frm (6))
d2
...
...
v(frm (8)), onn) with P(1s Tm) (o 1 (s'))
d/2
a'
Eny' V(
fr. (s')
Ely) wrt. a and
with P(u =1)
Ey ((Ju(s')
The variance of u(fu(s'),
a') with respect to and a' is
V "n.a' ((((a(''), a')]
Σp(w) Σ (l0(
En (
-P(()) Σ {((d'((a>')))
[V((a())).d) -
(76)
ΣP(w) Σ (CNN(()((((((())))))))))))))) (
+ 2(u(fa(s)) - E. -o (Bar a')l Ey, (V(fu(s'),
+ (Ea (s') B,,a'
The first term of Equation 76 is the expectation of squared advantage.
The second term of Equation 76 is because
- ΣP() न (a'l(fm(s') - (E
(Far (y( a')) - (
= =2Σ (P()) ((. Em. (m)
(((1150(())))00))) E. (o. (a
(77)
Σ (P(#) (E.. d')l
The third term of Equation 76 is the variance of te (s)(( with respect to J.. Both the variance V Be a a')]l and the variance of the empirical mean
24

---

Published as a conference paper at ICLR 2024
V Bi fo(()) T(. (s)))) for u. Pf Q2((f(ss)) a') is invariant with respect to for all of E A.
bounded by the KL divergence Dy, (x
Proof.
WilBer-rtsnoo))((((()))) En (Ba (Barl((((())) ( {
(78)
E (n(((((())) -nn
(((11((()))(((00))000)) P(n) / x((1116))))(((())))
P(n)
Σ P(1)
7 {> P(n) =1
+
- (>(2)() -1(
#(dd116(6)))00000))))) - -Q,(,(6),)))
(79)
The second term Equation 79 ? E, P(n) -1ATe1hhhA))0Q000000000000000000
is
related to the difference (s)) a') and (o(s((')), a'), which governed by the critic loss.
(s"),
When (s(f((s'), a') is invariant with respect to μ for all a' A. this term is zero.
For the first term of Equation 79
P(T) (Taa((((<')) - - (1d1))))=a11110)00000000
< Σ P(n) Σ (<(d'((()))) ((10)))))))1111(60)00000000
(80)
max Qa(Tn('), Σ P(m) Σ -
< max Qx(()('),
Σ P(m) (2
where in the first inequality absolute values T (a'lff - inequality (f(fn(s'), a') is replaced with max, (s( a²) and Pinsker's inequality is applied in the third inequality.
-(( (s') are applied. in the second
Similarly, lower bound can be derived.
Σ P(n) (
(81)
max Qz(sn(5')aa) PD
Therfore,
V Ea (1, E, n²2 max Qa(fa(s'), ('ssss)')'))). d') (22D
(82)
25

---

Published as a conference paper at ICLR 2024
Let (s', Bs - (1 (1n6))))
From Equation 61
Vu(P(''. 1 ((
(83)
where n is the number of samples to estimate the empirical mean P(d'
Therefore, if (a(( d') is invariant with respect to μ for all a' E A, the variance of ( 14) with respect to is bounded by the KL divergence Dn = DKz(T(- fr((s)) al fs(())) for 1. P.
1 Bu [79 Va .J]l
(mamxx max (a(fx((')) d'),, (22 [ 1)
(84)
n
For DDPG, minimizing the KL divergence between policy distributions two augmented states (xI(T() f,((s)) T(- fs())) is equivalent to minimizing the mean squared error between two deterministic actions ||=(S((s') =
SAC target value with the entropy term If the entropy term is added the target value, the variance of the empirical mean V, divergence Dr (= (s())) (((())))))) for u, P Qo(sp((), a') is invariant with respect y for all d' E A.
is still bounded by the KL
1 Fy ( V (Ln S
max(v((2(6)))) 22 + Drnss ) 2
(85)
where n is the number of samples to estimate the empirical mean, r the reward this transition and is the entropy coefficient.
Proof.
After add the entropy term, the target value becomes
v(fa('), = ((r())))))=+++0((()))))) (ff
(86)
where a' ~ (=)) and a the entropy coefficient.
Let
vn = - --
a log (())
(87)
Since is constant value, we can drop when calculating the variance.
V (B -uuBer
(B. a m (a mag - - (Qg(f - log (a')s,
(88)
26

---

Published as a conference paper at ICLR 2024
Ee yn (Ju(b)), a')] (n
Σ T(a'ls 7lll(((666))))(())))))) -P((1) / ((d(I[(6)))))((()))
P(n)) ((dd E DDaAAAAAAODDDDDaaata [(ddlf(s)))(())
EP(1) IxldIm(())) - ((d)))()))))))))))
"
P(1)) Σ
= -
+ (
= EP()
((d((((()))
(89)
+ +=(l1x4011QQ(A110000000000001000
EP(M Σ =muuaa))))/((((1111))11101111100
+ P(n) Σ [xddd(ii)))eeoos((((((11 zd'l((
Σp(N) -m(AAsS-))T1((((11111)))-------1 <(d'((,(())))
7 P(m) Σ IT )
+ Σ P(n)a (x(
Similar to Equation 80 and Equation 81 we apply Pinsker's inequality and obtain the lower and the upper bounds for the first term of Equation 89
max (fx(s(f)())) [Pnny/aattt((se)))))((C00 a
P(n) - (adolaa66)--mddsse))))eeeee))
(90)
<axx((f a') Ex
E, P(n) (1) Qz((((s))) is related the difference of (s),, and Qz(s((s', a'), which governed by the critic loss. When (u a') is invariant with respect y for all a' A. this term is zero.
The second term of Equation
89
27

---

Published as a conference paper at ICLR 2024
Therefore,
Σ P(n)) ( ta Daaletell1A(())))aa000)))1)11)11
/eDDaatttLGGGe))))(C(C Bo SolbU(ihhhc])---nllll(sss.l
Σ P(n) (max (aDp://))ooooa((((/(()))))////a a {u
(91)
Plug the above inequalities into Equation 88 we obtain
max yn (f=(s'), 2D ta a. Dmnt)) 1) (92)
V Be (L = B En
o'
(5(s(()), d') invariant with respect to jr for all a' E A. the variance of ÿ ,1) with respect
to μ is bounded by the KL. divergence D = DKL(T(( fn((')) T(. fa(('))) for u. ~~70~1 ~ P.
VaDY () E. ( [Bnnn_ax((((L - r) / + a Dr.sa) 1
(93)
n
F CALCULATING TARGET WITH COMPLEX DATA AUGMENTATION
In this section, we experimentally analyze using complex image transformations in calculating the target and show that cosine similarity of the augmented features the early training stage can be used as criteria for judging if an image transformation is complex or not. Sufficient updates is the key condition for good performance when using complex image transformations in calculating the target.
In contrast to the analysis in SVEA (Hansen al. 202T). observe that even using complex image transformation such as random conv in the target does not induce large variance in the target. Instead, much larger bias is observed for the trained agent, as shown in the Table This can be solved by increasing the number of updates, shown in Figure
Furthermore, we test with other image transformations which are regarded as complex image trans- formations in SVEA (Hansen et al. 2021). In order to show whether it's easy to enforce the in- variance of a image transformation, we record the cosine similarities of encoder outputs for two augmented images transformed by this image transformation, formations such as random overlay or gaussian blur, the invariance is easy to enforce and the cosine similarities large. When using this kind of image transformation in calculating the target values, won't t hurt the performance. Otherwise, for image transformations such as random convolution or random rotation, the invariance is relatively harder to enforce during training and the cosine similar- ities are small. Then directly applying this kind of image transformations in calculating the target values will decrease the learning efficiency. To resolve this issue, need more updates for each training step. The evaluation results for SVEA with random overlay, random convolution, random rotation and gaussian blur are shown Figure
shown in Table For image trans-
G HYPERPARAMETERS
Hyperparameters used in experiments on DMControl (drq). DMControl (drqv2) and DMGB can be found in Table 6 Table and For experiments in (dControl(drqvv)) and DMGB, when applicable, we adopt hyperparameters from the official implementation of drqv2 by Yarats et al. 202T) and SVEA by Hansen et (202T) respectively.
H ADDITIONAL RESULTS
H.1 ABLATION STUDY
The performance profile is shown in Figure and the training curves in different environments are shown in Figure 7
28

---

Published as a conference paper at ICLR 2024
Statisties
svea(DA=blur)
svea(DA=overlay)
svea(DA=conv)
svea(DA=rotation)
actor sim (shift)
0.919:10.007
0.011
0.910:1.001
0.936:10.006
actor sim (DA)
0.998.100001
0.906110.006
0.854.10.019
0.536:10.069
critic sim (shift)
0.938=0.010
0.942=00.003
0.922#00010
0.962:00005
critic sim (DA)
0.9981100011
0.939-10.003
0.883110.014
0.660.10.057
Table 4: Recorded cosine similarity for latent features 100k steps in walker walk environment for SVEA trained with different complex image transformations. Here, each column corresponds to SVEA with different image transformations and each row corresponds a cosine similarity recorded at 100k steps. For example, the first number is calculated by the cosine similarity between the latent features E.y (Jehhaf(())) in which Fg the encoder of the actor in SVEA trained with gaussian blur and fshift is the random shift. Considering that random shift is always applied in SVEA, the cosine similarity with respect to recorded as the baseline for comparisons. For gaussian blur and random overlay (second and third column), the cosine similarities of latent features higher or similar to the cosine similarities between the latent features from two randomly shifted images which means they are complex image transformations. In contrast, random conv and random rotation (last two columns) leads to smaller cosine similarities of latent features which indicates that they relatively complex image transformations in this environment.
Statistics
Step 100k
Step 200k
Step 300k
Step 400k
Step 500k
Mean (w/ conv)
64.05
112.44
140.78
166.42
185.92
Mean (w/o conv)
84.96
148.46
197.10
221.67
239.35
Variance (w/ conv)
1.228
1.109
1.148
1.323
1.306
Variance (w/o conv)
0.882
0.812
0,885
0.757
0.795
Bias (w/ conv)
52.88
67.44
54.20
63.89
55.36
Bias (w/o conv)
77.40
60.48
50.40
43.22
28.75
Table 5: Mean, variance and bias the target Q-values for the agent trained with/without using random conv in calculating the target. Here, the mean and variance are calculated by the mean and variance of a set of sampled target Q values and the bias calculated by the mean-squared error between the targets used in the training and the true targets estimated by the rewards from sampled trajectories. At the end the training, the increase in the variance when using random conv is not significant compared the mean of the target. However, the bias is much larger at the end.
of discounted
29

---

Published as a conference paper at ICLR 2024
walker walk normal eval
episode. reward (eval)
600
400
svea
svea_ wea_update2
200
aea_conv_targee
svea_conv_tat har__oooottttttuuue
www__arrget_upatee2
svea svea_conv pat_ccootttttetttteeeeee www__trrett_updaeee update4
svea_conv wwwccoottteetuueeeeeeeeoe updates
100000 200000 300000 400000 500000
evaluation steps
Figure 4: Performance of increasing the number of updates with/without using random conv in calculating the targets.
X2003 2000 23005 40000 J000E
endtarton Mays
Figure 5: Performance of increasing the number of updates in walker walk environment when using complex image transformation in calculating the targets. For random convolution and random rota- tion, "update more" stands for doing 4 updates for each training step.
RAD
RAD
DrQ
DrO
ours
Fraction runs with score
1.00
0.75
0.50
0.25
0.00
0.0
0.6
0.8
1.0
Normalized Score
Figure 6: Performance profile of different methods.
30

---

Published as a conference paper at ICLR 2024
Table 6: Hyperparameters used in experiments on DMControl (drq)
Hyperparameter
Value DMControl
frame rendering
84
84
stacked frames
3
action repeat
2
replay buffer capacity
100,000
seed steps
1000
250,000 in reacher easy
250,000 in finger spin
environment steps
250,000 ball 500,000 others
batch size N
256
discount ?
0.99
Adam (31 0.9, /2 0.999)
optimizer (ø. 8)
Adam
optimizer (a of SAC)
(G 0.9, 02 0.999)
learning rate (6.0)
le-3
learning (ax of SAC)
le-3
target network update frequency
2
target network soft-update rate actor update frequency actor log stddev bounds init temperature tangent prop weight Ctep
0,01
2
[-10,2]
0.1
0.1
actor KL weight OKL
0.1
Table Hyperparameters used experiments on DMControl (drqv2)
Hyperparameter
Value on DMC
frame rendering
84 x 84
stacked frames
3
action repeat
2
replay buffer capacity seed frames exploration steps n-step returns batch size N discount y optimizer (ó, 0)
106
4000
2000
3
256
0.99
Adam
learning rate (6.0)
le-4
agent update frequency target network soft-update rate exploration stddev clip exploration stddev schedule tangent prop weight Oup actor KL weight ax
2
0.01
0.3
linear(1.0 (1.0, 0. 500000)
0.1
0.1
H.2 CASE STUDY
The measures for invariance in the latent space shown in Figure
31

---

Published as a conference paper at ICLR 2024
sKy
troer
140
800
30
203
10000
3009K
25000
5000
150000
24000
209
140000
100000
20000
3400
350
001000
100
2000
200000
130000
1S/080
20000
.0000
8.30
Figure 7: Full results of validating propositions.
tore 2is JCD features
tone, dec Jeatures
005 00.55LL actor Seatures
200 Sin CYiiC Freture:
0.50
0.4
300
677
200
Figure 8: The figure shows the learned invariance in the feature space of the and critic. Two measures of the invariance are provided in this figure; the distances between projected points of the augmented features by t-SNE and the cosine similarities between augmented features.
32

---

Published as a conference paper at ICLR 2024
Table Hyperparameters used in experiments on DMControl Generalization Benchmark (DMGB)
Hyperparameter
Value on DMGB
frame rendering stacked frames
84 84
3
2(finger) 8(cartpole) 4(otherwise)
action repeat
replay buffer capacity seed steps environment steps batch size discount 7 optimizer (ó,
500,000 / action repeat
1000
500,000
128
0.99
Adam
(3 = 0.9, A8 = 0.999)
Adam
optimizer (a of SAC)
(G: 0.5, B2 0.999)
learning rate (d.00
le-3
learning rate (a of SAC) target network update frequency
lc-4
2
0.01(critic)
target network soft-update rate
0.05(encoder)
actor update frequency k actor log stddev bounds init temperature a tangent prop weight otp actor KL weight KKL
2
[-10.2]
0.1
0.5
0.1
H.3 MORE EVALUATIONS
Here, we include more evaluations of our proposition. The results of comparing our proposition with DrQ are shown in Figure The results of comparing our proposition with DrQv2 are shown in Figure 1O
H.4 RESULTS OF GENERALIZATION ABILITY DMCONTROL GENERALIZATION BENCHMARK (DMGB)
The comparison of generalization performance in DMGB between SVEA and our method using random overlay as data augmentation is shown in Figure 11
H.5 RESULTS OF RECORDED STATISTICS
The curves for the recorded statistics, including standard deviation of the empirical critic loss, stan- dard deviation of the target Q-values, and empirical mean of KL divergence between policies for two augmented samples along the training are shown in Figure 12 and Figure 13
I LIMITATIONS
We try to provide some recommendations on how to apply theoretically-soune sound data augmentation method in DRL. However, the analysis can still be further refined to be more comprehensive such as including the theoretical analysis of using different distributions for the image transformation and providing a thorough analysis on tangent prop regularization. Moreover, our method naturally requires the knowledge of some effective image transformations for given task. Without such knowledge, the invariant transformations for a problem would need to be learned, which is currently an active research direction. Finally, image transformation may rely
implicit assumptions,
33

---

Published as a conference paper at ICLR 2024
male
2000
010
D-C
102
K10
200
239
54T4
33000C 350000 200033 206030
250044
LEC300 158000
300006
warher hard
DXC
100
030
KCC
0-
500003
220000
200000
M00000
500003
2000C3
200000
300030
40000C
5000CS
130000 200093
300009
400000
500M6
Sriger
cheabeb
960
PAD
230
RAD
DrAC
DxrCC
SHIA
2154
Figure 9: Comparison between different methods in DMControl with normal background.
WORCEE
200
200
150
100 1900
Figure 10: Results of running experiments with DDPG as base algorithm.
34

---

Published as a conference paper at ICLR 2024
100
300030 200000 200300 AOOOCO 103000
TIDRE DOOOCC S0000C RC0000
JU300 FROOUE 10:00
1.000 JINOUS JECOCC
Boger 50 mOTEE atal
Prgger VIDNO Paand
1320
200
stand ECTTEE
1000
100
420
600
500
3000 20000 20000 40000 50000 13000 20000 300000 40000 50000
Figure 11: Comparison between SVEA and our method in DMControl with normal and video- hard backgrounds. Both methods use random overlay image transformation. We can see the improvement in generalization ability especially in environments such as ball in cup catch, finger spin and walker walk. Since the evaluation curves not stable even at the end of training, the recorded score in Table 2 is the average over the last 15 evaluation scores.
35

---

Published as a conference paper at ICLR 2024
DO--6 Surpp
0015.
mean
139330
23000
<2008
103030
mean ald Iarge for avamenee sample
3.5
322
6.-
63
33000 20000 30000
40009
500000
Figure 12:
Some important statistics recorded along the training. The variance of critic loss and target values decreased after using more augmented samples in the training of the critie, Adding the KL divergence term to the loss can quickly enforce the invariance of the actor even at the beginning of the training.
which may lead lower/bad performance if they are satisfied the real application domain. For instance, random shift/crop, which has been shown to be very effective in DMControl tasks, may yield worse performance if the agent not well-centered in the image, according to the empirical results from Tomar et al. (2022). A better understanding of why data augmentation transformation works DRL is needed.
36

---

Published as a conference paper at ICLR 2024
mean of ertic loss
mean std the target
0.8
420000
mean between augmented samples
0.6
(omtocenter Costovettay
03
00
200000
300000
500000
Figure 13: Some important statistics recorded along the training of SVEA and our method. With the help of KI. loss and tangent prop loss, the variance of critic loss and target values are lower. Applying KL loss can quickly enforce the invariance the actor.
37


Reviews:
[{"review_id": "s7vSCVYzqY", "reviewer": "Reviewer_GnUL", "summary": "The paper revisits state-of-the-art data augmentation methods in Deep Reinforcement Learning (DRL). It offers a theoretical analysis to understand and compare different existing methods, providing recommendations on how to exploit data augmentation in a more theoretically-motivated way. The paper introduces a novel regularization term called tangent prop and validates the propositions by evaluating the method in DeepMind control tasks. The experimental results are to be released after publication, and the paper also includes theoretical proofs to support the proposed methods and analysis. However, it is important to note that the paper is highly technical and may be challenging for readers without a strong background in DRL and computer vision to understand. Additionally, it focuses on image-based DRL and does not address the ethical implications of using data augmentation techniques in DRL.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The paper provides a comprehensive analysis of existing data augmentation techniques in DRL and offers recommendations on how to use them more effectively.\n2.  The authors introduce a novel regularization term called tangent prop and demonstrate its state-of-the-art performance in various environments.\n3.  The experimental results are presented in a clear and concise manner, and the code with comments on how to reproduce the results will be released after publication.\n4.  The paper provides theoretical proofs to support the proposed methods and analysis.", "weaknesses": "1. More insights into the limitations and potential failures of the proposed method should be discussed. This would provide a more balanced perspective and help readers better understand the practical considerations when applying the proposed approach.\n\n2. Further analysis and comparisons with a wider range of existing techniques should be conducted to showcase the advantages and limitations of the proposed method in different scenarios. This would provide a more comprehensive view of its effectiveness and contribution to the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tvcc5cTZ5S", "reviewer": "Reviewer_BeKW", "summary": "This paper theoretically analyzes and compares the impact of current data augmentation techniques on Q-target and actor/critic loss in visual reinforcement learning. The regularization term \"tangent prop\" proposed in computer vision is also applied to reinforcement learning to learn invariant.", "questions": "1) Please check that the two terms in the KL divergence in Eq. 8, and there seems to be an ambiguity with Eq. 4 and Eq. 9.\n2) How to prove that the distributions of $\\nu$ and $\\mu$ in Eq. 11 and Eq. 12 are the distributions of $\\hat{\\nu}$ and $\\hat{\\mu}$ satisfying Lemma 1?\n3) How to show that \"even using complex image transformations such as random convolution in the target\". We did not find an experiment in the paper that verifies this conclusion.\n4) In the experimental part, the grid search for $ \\alpha_{KL}$ and $ \\alpha_{tp}$ is too sparse, and the final choice of 0.1 leads to curiosity about the results within [0,0.1]. A similar situation occurs with the SVEA comparison experiment of 0.5 selected from {0.1, 0.5}.\n5) I'm curious about the method of determining $ \\alpha_i$, which seems to be missing the reason(s) in the paper. Does it have a specific value for each augmentation in the set, or does it choose an average weight?\n6) In Figure 2, the batch size of DrQ reproduced in the results is 256 instead of 512 in the original DrQ. The scores of DrQ will decrease under some environments when using a smaller batch size. Compared with the official results of DrQ (https://github.com/denisyarats/drq), some results of DrQ shown in Figure 9 are lower. Such as ball_in_catch , walker_walk, walk_run. Therefore, to make a fair comparison, the author should completely use the hyperparameter settings in DrQ to reproduce, or use the scores in DrQ.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The theoretical analysis of the paper is sufficient, which is difficult to see in many similar works. At the same time, the experimental data are also considerable.", "weaknesses": "1) The work in this paper seems to be equivalent to adding an explicit regularization to implicit regularization, is it equivalent to DrQ combined with DrAC in terms of functionality?\n2) How the proposed algorithm addresses the initial question \"Although they empirically demonstrate the effectiveness of data augmentation for improving sample efficiency or generalization, which technique should be preferred is not always clear.\". The main idea of the paper is not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DYYZ2hcImE", "reviewer": "Reviewer_9PSY", "summary": "This paper delves into the realm of data augmentation methods in Deep Reinforcement Learning (DRL), conducting a thorough analysis of existing techniques. The authors provide a theoretical framework to understand and compare these methods and propose a new regularization term, tangent prop, to enhance invariance. Their contributions lie in offering a theoretical understanding of data augmentation methods in DRL, suggesting how to use data augmentation in a more theoretically-driven manner, and introducing a novel regularization approach. Through extensive experiments, the authors validate their theoretical propositions.", "questions": "Refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality:\nIt conducts a comprehensive analysis of existing data augmentation methods, initially based on the author's introduction of assumptions regarding uncertainty in reinforcement learning and existing image-based online Deep Reinforcement Learning (DRL) data augmentation techniques. The paper establishes an integrated AC framework incorporating data augmentation. Within this framework, the mainstream data augmentation methods are analyzed and categorized into explicit and implicit regularization techniques. Qualitative and quantitative analyses of these augmentation methods are performed. Building upon these two types of regularization, the paper introduces a novel regularization technique – tangent prop – providing theoretical support for addressing uncertainty in the Critic component.\n\nQuality and Clarity:\nI think the quality of the paper is very high. Each proposition and hypothesis presented in the paper is accompanied by corresponding formulas, and the appendix contains detailed derivations of these formulas. Additionally, I find the paper to be very logically structured.\nFirst, the paper explains the process of the Data-Augmented Off-policy Actor-Critic Scheme and introduces the key hypotheses regarding the uncertainty in Q-values and policy π. The subsequent proofs and derivations are based on these definitions. The paper then elaborates on how explicit and implicit uncertainties are defined within the A-C framework and how they affect the Loss function of Actor and Critic. After providing the theoretical background, the paper proposes its own generic algorithm and provides detailed derivations and proofs.\n\nFurthermore, the paper analyzes the effects of applying different image transformations when calculating target Q-values, as well as the empirical actor/critic losses estimated under data augmentation. In implicit regularization, the author, unlike the previous SVEA method, incorporates KL divergence into the training process for policy. The paper explains and proves that, under the premise of Critic invariance, introducing KL divergence helps the model better learn the invariance of the Actor. Finally, the paper introduces the innovative concept of Tangent Propagation to further demonstrate how this newly introduced additional regularization term promotes Critic invariance.\n\nThe logic throughout the main body of the paper is very clear. It systematically proves its hypotheses about uncertainty and effectively addresses the initial problems posed in the paper.\n\nSignificance:\nThe paper holds significant value for the DRL research community. By offering a theoretical framework and introducing a novel regularization approach, the paper addresses a key challenge in DRL, enhancing the understanding and application of data augmentation techniques. The experimental validation across various environments supports the significance of the proposed methods. The findings are likely to influence future research directions, providing valuable insights for researchers and practitioners in the field.", "weaknesses": "The theoretical derivation in the paper is very thorough. However, I believe the experimental section of the paper is somewhat lacking. It compares the performance with the previous statistically trained model, SVEA, providing detailed experimental data and theoretical analysis. Nevertheless, there is a lack of in-depth analysis of the shortcomings of the previous algorithms and the advantages of the proposed algorithm. Moreover, I think the algorithm should be further compared with more methods and applied in various domains to validate its generalizability.\n\nThe entire article is dedicated to theoretically proving the reliability of its algorithm, and it has obtained favorable experimental results. However, throughout the entire text, there is no discussion about the shortcomings and limitations of the algorithm, nor is there any mention of how to extend the algorithm or areas that need further exploration in the future.\n\nThe article discusses how different complex image augmentation techniques have varying impacts on the invariance of target Q-values. The paper proposes using cosine similarity measured from encoder outputs for this evaluation. It is mentioned in the article that techniques such as random addition or Gaussian blur easily achieve invariance through these transformations because they result in high cosine similarity. When using these types of image transformations in computing target values, it does not affect performance. In contrast, image transformations like random convolution or random rotation are relatively challenging to achieve invariance during the training process, as they result in lower cosine similarity. The solution proposed in the paper is to enforce this invariance by performing more updates at each training step. However, the paper does not address whether there might be potential overfitting issues under these specific conditions and how to handle them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qJ4wa2Mafw", "reviewer": "Reviewer_pcif", "summary": "This paper presents an analysis of various data augmentation techniques applied in image-based deep reinforcement learning.  \nThe authors classify these techniques into two primary categories: implicit and explicit data augmentation where implicit DA consists of directly applying transformations on the observations during training, oppositely to explicit DA where transformations are applied on samples used for an auxiliary regularization term.  \nFollowing this classification, the authors then put forth a generalized formulation for integrating DA within actor-critic methods, covering both the previously mentioned implicit and explicit DA techniques. One of the paper's most salient contributions lies in the analysis of how DA impacts the variance associated with the actor and critic losses.  \nThe insights drawn from this analysis guide the authors towards designing a principled data augmentation strategy.  \nA notable aspect of their proposed method is the integration of a computer vision regularization called tangent prop into the learning objective for the critic. This modification is suggested to potentially improve the stability and efficiency of the learning process.  \nThe experimental segment of the paper includes tests on the DeepMind control suite benchmark to evaluate the proposed data augmentation scheme. Additionally, the authors test the generalization capabilities of their method using the dmcontrol-generalization-benchmark.", "questions": "* Based on the data from Table 2, RAD+ exhibits lower variance on the critic loss compared to DrQ. Yet, it underperforms DrQ in terms of overall results. Given that both methods utilize the SAC algorithm, this seems to counter the claim made by the authors that reducing critic loss variance leads to more stabilized training. Can the authors provide insights or explanations for this observation?\n\n* My understanding is that the only discernible difference between the authors' method and DrQ+KL is the introduction of the tangent prop regularization. Drawing from Table 1, it appears that this regularization is the primary factor that reduces the critic loss variance, rather than the authors' generic algorithm. Could the authors comment on the role and impact of tangent prop in this context?\n\n* The authors assert that employing random convolution in SVEA doesn't induce significant variance. However, the empirical results from Table 5 seem to suggest otherwise. Could the authors clarify?\n\n* What drives the intuition that the KL term assists in reducing the variance of the target critic?\n\nFor a more comprehensive analysis, would it be feasible for the authors to include SVEA in the experiments of the initial experimental setup?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Unified Framework:** A significant strength of this paper is its development of a unified framework that facilitates the direct application of data augmentation in RL. This holistic approach serves to streamline various data augmentation techniques within the context of reinforcement learning.\n\n **Analysis on KL regularization direction:** The paper offers a commendable examination regarding the optimal direction for applying the KL regularization term. While the theoretical insights are appreciated, an empirical evaluation would have added further depth to this assessment.\n\n **Variance Analysis in SAC:** The comprehensive analysis of the variance associated with the different terms used in SAC when DA is applied is insightful and sparks intriguing questions that could guide future research in the domain.\n\n**Tangent Prop regularization:** Although the tangent prop regularization has its roots in the computer vision world and might not be a groundbreaking addition to RL, its introduction in this context is promising. The authors successfully empirically demonstrate its potential benefits.", "weaknesses": "**Oversimplified Unified Framework:** While the proposed unified framework is a step forward, it seems overly simplified. Specifically, the representation of implicit data augmentation as solely reliant on KL regularization seems limiting. Other methods like SODA [1] and SGQN [2] incorporate data augmentation using auxiliary objectives, drawing parallels with techniques in self-supervised learning. Such nuances should have been addressed for a more comprehensive overview.\n\n**Excessive Reference to Appendix:** The frequent deference to the appendix, especially in sections 5.2 and 6, detracts from the paper's flow. Readers are forced into continuous back-and-forth toggling, making the narrative harder to follow.\n\n**Unverified Claims:** The assertion that introducing image transformations in the target doesn't lead to significant variance seems unsupported, particularly when considering table 5 in the appendix. Such claims should ideally be substantiated with empirical evidence.\n\n**Comparison with RAD and SVEA:** In the original paper, SVEA, which does not employ data augmentation on the target, significantly outperforms RAD. A comment or analysis from the authors on this discrepancy would have been informative.\n\n**Unclear Table Presentation:** Table 4 in the appendix is a bit perplexing. The logic behind measuring variance in relation to augmentations not employed during SVEA training (such as overlay, randomconv, rotation, or blur) is unclear. Additionally, the specific \"DA\" used here is not explicitly mentioned, leading to further confusion.\n\n**Novelty and Contribution:** A major point of contention is the actual novelty and impact of the paper's contributions. Its principal contribution seems to revolve around the unified framework and the introduction of tangent prop in RL. Yet, the proposed principled algorithm appears to be a minor extension of DrQ by merely incorporating a KL term. Based on figures 1 and 2, the tangible benefits seem largely attributed to the tangent prop, emphasizing its crucial role. While integrating this regularization in RL is commendable, its novelty in the broader context appears somewhat limited.\n \n**Minor comments**\n Bolding in Tables: The rationale behind the use of bolding in tables (specifically tables 4 and 5) needs clarification. \nLimitations are located in the appendix these and should be explicitly referenced in the main body.\nMissing legend in figure 7, what is the green line?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)