rQdia: Regularizing Q-Value Distributions With Image Augmentation
Authors: Samuel Lerman
Jing Bi
Chenliang Xu
Venue: ICLR 2022 Submitted (2022)
Keywords: deep reinforcement learning
regularization
q-value distributions
invariance
image augmentation
continuous control
Atari

Abstract: rQdia (pronounced “Arcadia”) regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning. With a simple auxiliary loss, that equalizes these distributions via MSE, rQdia boosts DrQ and SAC on 9/12 and 10/12 tasks respectively in the MuJoCo Continuous Control Suite from pixels, and Data-Efficient Rainbow on 18/26 Atari Arcade environments. Gains are measured in both sample efficiency and longer-term training. Moreover, the addition of rQdia finally propels model-free continuous control from pixels over the state encoding baseline. Additional results, namely more random seeds, pending.

Under review as a conference paper at ICLR 2022
RQDIA: REGULARIZING Q-VALUE DISTRIBUTIONS WITH IMAGE AUGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
rQiia (pronounced "Arcadia") regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning. This simple idea, to equalize Q-values across statistical distributions of actions and states, affords image aug- mentation techniques like DrQ additional sample efficiency and better final per- formance, while propelling discrete-action action DER nearly 1.5x the performance DrQ and nearly 2x that base DER. Representation learning is a major hur- dle in deep RL, which notoriously requires far too many environment interactions for real-world use cases. rQdia decreases this data hunger and increases overall scores, bringing deep RL closer to real-work applicability.
INTRODUCTION
Human perception is invariant to and remarkably robust against many perturbations, like discol- oration, obfuscation, and low exposure. On the other hand, artificial neural networks do intrin- sically carry these invariance properties, not without regularizers or hand-crafted inductive biases like convolution, kernel rotation, dilation, attention, and recurrence. In deep reinforcement learning (RL) from pixels, an agent must learn to visually interpret a scene in order to decide on an action. Thus, recent approaches RL have turned to the self-supervision and data augmentation techniques found in computer vision. Indeed, such contrastive learning auxiliary losses (Srinivas et al. 2020) or data augmentation regularizers (Yarats al. 202Tb) have afforded greater sample efficiency and final scores in both the DeepMind Continuous Control Suite (Tassa et al., 2018) from pixels and Atari Arcade Learning Environments (Bellemare et al. 2013).
Nevertheless, pixel-based approaches continue lag behind models that learn directly from state embeddings, not just in terms of sample efficiency, but also in longer-term asymptotic performance. For example, the recent DrQ (Yarats et al. 2021b), image augmentation-Bee based regularizer added to SAC HHarnoja et al., 2018). reports falling 14.5% short of its state embedding- based SAC coun- terpart the Cheetah Run task. Such discrepancies indicate that visual representations up to par with state embeddings, at least not for locomotive continuous control. State embeddings have many properties that facilitate generalization, such as location invariance, and to a degree, variance between morphological relations. If a subset of the dimensions of state embedding always indicates feet position, then the relation "one foot in front of the other" will be represented by those dimensions invariant to the placement of the robot's arms, head, or other body parts. Parametric vi- sual encodings are not guaranteed to learn such invariances with respect to the robot's morphology.
yet
What other signals in deep RI. can guide visual representation learning toward more invariant en- codings? We propose Q-value distributions, sets of cumulative discounted rewards, as such signal. For state s, Q-function Q, and actions (0)) ....... (m) D(A) from some statistical distribution D over action space A, we define Q-value distribution simply as:
a am),
Since Q-values are optimally proportional to action probabilities, this "distribution" is representative of the actual policy distribution when D Furthermore, it is measure of the current and future value of each action for that state. We review the MDP framework in Section 2.1 This signal is amenable to many of the same invariances afforded by state embeddings, if not additional ones. For

---

Under review as a conference paper at ICLR 2022
All batch actions
Original
Augmented
i
a
MSE
a.j
jjj
Q
a
rQdia
"Arcadia"
Figure 1: "rQdia in nutshell" rQdia regularizes Q-value distributions across augmentations.
example, if the optimal action is "put one foot in front of the other," then the Q-value distribution reflects this action's relation to other actions regardless of (invariant to) where agent is located.
While recent work shows individual Q-values benefit from regularizing across augmented im- ages (Yarats et al. (022b), we consider that Q-value distributions are also important, as they contain information not just about one action in isolation, but multiple actions in relation to one another.
BACKGROUND
2.1 DEEP RL FROM PIXELS
Markov Decision Process (MDP) consists of an action a E R state s E R and reward IR. "From pixels" assumes that state is image frame multiple image frames stacked together. The action is sampled from a policy at any t time step a ~ T(()). Taking such actions yields a trajectory
(so, ao 81, a1 01, via the dynamics model 5 ~ f(s,, az) of the environment and its rewards Tr+1 = R(s5, 0t). The goal is to maximize cumulative discounted reward R(T) Ei= Toy" where y is the temporal discount factor. The optimal action for state a" (s) = arggmar, Q" (s, a) thus depends on the state-action value function, also known as the Q- value, (s, E [[()]ss 8, dg a).
2.2 SOFT ACTOR-CRITIC & DRQ
further en- Soft-Actor Critic (SAC) (Haarnoja et al. 2018) is an RL algorithm which learns min-reduced ensemble of Q-value functions (s, a) 1in Qo, (s, optimized with one-step Bellman error, and stochastic Gaussian policy nH (s a) optimized by maximizing (s a) and entropy (s))
made differentiable via the reparameterization trick. courage exploration and avoid premature policy collapse, entropy H(s) is also added as part of the agent's 's reward. In visual domains, Qo and тee are typically equipped with a shared convolutional neural net encoder. DrQ (Yarats al.) (2021)) mented and non-augmented next-state targets, and minimizes Bellman error for both Q0 a) and (aug((s), a) where aug is the random augmentation.
the Bellman target the average of the aug-
2.3 EAFA-EFFICIEN RAINBOW (DER)
Rainbow (Hessel et al. 2018) maps directly to Q-value estimates for discrete actions. Compared to vanilla DQN MMnih al. 2013), several refinements are used: Q-values are sampled from multivariate Gaussian probabilistically (Dabney et al. 2018), noise is injected into network parame- ters [Plappert et al.) 2017). double Q networks (Van Hasselt et al. 2016). dueling DQNs (Wang et al.) 2016), n-step returns Watkins 1989). and replay (Schaul et al. 2015). "Data-efficient" refers to the Atari sample limit 100k environmental interactions, much challenging setting for notoriously inefficient RI.
are sampled from prioritized experience
2

---

Under review as a conference paper at ICLR 2022
3 RELATED WORK
3.1 DATA-EFFICIENT RL
Data-efficient RL is a paramount concern to the practicality of RL in real-world use cases. Image augmentation has proven an extremely effective regularizer for improving the sample efficiency of model-free off- off-policy RL algorithms (Yarats et al.) 2019a Srinivas et al. 2020 Sermanet et al.) 2018 Dwibedi et al. 2018). so much so that basic augmentation techniques suffice to rival or sur- pass model-based RL algorithms (Hafner al. 2019e; Lee et al. 2019b Hafner et al. 2019b) in the sample efficiency metric. Curiously, these methods have become progressively simpler. CURL (Srinivas et al., 2020) employed contrastive learning, using positive and negative samples extracted from the mini batch, requiring computation of quadratic complexity w.r.t. the mini-bate batch size. RAD Laskin et al. 2020) and DrQ (Yarats al.) 202Tb) showed that simpler methods work just as well or better, either by augmenting images naively, augmenting and averaging their Q Q-values, respectively. Even more recently (contemporaneouslly) the as-yet unpublished DrQv2 (Yarats al.) 202 found basic augmentation alone suffices under DDPG-based algorithm, with significant efficiency improvements over prior methods despite the exceptional simplicity. While these recent methods have traded CURL's mini-batch statistics for mere augmentation, rQdia marks the first combination of the two that uses mini-batch statistics to enforce consistency across Q-value distri- butions, in manner both simple and complementary to the above implements.
3.2 MINI-BATCH REGULARIZATION IN RL
Mini-batch-baad regularization has not com- monly been used in RL. For example, Batch Norm (loffe & Szegedy, 2015). common reg- ularizer in computer vision, is not as notably employed RL. This type of regularization re- mained unexploited in RL until CURL (Srinivas ctt al., 2020) showed that contrastive learning via image augmentation (Chen al. 2020 He et al. 2020 Misra & Maaten 2020 Henaff, 2020) greatly improved RL data efficiency. CURL con- trasts state's "positive" augmentation sample with the of the mini-batch's "negative" aug- mentation samples. These negative samples can be thought of as sampled from Uniform dis- tribution over the agent's experience replay, in- spiring rQdia. However, contrastive learning en- forces non-guaranteed ground truth, disassociating negative samples regardless of actual similarity. See Figure 2 for an example of this negative samples problem, where similar states are contrasted to have dissimilar encodings as inadvertent byproduct of the uniform randomness. rQdia bypasses this flaw by only enforcing guaranteed constraint: that the Q-value for any sampled action, regard- less of statistical distribution, be consistent across the same states invariant augmentation. This indeed should always be the case, thus yielding gains over CURL while remaining complementary to methods like DrQ and DrQv2.
Confesdivo
Leaening (TA)
embell(s
Sleary chastherat
euthed/supp)
complad
libel nagativ deapra lurty
Beakt Action: Paddie UP
Best Actions Praddie
Figure 2: Contrastive sampling has disadvan- tages that Q-value-based equalization does not.
3.3 IMAGE AUGMENTATION
Image augmentation is commonly used to counter over-fitting computer vision. Techniques in- clude color shift, affinity translation, scale, etc. (Ciregan et al., 2012] Ciresan et al. 2011 Simard cet al. 2003; Krizhevsky et al., 2012] Chen et al. (020)). In Yarats et al. (2021b). the authors in- vestigated several common image transformations and concluded that random shifts strike good balance between simplicity and performance for the MuJoCo environments. A variety of differ- ent augmentations are useful for different games in the Procgen benchmark (Raileanu et al. 2020), and Yarats et al. (2021b) used Intensity variation for the Atari environments. These techniques have proven critical to MuloCo from pixels.
3

---

Under review as a conference paper at ICLR 2022
4 METHODS
rQdia is the first RL Q-value regularizer that does not necessarily depend on either the on-policy or the off-policy states and actions. n, such states and actions instead sampled from arbitrary distributions, let's call them D and 22, of state space S and action space A, respectively:
( ( (S)
(m-1) ~ (2)...
Then, the following constraint is enforced for Qo(-). the neural network that models QT(
Q-value function:
Qo(s(9) Colaug(s!)) al)) Vi,j.
(2)
where aug(-) an arbitrary augmentation transform.
aug(.) and D1, D2 could vary.
For aug(.), ), in line with Yarats et al. (0211), we use translation for MuJoCo and intensity jittering for Atari. Specifically, to translate, pad by 4 pixels, then crop randomly inward by pixels; to intensity jitter, each image multiplied by some random noise s = 1.1 xclip(r, -2,2), ~ (o 1).
For D1, D2 in MuloCo continuous action spaces, we implement simple approach analogous to CURL'S negative-sample sampling, except sampling both states and actions rather than just states. That is, states and actions directly lifted from the mini-batch B (n from a Uniform distribution over the agent's experience replay.
|B), in effect sampled
By using historical states and actions opposed random noise, we compute Q-value distributions over state-action pairs that could more feasibly be encountered in a deployed roll-out.
In discrete Atari, actions may be lifted from the action space directly, that is, (0) (m- A.
Thus, given mini-batch states s(0) ,.., s(..,((-11)) ((n-1)
and mini-batch (or action space) actions (0) (m-1), the following auxiliary loss is proposed to constitute a basic implement of rQdia:
CrQdio
Σ (, Qo(aug(s())) a01)
(3)
TUTI
Then this auxiliary loss is simply added the RL agent's standard loss term. Voila, rQdia (visualized "in nutshell" in Figure 1. This is applied in parallel for each g(6), al)) pair. If mini-batches or actions spaces are very large, it is possible to convolve a smaller subset of n states and m actions.
In MuJoCo, we note that while D1, D2 are treated as Uniform distributions over an agent's history similar to CURL's negative sampling. Dr. D2 could be more sophisticated. For example, the proba- bility sampling an action could be proportional to state similarities. Or, like MPO (Abdolmaleki et al. 2018), actions could be sampled directly from the policy itself.
Algorithm provides pseudo-code for rQdia in continuous control algorithms like SAC-AE and DrQ. All code for rQdia will be released open-source.
5 EXPERIMENTS
To measure the data-efficiency and overall performance of rQdia, we conducted experiments at 100k and 500k steps in the DeepMind Continuous Control Suite (MuJoCo) from pixels and 100k interaction steps in the Atari Arcade Learning Environments.
In the following, (i) are all of the actions in the mini-batch (m = B)) for MuloCo, a are all of the actions in the action space (m = |A] for Atari, with (i)) as all of the states in the mini-batch
(n Bl) and the same MSF-based loss defined in Equation for both.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 rQdia (blue) added to Soft Actor-Critic (SAC), pseudocode courtesy of Achiam (2018).
SAC a good base framework and can be expanded easily to DrQ (see Section 2.2
Input: initial policy params 0, Q-function params Φ1, d2, empty replay buffer D. Set target params equal to main params Orar.1 f- Φ1. @rarg.2 2 <- 02
Denote augmentation function aug(-).
repeat
(1) Observe state 8 and select action a ~ To (-))) (2) Execute a in the environment, (3) Ob- serve next state s', reward r, and done signal d to indicate whether s' is terminal, (4) Store (s, r, s', d) in replay buffer D, (5) If s' is terminal, reset environment state.
if it's time to update then
for j in range(however many updates) do Randomly sample batch of transitions. B ((s,,,r,'',d) from D. Compute targets for the Q functions:
;d)=
Qouan (f, - log log (o)))) d' (-(66') 1=1,2
min
Update Q-functions (minimize Bellman error) by one step of gradient descent using:
V
Σ
(Q.. a) - s'r))
for = 1,2
B
(s,a.T,"',
Update rQdia by one step of gradient descent using:
Σ
min Qui (s(0) (s(0,ab)) min Qøa (aug(s a(1)) k=1,2
|B (a(t (00 ....) JEB
k=1,2
Note that g(0 and a are necessarily the historically paired state and action. Update policy by one step of gradient ascent using:
V
Σ SEB
min Qo. (s, (s)) log my (ao()) s))
where (8) is a sample from x-(sss) which is differentiable wrt via the reparametriza- tion trick. Update target networks with:
Duag. f- porarg, (1 p)o,
for i 1,2
until convergence
Most deep RL benchmarks report point estimates of aggregate performance such mean and me- dian scores across task runs, ignoring the statistical uncertainties that are a natural consequence of training with a finite number of random seeds. the recent analysis by [Agarwal al.] (2021), the authors observe that viewing reported mean scores as random quantities that depend on small number of sample runs exhibits substantial variability, and demonstrate that lot of the reported improvements from previous works disproportionately benefited from randomness in the experi- mental protocol. To account for the variability of results in RL, they propose number of statistical best-practice protocols. We followed these best recommended practices closely as possible and report results in accord with their measured benchmark performances using their sourcs library for RL statistical analysis, rliable (https : github....... com goog www.eeeeeeeecchheeeeeeeeee irlriaabllll Compared to previous works, we evaluate our method with this more thorough statistical analysis and prove rQdia with the recommended robust and efficient aggregate metrics Figures and
Sans stats, raw results show rQdia boosts DrQ's 100k MuJoCo sample efficiency on 4/6 tasks by wide margins, despite a smaller batch size, also surpassing ground truth state embeddings 4/6. In the already-saturated 500k setting, rQdia boosts DrQ on 4/6 tasks, and additionally surpasses state embeddings on the Cheetah Run task. In Atari 100k. rQdia affords DER clear gains over DrQ and CURL, superseding mean bascline human-nome scores especially by near 200%.
5

---

Under review as a conference paper at ICLR 2022
5.1
DEEPMIND CONTINUOUS CONTROL 500K & 100K
5.1.1
STATISTICAL ANALYSIS
100k steps
0.75
0.25
100k steps
500k steps
Fraction of runs with
0.00
DrQ+rQdia
0.0
0.2
0.4
0.6
0.8
1.0
DrQ RAD
500k
steps
1.00
0.75
PISAC
0.25
SAC+AE
0.00
SLAC
0.0
0.2
0.4
0.6
0.8
1.0
0.4
0.6
0.8
0.8
0.9
Normalized Score (T)
Normalized Scores
Normalized Scores
a) Performance Profile
b) Interval Estimates
SLAC
SAC+AE
Dreamer
PISAC
RAD
DrQ
DrQerQoia
cheetal
Distribution
6
234467
1234567
1234567
1 Ranking
Ranking
Ranking
Ranking
Rarking
Ranking
c) Rank Comparisons On Individual Tasks
Figure 20 seeds DM Suite, aggregated across tasks, on 100k and 500k benchmarks. We compare DrQ+rQdia with SAC+AE (Yarats et al. 2019b). SLAC (Lee et al. (019a). Dreamer (Hafner et al. 2019a), CURL (Srinivas et al., 2020). RAD (Laskin et al.), DrQ (Yarats et al. 2021b), and PISAC (Lee et al. 2020). Due to our own computational limits, we used a batch size 128, while other methods used 512. According to Yarats et al. (2021a), bigger batch sizes yield bigger improvements. At 1/4 the batch size of other methods, rQdia still surpasses matches benchmarked scores. (a) Although pure stochastic dominance is rarely observed (Dror al. 2019). rQdia outperforms others 100 steps, meaning achieve higher scores per number of 500k steps, we still reach SOTA performance despite the hampered, more-efficient batch size. (b) the lower bound of algorithm's 's interval is higher than another algorithm's 's upper bound, there is a high confidence that the algorithm is better. rQdia not only outperforms the others but also has a relatively small interval, meaning the result is better and also
At
consistent across different runs. (c) rQdia not only ranks first, but has a high probability of being ranked first on 5/6 tasks, indicating the holistic statistical performance exceeds baselines by decisive margin.
In line with recent works, we evaluate 6 tasks the DeepMind Continuous Control Suite at 100k and 500k steps to measure data efficiency and asymptotic performance respectively. The reported improvements are based on higher mean scores per task, with large variability across random seeds. When accounting for this variability, turns out that many previous algorithms do not consistently rank above the algorithms they claim to improve on (Agarwal et al. 2021). Therefore, the results and metrics aggregated by [Agarwal al. (202I) ) and reported Figure Fare not necessarily consistent with the ones reported in previous papers. We followed the protocol of [Agawal et al. (2021) and instead stick to their benchmarked curves for our analysis. Unlike prior works, our computational constraints required that smaller batch size. Yarats et al. (202 report that smaller batch sizes are disadvantaged.
smaller batch size. All of our rQdia results reported with this
Performance Profile The performance profile shows the tail distribution of scores on combined runs across tasks, thus allowing us to compare different methods at a glance. If one curve strictly higher than another. it is said to "stochastically dominate" (Dror al. 2019). Figure 3a) indicates that at 100k, rQdia can achieve the same score as other algorithms with fewer runs and a higher score with equal runs. At 500k, rQdia matches SOTA performance despite 1/4 the batch size.

---

Under review as a conference paper at ICLR 2022
Table 1: rQdia is robust batch size. At 1/4 the batch size, rQdia surpasses or matches baseline
models in data-efficiency (100k) and asymptotic performance (500k) given by mean episode reward averaged over 20 seeds. It is reported that larger batch sizes of 512 were necessary to achieving the performances of prior works (Yarats et al. 2021a), while DrQ+rQdia uses a batch size of only 128.
From Pixels
State Emb
500k Step Scores
DrQ+rQdia-128
DrQ-512
CURL
RAD
SAC+AE
SAC State
Ball In Cup Catch
919.69
963.94
958
970.36
810.85
979
Cartpole Swingup
864.75
868.82
861
858.09
730.94
870
Cheetah Run
777.25
679.91
500
774.96
544.3
772
Finger Spin
939.05
938.77
874
907.4
914.3
929
Reacher Easy
950.01
945,4
904
930.44
601.36
975
Walker Walk
934.47
924.16
906
917.58
858.16
964
100k Step Scores
Ball In Cup Catch
910
913.8
772
950.22
338.42
957
Cartpole Swingup
867.42
759.37
592
863.69
276.63
812
Cheetah Run
502.77
360.97
307
499.06
240.58
228
Finger Spin
842.47
901.41
779
813.05
747.01
672
Reacher Easy
905.34
600.42
517
772.44
225.14
919
Walker Walk
721.78
633.57
344
644.78
395.87
604
Interval Estimates We resampled with replacement independently for each task to construct an em- pirical bootstrap sample in which we computed 95% stratified bootstrap confidence intervals (CIs). This process is repeated 50000 times approximate the real sampling distributions. Normalized scores are computed by dividing by the maximum score (= 1000). rQdia yields a high confidence with a small interval, meaning its performance is both statistically better and more consistent.
Rank Comparisons show the probability that given method is assigned rank i, averaged across all tasks. The ranks are estimated using 200,000 stratified bootstrap re-samples. rQdia ranks highest on 5/6 tasks with high probability.
5.1.2 ROBUSTNESS To BATCH SIZE
We report tabular results in Table rQdia excels at sample efficiency, achieving SOTA results at the 100k benchmark, even rivaling the state embedding ground truth. At 500k, where results already saturated, rQdia yields less pronounced improvements in terms of final score, but notably attains SOTA scores despite smaller batch size. Yarats al. (2021a) report that performance hinged on the whopping 512 batch size. Due to computational limits, we could not reproduce this, but achieved competitive results nevertheless 1/4 the batch size, robustly using just 128.
5.2 ATARI ARCADE 100K
To conduct thorough statistical analysis of Atari 100k performance, we evaluated rQdia with the performance profile described in Section 5.1 We further scaled the x-axis such that the space between any and 72 is proportional to the fraction of runs averaged across algorithms between and 72. This scaling shows the regions of the score distribution where most of the runs lie as opposed to comparing tail ends of the distribution. In addition to median and mean, we further report two additional aggregate metrics as recommended in (Dror et al., 2019), where scores are computed with 95% stratified bootstrap confidence intervals. Our case study compares the performance of five recent deep RL algorithms, namely: (1) DER van Hasselt et al. 2019), OTR(Kielaki 2020), SimPle (Kaiser et al. 2019), DrQ Yaraat et al. 2021b), and CURL (Siiivass et al.) 2020). We also include SPR (Schwarzer et al.) 2020). a fiightly-apples apples-to-orangern tow-oroange-bbseliiinn baseline which learns a self- supervised environment dynamics model, which rQdia is orthogonal and amazingly approaches the performance of despite the substantial difference in simplicity. Raw tabular results for Atari 100k presented in Table 2 with benchmarks likewise pulled from rliable.

---

Under review as a conference paper at ICLR 2022
Table DER + rQdia rQdia augmented Data- Data-Efficient Rainbow (DER) yields performance gains competitive with SOTA models in the 100k data-efficient Atari benchmark (mean per 10 random seeds, scores pulled from rliable (Agarwal al. 202T)). The most apples-to-appies comparison is rQdia and DER, since we build on top of DER. rQdia is also orthogonal to the other reported methods, and could feasibly yield even more striking improvements augmented to those.
Atari Arcade Environments
DER rQdia
DrQ
CURL
DER
Random
Human
Alien
1188
734.076
711.033
802.346
227.8
7127.7
Amidar
208.9
94.195
113.743
125.905
5.8
1719.5
Assault
649.9
479.536
500.927
561.46
222.4
742
Asterix
890
535.645
567.24
535.44
210
8503.3
BankHeist
64
153.412
65.299
185.479
14.2
753.1
BattleZone
19000
10563.6
8997.8
8977
2360
37187.5
Boxing
12.3
6.631
0.95
-0.309
0.1
12.1
Breakout
8
15.406
2.555
9,214
1.7
30.5
ChopperCommand
1500
792.39
783.53
925.87
7387.8
CrazyClimber
23970
21991.55
9154.36
34508.51 57
10780.5
35829.4
DemonAttack
1833
1142.448
646.467
627.599
152.1
1971
Freeway
26.8
17.778
28. 268
20.855
0
29.6
Frostbite
2874
508.08
1226.494
870.975
65.2
4334.7
Gopher
896
618.014
400.856
467.02
257.6
2412.5
Hero
7261
3722.64
4987,682
6226.044
1027
30826.4
Jamesbond
985
251.765
331.05
275.66
29
302.8
Kangaroo
670
974.45
740.24
581.67
52
3035
Krull
4193
4131.377
3049.225
9256.886
1598
2665.5
KungFuMaster
16310
7154.51
8155.56
6580.07
258.5
22736.3
MsPacman
1598
1002.926
1064.012
1187.431
307.3
6951.6
Pong
-14.8
114.251
-18.487
99.11
-20.7
14.6
PrivateFye
12.9
24.844
81.855
72.751
24.9
69571.3
Qbert
2112.5
934.242
727.01
1773.54
163.9
13455
RoadRunner
8840
8724.66
5006.11
11843.35
11.5
7845
Seaquest
386
310.494
316.186
304.581
68.4
42054.7
UpNDown
4154
3619.133
2646.372
3075.004
533,4
11693.2
Mean Human-Norm Score
59.146 %
36.912%
26.149%
30.03%
0%
100%
Med Human-Norm Score
25.750 %
21.198%
9.235%
18.9%
0%
100%
Interquartile Mean (IQM) discards the bottom and top 25% of the runs and calculates the mean score of the remaining 50% runs, which serves an interpolation between mean and median across runs. IQM is robust to outliers compared to mean and has considerably less bias than median.
Optimality Gap is the amount by which the algorithm fails to meet minimum score (human score), which serves as robust alternative to mean. This metric assumes that a score of human- level performance is desirable target beyond which improvements not very important.
Probability Of Improvement designed to measure how likely it is for X to outperform Y on randomly selected task, which is computed by the Mann- Whitney U-statistic (Mann & Whitney) 1947). The interval reported estimates are based on stratified bootstrap with independent sampling with 2000 bootstrap re-samples.
DISCUSSION
Limitations One limitation of rQdia is that it assumes benefit to certain augmentation invariance. Translation invariance for example might not be useful in environments where most objects held within a consistent axis.
Moreover, rQdia ensures consistency between Q-value distributions across such perturbations, which means that models in environments that do not require such visual invariances are needlessly expected to learn more complex, more general function.
8

---

Under review as a conference paper at ICLR 2022
Smpin
DER
bPQ
Diserlyds
SFF
0-1
Score Distitttoona
Seors Clarbuzton with her Scalig
6/0
Trancion casks with sc.0r
Smha
0.75
0.75
0.90
Nord
nom
589
0.25
0.00
10:00
Human Nommlzzd Soore
Haman Normslized Score (1)
a) Performance profile w/ linear and non-linear scaling
b) Probability of Improvement
Optimality Gap
DER+rQdia
DrQ
CURL
III
OTR
DER SimPLe
0.56
Human Normalized Score
c) Aggregate metrics on Atari 100k
Figure 4: 10 seeds Atari ALE, aggregated across 26 environments. (a) Performance profiles with pointwise 95% confidence bands show rQdia outperforms others with a large margin especially when E [0, 1]. namely relative human-level performance. After non-linear scaling, the improvement of our algorithm is more pronounced. The gap between DER+rQdia and DER suggests rQdia can majorly improve learning. (b) The bottom-right subplot shows that rQdia has very high chance of improvement over all baselines, and no other baseline can have > 50% chance of outperforming rQia.. (c) Higher mcan, median, and IQM scores and lower optimality gap are better. rQdia has the best performance across all four metrics, indicating more certain and substantial improvement. All results are based on 10 runs per environment, except SimPLe, for which we use their reported 5. Notably, DrRaa with simple image augmentation, rivals the SOTA results of the orthogonal- SPR (Schwarzer et al.] 2020), which learns computationally intensive environment dynamics model. We include SPR just for reference.
On the other hand, such environments where these invariances are not useful or important may leverage rQdia learn more invariant representations that could potentially better generalize to different, more complex environments.
Ethics rQdia is simple regularizer that contributes the generalization of deep reinforcement learning models. While we hope deep RL continues to improve and its applications and abilities ex- pand, we would be remiss weaponry to economic exploitation. However, we are optimistic that RL can do much more good than bad for society. Autonomous agents that can interact with the real world via RL-based, more- streamlined robotics opens the door for countless medical, social, and economic benefits well.
note the destructive potential of the field, ranging from autonomous
Reproducability We will release code open-source; we have submitted code together with the paper; relevant code snippets are shared in Appendix [ continuous control pseudocode provided in Algorithm [ hyperparams specified in Appendix
CONCLUSION
We presented a simple regularizer for model-free reinforcement learning that may easily be inte- grated into existing reinforcement learning frameworks. With the inclusion this auxiliary loss, we attain strong performance compared to baseline models, including recent state of the arts. By reg- ularizing Q-value distributions, we further enforce the invariances afforded by image augmentation techniques such that Q-value distributions we observe improvements in sample efficiency and final reward in the DeepMind Continuous Con- trol Suite and environments in the Atari Arcade Learning Environments.
preserved under these perturbations. Consequently,

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abhas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv:/8000 88060069660 2018.
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Informa- tion Processing Systems, 2021.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. In 2012 IEEE conference computer vision and pattern recognition, pp. 3642- 3649. IEEE, 2012.
Dan C Ciresan, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High- performance neural networks for visual object classification. arXiv preprint arXiv:1102.0183, 2011.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 32, 2018.
Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance-how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 2773-2785, 2019.
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable rep- resentations from visual observations. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1577-1584. IEEE, 2018
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05900 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1912v:19/2.01600 2019a.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXiv:1912.01603. 2019b.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019c.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pp. 4192-4192. PMLR, 2020.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. PMLR. 2015.
Lukasz Kaiser, Mobammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint arXiv: (90::::00000777 2019.
Kacper Piotr Kielak. Do recent advancements model-based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003.10181, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learming with augmented data. arXiv:2004149990
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. arXiv preprint arXiv:2004. 140::0 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic:
Deep reinforcement learning with latent variable model. arXiv preprint MXXi::997.0099. 2019a.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint (0tiv:9907.0000 2019b.
Kuang-Huet Lec, Ian Fischer, Anthony Liu. Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11890-11901. Curran Asso- ciates, Inc., 2020. URL
neurips httpsc/pppp///2//EEE... pdff
Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochas- tically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6707-6717, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 1706.00900 2017.
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. 12862, 2020.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1rXiv:1511..0335 2015.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR, abs/2007. 00599,, 2020.
h/wtwiv.//////////aae/a wwtp:/wwwwww/////////o/// 05929
11

---

Under review as a conference paper at ICLR 2022
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-super learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1144-1141. IEEE, 2018.
Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar, volume 3. Citeseer, 2003.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: contrastive unsupervised representa- tions for reinforcement learning, CoRR, abs/2004.04136, 2020. UtLhttss://arxx
abs/2004. 04136
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1rXiv: .0690, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforce- ment learning? arXiv preprint arXiv:1906.0522 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference machine learning, pp. 1995-2003. PMLR, 2016.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019a.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv: arXiv:1910.01741 2019b.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09644 2021a.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representa- tions, 2021b. URL
A ARCHITECTURE
The SAC-AE base architecture and distinct policy and Q-function heads. The CNN encoder is shared by the actor and critic; critic consists of two Rel.U-activated 3-layer MLP Q-networks; and the actor is single ReLU-
for DrQ is the same [38], consisting of shared encoder
activated 3-layer MLP Gaussian policy head. We modify the code provided by [30]: https: 7/github. com/ Mi Mishaasskin//am curl
Atari environments were tested with a Rainbow architecture inspired by [35] and built on the variant implemented in tandem with CURL in [30]. We added the rQdia auxiliary loss to their code sans the CURL-related portion. This code may be found here: htttps://githim com/ aravindarini wttpovaaaacurrrraabbbae
B TRAINING
All hyperparameters were preserved from the original implementations discussed above. They reviewed in Tables 3 and y except for the substitution of batch size since we used a batch size of 128 while [30, 37] used 5T2, reportedly giving those models decent advantage.
12

---

Under review as a conference paper at ICLR 2022
Param
Value
Observation Size
(84, 84)
Replay Buffer Size
100000
Param
Value
Frame Skip
Observation Size
(84, 84)
Action repeat
Replay Buffer Size
100000
Q-network Channels
32, 64
Initial Steps
1000
Q-network Filter Size
5, x
Stacked Frames
3
Q-network Stride
5,
Action Repeat
finger, spin;
Q-network Hidden Units
256
walker, walk
Momentum
0.001
8 cartpole, swingup
Non-Linearity
ReLUU
4 otherwise
Reward Clipping
[-1, 1]
MLP Hidden Units
1024
Multi Step Return
20
Evaluation Episodes
10
Min replay size
Optimizer
ADAM
for sampling
1600
Learning Rate (Jo, Tatt Q²)
0.001
Max Frames Per Episode
108K
Learning Rate (a))
0.0001
Target network
Batch Size
128
update period
2000 updates
Q Function EMA
0.01
Support Of Q-dist
51 bins
Critic Target Update Freq
Discount ?
0.99
Conv Layers
Batch Size
32
Number of Filters
Optimizer
ADAM
Non-Linearity
ReLU
Learning Rate
0.9
Encoder EMA T
0.05
(81,32)
(0.9, 0.999)
Latent Dimension
50
Optimizer
0.000015
Discount Y
0.99
Max Grad Norm
10
Initial Temperature
0.1
Noisy Nets Parameter
0.1
Priority Exponent
0.5
Table 3: Hyperparameters for rQdia-DrQ
Priority Correction
0.4 - 1
Table 4:
Hyperparameters for rQdia-
Rainbow
C
CODE
Code for continuous control and discrete Atari will be released on GitHub and is provided in the supplementary material.
The rQdia loss in Rainbow is a sim- ple mean squared етror between the
r@dia, Loss
torch.nn, www.e.inn.unnttiiooooo functional.co ase, watpll..sse______e_____oo_oo_o Loss(log.avg dist, Log_anchor (og_anchor_dist) rQdia,
anchor and augmentation's 's respective
Q-value distributions (Figure 5).
Figure Pytorch code for rQiia in Rainbow Atari.
Continuous control involves a bit more handiwork, but is also simple to tweak into an exist- ing RL library (Figure 6). First, mini-batch actions and states have to be convolved in pairs with one another. A scaling factor E (0, 1] can be modified for efficiency to determine how many such pairs should be used, Then the double-critics predict a Q-value distribution convolved pairs, which is mini- mized w.r.t. the augmentation.
rinia (Rogotarizing e-Veuee oistributions
Inage Augnentation)
sutch. astion.amspulo
scaling 2.15 Lowar afficiant
sba ahwos[1]
action. astian.ansoual
umsquesze()) aspans(-1, settiooo -1).mnhhape-22 cla) unsoweaau(1) exourdt
sctton_patre acttonl nuu_oottore) unsquesze(s) expant thatchhahcccstee -1) restapec-i, a0tton_dti)
0 dists tbs tbt_ tes.entt.0 02 dutt ent oorsttteie wwweeeeeoon.orreeee orto. patre, sctton_patre) sbu
dist selt crataefate action suirs)
the
Figure 6: Pytorch code for rOdia in continuous control.
13


Reviews:
[{"review_id": "lHaT3witmRr", "reviewer": "Reviewer_n4Pr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nSimple Idea\nWell explained\nStandard benchmarks\n\nWeaknesses:\nNot a significant improvement compared to prior work.", "summary_of_the_paper": "Image augmentations have recently become a standard component of deep RL algorithms.\n\nPrevious work has enforced consistencies at a sample-level.\n\nThis paper proposes to look at the distribution of statistics at a minibatch-level in order to enforce consistencies.\n\nPaper shows results on standard benchmarks in discrete and continuous control (Atari and DMC).", "summary_of_the_review": "The idea is simple and easy to implement and people can try to fork this in whatever they're doing. However, the improvements shown aren't significant to strongly push for accepting it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SVAYv60fjEK", "reviewer": "Reviewer_tset", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Data-augmented methods for reinforcement learning have been increasing in popularity and this paper demonstrates another way of using augmentation to regularize the Q values that empirically performs well.\n\nThe paper is generally well-written and easy to understand the contribution. The comparisons to the baselines of DrQ/CURL/PlaNet/SAC-AE in Tables 1 and CURL/Eff Rainbow are, to the best of my knowledge, the relevant baselines on these challenging tasks. To the best of my knowledge, the data augmentations are similar to the ones used in DrQ/CURL, but used to regularize the Q function on many actions sampled rather than in the standard updates.\n\nI found two minor parts confusing:\n1. The definition of the Q-value distribution at the bottom of page 1 is not a proper statistical distribution. It instead appears to be pointing out that for a fixed state, we can consider a collection of values of applying different actions from that state.\n2. After eq (1), the paper says the actions $a_i$ are selected from the minibatch for a fixed state $s_t$. This also seems consistent in Alg 1. Do I understand correctly that this uses the actions from irrelevant states in the minibatch to use for this regularization term? If so, is it interesting to consider other distributions over the actions to be sampled here?", "summary_of_the_paper": "This paper proposes to use data augmentation to regularize the Q distribution by matching the Q values between augmented and unaugmented states (Alg 1). The results are competitive on the DeepMind control suite tasks (Tab 1) and Atari (Tab 2).", "summary_of_the_review": "It's an interesting new usage of data augmentations that is nicely demonstrated", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CgpIqptpASz", "reviewer": "Reviewer_5Bt1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is very simple and effective but there has been a lot of prior work, some of which is not cited or compared. For example, RAD (Laskin et al. Reinforcement Learning with Augmented Data) is a simpler method which is extremely relevant but there is no mention of it. \n\nTable 5 in the RAD paper referred above is inconsistent with Table 1 highlighted in this paper. They both refer to the same 500k and 100k scores in the DeepMind Continuous Control Suite. Scores in RAD are sometimes similar or better than the proposed method. But perhaps more surprisingly, the CURL scores are different in these two tables. I am not sure why this would be the case. This is a significant issue that needs to be addressed by the authors. ", "summary_of_the_paper": "This paper proposes a very simple auxiliary loss to regularize Q-values given mini-batches of actions between images and their affine augmentations.\n\nThis approach is then validated on continuous control and ALE environments (standard benchmark).", "summary_of_the_review": "The authors should provide an explanation regarding the discrepancies in results highlighted in this paper vs the RAD paper for the same benchmark tasks. The experimental validation and baselines need more work before this paper is ready for publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-VT2PzQeCbZ", "reviewer": "Reviewer_dutv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the one hand, the proposed method is very simple. It can be added to many RL methods as a regularization term. Experiments show the validity of this method. \n\nOn the other hand, however, the method that pulls the output of an image and its transformed counterpart close to each other is already commonly used in computer vision methods, thus, the novelty of this paper is limited. More study and analysis on the regularization is expected, such as different functions for loss computation besides MSE and KL, how does regularizing Q-value differ from other methods.\n\nAdditionally, there are several small issues:\n1)\tFigures in this paper are far from elegant. Symbols in Fig.1 is too large, while text in Fig.3 is too small.\n2)\tI suggest reorganizing Sec.4. Some discussion can be set in a single subsection.\n3)\tIn Sec.4, “Equation 2” looks like a typo.\n4)\t$n$ used in Eq.1 denotes the number of actions, not the batch size.\n", "summary_of_the_paper": "This paper proposes a regularization method for reinforcement learning that encourages the Q-value of the original image (i.e., original state) and the Q-value of the transformed image (new state) to be the same. This method enhances the robustness of RL methods against environment variation. This paper introduces the background and the motivation of the proposed method. Discussion and comparison of the difference between related works, such as SAC and DrQ, is also provided. Experiments show that the proposed method can improve the performance of image-based methods, even outperforming several state-based methods.", "summary_of_the_review": "This paper proposes a simple method to improve the performance of image-based RL. My main concern is about the novelty as proposed method is commonly used in CV methods. More further discussion and study will improve the quality of this paper.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple modification to how data augmentation is done in image-based RL. This results in some improvements on benchmark tasks. The change essentially amounts to adapting data-augmentation strategies that are already understood in other fields to deep RL. However, the effect of data augmentation in simple image-based deep RL tasks is already known. As such, I think the contribution in this paper is quite incremental -- the notion that data augmentation in deep RL helps is already known, and the particular augmentation strategy proposed here is not especially novel. So while it's good in terms of producing improved results on some benchmark tasks, it doesn't seem to be of high significance to the study of reinforcement learning or machine learning more broadly. As such, I think it could be a valuable contribution to a more narrow venue, or as a technical report, but is too incremental and narrow in scope for ICLR.

A note to the authors (this did not impact the paper decision): due to the unfortunately lackluster quality of the reviews, I read and reviewed the paper myself as well to be able to produce a more accurate meta-review. In the balance, I see the point the authors make in the response that some of the results in prior work (e.g., CURL) are unfortunately unreliable. That's not the fault of the authors, it's the fault of the prior works. I took this into account in my assessment. In this sense, I do think the comparison to prior work is sensible. On the other hand, I think the practice of reporting only very specific checkpoints (e.g., 100k and 500k), though borrowed from prior work, is not a good way to report results, as it hides the real performance of the methods.
Decision: Reject