On the Data-Efficiency with Contrastive Image Transformation in Reinforcement Learning
Authors: Sicong Liu
Xi Sheryl Zhang
Yushuo Li
Yifan Zhang
Jian Cheng
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Data Augmentation
Self-Supervised Learning
Representation Learning

Abstract: Data-efficiency has always been an essential issue in pixel-based reinforcement learning (RL). As the agent not only learns decision-making but also meaningful representations from images. The line of reinforcement learning with data augmentation shows significant improvements in sample-efficiency. However, it is challenging to guarantee the optimality invariant transformation, that is, the augmented data are readily recognized as a completely different state by the agent. In the end, we propose a contrastive invariant transformation (CoIT), a simple yet promising learnable data augmentation combined with standard model-free algorithms to improve sample-efficiency. Concretely, the differentiable CoIT leverages original samples with augmented samples and hastens the state encoder for a contrastive invariant embedding. We evaluate our approach on DeepMind Control Suite and Atari100K. Empirical results verify advances using CoIT, enabling it to outperform the new state-of-the-art on various tasks. Source code is available at https://github.com/mooricAnna/CoIT.

Published as a conference paper at ICLR 2023
ON THE DATA-EFFICIENCY WITH CONTRASTIVE IMAGE TRANSFORMATION IN REINFORCEMENT LEARNING
Sicong Liu 23" Xi Sheryl Zhang 235 Yushuo Li Yifan Zhang
Jian Cheng 234
NJUST, Institute of Automation, Chinese Academy Sciences, 3 AIRIA, Sccooo of Future Technology, University of Chinese Academy of Sciences, University of Chinese Academy of Sciences, Nanjing
Isicongliu1014
@eggmaall..... com,
yushuo. 1i@1 via ac. en, (yfzhang; jcheng] @nlpr. iaaiiaaa.. aa.ac.cc..
ABSTRACT
Data-efficiency has always been an essential issue in pixel-based reinforcement learning (RL). As the agent not only learns decision-making but also meaning- ful representations from images. The line of reinforcement learning with data augmentation shows significant improvements in sample-efficiene However, it is challenging to guarantee the optimality invariant transformation, that is, the augmented data are readily recognized completely different state by the agent. In the end, we propose a contrastive invariant transformation (ColT), a simple yet promising learnable data augmentation combined with standard model-free algorithms to improve sample-cfficieney. Concretely, the differentiable ColT lever- ages original samples with augmented samples and hastens the state encoder for a contrastive invariant embedding. We evaluate our approach on DeepMind Control Suite and Atari100K. Empirical results verify advances using ColT, enabling to outperform the new state-of-the-art various tasks. Source code is available at https://github. wwtp:/wwww..ooooo..e.o..ooo.oc.e http://tw......coooooooooccc///a
INTRODUCTION
Improving data-efficiency to accomplish sequential decisions has always been a crucial problem in pixel-based reinforcement learning. As the agent has learn optimal policy with a meaningful information abstraction from observations parallel. Unlike supervised representation learning with strong supervised eigh-dimessionl signals, the training process in RL is fragile. It could be harmful to the training process and cause performance degradation consequently using inappropriate manners. Hence, it is an urgent request to seek subtle representation learning methods for visual RI..
Previous works have been proposed in the literature to demonstrate that introducing auxiliary loss functions such as pixel reconstruction (Yarats et al. 2019) and contrastive learning (Laskin et al.. 2020b alleviates this issue. In particular, data augmentations have already proven beneficial to data- efficiency. RAD (Laskin et al. 2020a) performs an extension of experiments and widely analyzes the impact of various techniques data augmentation. DrQ (Yarats et al. 2020 and DrQ-v2 (Yarats ct al. 202I) make use of appropriate image augmentation with great success. Also, previous works have carried out the potential of data augmentation in terms of generalization (Hansen al. 2021 Raileanu et al.
& Guo 2021 Hansen & Wang 2021 Fan et al., (202)).
Despite the mentioned efforts, it is pretty hard guarantee that the augmented representations sufficiently diverse yet semantically consistent. To this end, we explore the underlying condition for representation learning in RL. It is rational to hypothesize that there is an optimal transformation enabling an encoder to abstract informative latent space. This line of works belongs to the regime of
state abstraction (Du et al., 22199; Zhang et al.) 22020b; Tomar al. 2021| Wang et al. (022).. which
derives from grouping similar world-states for descriptions of the environment (Dietterich) 2000
Andre & Russell 2002 Castro & Precup 2010). Inspired by spatial transformer networks (STN)
Work done during Institute of Automation, Chinese Academy of Sciences internship.
Cooresponniig Author.

---

Published as a conference paper at ICLR 2023
(Jaderberg et al. 2015). data augmentation model in the vision domain, consider that merging the parameterized transformation with visual RL could be beneficial. The designed transformation not only discovers the optimality of state abstraction but also produces diverse virtual samples for the agent. To do so, we enforce a learnable data augmentation that updates its parameters along with the RL objective.
To understand parameterized augmentation and its relation to representation learning in RL, we focus on fundamental data manipulation by generating augmented data from a learnable Gaussian distri- bution. To be clear, we present the image transformation to control the margin of the augmentation under an RL gaining-friendly data distribution. Since changed data distributions meanwhile being controlled by learning algorithms would be helpful in high- nigh-dimensional cases (Balestricro et al. 2021 Here we raise our idea:
Can we parameterize the data augmentation by sampling from a dynamic distribution to obtain a training-friemn state dis- tribution along with model-free RL?
Method
ColT
133%
In light of this challenge, we present a contrastive invariant transformation (ColT), novel contrastive learning to ame- liorate the data-efficiency for visual RI. ColT integrates a learnable transformation for model-free methods with minimal modification to the architecture and training pipeline, Specif- ically, we parameterize the mean and variance of Gaussian distribution for transforming data and update the parameters together with RL by using constraints urge faster algorithm convergence empirically. As the learning goes on, the agent approximates the TRANSFORM distribution that is optimal for the task at hand to solve the task. In addition, we evaluate ColT on Deep Control Suite and Atari and exper- imental results demonstrate that the learnable transformation outperforms the current SOTA methods. Besides, our method does not claim any custom architecture choices and is essential for reproducing end-to-end training. Based on these results, we demonstrate that learnable transformation improves data- efficiency effectively for visual RL.
DIQ-2
100%
CURL
65%
% Tasks solved
Method
ColT
DrQ-v2
100%
CURL
112%
% Tasks solved
Figure 1: Percentage (%) of score solved in the DMC. We set the score of DrQ-v2 as 100% and report the result of ColT and CURL: Up: in 500K steps. Bottom: in 100K steps. The task is solved when re- turn nearly reaches the upper bound.
Key Contributions: (i) We present ColT, simple yet ef- fective framework with a learnable image transformation that integrates invariant representations with model-free RL im- prove data-efficiency. (ii) We propose a theoretical analysis of how our method can approximate stationary distribution over the transformed data by the optimal invariant metrie, thus learning better representations. (iii) We evaluate ColT on popular benchmarks and show that our method outperforms previous state-of of-the-art methods on data-efficiency and stability.
2 RELATED WORK
Several concurrent methods have been proposed for improving data-efficiency whose common ingredients containing data augmentation and self-supervised learning are listed.
Data augmentation in RL. Like the success of data augmentation in computer vision ZZhon et al. 2020 DeVriss & Taylor 2017 Yun et al. 2019 Zhang al.) (017). these methods have played a key role in improving the data-efficiency of visual RL problems Mnih et al. (2013); Yarats al. (2019); Hafner et al. (2019); Lee et al. (2019). RAD (Laskin et al.) 2020a) conducted mounts of experiments and finds out that different data augmentations lead to entirely different results. It provides a broader perspective for the follow-up study of data augmentation in RL. DrQ (Yarats et al. 2020) proposed an effective augmentation method called random shift and introduced regularization term for Qllearning Based on DrQ, the DrQ-v2 Yarats et al., 202T) conducted minimal changes and demonstrated that merely a simple augmentation method could match the state-of-the-a model-based algorithm on data-efficiency and performance.
Self-supervised learning in RL. Motivated by the breakthrough in self-supervised learning (Chen et al. 2020 He al. 2020 Caron et al. 2020 Grill et al. 2020), it is natural to combine these
2

---

Published as a conference paper at ICLR 2023
methods with visual RL to learn rich representations. CURL Laskin et al.) 2020b) introduces a framework similar to SimCLR (che) into visual RL. CoBERL (Banino et al. [022] also tried
to minimize the consistency between positive samples by semantic preserrireeeeegggeee data augmentation. Besides, STDIM (Mazoure et al. 2020) and PI-SAC (Lee et al. 2020) maximize the temporal mutual information (MI) between the nearby states. SPR (Schwarzer al. 2020) and Play Virtual (Yu et al. 2021) follow their idea, but they utilize the dynamics model to predict nearby states in latent space. DBC (Zhang al. 2020b) and PSM (Agarwal et al. 202T) focus on learning task-relevant information. They utilize signals in the environment to achieve an invariant representation learning and thereby generalize the agent to unseen environments.
PRELIMINARIES
3.1 REINFORCEMENT LEARNING FROM OBSERVATIONS
Visual RL control is formulated as an infinite- Markov Decision Process (MDP) (Bellman
1957), as the observations can not fully describe the underlying state. To address this problem, we stack multiple consecutive frames together to represent the current underlying state s (Mnih et al. 2013). In this mind, the MDP M is 5-tuple (O, S, A, r, 7). Here, the observation space O generally consists of multiple-staee stack frames. The state space S is either observable or unobservable (Silver et al., Zhang et al,, 2020a). The agent uses observations O to sample actions from the action space A. Every time the agent interacts with the environment, it obtains reward r. The end goal is to train an agent to maximize the cumulative reward R. The policy evaluation used as estimating the performance of the policy Tg is normally defined by rewards in infinite-horizon tasks,
B E
Σ
('r ar. S141)
(I)
where y E [0, 1) is the discount factor and denotes the reward at time t.
3.2 Q LEARNING
The state-action value function Qo is trained by minimizing the Bellman error to estimate the cumulative reward at the current state:
Ja(D) = (Q- - r. (.
(2)
where e is a transition from the replay buffer D. And 0 denotes an exponential moving average of 6. For the continuous control tasks, we utilize an actor-critic algorithm called Deep Deterministic Policy Gradient (DDPG) (Silver al., [2014] Lillicrap et al.) 2015) which consists of the aforementioned state-action value function Qa and a deterministic policy Tg. The policy то aims at maximizing (DD = ( To (s:)). Various effective improvements have also been lead DDPG. The Q- learning process incorporates n-step returns [Watkins] 1989 Peng & Williams, 1994). The scheduled exploration noise is produced by a linear decay (tt) for the variance 0² which provides different levels of exploration at different training steps: (t)) = Ōini + (1 min(t/T, ,, 1)) (ā 0. The
initial and final value for standard deviation are defined by Finit and Ffmal and the decay horizon T is related to the total training steps of the environment.
2019) which applied multiple improvements on top of the original Nature DQN (Mnih et al. 2015).
For the discrete control, we use the data- fata-efficient Rainbow DQN (Van Hasselt et al.
3.3 STATE ABSTRACTION
While visual RL has achieved many successes in simulated tasks, it remains challenging to learn robust representations from real vision, where images reveal detailed scenes of a complex and unstructured world (Zhang et al. 2020b Agarwal et al. 2021 Wang et al.) (022). Therefore, abstracting meaningful elements from the visual scene to present the underlying state is significantly important for visual RL.
We follow the Block Markov Decision Process (BMDP) (Du et al. 2019). which refers episodic learning tasks via an unobservable latent space S and an observable context space X. The environment generates context by p(-Is). They present fundamental assumption as: each observation
3

---

Published as a conference paper at ICLR 2023
uniquely determines its generating state Similarly, in model-free RL without modeling dynamics, the manipulated context x can be conditioned on a certain probability given an environment transition which is x p(-le).
4 THE COIT
4.1 LEARNABLE INVARIANT TRANSFORMATION
Following the motivation of smoothing training experiences to stabilize the target Q network (Mnih et al., 2013; Lillicrap et al., 2015), the transformed x' is required to satisfy x' ~ p(-le), where environment transition e is ideally in the replay distribution D. Note that e is a random variables. Formally, we are ready to introduce the optimal invariant metric to reach the stationary distribution 'D over the augmented context x', through the definition,
Definition 4.1. (Optimal Invariant Metric), Given a transition distribution D for tuples in the replay buffer, suppose the block structure assumption holds, the shift between transitions x and its context x' can be measured by a conditional divergence:
d(xx EEED [dKr. (p(xle e))lp(xx =c)]] - where d& (-11-) the Kullback-Leible (KL) divergence. fi indicates the expected distance between x and x' conditioning on e. One may argue that the dynamics cannot be assumed as fixed distribution when it comes to new observations, especially after data manipulation. Nevertheless, generally claims that the experiences are uniformly sampled in a replay memory (Mnih et al. (013). Also, the fact that the given conditions of the transition are consistent for the observed data as well as the transformed data, makes the above definition reasonable. Next, we will show why the conditional divergence defined in Eq.(3) is an optimal invariant metric from theoretical perspectives.
+)]] [ dxr. (p(xle = ()]lp(x'ee= =)) dp(e) (3)
We employ the Bayes' rule the conditionally distribution p(x|e) p(c(x))((x//p(() Vx O, e E DD Then the transition operator p(c|x) can be further divided as p(cis)p(ssx) for any x E O, if e and x are conditional independent given + Eq. 33 is rewritten as, The [dxz (p(xle) |p(x [[x]ee))] = Feja [dRz (p(sixx)((<<]lp(xxxxxxx))>>
(4) Therefore, minimizing the conditional divergence leads to encoding the observation x and the transformed context into an invariant latent state space S. As a consequence, the learnable pixel transformation is optimality invariant combining qualified encoder.
Now we have the observation encoder g O + S mapping from the observed state O to the latent state S by non-trivial function 9 such that g(x) p(s)x), Vx. Traditionally, there is only one encoder dubbed feature backbone in RL models. Since the pixel transformation could drift away, e.g., supported by different components with those supporting (Du et al.] 2019). To enforce the invariant hidden states, another state encoder g' that can map x' to s should exist, which is g'(x') pxs)v(xx. )), Vx', p(x,-) -) is the transformation.
So far, the goal of learning the optimal transformed data and encoders boils down to minimizing the distance between representations gxx) and g' (x'). To tackle the issue, we first provide two definitions introduce measurements as follows,
Definition 4.2. (e- (e-Approximation) Given a distance metric d' : OxS + R4 satisfies d(s, = 0, Vs.. and let 9.9'= O + S be two functions. Let 2 O, given a distribution D on O, then and g' are e-upproximate w.r.t. (d, D) if E. ExxD [d(g(x), (x)) S.
Definition 4.3, (B-Similarity). Given distance metric d : O x S >> R.4 satisfies d(s, s) = 0, vs. There exists O + S. Let B > 0. given distributions D and D' on O, then and x' are 8-similar E BB- D [g(g(x), g(x))) B.
Without loss of generality, the distance between the encoded states gxx) and (x') can be expressed as the following triangular inequality. To obtain metric, Kullback-Leible divergence is rewritten in a form of the square root of Jensen-Shannon divergence. Therefore, we have,
dxg(x). g' (x')) s d(g(x).g 9'xx)
d(g' (ll'(xx)'''' (x')
(5)
encuding: A-pprxxiiaaionn
angmentatiio:: 9 9-Similurtt
'The tuples in reply buffer can he written (s, Te, Se+1) after encoding, which makes an intermediate random variable.
4

---

Published as a conference paper at ICLR 2023
From the view of invariant learning, minimizing the right side of the inequality upper bound our problem. The first term on the right side is the so-called e-approximation to measure the functional similarity after state abstraction, whereas the second term exists based on the procedure of data augmentation. Thus, we learn the encoders and shifted data simultaneously through the upper bound.
4.2 OPTIMAL STATE ABSTRACTION
To restrict the functional similarity of Eq. from the perspective of learning a good encoding function with consistent semantics, the approaches formulating the main and momentum feature learning are utilized, motivated by contrastive learning (He et al. 2020). In particular, we enforce the
encoding functions with exactly the same architecture, and use 5 (I- Tm 2 + =ms at timestep t to update the parameters of momentum function 95 with gE. where
E [0, is the updating
rate. Furthermore, we design projection that is f S + + using ReLU network (Petersen & Voigtlaender 2018) to upper bound the divergence by minimizing the distance in the projected space y. Previously, the projection has been proposed by [Chen al. (2020), while the theoretical guarantees of the underlying mechanism with momentum updating for model-free RL are explained this work.
Suppose the Markov chain O 0 S 4 y holds. For two functions g and f in the compatible ranges, we use fo g to denote the function composition f()()). Before showing the proposed data transformed method, we introduce technical lemmas to take advantage of the designable projection function by leveraging the convexity. The momentum updating paradigm is capable of turning into momentum feature updating through convex function an equivalence of the convex function.
Lemma 4.1. Assume that h I S - TR| can be written as h(g) = f(< č,s >), for some E RRIS|
and f x + RP' with parameter g. Then, convexity of f implies the convexity of h.
Lemma 4.2. Given the dynamical updating: eL (1 Tm ) 1 Tm By Lemma 4.1 Ss -
7m)
holds after convergence. As result, the problem of min Ex (x(x) fg o (x equivalent to the problem of min (x] (x)x))
To meet the requirement of small upper bound, we state a theorem that provides some insights into why it is necessary to learn optimal transformed data together with the encoders.
Theorem 4.1. (ColT) Suppose that Lipschitzness holds for functions gt- It. f& and fę. respectively. (1 Tm ) + Tm Tm E [0, 1]. For any input x D and
The updating dynamics is: transformed x' obtained via the transform operator v(x, -), optimizing the conditional divergence in Definition means to minimize the upper bound as follows,
Ex [d[f 98 (x(x), Se gg (x(x'))] pl [[x x'I]
(6)
where =
Ly (CL, + |k, = , = 1 - Tm are constants. Ly and Ly are Lipschitz constants of the functions (s) and (x),, respectively.
The upper bound of the right side measures the margin of augmentation between original and transformed data. The left side measures the distribution changes in the latent space. Since both transformed data x' and encoder are updating, incorporating augmentation directly cannot well meet the basic stationary environment. Hereby, the theorem suggests that the automatic transformation is used to bound the representation learning so that the abstracted states enhance the stationary distribution of tuples e and facilitate efficient training. Proofs are given in Appendix [A] comparisons of the projection network are presented in Appendix D
The empirical
4.3 PARAMETERIZABL OBSERVATION
From the Theorem 4.1 we know the relation between the parameterized augmentation and the learnable latent state. That is, image transformation needs be constrained by the distance between an observation and its associated augmentation. The optimal embedding can be obtained by minimizing of Eq. 5). Particularly, we parameterize the transformed data x' as v(x, G), where G is a Gaussian distribution dynamically changed along with the RL training.
In Algorithm 1 it defines the MDP M with Gaussian random variables Go ~ glo for initialization. The TRANSFORM operator is fulfilled by the aforementioned pixel transformation v which is shift
5

---

Published as a conference paper at ICLR 2023
Tansoomation
Polryy Model
Enozder
Reward Model
Makus
Replay Burter
Projecton
Encoder
Projection
Invariant Learner
Observatonx
Figure 2: Overall architecture of ColT. The observations are transformed following Gaussian distribution ((s, σ) and encoded by the state encoder Рę. The observation encoder 9 and projection fg are the exponentially moving average version of the state encoder de and projection fe.
subject following Gaussian distribution Ge (Pe, σx) on the top of data interpolation. The transformation x' (x, G) is normalized according to both x and learned distribution G, and thereby contributes to the cumulative reward maximization in interactive way. We use the scope to the observation sampling for an optimal state abstraction. can be regarded as data sampling from the replay buffer reach training-friendyy distribution, dubbed Contrastive Invariant Transformation.
4.4 STABILIZING REWARD FUNCTION
It identifies of the key impacts that ColT in the RL training procedure as parametriz- ing the underlying invariant optimization smooth the distribution 'D in the replay buffer. To further stabilize the reward function, we propose mixed ColT that samples multiple transformed data from the learned distribution ((,,)), and then mix up the learned obser- vation x'. Similarly, we provide the invariant learning guarantee by optimizing the right side transformation in Theorem
Algorithm Parameterized Transformation in RL.
I: Initialization: Draw distribution Go with given mean 14 and variance o Set cumulative reward R = 0. Training:
glol
3: for each timestep / in o.... ... T do
X TRANSFORM(X,, 9r)
5: R = R + , , az)
Adjust to an optimal Gi (Mc, 0t).
Theorem 4.2. (Mixed ColT) Suppose that
I Lipschitzness holds for functions 98. 9g. Je and fe. respectively. The updating dynamics is: EL = (1 Tm -
7: end for
Details about learning f4r and dı are given in Algo- rithm Appendix
1 + Tm, For any input ~ 'D and transformed x', the divergence with mixed transformed observation can be bound by,
Ex [d(Se o o (Ex((x)) S pxxExx [ [[x
x'Il]
(7)
where p Ly (CL, - = functions fs) and g respectively. The proof of Theorem 4.2 is straightforward based Theorem and the details are presented the supplement.
Tm L, and Lg are Lipschitz constants of the
4.5 LEARNING CONTRASTIVE INVARIANT TRANSFORMATION
With theoretical analysis of invariant transformations, presented new framework with normaliza- tion variants to ensure above discussed learning guarantees by optimizing parameters. We initialize a distribution (1, σ)) and use the TRANSF ORM operator produce different views of (Algorithm Dp. The transformed data x/ is viewed as the positive pair of x,. We also utilize a similarity metric to learn contrastive invariant transformation for the encoder (-).).
We first apply the bilinear interpolation to x, and sample shift terms from G, to produce multiple positive samples x! xf,,,, -.,,," and mix them together x following Theorem 4.2 To prevent the

---

Published as a conference paper at ICLR 2023
distribution of transformed data from shifting too far away from the replay buffer D. we borrow a similar idea from Yin et al. (2020) to regularize and smooth the distribution shift between the transformed and overall data. We use the statistical data stored in the BatchNorm layers to approximate the distribution of the overall data. In this way, the distribution shift between the transformed and overall data be estimated by the following formulation,
K. (x,)
Σ ||ā(x)) - ((((())))) + + [1100(x) [()
where ū(x() and (x() are the mean and variance of the transformed data and represents parameter collection (Me of the Gaussian distribution o). The expectation terms B(i (x)]O) and (x)]()))) respectively denote the estimation of the batch-wise mean and variance for the feature map corresponding to the (-th convolution layer, and O is the given observations.
Second, we utilize the similarity metric d proposed by Chen et al. (2020) for learning the encoder ge (+) which maps high-dimensional observation to embeddings to meet the invariant transformation in 5 Given a positive observation pair (x,, x(), the loss is given by (eelo(()),,fll(((x))))
W A (ee( - f((9g(xx)) || = 2 I[fe(e(x()M) ([
(9)
Here denotes the momentum version of parameters & and fe(.) is non-linear projection of the representations embedded by (-). DD indicates the tuples stored in the replay buffer.
Next, we update the critic network Qo with transformed data x/ and x44n minimize the TD error for n-steps returns. This regarded as a regularized Q learning by Yarats et al. (2020) where the regularized representation learning is beneficial for optimal action taking (Zhang al. (020a).
Jo(D:O,w.E) (Qo (gt (xi),
Σ 7're4 7 Qu (x)4))) T-199 (si
(10)
Eventually, we give the unified objective function as the full version of the ColT,
(D) = (D( + + + (D)
(1I)
where a and À are hyper-parameters and the overall architecture is presented in Figure 2 We replace the vanilla Q-learning by Je.g.w and the entire algorithm presented in Algorithm in Appendix Then, we evaluate ColT on popular benchmarks to demonstrate the benefits of our method.
5 EXPERIMENTS
In this section, we benchmark our method on the DeepMind control suite and Atari100K, We compare ColT with prior model-free methods first, then we present ablation studies show the details of our method. Implementation details can be found in Appendix
5.1 ENVIRONMENTS
DMControl. DeepMind control suite (Tassa al. 2018)
widely used benchmark with several robot control tasks. Each episode is set to be 1, 000 frames and we use the total experienced frames to measure the data-efficiency. The per-frame reward in the unit interval [0, 1). so each episode contains a total reward of no more than 1, 000. Considering the different difficulties depending on tasks, we refer to setting more episodes with hard tasks for better evaluation.
Atari100K. There have been a number of prior papers that have benchmarked data-efficiency on the Atari 2600 Games for discrete control. Van Hasselt al. (2019) and Kielak (2019) propose the data-efficient version of Rainbow DQN (Hessel et al. [2018) compared with human performance (Kaiser al. 2019) within 100K time steps (400K frames, frame skip of 4). This sample-constrained evaluation the so-called Atari I (0OK and we benchmark ColT all 26 Atari Games.
5.2 BASELINES
DMControl. For continuous control we present several baselines, including methods of using data augmentation and contrastive learning improve data-efficiency: (i) DrQ-v2 Yaratt al. 2021),
7

---

Published as a conference paper at ICLR 2023
Corppoe Swingup Sparse
Pendulum
Hopper Stand
Quacruped Walk
12000 Beun
Walk
Finger
Peopper
Figure 3: Results of complex tasks in DMControl. These tasks are chosen to offer multiple degrees of challenges, including complex dynamics, sparse rewards, hard exploration, and more.
CURL. (Laskin et al., 2020b), (iii) Pixel SAC, and (iv) Pixel DDPG Vanilla SAC and DDPG training directly from pixels. All methods are evaluated with the same periodicity of frames and average over 10 episodes return for evaluation query.
Atari100K. To benchmark the data-efficiency of ColT for discrete control tasks, we compare our method to (i) DrQ Yarrats al.) (202)), (ii) CURL (Laskin et 2022b). (iii) SPR (Schwarzer et al., 2020), (iv) Random Agent, and Human Performance (Kaiser et al. 2019). All the algorithms are evaluated within 100K time steps for interaction. We average Coll's performance over 10 random seeds and report the best score for each game following prior works.
5.3 MAIN RESULTS
DMControl. We choose 8 complex tasks from the DMControl for evaluation and present the results
in Figure [3 and Table in Appendix [ We also report the percentage (%) of score solved in the
DMControl for baselines and ColT in 500K and 100K steps in Figure [ Below are key findings: (i) wide range. (ii) We also compare ColT with DrQ-v2, remarkable method for continuous control, to better demonstrate our method's 's data-efficiency. (iii)
ColT outperforms vanilla DDPG and SAC
From general trends of the learning curves, ColT improves or keeps the data-efficiency in a more stable manner which is not trivial on DMControl tasks.
Atari100K. We present results for Atari1008 in Table 1 and below are key findings: (i) ColT achieves top-performance on 10 of 26 games while still being competitive in the rest. (ii) ColT surpasses superhuman performance on 6 games the basis of Rainbow DQN (iii) We also report the mean and std of the scores achieved by ColT in 10 of 26 games which are top-performance. We present the results of two versions of ColT (mixed & no-mixed) in Figure in the appendix. From the histogram, we find that ColT has much better stability, which is similar to the observation in the DMControl.
5.4 ABLATION STUDIES
We first visualize the TRANSFORM operator to demonstrate that there exists an invariant transforma- tion for each task. We initialize the Gaussian distribution Gr Ge αr) based on the range of pixel shifts and plot the curves of the mean μ and std during training in the DMControl in Figure
Cortpore Swirgte
Hagge
Sore
Welbhe Meb
"esper Hap
Mick: Fan
Figure 4: Visualization of the parameters of the Gaussian distribution for TRANSFORM.
8

---

Published as a conference paper at ICLR 2023
Table Mean episodic returns achieved by ColT and baselines on 26 Atari games benchmarked 100K environment steps. The results are recorded and averaged over 10 random seeds.
Game
Human
Random
SimPLe
CURL
DrQ
SPR
ColT
Aline
7127.7
227.8
616.9
558.2
771.2
801.5
1206.7 7
Amidar
1719.5
5.8
88.0
142.1
102.8
176.3
182.3
Assault
742.0
222.4
527.2
600.6
452.4
571.0
635.7
Asterix
8503.3
210.0
1128.3
734.5
603.5
977.8
709.0
Bank Heist
753.1
34.2
131.6
168.9
380.9
124.8
Battle Zone
37187.5
2360.0
5184.4
14870.0
12954.0
16651.0
13760.0
Boxing
12.1
0.1
9.1
1.2
6.0
35.8
23.6
Breakout
30.5
1.7
16.4
4.9
16.1
17.1
16.1
Chopper Command
7387.8
811.0
1246.9
1058.5
780.3
974.8
1338.0
Crazy Climber
35829.4
10780.5
62583.6
12146.5
20516.5
12923.6
17538.0
Demon Attack
1971.0
152.1
208.1
817.6
1113.4
545.2
84644
Freeway
29.6
0.0
20.3
26.7
9.8
24.4
29.6
Frostbite
4334.7
65.2
254.7
1181.3
331.1
1821.5
2069.8
Gopher
2412.5
257.6
77L.0
669.3
636.3
715.2
746.8
Hero
30826.4
1027.0
2556.6
6279.3
3736.3
7019.2
7572.8
Jamesbond
302.8
29.0
125.3
471.0
236.0
349.0
336.0
Kangaroo
3035.0
52.0
323.1
872.5
940.6
3276.4
4116.6
Krull
2663.5
1598.0
4539.9
4229.6
4018.1
3688.9
3426.2
Kung Master
22736.3
258.5
17257.2
14307.8
9111.0
13192.7
9250.0
Ms Pacman
6951.6
307.3
1480.0
1465.5
960.5
1313.2
1509.6
Pong
14.6
-20.7
12.8
16.5
8.5
-5.9
1.5
Private Eye
69571.3
24.9
58.3
218.4
-3366
124.0
145.7
Qbert
13455.0
163.9
1288.8
1042.4
854.4
669.1
2117.5
Road Runner
7845.0
11.5
5640.6
5661.0
8895.1
14220.5
11758.5
Seaquest
42054.7
68.4
683.3
384.5
301.2
583.1
554.0
Up N Down
11693.2
533.4
3350.3
2955.2
3180.8
28138.5
4734.2
According to the curves below, we find that the mean and std converge an interval the training goes on. These results demonstrate that the gaussian distribution proposed ColT could automati- cally find TRANSFORM smooth the distribution shift between the different views of the same observation, therefore being beneficial to the representation learning.
Then, study the effects of different components in (I. This object function is composed of two parts: (x() for regularization and Leg (D) for similarity metric. On this basis, we divide ColT into 4 versions: (i) Critic. Transformation is only updated with the critic. (ii) X-stats Critic. Transformation updated by critic and (x()) together. (iii) H-dist & Critic. Transformation is updated by critic and Le, (D) together. (iv) Unified Objective. We evaluate all of these versions on 8 representative tasks from the DMControl and present the results in Figure 9 in the appendix.
Compared Critic with other variants, demonstrate that both of the components the performance. Though Critic is data-efficient on most tasks, it may fall into trivial solutions. To solve this issue, we utilize the regularization the invariant transformation. Thus the Unified Objective's performance leads ahead of all tasks. See Appendix C for extra ablation studies.
beneficial
(8) with the similarity metric in Eq. to meet
CONCLUSION
A novel pixel transformation ColT under model-free RL algorithms that significantly improves the data-efficiency and stability for visual tasks introduced in this work. We theoretically analyze how the learnable transformation constrains the distribution of transformed data, and dissect its benefits to representation learning. ColT no need for any additional modifications to the backbone RL algorithm and is easy to implement. We compare ColT SOTA methods popular benchmarks and certify that it gains promising performance with advanced stability. Hopefully. contrastive invariant transformation lead to new branch for representation learning in RI.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work was supported in part by the National Key Research and Development Program of China No. 2020AAA010344 National Key Research and Development Program of China No. 20212D0201504, National Natural Science Foundation China under Grant 62273347, and CCF- Tencent Open Research Fund RAGR20220104, We thank anonymous reviewers for their discussions and feedback on the paper.
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265. 2021.
David Andre and Stuart J Russell. State abstraction for programmable reinforcement learning agents. In Aaaifiaai, pp. 119-125, 2002.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1671v:160.0064500 2016.
Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv:21 1tt::2110.09//00 09.45, 2021.
Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. arXiv preprint arXiv:2107.05431, 2021.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, 6(5): 679-684, 1957.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0988 09882, 2020.
Pablo Samuel Castro and Doina Precup. Using bisimulation for policy transfer in mdps. In Twenty- Fourth AAAI Conference on Artificial Intelligence, 2010.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv 2017.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of artificial intelligence research, 13:227-303, 2000.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning. pp. 1674. PMLR, 2019.
Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Anima Anandku- mar. Secant: Self-expert cloning for zero-shot generalization of visual policies. arXiv preprint arXiv:2106.09678 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Saan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006. 07733, 2020.
10

---

Published as a conference paper at ICLR 2023
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on PMLR, 2019.
Machine Learning, pp.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data aug- mentation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 13611-13617. IEEE, 2021.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. Advances in neural information processing systems, 28:2017-2025, 2015.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: 1903:0/0370777 2019.
Kacper Piotr Kielak. Do recent advancements in model- based deep reinforcement learning really improve data efficiency? 2019.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, arXiv preprint arXiv:2004. 14990, 2020a.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020b.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953 2019.
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 18901199-,, 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
Bogdan Mazoure, Remi Tachet des Combes, Thang Doan, Philip Bachman, and Devon Hjelm. Deep reinforeement and infomax learning. arXiv preprint arXhv:2006.07217, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller. Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):5295533 5333, 2015.
11

---

Published as a conference paper at ICLR 2023
Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceed- ings 1994, pp. 226-2222222222 232. Elsevier, 1994.
Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296-330, 2018.
Roberta Raileanu, Max Goldstein, Denis Yarats, llya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. /2862,
2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self seff-predictive representations. arXiv preprint arXiv:2007.05929, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp. 387-395. PMLR, 2014.
David Silver, Hado Hasselt, Mattco Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Arnod,, David Reichert, Neil Rabinowitz, Andre Barreto, al. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, pp. 3191-3199. PMLR, 2017.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 10XIv:1801.0000 2018.
Manan Tomar, Amy Zhang, Roberto Calandra, Matthew E Taylor, and Joelle Pineau. Model-invariant state abstractions for model-based reinforcement learning. arXiv preprint arXiv:2102.09855/ 2021.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-
learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
Tongzhou Wang, Simon S Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. De- noised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206. 15477 2022.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1rXiv:1910.0/744. 2019.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.0966 2021.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724, 2020.
Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34:5276-51 5289, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 6023-6032, 2019.
12

---

Published as a conference paper at ICLR 2023
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In International Conference Machine Learning, pp. 11214-11224, PMLR, 2020a.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint
arXiv:2006.10742. 2020b.
Hanping Zhang and Yuhong Guo. Generalization of reinforcement learning with policy-aware adversarial data augmentation. arXiv preprint arXiv:2106. 15587, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 170,0411 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data aug- mentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13001-13008, 2020.
13


Reviews:
[{"review_id": "q02AcZyNxf", "reviewer": "Reviewer_yPYT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple yet effective learning module upon the framework of contrastive reinforcement learning. The module could automatically learn a optimal transformation for data augmentation. The algorithm is theoretically founded and also demonstrated by expeiments on DMControl and Atari 100k.", "summary_of_the_review": "I like the proposed algorithm for its simplicity and the theoretical analysis is also somewhat interesting. The main concern for me is (a) whether the main theorem is constructive for the algorithm and (b) whether the objectives really help generally for different tasks or it is carefully tuned for each single task to make it work. The detailed questions are mentioned before, which I hope the authors could address.\n\n", "strength_and_weaknesses": "Strength:\n\n- The idea is simple but effective, which might be easy to implement and follow.\n\n- The authors make theoretical efforts in justifying the framework.\n- The experiments are extensive, conducted on DMControl and Atari 100k. Also the baselines (DrQ v2 and CURL) are decent.\n\nWeakness:\n\n- The theoretical analysis in this paper uses several inequalities to achieve the final objective, while some reasoning looks not very clear to me. See following comments for more details.\n- In Equation 5,  the authors use the triangular inequality $d(g(x),g'(x'))\\leq d(g(x),g'(x))+d(g'(x),g'(x'))$ so that they could minimize the upper bound. However, the LHS $d(g(x),g'(x'))$ could be optimized directly. So what is the necessarity of using the upper bound? And it is better that the necessarity be illustrated  from both the theoretical perspective and the empirical perspective. I am very willing to take feedbacks from authors.\n- In Equation 6, the authors still try to prove that minimizing the LHS means minimizing the RHS. However, the LHS is the lower bound and thus it seems not meaningful to minimize the lower bound. If I do not misunderstand, the main theorem might not be able to motivate the CoIT algorithm. There is a chance I misundertand and there could be more efforts in detailing the theorem.\n- The writing could be improved. There are many variables not well stated. For example,  in Theorem 4.1, the variables L_g and L_f are not stated in the main paper. \n\n- The authors propose a multi-task objective that combine three loss functions in total. Though it is common that different objectives are used with different weights, the hyperparameters given in appendix seem to show that the weight for objectives must be carefully choosen for each tasks to make it work. More explanations about the hyperparameter selection and the experiment observation about the hyperparameter might be necessary.\n\n\n\nTypos:\n\n- related work: DRQ -> DrQ\n- main results: Table 2 -> Table 1\n- Some theorem links are not correct. One careful check might be great.\n\nnote: this is an emergency review", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h5OvIHgp13P", "reviewer": "Reviewer_ao8s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a learnable data augmentation framework for visual RL, which learn the parameterized image augmentation together with model-free RL. Some theoretical interpretations are provided to show the rationale of learned augmentation. Experiments on DMC and Atari100 benchmarks demonstrate that the proposed method CoIT achieves better performance than SOTA augmentation methods for model-free RL.", "summary_of_the_review": "This paper presents a relatively novel idea and show good empirical success of the method. But there are several clarity issues of the theory and critical claims, making it hard to justify the quality of the paper. I would like to increase my rating if the authors can address my confusion with convincing evidence.", "strength_and_weaknesses": "### Strengths\n\n- The idea of learnable augmentation is relatively novel.\n- The authors provide some theoretical interpretation for why learnable augmentation is desired.\n- The proposed method achieves relatively promising results on several benchmark environments. \n\n### Weaknesses\n\n- I think the proposed algorithm seems simple yet effective. I like the idea of learning augmentations. However, the theoretical analysis sections are a little confusing and not very to the point. First, why is the replay buffer fixed? During training of model-free RL, I would expect the replay buffer to be consistantly updated, with imperfect data coverage. It is weird to me that the paper motivates by \"dynamic distribution\" and \"training-friendly\" but does not discuss much on the distribution shift problem during training. Second, I don't think Section 4.1 and 4.2 have provided sufficient evidence for the \"optimality\" of the proposed metric and the algorithm. Some claims are not very convincing. For example, why does Theorem 4.1 suggest that \"incorporating augmentation directly cannot well meet the basic stationary environment\"? \n- There exist a bunch of grammar errors and typos, which may cause confusion. Some notations are not clearly defined in the context. For example, in Theorem 4.1, what does \"Lemma 4.2 holds for functions $f_{\\xi}$ and $f_{\\bar{\\xi}}$ respectively\" mean? Lemma 4.2 is a lemma, not a condition for $f$. Also, $L_g$ and $L_f$ appear in the theorem without definitions. \n- The ablation study also confuses me. Why does the convergence of $\\mu$ and $\\sigma$ mean that TRANSFORM successfully smooths the distribution shifts? Can the authors provide more explanation?\n\nQuestions:\n- Why is Gaussian distribution used to generate augmentations? Would a deconvolutional network work here?\n- How is \"tasks solved\" defined in Figure 1?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GwxhK4CCB0l", "reviewer": "Reviewer_7fsn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learnable data augmentation procedure for speeding up reinforcement learning from pixels. In particular in applies a transformation to the input images, which is parameterized by a Gaussain distribution. The proposed method combines three main components:\n\n1. Standard Q-learning approach with data augmentation, similar to DrQ\n2. Contrastive learning loss for the encoder\n3. Smoothing objective for the transformation parameters\n\nThe paper proposes a theoretical justification for this approach and shows state-of-the-art results on DM Control and Atari. ", "summary_of_the_review": "Well written, intuitive paper with good justification and results. ", "strength_and_weaknesses": "Strengths:\n\n1. While practical the proposed method is also justified from a theoretical perspective as well.\n2. The approach is intuitive and straightforward.\n3. Produces state-of-the art results on some standard benchmarks. \n\nWeaknesses:\n\nOnly wandering what results are on the Humanoid task, which has now been successfully solved from images. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3WSHPvyfzy8", "reviewer": "Reviewer_RdJD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new framework to improve the sample efficiency of model-free RL algorithms through combining them with a learnable data augmentation approach. The paper provides a theoretical analysis of how their method works to improve the representation learning and reports results on DMControl and Atari100K.", "summary_of_the_review": "This paper proposes a new solution for the problem, and experiments on two benchmarks demonstrates its effectiveness. The major concerns are limited contributions and the lack of comparison with more recent approaches.\n", "strength_and_weaknesses": "Strength:\n1. The idea proposed in this paper is novel.\n2. The proposed idea is presented clearly and the paper is globally well-written.\n3. The theoretical analysis is provided.\n4. The proposed framework is easy to implement.\n\nWeakness:\n1. Although the idea seems to be new, the contribution of this paper seems to be limited. According to Table 1, the improvement resulted from the proposed framwork is quite limited. Second, it is not discussed whether the proposed method can also contribute to improved performance in addition to sample efficiency. Besides, the improvement on stability is not very obvious, more explainations should be provided.\n2. The paper lacks comparison with more recent methods, such as soda, svea and so on.\n3. The paper lacks parameter analysis. For instance, how are hyperparameters α and λ determined?\n\nMinor:\n1. The details of how to adjust to the optimal Gt(µt, σt) can be mentioned in Algorithm 1.\n2. Lf and Lg should also be mentioned in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster