Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?
Authors: Xiang Li
Jinghuan Shang
Srijan Das
Michael S Ryoo
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Self-supervised learning
evolutionary search
representation learning
representation analysis

Abstract: We investigate whether self-supervised learning (SSL) can improve online reinforcement learning (RL) from pixels. We extend the contrastive reinforcement learning framework (e.g., CURL) that jointly optimizes SSL and RL losses and conduct an extensive amount of experiments with various self-supervised losses. Our observations suggest that the existing SSL framework for RL fails to bring meaningful improvement over the baselines only taking advantage of image augmentation when the same amount of data and augmentation is used. We further perform evolutionary searches to find the optimal combination of multiple self-supervised losses for RL, but find that even such a loss combination fails to meaningfully outperform the methods that only utilize carefully designed image augmentations. After evaluating these approaches together in multiple different environments including a real-world robot environment, we confirm that no single self-supervised loss or image augmentation method can dominate all environments and that the current framework for joint optimization of SSL and RL is limited. Finally, we conduct the ablation study on multiple factors and demonstrate the properties of representations learned with different approaches.

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?
Xiang Li Jinghuan Shang' Srijan Das 1.2 Michael Ryoo'
Department of Computer Science
'Stony Brook University, UUniversity of North Carolina at Charlotte (xiangli8, jishang, aryoo)@cc..tonnbrook edu, Psdas240uncc.eds
Abstract
We investigate whether self-supervisee learning (SSL) can improve online rein- forcement learning (RL) from pixels. We extend the contrastive reinforcement learning framework (e.g.. CURL) that jointly optimizes SSL and RL losses and con- duct an extensive amount of experiments with various self-supervised losses. Our observations suggest that the existing SSL framework for RL fails to bring meaning- ful improvement over the baselines only taking advantage image augmentation when the same amount of data and augmentation is used. We further perform evolutionary searches to find the optimal combination of multiple self-supervised losses for RI, but find that even such loss combination fails to meaningfully outperform the methods that only utilize carefully designed image augmentations. After evaluating these approaches together in multiple different environments in- cluding a real-world robot environment, we confirm that no single self-supervised loss image augmentation method can dominate all environments and that the current framework for joint optimization of SSL and RL is limited. Finally, we conduct the ablation study on multiple factors and demonstrate the properties of representations learned with different approaches.
1 Introduction
Learning to act from image observations is crucial in many real-world applications. One popular approach is online reinforcement learning (RL), which requires human demonstration expert trajectories. Since all training samples are collected by the agent during policy learning in online RL, the collected data often has strong correlations and high variance, challenging the policy learning. Meanwhile, the cost of interacting with environments requires the RL algorithms to have higher sample efficiency. Compared RL using state-based features, pixel-based RL continuously takes images as inputs, which usually come with a much higher dimensionality than numerical states. Such properties pose serious challenges image representation learning in RL.
Several recent works studied such challenges from various directions, including: (1) Inspired by
the great success of self-supervised learning (SSL) with images and videos (e.g., 8
10
12
16
17 19 23
34 35
43 55
57
58 64
73)), some RL methods 1
45
49
62
66
90
take advantage of self-supervised learning. This is typically done by applying both self-supervised loss and reinforcement learning loss in one batch. In this paper, we dub such joint optimization of the self-supervised loss and the RL loss as the joint learning framework.
(2) On the other hand,
many papers [3]] 46) 61 63 78 83 85 investigate how online RL can take advantage of image augmentations. Among them. RAD 46 and DrQ
83 35 show significant improvements by applying relatively simple image augmentations to observations of RL agents.
Our objective is to study how well single combination of self-supervised losses and augmentations work under the current joint learning frumework and to empirically identify their impact on RL
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

systems, In this paper, we extend such joint (SSL + RL) learning framework, conduct experiments comparing multiple self-superrised losses with augmentations, and empirically evaluate them in many environments from different benchmarks. We confirm that a single self-supervised loss under such a joint learning framework typically fails to bring meaningful improvements to existing image augmentation-onf, methods. We also computationally search for a better combination of losses and image augmentations for RL with the joint learning framework. The experiments in different environments and tasks show inconsistency in self-supervised learning's 's capability to improve reinforcement learning. Given a sufficient amount image augmentations, under the current framework, self-supervision failed to show benefits augmentation-only methods regardless how many superviede losses are used.
With all our findings, present this work thorough reference for investigating better frameworks and losses for SSI. RI, and inspiring future research. Our contributions can be summarized as follows:
1. We conduct extensive comparison of various self-supervised losses under the existing joint learning framework for pixel- based reinforcement learning in many environments from different benchmarks, including real-world environment.
2. We perform evolutionary searches for the optimal combination of multiple self-supervised losses and the magnitudes of image augmentation, and confirm its limitations.
3. We conduct the ablation study multiple factors and demonstrate the properties of repre- sentations learned by different methods.
2 Preliminaries
2.1
Reinforcement Learning
In this paper, extend the configurations of previous work 45 Critic) [27 (8 and Rainbow DQN (6) for the environments with continuous action space and discrete action space respectively.
84 and exploit SAC (Soft Actor
Soft Actor Critic 27
28 is an off-policy actor-critic algorithm that takes advantage of the maximum entropy to encourage the agent explore more states during the training. It maintains policy network Ty and two critic networks Qo, and Qoz The goal of T is to maximize the expected sum of rewards and a r-discounted entropy simultaneously, where the entropy encourages the agent to explore during learning.
36 is a variant of DQN 54 with bag of improvements such as double Q- Rainbow DQN
learning 32 works 80 and multi-step reward.
prioritized sampling 65 noisy net 21, distributional RL [ dueling net-
2.2 Pairwise Learning
We coin the term "pairwise" learning for the frameworks that learn visual representations based on semantic invariance between dual-stream encoder representations. A general pairwise learning method first generates multiple augmented views by applying series of random image augmentations the input sample, then clusters views with the same semantics in the representation space. Optionally in such frameworks, methods using contrastive losses repel samples with different semantics. In this paper, we focus on four representative pairwise learning methods, MoCo [3] 14 34 BYOL (2)). SimSiam [12 and DINO S We have detailed explanation and comparison of these methods in Appendix A.1.
2.3
Representation Learning for Pixel-based RL
Previous works explore the possibility of learning better visual representation which may finally benefit policy learning. One direction using image augmentation for policy learning 46 51
61 63
78
83 85,, where RAD [46] and DrQ 83 (5) achieve significant performance using simple
image augmentation. Another direction combine SSL with RI. I
90] in
which there are two representative methods, SAC+AE [84] and CURL
45

---

Actor Critc Loss
Other SSL Losses
Painwise Loss
Rainbow Loss
Other SSL Lasses
Palnwise Loss
Actor Critic d Mg 10.
Other SSL Head fs
Pairwise Head Vs W.
Other
V
Orline Encoder /
EMA
Momentum Enooder /s
Torget Eruzder
Crtine Encader
EMK
Mormentum
Enarcer
Imsgae Augi
Image Aug.
Image Aug.
Image Aug.
SSL Both
SSL Both
9g
5g -
With SAC the RL method
(b) With Rainbow as the RL method
Figure 1:
General joint learning framework for SSL + RL. The red solid arrow represents the RL flow; the blue one represents the SSL flow and the black one means a shared flow for both; sg stands for stop gradients.
investigates the impact of different types of image augmentations for both image and state inputs. By applying random translation or random crop to the input image, RAD significantly improves data efficiency solely through image augmentation without any auxiliary losses.
RAD (Reinforcement Learning with Augmented Data) 46
DrQ (Data-regularized Q) 83 further investigates the possibilities utilizing image augmentation. DrQ applies image augmentation twice the input images and averages the value over two augmented images which is assigned as the Q value of the input images. DrQ v2 [85) which is the successor of DrQ, switches DDPG (Deep Deterministic Policy Gradient) (50) the RL method, brings scheduled exploration noise to control the levels of exploration at different learning stages, and introduces faster implementations of the image augmentation and the replay buffer.
S4 takes advantage of RAE (deterministic Regularized AutoEncoder) 22 in improve learning stability. The RAE is jointly trained with SAC by performing both SAC update and RAE update alternatingly in one batch.
SAC+AE
replacement B 3-VAE [77
CURL (Contrastive Unsupervised Represennaatinss for Reinforcement Learning) contrastive learning with an online RI. algorithm by introducing an additional contrastive learning head the end of the image encoder. Similar to the aforementioned SAC+AE, here the contrastive loss and reinforcement learning loss are applied alternatielyy at training.
45 combines
3 Self-supervision for Reinforcement Learning
To effectively evaluate different self-supervised losses, we extend the well-known joint learning framework widely used in previous papers [] 45
49
66 84 by adding general self-supervised
learning head to the RL framework. We keep the same RL method in CURL [45]: we use SAC (28)
in tasks with continuous action space and use Rainbow DQN (36] in tasks with discrete action space.
3.1 General Joint Learning Framework
With SAC Fig. shows a general joint learning framework, using SAC as the RL method. The unmodified SAC contains online encoder f.. target (or momentum) encoder fa. and actor head Ap. Each encoder is also followed by two critic heads. Besides that, we attach an additional self-supervised head 9g after the online encoder. For pairwise learning losses, concatenate momentum SSL head Jt after the target encoder when needed.
For every sampled batch of transitions, we first apply image augmentation to both the current state and the next state s' and update the SAC model (Gy. Uq:Qu=11 Qi=L22 A,) using the augmented images. Note that for stability concerns, do not update the parameters of the image encoder when updating

---

the actor head Ag. Then, the target networks are updated by Exponential Moving Average (EMA). This is followed by also performing an EMA update of the SSL head if required. Finally, the online encoder f and the self-supervised head Bg are updated by the self-supervised loss. By alternatingyy performing RI. and SSL. in every batch, we jointly train all the components the framework. The pseudo-code of SAC update alternating RL and SSL is provided in Algorithm
Algorithm 1 Update SAC with Self-supervised I Losses
Green: additional operations for SSL; Orange: only for BYOL and DINO.
procedure UPDATES AAFFFSSCCWI ACWITHSS current state, a': next state, a: action, r: reward, done signal, step: model update step counter, fa: online encoder, fs: target/momentum encoder, Ap: actor head, Q online critic head, Q": target critic head, T target/mommetuum network update rate, 9q: online SSL head, ga: momentum SSL head)
SuySa S. < IMAGEAUGMEETTTIONN s), IMAGEAUGMENTT Sp IMAGEAUGMENTAAIONN 9), INAGEAUGMENTATION 6,, 0, r. fk. Qn==11 < T.g Q. 1:) + (1 - (Jx,, 1,2)
(s')
fq: Q:-1.2 Ap + UPDATTSSOFTACTORCCTTIC
EMA update of SAC b EMA update of the momentum SSL head
9k < -9o + (1-7)9m
fq: +- UPDATESSL (8a: & Sp: 8/s ar
end procedure
With Rainbow DQN Fig. 1b demonstrates how to jointly apply SSL. to Rainbow DQN. The unmodified Rainbow DQN maintains an online encoder fg and target encoder f', followed by two state value heads Qu and Q'. We introduce heads 9n and 9l as suggested in CURL. For each batch, the self-supervised losses are computed using augmented images, while the RL loss is computed using the original data. Finally, the online encoder fq and the self supervveee head Oy are updated by the self- self-supervised loss. The pseudd-code of Rainbow DQN update can be found at Algorithm 2
additional momentum encoder fx and self-supervised
Algorithm 2 Update Rainbow with Self-supervised Losses
Green: additional operations for SSL; Orange: only for BYOL and DINO.
current state, s': next state, a: action, reward, d: done, step: model update step counter, fg: online encoder, f': target encoder, Qgi online value head, Q': target value head, fu: momentum networks, momentum network update rate, 9ot online SSL head, gxt momentum SSL head, WSSL: weights of self- supervised losses)
procedure UPDATER BOW
8a, 8 < IMAGEAUGGENTTTIONN (s), IMAGEAUGMENTATIONN 8p: by <- IMAGEAUGGENTATUU
6), IMAGEAUGMENTATION su, Sp, 8/p r) LRainhow +- CALCULAVERAINNONWOSS(/
LSS1. < CAL.CULATESS
aar,
L C LRainhow + WSssI.Css.
Ja.Qu:99 +- ONLLENEEWORKKUUPPTTEEE
8,Q' fq:Q0
Copy parameters from online networks to target networks EMA update of momentum networks and SSL head
fu.9k 9h -T(fe:9)) + (1 9N)
end procedure
3.2 Losses for Self-supervised Learning
The self-supervised losses we investigated can be categorized into four classes: pairwise learning, transformation awareness, reconstruction, and reinforcement learning context prediction.
Pairwise Learning We investigate three representative pairwise learning methods: BYOL 24
DINO 8 and SimSiam 12), along with existing CURL whose framework is similar to MoCo 34] BYOL, DINO, and SimSiam only explicitly pull positive samples closer without the need for large number of negative samples. CURL uses a contrastive loss taking both positive and negative samples into consideration.
Given the general joint learning framework described in Sec. 3.1 by substituting the self-supervised head and loss, casily formulate different agents w.r.t. self-supervised losses. For BYOL, as

---

shown in Fig. 10a, projector and a predictor are appended to the online encoder sequentially while momentum projector is attached on top of the targetmomeentu encoder. DINO (Fig. 10c) maintains only projector in both online and target branches. Similar to BYOI,, the momentum projector in DINO also updated by EMA. The two encoders in BYOL and DINO operate on two augmented views of the data respectively whereas SimSiam (see Fig. 10b), uses only the online network and a projector for processing both the augmented views.
We also test two methods that introduce RL-specific variables to this pairwise learning framework, CURL-W-Action and CURL-w- --Critic CURL-W-Action is based on CURL while the contrastive loss is applied to the concatenation of image representation and output of the actor network, instead of the image representation only. Similarly, CURL-w- Critic concatenates the critic network output with the existing image representation for contrastive loss.
Transformation Awareness Recent works (e.g.. 15
55 have shown that the awareness of transformations (like rotation, Jigsaw puzzle, and temporal ordering) improves many downstream tasks in computer vision like image classification and action recognition. Typically such awareness can be acquired by explicitly asking classifier to identify the applied transformation from the pixel representation. Therefore, we investigate two simple classification losses, rotation classification (RolationnI.S)) and shuffle classification (ShuffleCL.S), and set a two-layer MIP classifier as the self-supervised head in the joint learning framework.
RotationCLS represents the methods that encourage spatial transformation awareness. Inspired by
RotNet 23 and E-SSL [15., we rotate the input image after augmentation by 0% 90", and 270". The classifier predicts the rotation angle from the visual representation and it is trained by cross-entropy loss.
Shuffle Tuple (53] encourages the encoder to develop an awareness of action causality by predicting if two frames appear in order. We adapt Shuffle Tuple by randomly shuffling the current state image and next state image in a state transition tuple and predicting whether it is shuffled or not. The classifier also takes action into consideration because some of the transitions are reversible. The overall architecture of SLuffleLSS is shown in Fig. 2
Reconstruction Reconstructing the input image with hourglass architecture has been shown to be an effective way to learn image representation [22 37 43 We simply extend SAC+AE by
changing the input and reconstruction target to be augmented images. The reconstruction loss and regularization from RAE [22] are left untouched.
Recent study on Masked AutoEncoder (35) (MAE) adapts the reconstruction task for patch-based Vision Transformers [20). The objective in MAE includes reconstructing the entire image from input masked image patches. Inspired by this, we adapt SAC+AE into SAC+MAE by replacing the augmented input image with its masked version and only penalizing the reconstruction error for the masked patches.
RL Context Prediction Besides the self-supervised learning methods that are specifically designed for pixels, we investigate the losses using attributes naturally collected during the RL process. For any state transition that is not the end of a trajectory, it contains four components: current state s, next state s', action and reward r, with the trajectory termination signal omitted. Inspired by Shelhamer et al. [70), we concatenate the visual representation of the current state s and another representation the input. Without loss of generality, the second input representation h can be any of these three representations of s', a, and r. Then, we predict the remaining components using two-layer MLP. For continuous outputs, mean-squared error (MSE) loss is applied, while for the discrete target (e.g., action in discrete action space), we use cross-entropy loss. The architecture of this group of self-supervised losses is shown in Fig. 3 From the combination of inputs and outputs, we define nine losses whose I/O specifications provided in Table For those losses whose outputs include two components, two target prediction networks share the same SSL head except the last task- ssecifpc layer.
3.3 Evolving Multiple Self-supervised Losses
Besides a single self-supervised loss or handcrafted combination of two losses, we further investi- gate how multiple self-supervised losses affect the policy learning together with the joint learning
5

---

Cross-Ente Entropy
MSE
MLP
on exact algoriths
Celine Crazdler
Encoder
FC Layat
Encocer
Ebooder
FG Layer
FC Layer
Image Aug,
Image Aug
Image Aug
Image Aug
concomly
State
Next state
Action
Reward
Figure 2: ShuffleCLS
Figure 3: General RL context prediction
Table 1: 1/O of RL context prediction losses
Extract-A
Extract-R
Guess-A
Guess-F
Predict-F
Predict-R
Extract-AR
Guess-AF
Predict-FR
Rep. of s' Action
Input
Inpur
Output
Output Input
Input
Output
Output
Output
Output
Input
Oulput
Ouiput
Input
Reward r
Output
Input
Input
Output
Output
Input
Output
framework. In such a configuration, the agent maintains multiple SSL heads the same time and we apply losses to their corresponding head individually. We formulate the combination of multiple losses weighted sum LCmbbo Ei w; La where is the weight of a specific loss Li and Ni is the total number of losses in the search space. In the joint learning framework, we apply both self-supervised LCombo and RL losses jointly to the networks for every mini-batch. Considering that the policy learning is quite sensitive to hyper-parametent it is non-trivial to find each weight for every SSL loss.
ELo (Evolving Losses) 660 shows promising results in unsupervised video representation learn- ing [58 731, by using evolutionary search to automatically find optimal combination of many self-supervised losses. In the spirit ELo, we turn to evolutionary search to automatically find the optimal solution. Assume an unknown objective function whose inputs are weights of multiple losses u; and the magnitudes of image augmentation mý=1,2 for the online encoder and momentum encoder. The function output is the score achieved by the trained agent in its environment with certain random seed: Reed (mj= j=12, N. Essentially, the objective function maps a set of w and my to the reward achieved by corresponding agent, and w; and my stay unchanged during the agent learning process. The optimization algorithm approaches the maximum value of the objective function by repeatedly updating Wi and m, and testing the value of the objective function, which in our case is the training and evaluation of an agent with the given parameters (i.e., the input of the objective function). We choose an off-the-sher optimization algorithm PSO (Particle Swarm Optimization) [11) for its simplicity. For each set of inputs, we find it critical to run with multiple random seeds and report IQM (interquartile mean for stable and robust search. The optimization process is presented as:
argmaa
IQM IQM sead=
( (m1=112; Wi Wi=l....N.)
(I)
my 1.2
Note that we are also implicitly searching for the balance between the self-supervised loss and the RL loss by performing this search, it has the capability to adjust the absolute weights of the self-supervised losses overall. We search on DMControl (72) with SAC using three different configurations named ELo-SAC, ELo-SACv2 and ELo-SACv3 respectively. ELo-Rainbow performs a search on Atari with Efficient Rainbow, Please refer to Sec. A.2.4 for our detailed configurations and search results.
MMaa using only the data between the first and third quartiles
81

---

4 Experiments
We conduct experiments three major directions, in order to better understand how we should integrate SSL with RI. First, we demonstrate how different self-supervised losses affect the RL process, by trying them on multiple challenging tasks. Then, we dive into detailed ablations on multiple factors, and finally, we perform empirical analysis the visual representations learned with the joint learning framework (Sec. A.6). In addition, we benchmark a pretraining framework as an alternative to the joint learning framework (Sec. A.7).
Evaluation Scheme Thorough evaluation of reinforcement learning algorithms is challenging due to the high variances between each run and the extensive requirement of computation. Consequently, we run all experiments with multiple different random seeds and report the interquartile mean and the standard deviation of the scores suggested by Agarwal et al. [2]. For quantitative comparison of the different methods mentioned in Section 3.2 in addition to the absolute scores, we assign a Relative Score to each method. We denote the interquartile mean of scores achieved by agent A in environment E E as IQM and denote the collection of all interquartile mean scores achieved in environment e by different agents as IQM". The Relative Score of agent A is computed as S:A Relative
EeEF [I mean (raan(I (IQM".
DMControl Experiments DMControl (DeepMind Control suite) [72] contains many challenging visual continuous control tasks, which are widely utilized by recent papers. We evaluate all the methods introduced in Sec. [3] along with two important baselines, SAC-NoAug and SAC-Aug( 100), in six environments of DMControl that are commonly used in previous papers 45
X3
Other methods that only take advantage of image augmentation, like RAD
46 and DrQ 83
are also benchmarked for comparison. the case of SAC+AE (84) we provide the augmented images for fair comparison, which is different configuration to the original paper. Please refer to
Appendix A.2.5 for a detailed comparison of method variants and the exact data augmentation they applied.
We mainly follow the hyper- paremmearetters and the test environments reported in CURL, except that we use the same learning rate 10 in all environments for simplicity. All the methods are benchmarked 100k environment steps, with training batch size 512 under 10 random seeds, and they share the same capacity of policy network. The relative score of each tested algorithm on DMControl is We also strongly encourage readers to check full results Table 11 and results in two additional harder environments at Table 12 for a full picture.
reported as Fig. 4
From the first glance Fig. no tested SSL-based method within the joint learning framework achieves better performance than DrQ and RAD which are carefully designed to take the best advantage of specific image augmentations. Compared to the baseline SAC-Aug( (100), approaches with self-supervised loss frequently II out of 19) fail to improve reinforcement learning. Some SSL methods (like SimSiam, ShufflcCLS) ruin the policy learning resulting performance even worse than SAC-NoAug, which suggests that improper use of self-supervised loss can damage the benefits brought by image augmentation. Then, regarding combining losses, Guess-AF and Predict-FR, which are manually designed to combine two individual losses, are not better than the single self-supervised loss in their combinations (check Guess-Acte Accsss-Actionn and Predict-Reward in Fig.
ELo-SAC and ELo-SACv2 find the desired combination by searching in one task. Such combination generalizes to other environments on DMControl with better overall performance than any approach in the search space. In the 'cheetah run' where the search was performed, they obtained the best result among the approaches with SSL (Table 12). This demonstrates the feasibility of ELo-SAC and implies that the obtained combination through cvolutionary search has the potential to generalize to other environments in DMControl. However, weaker performance in 'finger, spin' and 'reacher, easy' made ELo-SAC relatively worse than DrQ (which does not use any supervviervis on average. Interestingly, there similar performance pattern between ELo-SAC and ELo-SACv2 though they have different search spaces. By contrast, ELo-SACv3 finds an overall better combination by searching in six environments simultaneously. Though achieves highest score in 'walker, walk' and 'reacher, easy', it performs worse in 'cartpole, swingup' and 'cheetah, run' than ELo-SAC and ELo-SACv2. Such observations could be clue to the properties of different tasks and sufe-supe methods.
7

---

5.0
Relative Score
0.0
2.5
-7.
-0.0
-125
SAC.HoRug
SAC SAC-Augi88) Augigs)
SAC Augi100)
RAD
Drg
Drq/8a)
Dra(100)
CURL CURL-w Actinn
CURL
CURL - Critc
mOL
DINO
SimSlam
Rotation
ShuffeeCL
+AC+
Extract Extract-Action Action
MAE
Extract Petract-Reward Reward
Guess Guess-Action Actian
Guess Guess--utuus Futura
Predict Rect-Futtrr Future
Predict Reward
Extract-Ae
Guess Ap
Predict
Elo SAC
ELo BLo-SAC22
BLO-SACV3
Figure 4: Relative Scores on six DMControl tasks, environment step=100k, batch size=512, Number of sceds=10. SAC-NoAug uses image augmentation, while all the other methods benefit from image augmentation; The methods in blue (like DrQ) only take advantage image augmentation without any SSL,; the methods in black (like CURL) apply one self-super loss; the methods in orange (like Extract-AR) manually combines two self-supervised losses; ELo-SAC, ELo-SACv2 and ELo-SACv3 combine multiple self-supervised losses with specific weights from evolutionary search. From this figure, No existing SSL-bbased method with the joint learning framework achieves better performance than DrQ which only use well-designed image augmentation. ELo-SAC methods achieve higher Relative Scores than all the self-supervised methods, but it still performs worse than DrQ and RAD, with an exception of ELo-SACv3 which marginally better than RAD.
SAC-NoAug
107
SAC NoA.g
SAC-Aug
Rash-et
SAC-AUg
CURL
RAD
.00
BYOL
RetationCLs
BYOL
Score
233
Extract Action
SAC-AL
RotationCLS
Peedit-FFR
SAC+AE
ExtractAcction
Pradict-FR
Scoral
Image Size before Random Crop
Figure 5: Ablation encoder backbone
Figure 6: Ablation on random crop augmentation
Ablations Our observations with SAC-Aug(88), SAC-Aug(100), and RAD suggest the importance of augmentation hyper-parmeters, given the only difference between these three methods is the augmentation applied. We conduct an ablation study the image augmentation random crop [45] in cheetah run, DMControl. All the sapeer-rrametrrs are as noted in Table 2 except that the environment step is set to 400k and the batch size is reduced to 1 128. Fig. shows how the magnitudes of random crop and translate contribute to the score that the agent achieved. The image size before the random crop is linear to the magnitude of the random crop when using fixed crop size: the larger the image size, the stronger the augmentation. There a trend that the score first increases and then decreases as the image augmentation gets stronger. In summary, it critical to engineering image augmentation carefully when designing an RL system with or without SSL.
Then investigate different visual encoder backbone ResNet 33 by replacing the last two convolutional layers with a residual block that has the same number of layers and channels as the CNN baseline. The ResNet backbone slightly improves all these methods (see Fig. 55) We also encourage the readers to check more ablations regarding image augmentation (e.g. random translate), learning rate, encoder layers, and activation function in Appendix A.3.
Atari Game Experiments Atari 2600 Games are also challenging benchmarks but with discrete action space [4) We choose seven games in this benchmark for selected methods. All the methods use Efficient Rainbow 75 as the RI. method, which is a Rainbow (36) variant with modifications for better data efficiency. Note that Efficient Rainbow, as a baseline, does not take advantage of image augmentation. Therefore, we also benchmark Rainbow-Aug which is essentially Efficient Rainbow

---

Atari
Relative Score
Eff. R-RRinboow
Rainbow-Aug
DrQ*
CURL
BYOL
RotationCIS
Rainbow AAE
Extract-Action
Extract Extract-Reward Reward
Predict-Futurg Future
Predict- Reward
Predict- Wredict-FR-Balancee Balanced
Predict-FR
ELo-Rainbow
Figure 7: Relative Scores seven Atari games, environment step=400k, batch size=32. Number of seeds=20. The color of method reflects its category same as Fig. 4 The overall results show that image augmentation for RI does not benefit policy learning from DMControl. Most of the self-supervised losses fail to bring improvements even given more computation and extra model capacity from the SSL head. Only Rainbow+AB outperforms Efficient Rainbow, which is inconsistent with SAC+AE. ELo-Rainbow achieves worse results even than some of the SSL-based methods in the search space like BYOL and Rainbow+AE, The high variance and the image domain gap between different games make it extremely challenging for GLo-Rainbow to find the combined loss that generalizes to all environments.
Atari which is quite different
taking the augmented images for policy learning instead. We the same image augmentation and hyper-parametter reported by CURL for all applicable methods. For a fair comparison, the augmentation for DrQ is also adopted from CURL, which is different from what the original DrQ paper suggested. We denote our setting as DrQ" to distinguish it from the original DrQ. Similarly, Rainbow+AE takes augmented images. For each game,
20 random seeds and benchmark the agent at 400K environment steps (100K model steps with frame skip of 4). We report interquartile mean, standard deviation, and Relative Scores same as DMControl (See Table 13).
Figure 7 shows a summary of the seven different tasks Relative Score. Firstly, compared to vanilla baseline Efficient Rainbow which does not have any image augmentation self-supe learning, Rainbow-Aug performs worse overall with additional image augmentation for RL. This suggests that the image augmentation used for self- superviee learning in CURL does not easily transfer. Similarly, DrQ achieves compromised performance than Efficient Rainbow, showing that using image augmentation for Rainbow Atari does not benefit policy learning unlike SAC on DMControl. Based on the inconsistent impacts of image augmentation, further investigation is required when applying image augmentation to RL Atari.
As for the self-supervised losses, BYOL,, Rainbow+AE, Extract-Reward and Predict-Rewadd gain better performance than CURI.. However, only Rainbow+AF shows significant improvement on Efficient Rainbow and outperforms the other tested methods, which interestingly is inconsistent with SAC+AE DMControl. Predict-FR Fecdiit-FR-Balanced,, which shows considerable improvements on DMControl by manually combining two self-supervised losses, even fails to surpass Predict- Reward on Atari. ELo-Rainbow, which searches in Frostbite, improves the baseline only in demon attack and frostbite. The high variance on this benchmark made the evolutionary search extremely difficult. Further, there huge image domain gaps between games, which makes it even harder for ELo-Rainbow to work across multiple games on Atari.
Real Robot Experiments We further conduct experiments in a real-world robot environment, uArm reacher. Similar to Burgert et al. [6], the goal to move the actuator close to target object as fast as possible. Our autonomous training environment and results are shown in Figs. g and [ (Please check Appendix A.5 for environment setup details). We benchmark all methods with ten different random seeds, using the same hyper-parameters as DMControl experiments unless reported Table Results are shown as Fig. where ELo-SAC uses the optimal combination found in cheetah run shown Table 6.

---

Cam
SAC-hisAug SAC AugTLoD)
DrQ
Cam
DrO:100) CURL
ELo-SAC
25 75
100
125
155
175
uArm
Scora
Target
Figure 9: Scores on real robot uArm, reacher, environ- ment ttep=100k,, batch size=512. The agent fails to learn effective policy without image augmentation.
Figure 8: Real robot environment setup
Surprisingly, in this real-world environment, the agent fails to learn an effective policy without any image augmentation. The image augmentation alone (i.e., SAC-Aug(100)) was sufficient to outperform other methods including CURL and ELo-SAC using self-supprvision. SAC-Aug(100) (100) performs better than DrQ, which is quite different from our previous observations DMControl. From all three methods only relying on image augmentation (blue in Fig. 9). we conclude that it requires careful design of image augmentation that helps in specific task/environment.
5 Related Works
Self-supervised learning can fit in robot policy learning in multiple fashions and at different stages.
Some works (26] 67-69 76 82
88 use SSL for representation learning in a pre-training stage
before policy learning. Others 25
38
49 52 57 66
84 86
87 89 90 jointly optimize
the self-supervised loss with policy learning. Specifically, Transporter
and VAI [79] train an
unsupervised keypoint detector to discover critical objects in the image for control. RRL [68 and
VRL3 76 also benefit from pre-training deeper visual encoder on large datasets like ImageNet 45 take advantage of contrastive learning. After the agent is deployed, SSL can TCN 7 and CURL
be used to continuously improve the policy (30). Shelhamer al. (70 study several self-supervised losses within both the pretraining framework and the joint learning framework, while their selection of losses, the number of runs, and test environments are limited from a current point view. Chen al. III focus on imitation learning and test multiple SSL objectives for representation learning in various environments. They confirmed the critical role of image augmentation in imitation learning and showed inconsistencies in performance across environments. Our investigation supports some of their observations, beyond that, our evolving loss, real robot environment, and representation analysis provide unique perspectives for online reinforcement learning.
6 Discussion
From DMControl and the real robot experiments, we empirically show that compared to the image augmentation, the role of existing self-supervised losses with the joint learning framework usually limited, even with the help of evolutionary search. While results on Atari show a different trend from DMControl, augmentation that generalizes across environments. At the same time, it is usually challenging to conclude consistent trend that one method meaningfully better than others across multiple tasks. One should cautiously decide the design choice of image augmentation or self-supervised loss for specific RL task. We are excited to see future works that introduce more sufe-viiees losses designed specifically for RL, as well as novel training frameworks that can benefit policy learning.
again we confirm that there is golden self-supervised loss image
Acknowledgments
We thank Kumara Kahatapitiya, Ryan Burgert and other lab members of Robotics Lab for valuable discussion. We thank Hanyi Yu and Rui Miao for their helpful feedback. This work is supported by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Ministry of Science and ICT (No.201 8-0-00205, Development of Core Technology of Robot Tassk-nntlliigence for Improvement of Labor Condition. This work also supported by the National Science Foundation UI-2110440 and CNS-2104416).
10

---

References
[1] Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances Neural Information Processing Systems (NeurIPS), 34, 2021.
[3] Iretiayo Akinola, Anelia Angelova, Yao Lu, Yevgen Chebotar, Dmitry Kalashnikov, Jacob Varley, Julian Ibarz, and Michael S Ryoo. Visionary: Vision architecture discovery for robot learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10779-10785. IEEE, 2021.
[4] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 2013.
[5] Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 449 458. PMLR, 2017.
[6] Ryan D Burgert, Jinghuan Shang, Xiang Li, and Michael S Ryoo. Triton: Neural neural textures for better sim2real. In 6th Annual Conference on Robot Learning (CoRL), 2022.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NewrIPS), volume 33, pages 9912-9924, 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in superviee vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9650-9660, 2021.
[9] Lili Chen, Kimin Lee, Aravind Srinivas, and Pieter Abbeel. Improving computational efficiency in visual reinforcement learning via stored embeddings. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML), pages 1597-1607. PMLR, 2020.
[11] Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang- Lee, Neel Alex, Steven H Wang, Ping Luo, Stuart Russell, et al. An empirical investigation of representation learning for imitation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758, 2021.
[13] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.044 04297, 2020.
[14] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised visual transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[15] Rumen Dangovski, Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Che- ung. Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equivariance in representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
11

---

[16] Srijan Das and Michael S Ryoo. Ste-mix: Space, time, channel mixing for self-supervised video representation. arXiv preprint arXiv:2112.03906, 2021.
[17] Srijan Das and Michael S Ryoo. Viewclr: Learning self- seff-sueeviiee video representation for unseen viewpoints. arXiv preprint arXiv:2112.03905, 2021.
[18] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248 255. leee, 2009.
[19] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the International Conference Computer Vision (ICCV), pages 1422-1430, 2015.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[21] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[22] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[23] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[24] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self self-super learning. Advances in Neural Information Processing Systems (NeurIPS), 33:21271 1777712121 21284, 2020.
[25] Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predicte representations for multitask reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 3875-3886. PMLR, 2020.
[26]
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10122, 2018.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), pages 1861-1870. PMLR, 2018.
(28) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.59003 2018
[29] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning (ICML), pages 3555-2565 PMLR, 2019.
[30] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Aleny,, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[31] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
12

---

[32] Hado Hasselt. Double q-learning. Advances in Neural Information Processing Systems (NeurIPS). 23, 2010.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[34] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 9729-97329 2020.
[35] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, 2022.
[36] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018.
[37] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[38] Maximilian Igl. Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. In Proceedings of the International Conference on Machine Learning (ICML), pages 2117-21.6. PMLR, 2018.
[39] Longlong Jing, Xiaodong Yang. Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. arXiv preprint 1rXiv:1811.11338 2018.
[40] Kumara Kahatapitiya, Zhou Ren, Haoxiang Li, Zhenyu Wu, and Michael S Ryoo. Self- supervised pretraining with classification labels for temporal activity detection. arXiv preprint arXiv:2111.13672 2021.
[41] James Kennedy and Russell Eberhart. Particle swarm optimization. Proceedings of ICNN'95. international conference neural networks, volume 4, pages 1942-1948. IEEE, 1995.
[42] Dahun Kim, Donghyeon Cho, and In So Kweon. Self self-super video representation learning with space-time cubic puzzles. In Proceedings of the AAAI Conference on Arlificial Intelligence (AAAI), volume 33, pages 8545-8552, 2019.
[43] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
[44] Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[45] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. Proceedings of the International Conference on Machine Learning (ICML), pages 5639-5650. PMLR, 2020.
[46] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems (NeurIPS), 53:19884-19893 2020.
[47] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critie: Deep reinforcement learning with a latent variable model. Advances Neural Information Processing Systems (NeurIPS), 33:741-752, 2020.
13

---

[48] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation awwaee self-supervision. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[49] Kuang-Huee Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems (NeurIPS), 33:1 1890-11901, 2020.
[50] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[51] Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas, and Paul Weng. Invariant transform experience replay: Data augmentation for deep reinforcement learning. IEEE Robotics and Automation Letters, 5(4):6615-66)) 6622, 2020.
[52) Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems (NeurIPS), 13:3686-3699 2020.
[53] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 527-544. Springer, 2016.
[54] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 527555)):559-533 2015.
[55] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision (ECCV), pages 69-84. Springer, 2016.
[56] Edwin Olson. Apriltag: A robust and flexible visual fiducial system. In 201/ IEEE international conference on robotics and automation (ICRA), pages 3400-3407. IEEE, 2011.
[57] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 03744, 2018
[58] Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11205- 1 11214, 2021.
[59] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurpris- ing effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.0m 03580,, 2022.
[60] AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving losses for unlabeled video representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[61] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems (NeurIPS), 33:3976- 3990, 2020.
[62] Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforce- ment learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 8787-8798. PMLR, 2021.
[63] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
14

---

[64] Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Michael S Ryoo. Self-supervised video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[65] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[66] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[67] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pages 1134-1141. IEEE, 2018.
[68] Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[69] Jinghuan Shang and Michael S Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 214-221. IEEE, 2021.
[70] Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[71] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML). pages 9870-9879. PMLR, 2021.
[72] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 180X. 801000990, 2018
[73] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint 2rXiv:2203.12602 2022.
[74] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 30, 2016.
[75] Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to parametric models in reinforcement learning? Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[76] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vrl3: A data-driven framework for visual deep reinforcement learning. arXiv preprint 223.2 2022.
[77] Han Wang, Erfan Miahi, Martha White, Marlos C Machado, Zaheer Abbas, Raksha Ku- maraswamy, Vincent Liu, and Adam White. Investigating the properties of neural network representations in reinforcement learning. arXiv preprint arXiv:2203.1595: 2022.
[78] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Advances in Neural Information Processing Systems (NeurIPS), 33:7968-7978. 2020.
[79] Xudong Wang. Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6677-6687, 2021.
[80] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Duel- ing network architectures for deep reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), pages 1995-2003. PMLR, 2016.
15

---

[81] Wikipedia contributors. Interquartile mean, 2022. URL httth:t/ttee...... www.ee/wwwwiiiiiiaaomm waw.ipedda..rg/villi. wiki/ Interquartile. [Online; accessed 13-May-2022)
org/wiki/
[82] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control, arXiv preprint arXiv:2203.06177 2022.
[83] Denis Yarats, llya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[84] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 10674-10681, 2021.
[85] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[86] Pei Yingjun and Hou Xinwen. Learning representations in reinforcement learning: An informa- tion bottleneck approach. arXiv preprint arXiv: 2019.
[87] Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvir- tual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems (NeurlPS), 34, 2021.
[88] Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
[89] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[90] Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. arXiv preprint arXiv:2010.07470, 2020.
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. A.8
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Please check supplemental material
(b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)? [Yes] See Sec. A.2
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Sec. A.4
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sec. A.9
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
17


Reviews:
[{"review_id": "_HRjKD_tEzy", "reviewer": "Reviewer_GjgB", "summary": "The paper performs an empirical study of self-supervised learning for RL from pixels. Specifically, the study considers different SSL objectives (e.g., MoCo, BYOL, etc.), evaluates them using the RL + auxiliary SSL objective framework (like in SAC+AE and CURL), and compares them to RL with data augmentations alone (e.g., RAD and DrQ). The results suggest that auxiliary SSL objectives do not lead to clear gains in tested settings. The paper also performs experiments on using evolutionary search for combining different SSL losses and using SSL pre-training rather than auxiliary SSL objectives.", "questions": "Questions/suggestions:\n- It would be to expand the discussion of the impact of data augmentations (please see my comment in the previous text box)\n- Similarly, it would be good to include more experiments / discussion on the impact of different hyper parameter choices (we know from prior work, e.g. DrQ-v2, that this can have a very large effect)\n- It would be good to consider the impact of the neural network architecture on performance (e.g., different vision encoders, encoders of different size, etc.)\n- It would also be good to study what is the best way to pass / process vision embeddings to policy / critic heads (e.g., like the impact of the tanh discussed on L238)\n- It would be good to consider additional RL algorithms (e.g., DDPG, PPO, etc.)\n- The paper considers DrQ but not DrQ-v2. Is there are reason for not including DrQ-v2 in the study? (SAC vs DDPG?)\n- It seems that action information is not considered by the pairwise approaches but is used in the reconstruction and shuffle approaches. It would be good to try to study the impact of action information.\n\nMinor:\n- The term \"SAC backbone\" may be a bit confusing. People typically use backbone to refer to a neural network architecture used for feature extraction (e.g., common in vision). Maybe something like (learning) \"architecture\", \"framework\", or \"meta architecture\" would be clearer in this context.", "limitations": "Some discussion of limitations in the appendix (A.9). I think it would be nice to see a more detailed limitation sections perhaps focusing on potential confounding factors and/or methodology aspects that might influence the conclusions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper considers and important question and would be of interest to the community\n- The study perform extensive experiments across different settings\n- The paper report robust performance estimates \n\nWeaknesses:\n- Overall, it feels that the paper is trying to do too much by studying three related but different enough aspects and not developing any of them in sufficient depth: (1) RL with auxiliary SSL losses, (2) evolutionary search for combining SSL losses, and (3) SSL pre-training for RL. Most of the paper focuses on (1) and that component is the most thorough of the three (though still leaves a number of questions unanswered; please see the questions section). Sections (2) and (3), and (3) in particular, feel like they have not been explored enough in the current paper and should perhaps be studies of their own (e.g., most of the questions I asked in the context of (1) would equally apply to (3) + additional questions specific to (3), e.g., the choice of pre-training data).\n- I will focus on the weaknesses of (1). If my understanding is correct, the main findings of the experiments is that the RL + SSL techniques do not seem to outperform RL + data augmentations in tested settings. This general finding in itself is not new and was reported by DrQ and RAD. Thus, I think that it would be nice to provide a bit more insight on why that might be and/or what specific aspects may be the reason for that (see the questions section below).\n- Experiments on the impact of different data augmentations and their hyper parameters in the appendix are nice. It would be good to focus more on that and bring / discuss some of these aspects in the main text. And to try to develop that aspect a bit more (e.g., instead of just saying that data augmentations have a big impact on performance try to provide a bit more specific findings)\n\nOverall, the paper asks an important question and I like it. However, I feel that in its current form it is trying to address too many different aspects (1-3 above) and does not develop any of them in sufficient detail. I think that the paper would benefit from focusing more on RL + auxiliary objectives (e.g. including additional experiments from the questions section) and leaving the remaining ones for future studies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WICYiWssVoY", "reviewer": "Reviewer_Pnwo", "summary": "This paper presents an empirical study on the effect of self-supervised learning (SSL) in online and pretraining RL from pixels. Through the experiments on DMControl, Atari and a real-world robot environment, the authors make the following conclusions (i) the existing SSL framework for RL fails to bring meaningful improvement compared with the baselines using data augmentation techniques, while using the same amount of data and augmentation; (ii) the combination of SSL losses for RL also do not bring much gain; and (iii) no single self-supervised learning or image augmentation method dominates all environments. ", "questions": "Questions:\n1. Line 235: why study 86x86 and 87x87?\n\nSuggestions:\n1. Some experiments or methods are not insightful. For example, RotationCLS and ShuffleCLS can cause ambiguities. An input image and the rotated image (with 180 degree) both make sense in some environments, then how do the networks make the discrimination? Besides, \"tanh\" and many augmentation tricks are presented. They are a little bit off the point of the paper \"SSL impact on RL\".\n2. PSO seems to be an offline search method while RL is an online and dynamic learning process. The search results are likely to be sub-optimal.\n3. Studying which SSL benefits which type of tasks is more interesting and insightful.\n4. Line 230: \"Random\" ->\"random\"\n5. SPR [56] has published and the authors should cite the proceeding version.", "limitations": "The authors have addressed some limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors made a lot of experiments. Some results are interesting.\n2. The paper is easy to follow.\n\nWeakness:\n1. The conclusions are not very rigorous. For example, the authors test only six environments on DMControl and make the conclusion that augmentation is much more effective than SSL, e.g., CURL is much worse than RAD in Figure 4. However, I tested CURL and RAD on several hard DMControl tasks such as Hopper-hop and Reacher-hard and found CURL is more effective than RAD. Augmentation baseline like RAD may be better than SSL baseline such as CURL on only the six DMControl benchmark environments (in the paper), but that does not mean augmentation is always better than SSL on DMControl, especially on hard DMControl tasks. Therefore, I think the conclusion is biased.\n2. Most tested SSL methods in this paper are naively applied to RL. As far as I know, SSL methods applied to RL framework often need specific design. It would be much better if the authors use off-the-shelf SSL for RL methods. For example, when the authors study BYOL or predicting future, they should consider using SPR [56].\n3. Readers can hardly learn useful information from this paper. The conclusions that the authors make vary a lot on the test environments, e.g., on DMControl and Atari.\n4. The organization of this paper needs to be improved. Too many unimportant contents are put in the main text while some important figures/tables and analysis are in the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KJTFjaZxbL", "reviewer": "Reviewer_cRC9", "summary": "This work aims at answering the question of to what extent can self-supervised learning (SSL) objectives help with online Reinforcement Learning. The authors build a consistent experimentation framework around two popular RL algorithms designed for continuous and discrete action spaces respectively. Experiments in this work are performed over a significant collection of approaches, including state-of-the-art SSL methods, data augmentation methods, and a custom combination of multiple SSL objectives found through evolutionary search. The paper presents results from multiple synthetic domains and a real robot experiment. The authors come to the conclusion that there is not a single golden SSL approach that works the best in all cases, and that the performance of SSL approaches varies across different environments.", "questions": "- The paper only shows each model's final performance. Can the authors also include some reward vs. training-step plots? Are there other observations to make from those curves, e.g. ramp-up speed?", "limitations": "The authors did a great job making a fair comparison among different methods and not overclaiming their contribution. Some potential limitations are discussed above in the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well organized and well written.\n- The authors include an extensive collection of SSL and data augmentation methods for comparison in the experiments.\n- The real robot experiment is a good addition to showcase how these methods work with the presence of real-life camera/lighting noises.\n- The work provides valuable insight into how the instance-based SSL methods from state-of-the-art vision models don't seem to be super useful in the RL domain.\n\nWeaknesses:\n- This work lacks novelty in terms of methods. The authors utilize evolutionary search, but the resulting model is still a weighted mixture of existing methods.\n- The discussion on pretraining does not seem necessary because it relies on additional data at the very beginning, as the authors have pointed out, making its setting different from the rest of the methods in comparison. It's still good as an appendix section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept