Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learning
Authors: Manuel Goulão
Arlindo L. Oliveira
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Reinforcement Learning
Transformers
Self-Supervised Learning
Pre-training

Abstract: The Vision Transformer architecture has shown to be competitive in the computer vision (CV) space where it has dethroned convolution-based networks in several benchmarks. Nevertheless, Convolutional Neural Networks (CNN) remain the preferential architecture for the representation module in Reinforcement Learning. In this work, we study pretraining a Vision Transformer using several state-of-the-art self-supervised methods and assess data-efficiency gains from this training framework. We propose a new self-supervised learning method called TOV-VICReg that extends VICReg to better capture temporal relations between observations by adding a temporal order verification task. Furthermore, we evaluate the resultant encoders with Atari games in a sample-efficiency regime. Our results show that the vision transformer, when pretrained with TOV-VICReg, outperforms the other self-supervised methods but still struggles to overcome a CNN. Nevertheless, we were able to outperform a CNN in two of the ten games where we perform a 100k steps evaluation. Ultimately, we believe that such approaches in Deep Reinforcement Learning (DRL) might be the key to achieving new levels of performance as seen in natural language processing and computer vision.

Under review as a conference paper at ICLR 2023
PRETRAINING THE VISION TRANSFORMER USING SELF- SUPERVISED METHODS FOR VISION BASED DEEP REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The Vision Transformer architecture has shown to be competitive in the computer vision (CV) space where it has dethroned convolution-bass networks several benchmarks. Nevertheless, Convolutional Neural Networks (CNN) remain the pref- erential architecture for the representation module in Reinforcement Learning. this work. study pretraining Vision Transformer using several state-of-the-art self-supervised methods and assess data-efficiency gains from this training frame- work. We propose new sule-supe learning method called TOV- VICReg that extends VICReg to better capture temporal relations between observations by adding temporal order verification task. Furthermore, evaluate the resultant encoders with Atari games in a sample-efficiene the vision transformer, when pretrained with TOV-VICReg, outperforms the other self-supervised methods but still struggles to overcome a CNN. Nevertheless, we were able outperform CNN in two of the ten games where we perform a 100k steps evaluation. Ultimately, we believe that such approaches in Deep Reinforce- ment Learning (DRL) might be the key to achieving new levels of performance as seen in natural language processing and computer vision.
regime. Our results show that
INTRODUCTION
Despite the successes of deep reinforcement learning agents in the last decade, these still require a large amount of data or interactions to learn good policies. This data inefficiency makes current methods difficult to apply to environments where interactions are more expensive data is scarce, which is the case in many real-world applications. In environments where the agent doesn't have full access to the current state (partially observable environments), this problem becomes even more prominent, since the agent not only needs to learn the state-to-action mapping but also a state representation function that tries to be informative about state given observation. In contrast, humans, when learning a new task, already have a well-developed visual system and a good model of the world which components that allow us easily learn new tasks. Previous works have tried to tackle the sample inefficiency problem by using auxiliary learning tasks (Schwarzer et al.) 2021b Stooke al.) [022]] Guo al. 2020), that try help the network's encoder learn good representations of the observations given by the environments. These tasks can be supervised or unsupervised and can happen during pretraining phase or reinforcement learning (RL) phase in joint-learning or decoupled-leamming scheme.
In recent years, selervueer learning has shown to be very useful in computer vision, the increasing interest in this area has resulted in the appearance of new and improved methods that train network to learn important features from the data using only the data itself as supervision. approach to evaluating such methods is to train a network composed of the pretrained encoder, with the parameters frozen, paired with a linear layer in popular datasets, like ImageNet. These evaluations have shown that these methods can achieve high scores in different benchmarks, which shows how well the current theteatff methods are able to encode useful information from the given images without being task-specific. Additionally, it has been shown that pretraining network using self-supervised learning (or unsupervised) adds robustness to the network and gives better generalization capabilities (Erhan al. 2010).
common

---

Under review as a conference paper at ICLR 2023
Also recently, new architecture for rision-based tasks called the Vision Transformer (ViT) (Dosovit skiy et al.] 2020) has shown impressive results in several benchmarks without using any convolutions. This architecture presents much weaker inductive biases when compared to CNN, which can result in lower data efficiency. But the Vision Transformer, unlike the CNNs, can capture relations between parts of image (patches) that are far apart from each other, thus deriving global information that can help the model perform better in certain tasks. Furthermore, when the model is pretrained, using supervised self-supervised learning, manages surpass the best convolution-bass models in terms of task performance. Nonetheless, and despite these successes in computer vision these results are yet to be seen in reinforcement learning.
Motivated by the potential of the Vision Transformer, in particular when paired with pretraining phase, and the increasing interest self-supervised tasks for DRI,, we study pretraining ViT using state-of-the-are (SOTA) self-supervised learning methods and use it as the representation module in Deep RI. algorithm. Consequently, we propose extending VICReg (Variance Invariance Covariance Regularization) (Bardes et al. 2022) with temporal order verification task (Misra et al. 2016 to help the model better capture the temporal relations between consecutive observations. We named this approach Temporal Order Verification- -iCReg or in short TOV- TOV-VICReg. While could have adapted any of the other methods, we opted for VICReg due to its computational performance, simplicity, and good results in early experiments and metrics such the ones presented in Section 7 After our empirical results in the Atari games, we present a small study of the pretrained encoders using several metrics to understand if they suffer from any representational collapse and also analyse the learned representations using similarity matrices and attention maps.
Our main contributions are:
We propose a new self-supervised learning method which extends VICReg to capture the temporal relations between consecutive frames through a temporal order verification task, in Section 4
We pretrain a Vision Transformer using several SOTA self-supervised methods and our proposed method, and study them through metrics (Section Z. visualizations (Section 8) and fine-tuning in reinforcement learning Section 66, where we show that temporal relations learned by the model pretrained with method contribute relevant increase in data efficiency.
2 RELATED WORK
Pretraining representations Previous work, similarly to our approach, has explored pretraining representations using self-supervised methods which led to great data-efficiency improvements in the fine-tuning phase (Schwarzer et al. 2021b Zhan et al. 2020) or superior results evaluation tasks, like AtariARI (Anand et al. 2020). Others have pretrained representations using RL algorithms, like DQN, and transfer those learned representations to a new learning task (Wang et al. 2022).
Temporal Relations Other works have explored learning representations that have temporal in- formation encoded. ATC (Augmented Temporal Contrast) (Stooke et al.] [02T trains an encoder compute temporally consistent representations using contrastive learning. and the ST-DIM (Spa- tioTemporal DeepInfoMax) (Anand et al., 2020) captures spatial-temporal information by maximizing the mutual information between features of two consecutive observations.
Joint learning In recent years, adding an auxiliary loss to the RL loss, usually called joint learning,
has become a common approach by many proposed methods. Curl (Srinivas et al.) (022) adds contrastive loss using siamese network with momentum encoder. Another work studies different joint-learning frameworks using different self-supervised methods (Li al., 2022). SPR (Schwarzer et al. 2021a) uses an auxiliary task that consists of training the encoder followed by an RNN to predict the encoder representation k steps into the future. PSEs (Agarwal et al. 202 combines a policy similarity metric (PSM), that measures the similarity of states in terms of the behaviour of the policy in those states, and a contrastive task for the embeddings (CME) that helps to learn more robust representations. PBL (Guo et al.) 2020) learns representations through interdependence between an encoder, that is trained to be informative about the history that led to that observation, and RNN that is trained predict the representations of future observations. Proto-RL. (Yarats)
2

---

Under review as a conference paper at ICLR 2023
al. 2021 uses auxiliary self- supf-sueeeeieed objective to learn representations and prototypes (Caron et al. 2020). and uses the learned prototypes to compute intrinsic rewards which will push the agent to explore the environment.
Augmentations While we only use augmentations in the pre-training phase, their use during reinforcement learning has also been studied. Methods like DrQ (Kostrikov et al. (202)) and RAD (Laskin et al., 2020) pair an RL algorithm, like SAC, with image augmentations to improve data efficiency and generalization of the algorithms.
Vision Transformer for vision-based based Deep RL Recent works, also compare the Vision Trans- former to convolution- based architectures with a similar number of parameters and show that VIT is very data inefficient even when paired with an auxiliary task (Tao et al. 2022).
Self-Supervised learning for image sequences Multiple works propose simple pretext tasks to train encoders to capture information from image sequences. These pretexts tasks can be playback speed classification ( Yao al. 2020), a temporal order classification (Misra et al. [016]] Tee et 2017 Xu et al., 2019), jigsaw game (Ahsan et al., (219)) masked modelling task (Sun et al. 2019). A different approach consists of using contrastive learning. In this category, we can find methods that maximise the similarity between image sequences (Feichtenhofer al. 2021). autoregressive models to predict frames multiple steps in the future Lorre et al. (2020). and maximize the similarity between temporally adjacent frames (Knights et al. 2021).
3 BACKGROUND
3.1 VISION TRANSFORMER
ViT (Dosovitskiy et al. 2020) is a model, for image classification tasks, that doesn't rely on CNNs using only attention. The model wraps the encoder of Transformer, uses patches of the input image as tokens and adds a classification token which after the computation will serve as the image representation. When compared to CNNs, ViT presents weaker image-specific inductive biases which can impact the sample-efficiency of the model during learning (d d'Acooli et al.] 2021). however, has been shown that with enough data the image-specific inductive biases become less important (Dosovitskiy et al.) 2020)). Moreover, ViT can capture relations between patches that are far apart from each other, thus deriving global information that can help the model perform better in certain tasks
3.2 REINFORCEMENT LEARNING
The problem of an agent learning to solve task in a certain environment can be defined as a Markov Decision Process (MDP). A MDP M defined by the tuple (8, A, R, T), where S is the set of states, AA the set of actions, R the reward function, and T the transition function. At each timestep the agent state
and takes an action a A. Upon performing the action the agent receives from the environment a reward r E R and new state s' E S which is determined by the transition function T(s', s, a). The MDP assumes that the Markov property holds in the environment, i.e. state transitions are independent and the agent only needs to know the current state to perform action P(ay) po. 21....;) = P(ay(xr (x)). For the agent to decide what action to take uses a policy function which gives a distribution over actions given state, π (a, st). This policy is evaluated
using the function V (s), which estimates the expected total discounted reward an agent in a state and which follows a policy
3.2.
DQN AND RAINBOW
DQN Mnnih al. 2013 is value-based method and uses network with parameters that given state 8 outputs prediction of the distribution of Q values over actions, Q& (8, a). The network learns the Q function by minimizing the mean squared error: (y - as) where y + + mar. Q (s', a'), shown in Algorithm I at the Appendix.
3

---

Under review as a conference paper at ICLR 2023
Several works followed the DQN algorithm which introduced changes to improve performance. Rainbow (Hessel et al. 2017) combines six improvements, Double Q-Learning (van Hasselt et al.)
2016), Prioritized Replay (Schaul et al. 2016), Dueling Networks (Wang al. 2016), Multi-step Learning (Sutton & Barto 2018), Distributional RI. (Bellemare et al. 2017 and Noisy Nets (Fortunato et al. 2018) resulting in a more stable and sample efficient algorithm.
3.3
SELF-SUPERVISED METHODS
Recent self-supervised methods for vision tasks can be put in two main categories: contrastive and non-contrastive.
In contrastive learning. methods like MoCo (He et al. 2020) or SimCLR (Chen et al. 2020aa learn using a loss function that pulls the positive samples together and pushes the negative samples apart. These methods usually require very large batch sizes or auxiliary structures that allow for more negative samples. MoCo, in particular, has three iterations (He et al. 2020), v2 (Chen et al., 2020b). and v3rChen al.) 2021). In this work, we consider the more recent version (v3). This version uses siamese network, where in one path the augmented samples (queries) are computed by an encoder fo (backbone) and projector Is. and in the other the samples (keys) by momentum-encodee fo' and a projector Ip. The loss function is the InfoNCE loss, with temperature, of the dot product of the queries with the keys.
On the other hand, non-contrastive methods don't rely on the notion of positive and negative samples which results in vast number of different approaches. DINO (Caron et al. (02T] consists of a siamese network where each path fed with a random augmentation of the input and where the encoders learn minimize the cross-entropy between their normalized output probability distributions, computed using a softmax with temperature scaling. The teacher encoder is updated using an exponential moving average of the student encoder parameters and in its computation path is used an additional centring operation that contributes to an asymmetry that helps the method avoid collapse. Unlike, most methods, DINO creates more than augmentations of the same source. More precisely it creates set of views composed of two global views and several local views. All views are computed by the student network while only the global views are computed by the teacher network, which pushes the student to create a local-to-glod
correspondence.
VICReg, on the other hand, tries to learn representations invariant to augmentations by minimizing the L2 distance while maintaining some variance in the representation features and decorrelating features. more detailed explanation of the method will be presented in Section
For this study we selected DINO, MoCo, and VICReg since they currently considered state-of- the-art, their official implementations are available in PyTorch, and each represents a different type of approach.
4 TOV-VICREG
VICReg is non-contrastive method that trains network be invariant to augmentations applied to the inputs while avoiding a trivial solution with the help of two additional losses, called variance and covariance, that act as regularizers over the embeddings. While VICReg is agnostic concerning the architectures used and even the weight sharing, in this work we consider the version where paths are symmetric, the weights are shared, and each path is composed of an encoder (also called backbone) and an expander. The expander increases the dimension of the representation vector in non-linear way allowing the covariance loss to reduce dependencies and not only correlations of the representation vector. In addition, the expander also removes information that is not common to both representations.
VICReg uses three loss functions: invariance is the mean of square distance between each pair of embeddings from the same original image, as shown in Equation I where Z, and Z' are two sets of embeddings, of size N, that result from computing two different augmentations of N sources, and 2j denotes the j-th embedding in the set; variance is a hinge loss that computes, over the batch, the standard deviation of the variables in the embedding vector and pushes that value to be above certain threshold, as shown in Equation?? [ where d denotes the number of dimensions of the embedding vector, and Z is the set of the j-th variables in the set of embedding Z; covariance is function
4

---

Under review as a conference paper at ICLR 2023
that computes the of the squared off off-diagonal coefficients of covariance matrix computed over a batch of embeddings, as shown in Equation 3 to decorrelate the variables from the embedding. While the invariance loss function tries to make the model invariant augmentations, i.c. output the same representation vector, the other two functions regularize the method by pushing the variables of the embedding vector vary above certain threshold and decorrelating the variables in each embedding vector.
1 ((,
Σ
2j =3112
N
1
222
d Σ
max 7 (Var(Zi))
(2)
dz) d ビ
[ov(Z))
TOV-VICReg or Temporal-Order- Teempral--rdee--eiiiittttt----
extends VICReg better capture the tem- poral relations between consecutive observations and consequently encode extra information that can be useful the deep reinforcement learning phase. To achieve that we add new temporal order verification task, as seen in Shuffle-and-Less
(Misra et al. (016), that consists of binary classification task where a linear layer learns to predict three given representation vectors are the correct order or not. Like the other losses, we also employ coefficient for the temporal loss and in most of our experiments, the value is 0.1. Figure visually illustrates TOV-VIC VICReg.
humpeeal loss
Voniance
concel shuffin
Covor e0c
Contriinco lacs
Z
expandor
expander
3/.
1/1
3/4
T
T~
r'
Tt-1
E++1
Et..
Figure 1: TOV-VICReg architecture
At each step sample consecutive observations, (-,-1, Iy, }e+1), N, is processed by two different augmentations, and like VICReg these are the augmentations used in BYOL (Grill et al. 2020), while Tı-1 and 2441 are processed by two simple augmentations composed of color jitter and random grayscale. The 2, augmentations are computed by the VICReg computation path and the resultant embeddings are used for the loss functions, i.e. variance, invariance, and covariance. In the temporal order verification task we encode the augmentation of It-1 and I1+1. and concatenate those two representations with one of the representations of w,, in our case we used the one that was augmented without solarize, obtaining the vector [br-1, 1, Mr. Vt++ }. At last, we randomly permute the order the representations in the vector and feed the resultant concatenated vector to a linear layer with single output node that predicts if the given concatenated vector has the representations in the right order or not. The temporal loss used to optimize the model for this task is a Binary Cross Entropy loss. TOV-VICReg's pseudocode be found in Appendix

---

Under review as a conference paper at ICLR 2023
5 PRE-TRAINDE METHODOLOGY
We pretrained four encoders, one using proposed method TOV-VICReg and three using state-of- the-art self-supervised methods: MoCov3 (Chen et al. 202T), DINO (Caron et al. 202T) and VICReg (Bardes al., 2022). For this study, the encoder used Vision Transformer,
precisely the VIT tiny with patch size of 8. We chose this patch size based on experiments that show that this value performed well in terms of data-efficiency when compared to 6, 10, and 12 without being too computationally intensive (Appendix [B. Moreover, the implementation we use is an adaptation of the timm library (Wightman 2019) implementation, which can be found the source code of the DINO method. The dataset used is a set of observations from 10 of the 26 games in the Atari 100k benchmark, all available in the DQN Replay Dataset (Agarwal et al. 2020 For each game, we use three checkpoints with a size of one hundred thousand data points (observations), which makes up total of three million data points (-55 hours). The pretraining phase is 10 epochs with two warmup epochs. We used the official code bases of all the self-supervised methods and tried to change the least amount of hyperparameters. Appendix contains the tables with the hyperparameters used for each method.
6 DATA-EFFICIENCE
To test the pretrained Vision Transformers in reinforcement learning and compare data-efficiency gains, we trained in the 10 games used for pre-trate pre-training for 100k steps using the Rainbow algorithm (Hessel et al. (017). with the DER (van Hasselt et al. 2019) hyperparameters. The only difference between the agents at the start the representation module. We chose two networks to compare against, the Nature CNN (Mnih et al. 22155, and a ResNet with amount of parameters similar to ViT tiny (Appendix C (Schwarzer et al. 2021b) that has a size roughly similar to the ViT tiny. Moreover, use learning rate two orders of magnitude smaller for the encoder (1 10 9, which previous works and experiments performed by us show to be beneficial (Schwarzer et al.) 20215).
In this section, to report our results follow the rliable (Agarwal et al. 2021b) evaluation framework. where the scores of all games are normalized and treated as one single task.
6.1 RESULTS
Figure 2 shows the aggregate metrics of seven different encoders on 10 Atari games with training runs of TO0k steps. The first four (ViT+<method>) are ViT tiny models pretrained with four different self-supervised methods, while the last three (ViT, ResNet, and Nature CNN) are randomly initialized models. Starting with the randomly initialized models we can assess that the Nature CNN and the ResNet are the most sample efficient models, with ViT far behind. Regarding the pretrained models, ViT, when pretrained with our method, performs better than the other models and the non non-pretrained ViT in all metrics. worth noting that we report a higher variance in the results of our proposed method when compared to the remaining methods and non pretrained models. ViT+TOV-VICReg when compared to Nature CNN, which has far fewer parameters, and ResNet, with similar number of parameters seems to closely match their sample-efficinncy performance (Appendix Table 7). Furthermore, the difference between the non-pretrained ViT and VIT pretrained with TOV-VICReg shows that good self-supervised method that explores temporal relations and 3 million data points can help close the and capable model. Regarding the remaining self supervised methods, MoCo seems perform considerably well obtaining even a median very similar to TOV- VICReg and is then followed by DINO and VICReg, respectively. All pretrained ViTs show an improvement in comparison the non-pretrained VIT.
gap while remaining a more complex
METRICS
significant phenomenon when doing self-supervised training is the collapse of the representations, which can be seen in three forms: representational collapse, dimensional collapse, and informational collapse. Representational collapse refers to the features of the representation vector collapsing to single value for every input, meaning the variance of the features is zero, close to zero. In

---

Under review as a conference paper at ICLR 2023
MEAN
MEDIAN
IQM
Optimality Gap
VT-TOV-VICReg
VTT+MoCo VIT+DINO
MT+VICReg
VI
ResNet
Nature CNN
0.12
0.16
0.20
0.02
0.04
0.08
0.015 0.030 0.045 0.060
0.84
0.88
0.92
Human Normalized Soore
Figure 2: The eval runs across the different games normalized and treated as single task. The IQM corresponds to the Quter-Qurttlle Mean among all the runs, where the top and bottom 25% are discarded and the mean is caleulated over the remaining 50%. The Optimality Gap refers to the number runs that fail to surpass the human average score, i.e. 1.0.
dimensional collapse, the representations don't use the full representation space, which can be mea- sured by calculating the singular values of the covariance matrix calculated over the representations. Informational collapse defines the case where the features the representation vector are correlated and therefore representing the same information.
Dimensional Collapse All methods seem to avoid dimensional collapse, i.e. most dimensions have singular value larger than zero, observed in Figure 3 However, notice that some methods make better use of the space available since they present higher singular values. TOV-VICReg, in particular, seems excel in this metric, even improving the results obtained by VICReg. It is worth noting that both VICReg and TOV- VICReg employ a covariance loss that helps decorrelate the embedding variables which may be contributing positively these results. Furthermore, used a covariance coefficient of 10 for TOV-VICReg and for VICReg a change that according to our experiments culminates in the increase here observed.
log singular value
-5-
DINO
-15
VICReg
TOV-VICReg
0
30
100
125
100
175
200
singular value rank index
Figure 3: Logarithm of the singular values of the representation vector's covariance matrix sorted by value.
Representational Collapse Results in the first row of Table Ishow the computed standard deviation of the representation vector over batch of thousands of data points. DINO, VICReg and TOV- VICReg show value well above zero, meaning that none of the methods suffered from representation collapse during training. On the other hand, MoCo shows much smaller value of 0. 178, which is still, is far from a complete collapse. Both VICReg and TOV- VICReg use a hinge loss that pushes the representation vector to have standard deviation of above. While VICReg slowly converges to this value our method converges to roughly 1.65, which might be the result of adding a temporal order verification task.
Informational Collapse We report in the second row of Table I the comparison of the average correlation coefficients of the representation vectors. TOV- performs better than the other methods, including VICReg, which present very similar coefficients. Like in the dimensional collapse, this result is in part due to the higher covariance coefficient used in TOV-VICReg which by design
7

---

Under review as a conference paper at ICLR 2023
Metric
DINO
MoCo
VICReg
TOV-VICReg
Std
0.979
0.178
1.003
1.648
Corr. Coef.
0.1764
0.1538
0.1531
0.0780
Table 1: Average standard deviation and correlation coefficient of the representation vector
helps the model decorrelate the representation's 's features. Increasing the coefficient in ViCReg results in a lower correlation coefficient as well, but is still higher than TOV-VICReg.
REPRESENTATIONS
In this section, we present different visualizations better understand the representations learned by each of the methods. Our goal with the following visualizations is to help us better understand the learned representations and give some intuitions about their properties.
Cosine similarity Figure presents similarity matrix of the representations where observe that TOV-VICReg can better distinguish between observations of different games but also observations from the game, shown in Figure MoCo, the other hand, seems make good distinction between observations from the same game. However, we can observe in the colour that all the representations are very similar each other, which corroborates the results obtained in Section 7 Oppositely, VICReg and DINO manage to spread representations more, as we can see in the colour bars, but, the yellow squares in the diagonal show that the representations from the same game are more similar to each other which is corroborated by Figure Given the empirical results, we believe that this capacity to distinguish observations from the same game might good indicator.
MoCo
1440
ViCReg
TOY-ICREg
Figure 4: Similarity matrices of the representations computed by MoCo, DINO, VICReg, and TOV- VICReg respectively. There are a total of 64 data points, from 4 different games: Alien, Breakout, MsPacman, and Pong, where from 0-15 are from Alien, 16-31 are from Breakout and forth.
DINO
MoCo
ViCReg
TOWVICReg
M
439
5356
317
3305
850
43942
6.07
C200
Figure 5: Similarity matrices of the representations computed by MoCo, DINO, VICReg, and TOV. VICReg respectively, of observations from MsPacman.
Attention visualisation The research work that proposes DINO shows that the Vision Transformer is able to attend to important parts of the input after training using DINO. Inspired by these results, we try to make the same evaluation for the several self-supervised methods we are studying, including TOV-VICReg and try understand if any of the encoders attend interesting parts of the input.
8

---

Under review as a conference paper at ICLR 2023
the results of all methods for observation from the game of Pong, where each method produces three attention maps, one for each self-attention head of the last block of the Vision Transformer. All pretrained VIT seem to attend at some level to important game features like the ball and the paddles. However, TOV-VICRes other parts of the frame that don't consider important describe the current state of the game. When comparing to VICReg's s attention maps we believe that the temporal order verification task greatly helped the attention of the model. In more visually complex games, e.g. Freeway or MsPacman, these attention maps start to be more difficult to analyse but it is still possible to discern some important features.
In Figure 6
the only method that doesn't 't spread the attention
Observation
DINO
MoCo
ViCReg
TOV-VICReg
head 0
head 0
head 0
head
head 1
head 1
head 1
nead 1
head 2
head
head
head
Figure Attention maps produced by the pretrained ViTs. We fed a pretrained ViT with an observation from the game Pong and obtained the attention maps from the three heads in the last block.
DISCUSSION & CONCLUSION
In this work, we presented a study of VIT for vision-based deep reinforcement learning using self-supervised pretraining, and proposed self-supervised method that extends VICReg to better capture temporal relations between consecutive observations. Our results showed that the agent using Vision Transformer that pretrained with our method manages surpass all other Vision Transformers, pretrained and non pon-prtrriined in sample efficiency and also achieves results very close to convolution-basee based models with far fewer parameters. These results reinforce the importance of encoding temporal relations between observations in the representation model, as shown by previous vision models with weaker inductive biases and more parameters,
works, and also show that when well pretrained, can achieve similar results in sample efficiency, Furthermore, we show several metrics, evaluation tasks and visualizations which be of great value for future work.
The ability to use larger models, with millions of parameters, that are as sample efficient as some of the most popular CNN-based models (like Nature CNN or Impala ResNet), with thousands of parameters, is very important since it opens the door to using Deep RL in even more complex problems where smaller models tend to struggle to perform, without losing sample-efficiency. In this work, we try to advance the knowledge by studying the pretrain of vision transformer using self-supervised methods, This approach has seen successes in natural language processing (Devlin) et al. 2019: (Brown et al.) 2020), and computer vision (Radford et al. 2021) and we believe that similar approaches RL have the potential unlock new levels of performance never achieved
before (Baker et al. 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An Optimistic Perspective on Offline Reinforcement Learning. /9Xiv:/907.044 June 2020.

---

Under review as a conference paper at ICLR 2023
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare G. Bellemare. Contrastive Be- havioral Similarity Embeddings for Generalization in Reinforcement Learning. arXiv:2101.05265, March 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in neural information processing systems, 2021b.
Unaiza Ahsan, Rishi Madhok, and Irfan Essa. Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition. In 2019 JEEE Winter Conference on Applications of Computer Vision (WACV), pp. 179-189. IEEE, 2019.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. Devon Hjelm. Unsupervised State Representation Learning in Atari. Technical Report KXiv:1906.08226, arXiv, November 2020.
Bowen Baker, Igge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video Pre'rriiiin (VPT): Learning to Act by Watching Unlabeled Online Videos. arXiv:2206. 11795, June 2022.
Adrien Bardes, Jean Ponce, and Yann Lecun. Vicreg: Varaane-iiyarrane--cvvainae regulariza- tion for self-supervised learning. In ICLR 2022-10th International Conference on Learning Representations. 2022.
Marc G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 449-458 06-11 11 Aug 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym.
(6666000540)) June 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel perbert-Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33. pp. 1877-1901, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self Supervised Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Ting Chen, Simon Kornblihh, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607, November 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv:2003. 04297, March 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self- Sueerrupeeee Vision Transformers. arXiv:2104.0200 02055,, August 2021.
Stéphane d' d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. ViTV Improving Vision Transformers with Soft Convolutional Inductive Biases. arXiv:2103. 10697, June 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv: May 2019.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, September 2020.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why Does Unsupervised Pre-training Help Deep Learning? In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 201-208, March 2010.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2021.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy Networks For Exploration. February 2018
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo. Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised Learning. arXiv:2006.00 077333 September 2020.
Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap Latent-Predictio Representations for Multitask Reinforcement Learning, arXiv:2004.14666 April 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. 170.v217.0.02298. October 2017.
Joshua Knights, Ben Harwood, Daniel Ward, Anthony Vanderkop, Olivia Mackenzie-Roon and Peyman Moghadam. Temporally coherent embeddings for self self-supervised video representation learning. In 2020 25th International Conference on Pullern Recognition (ICPR), pp. 9914-89211. IEEE, 2021.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixcls. arXiv:2004.13644 March 2021.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement Learning with Augmented Data. In Advances in Neural Information Processing Systems, volume 33, pp. 19884-19895, 2020.
Hsin- Hsin-Ying Lee, Jia- Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE international conference on computer vision, pp. 667-676, 2017.
Xiang Li, Jinghuan Shang. Srijan Das, and Michael S. Ryoo. Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels? wrXiv:2206.05266, June 2022.
Guillaume Lorre, Jaonary Rabarisoa, Astrid Orcesi, Samia Ainouz, and Stephane Canu. Temporal contrastive pretraining for video action recognition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 662-670, 2020.
Ishan Misra, C. Lawrence Zitmick, and Martial Hebert. Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, Lecture Notes in Computer Science, pp. 527-544, Cham, 2016.
11

---

Under review as a conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602 December 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-lever control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, pp. 8748-8766 July 2021.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. In ICLR (Poster), January 2016.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-Efficies Reinforcement Learning with Self-Predietive Representations. arXiv:2007.0592. May 2021a.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining Representations for Data-Efficien Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 34, pp. 12686-12699, 2021b.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. CURL: Contrastive Unsupervised Representa- tions for Reinforcement Learning. arXiv:2004.04 04,366, September 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling Representation Learning from Reinforcement Learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 9870-9879, July 2021.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7464-7473, 2019.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, second edition: An Introduction. November 2018
Tianxin Tao, Daniele Reda, and Michiel van de Panne. Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixels. 2xXiv2204.04900 May 2022.
Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-Leamnig. In Thirtieth AAAI Conference on Artificial Intelligence, March 2016.
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, volume 32, 2019.
Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, and Adam White. Investigating the Properties of Neural Network Representations in Reinforcement Learning. arXiv:2203. 15955, March 2022.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1995-2003, June 2016.
Ross Wightman. Pytorch image models.
pytorch- inni--maee--ee models 2019.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotem- poral learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10334-10343, 2019.
12

---

Under review as a conference paper at ICLR 2023
Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception representation learning. In Proceedings of the IEEE/CVF
for self-supervised conference on computer vision and pattern recognition, pp. 6548-6488866 (148-6557,11 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In Proceedings of the 38th International Conference on Machine Learning, pp. 19920-11931, July 2021.
Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin, Framework for Efficient Robotic Manipulation. arXiv:2012.07975. December 2020.
13


Reviews:
[{"review_id": "p2TVmXbGBNQ", "reviewer": "Reviewer_Wuo3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to introduce visual transformers to extract feature representations for reinforcement learning. It proposed a method, which is a combination of VICReg and temporal ordering prediction, to pretrain the ViT network. Experiments are done on Atari 100K using the visual transformer pretrained by the proposed method. ", "summary_of_the_review": "The paper has a good motivation of using a well-pretrained visual transformer in deep RL. However, the method design seems to be generic and of marginal novelty. The experimental results are a bit confusing and do not seem to be staring enough to address the paper’s motivation. ", "strength_and_weaknesses": "** Strength: **\n1. It is reasonable to combine the image only pretraining method (specifically, VICReg) with temporal ordering prediction as the overall objectives covers both image and temporal information. \n2. The paper analyzes the property of representation from a few different aspects in the experiments. \n\n** Weaknesses: **\n1. It is unclear how the pretraining is specific to deep reinforcement learning. It looks like a generic video-based pretraining method, and its connection with reinforcement learning seems a bit loose. \n2. The novelty of the method is limited as combining two established self-supervised learning methods in the same training objective is not something difficult to come up with. \n3. Q: it seems Atari 100K has 26 games (please correct me if I am wrong), why only 10 games are used?\n4. The experimental results are not strong. Especially, using visual transformer does not seem to be that competitive to using CNN, as the paper claimed. \n5. The clarity of the paper (especially in the experiment section) can be improved. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ULVxcWrhNb", "reviewer": "Reviewer_9PJR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly targets usage of ViTs in Reinforcement Learning. The authors conduct several experiments with different training strategies for ViTs, showing the effectiveness of self-supervised learning for ViTs in RL. Besides, the authors propose a new self-supervised objective function based on VICReg, which involves the temporal order prediction for learning better temporal relations.", "summary_of_the_review": "The authors present good technical analysis of using ViTs for RL problems. The novelty and significance are limited.", "strength_and_weaknesses": "Strength:\n1. The authors present interesting and trials with ViTs for RL.\n\nWeaknesses:\n1. The novelty is limited. The proposed method is made up of several previous methods.\n2. Even though the proposed TOV-VICReg is more effective than the other self-supervised methods, it is still only comparable to the CNN based models with no significant advantage. In other words, the ViT based models remain impractical for RL.\n3. It would be better if the authors can consider reconstruction based self-supervision like MAE and SimMIM in their experiments, which adopts a quite different objective compared with DINO and MoCo.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JiL4Dwri0y", "reviewer": "Reviewer_rKFV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the effect of self-supervised pre-training for Vision Transformer based RL agents. It shows the effect of temporal order verification and VICReg on 10 Atari games. The proposed methods improves final return on Atari games.", "summary_of_the_review": "The submission is limited in terms of both theoretical novelty and empirical novelty. Especially, the empirical validity is pretty limited due to small set of experiments, missing experiment details, etc. More work should be done to warrant the publication of this work.", "strength_and_weaknesses": "[Strengths]\n- The paper is easy to follow.\n- The proposed method makes intuitive sense to me.\n\n[Weaknesses]\n- Most importantly, the main contribution of the submission is not very clear to me.\n- Also, the technical novelty and empirical novelty is quite limited. TOV and VICReg are existing self-supervised learning methods. Also, it is well-known that the use of pre-training is helpful for vision-based reinforcement learning.\n- The analysis in Section 7 (representation collapse, dimension collapse, etc) are very generic and do not provide much insight on using self-supervised learning for reinforcement learning.\n- It seems that the CNN result is without self-supervised training and ViT result is with self-supervised training (although it is unclear to me). Can authors add more on this?\n- I don't see the learning curve of return vs environment steps. This is pretty important for demonstrating sample efficiency in RL.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8TKM0fFDJb9", "reviewer": "Reviewer_X82N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies if Vision Transformers (ViTs) could outperform CNNs in vision-based RL tasks. The authors compared existing self-supervised strategies and proposed a new self-supervised approach designed for the sequential observations in RL. The newly proposed method combines previous ideas from VICReg (Bardes et al, 2021) and Shuffle-and-Learn (Misra et al, 2016). The authors show that the new approach TOV-VICReg outperforms other SOTA self-supervised learning methods in the RL setting. However, from the authors experiments, CNNs still outperform ViTs trained with TOV-VICReg in eight out of the ten tested Atari games. ", "summary_of_the_review": "I’m inclined to reject this paper mainly because: 1) the experiments do not justify the motivation of this work, i.e., use large models to solve complex RL tasks where smaller models struggle, as stated in the conclusion of this paper. Clearly, the chosen environments are not complex enough to demonstrate the potential benefits of pretrained ViTs. 2), the proposed method lack novelty and the results do not bring significant new insights to the community, 3) the paper does properly compare other self-supervised learning methods that aim to capture the temporal relations  4) the paper presentation needs further polishing.  ", "strength_and_weaknesses": "### Strength\n\n1. The proposed self-supervised learning method is well suited for RL tasks and experiment results show that it improves performance of self-supervised learning methods that does not capture the temporal relationship.\n\n### Weaknesses\n\n1. This paper lacks novelty in terms of the proposed self-supervised learning method. Also, the presented results are mostly well-known in the community.   \n2. Temporal order verification is one type of self-supervision loss that aims to capture temporal dependency. This paper does not justify the choice of Shuffle-and-Learn against other methods [1, 2, 3]. I’d recommend the authors to add a discussion on relevant self-supervised learning literature for temporal relationship learning and provide a justification on the choice of Shuffle-and-Learn.   \n3. The number of steps used for temporal order verification loss is an important hyperparameter that would affect the quality of the learned representations. This paper does not provide experimental results showing its impact. I’d encourage the authors to add an experiment varying the number of steps.\n\n### References\n\n[1] Lee, Hsin-Ying, et al. \"Unsupervised representation learning by sorting sequences.\" *Proceedings of the IEEE international conference on computer vision*. 2017.\n\n[2] Xu, Dejing, et al. \"Self-supervised spatiotemporal learning via video clip order prediction.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2019.\n\n[3] Yao, Yuan, et al. \"Video playback rate perception for self-supervised spatio-temporal representation learning.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6hgzpDMvtr", "reviewer": "Reviewer_Wxq5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "*I am a vision person and an emergency reviewer.*\n\nThis paper proposes to combine VICReg and a temporal verification loss for video representation learning, and inspect its impact on the data-efficiency of down-streaming reinforcement learning tasks. It is shown that the proposed method out-performs VICReg, MOCO and DINO while still under-performing (randomly initialized?) CNNs. Further analyses show that the proposed variant of VICReg shows better properties in terms of collapse and feature distribution.", "summary_of_the_review": "This is an empirical study yet I am not convinced by the motivation or the significance of empirical results.", "strength_and_weaknesses": "Strengths:\n+ The proposed training framework that combines VICReg and temporal verification is new to my knowledge.\n+ Experimental results show that the variant out-performs alternative representation learning methods in the inspected RL setting.\n+ A set of analysis shows that the method shows better statistical properties w.r.t. collapse and feature similarity.\n\nWeaknesses:\n- Most of the experimental results seem not related to reinforcement learning and the only RL evaluation reports inferior performance than (randomly initialized?) CNNs. I recommend the authors to consider generic video classification evaluation protocols in [A], which may be a more promising thing to pursue.\n- The language is fine but the organization is pool IMHO. Too much space is spent on describing standard things like MDP, DQN or VICReg.\n- Generally I am not convinced by the motivation. I think transformers are widely considered less data-efficient and I don't understand why authors expect it to behave the opposite way in reinforcement learning.\n- Typos:\ncomponents that allows us -> allow\nthe increasing interest -> and the increasing interest\nViT presents weaker image-specific inductive biases which allow the CNNs for much sample-efficient learning; allows CNNs?? what does this mean\n\n[A] VideoMoCo: Contrastive video representation learning with temporally adversarial examples, CVPR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject