Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the effect of self-supervised pre-training for Vision Transformer based RL agents. It shows the effect of temporal order verification and VICReg on 10 Atari games. The proposed methods improves final return on Atari games.
strength_and_weaknesses: [Strengths]
- The paper is easy to follow.
- The proposed method makes intuitive sense to me.

[Weaknesses]
- Most importantly, the main contribution of the submission is not very clear to me.
- Also, the technical novelty and empirical novelty is quite limited. TOV and VICReg are existing self-supervised learning methods. Also, it is well-known that the use of pre-training is helpful for vision-based reinforcement learning.
- The analysis in Section 7 (representation collapse, dimension collapse, etc) are very generic and do not provide much insight on using self-supervised learning for reinforcement learning.
- It seems that the CNN result is without self-supervised training and ViT result is with self-supervised training (although it is unclear to me). Can authors add more on this?
- I don't see the learning curve of return vs environment steps. This is pretty important for demonstrating sample efficiency in RL.
clarity,_quality,_novelty_and_reproducibility: - The core contribution is not very clear to me.
- Regarding the use of self-supervised learning for RL, "CURL: Contrastive Unsupervised Representations for Reinforcement Learning" is one of the work that popularized this. The authors should cite and compare against this work.
- In light of the existing work, more work should be done to advance the field. Just trying self-supervised learning on reinforcement learning is not sufficient for publication at ICLR.
- The contribution may be the use of self-supervised learning on ViT based image RL. The proposed method does not contain modifications for ViTs: TOV and VICReg are very generic methods, and they can be applied to any deep nets that take images (regardless of CNN vs ViT, RL vs supervised learning problems). Can authors provide why the proposed method is significantly novel or overcomes challenges in using self-supervised learning for RL?
- Also, could authors provide the results of applying the proposed methods on CNNs? If the CNN performance can't be improved, why so?
summary_of_the_review: The submission is limited in terms of both theoretical novelty and empirical novelty. Especially, the empirical validity is pretty limited due to small set of experiments, missing experiment details, etc. More work should be done to warrant the publication of this work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper mainly targets usage of ViTs in Reinforcement Learning. The authors conduct several experiments with different training strategies for ViTs, showing the effectiveness of self-supervised learning for ViTs in RL. Besides, the authors propose a new self-supervised objective function based on VICReg, which involves the temporal order prediction for learning better temporal relations.
strength_and_weaknesses: Strength:
1. The authors present interesting and trials with ViTs for RL.

Weaknesses:
1. The novelty is limited. The proposed method is made up of several previous methods.
2. Even though the proposed TOV-VICReg is more effective than the other self-supervised methods, it is still only comparable to the CNN based models with no significant advantage. In other words, the ViT based models remain impractical for RL.
3. It would be better if the authors can consider reconstruction based self-supervision like MAE and SimMIM in their experiments, which adopts a quite different objective compared with DINO and MoCo.

clarity,_quality,_novelty_and_reproducibility: This paper is generally clear. The novelty is limited due to no originated method is proposed in this paper. The TOV-VICReg is a combination of two existing methods. The proposed method should be easy to reproduce based on the authors' code.
summary_of_the_review: The authors present good technical analysis of using ViTs for RL problems. The novelty and significance are limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to introduce visual transformers to extract feature representations for reinforcement learning. It proposed a method, which is a combination of VICReg and temporal ordering prediction, to pretrain the ViT network. Experiments are done on Atari 100K using the visual transformer pretrained by the proposed method. 
strength_and_weaknesses: ** Strength: **
1. It is reasonable to combine the image only pretraining method (specifically, VICReg) with temporal ordering prediction as the overall objectives covers both image and temporal information. 
2. The paper analyzes the property of representation from a few different aspects in the experiments. 

** Weaknesses: **
1. It is unclear how the pretraining is specific to deep reinforcement learning. It looks like a generic video-based pretraining method, and its connection with reinforcement learning seems a bit loose. 
2. The novelty of the method is limited as combining two established self-supervised learning methods in the same training objective is not something difficult to come up with. 
3. Q: it seems Atari 100K has 26 games (please correct me if I am wrong), why only 10 games are used?
4. The experimental results are not strong. Especially, using visual transformer does not seem to be that competitive to using CNN, as the paper claimed. 
5. The clarity of the paper (especially in the experiment section) can be improved. 
clarity,_quality,_novelty_and_reproducibility: The clarity and writing quality of the paper can be improved. One example, the mode performance analysis in Sec 6.1 is unclear. It is not self-contained enough for one to understand the metric. And it is confusing if the results focus only on “sampling efficiency” or actually looks at the model’s game performance. 

The papers have a few broken links. 

While it could claim the method is novel for RL, but the specific connection between the proposed method and RL is not clear to me. And a generic method to pretrain a feature extractor on video. Such a combination of image-only objectives and temporal ordering prediction is marginally novel. 

The amount of details in the paper is sufficient for one to understand the main idea but may not be sufficient for one to reproduce the results fully. 
summary_of_the_review: The paper has a good motivation of using a well-pretrained visual transformer in deep RL. However, the method design seems to be generic and of marginal novelty. The experimental results are a bit confusing and do not seem to be staring enough to address the paper’s motivation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: *I am a vision person and an emergency reviewer.*

This paper proposes to combine VICReg and a temporal verification loss for video representation learning, and inspect its impact on the data-efficiency of down-streaming reinforcement learning tasks. It is shown that the proposed method out-performs VICReg, MOCO and DINO while still under-performing (randomly initialized?) CNNs. Further analyses show that the proposed variant of VICReg shows better properties in terms of collapse and feature distribution.
strength_and_weaknesses: Strengths:
+ The proposed training framework that combines VICReg and temporal verification is new to my knowledge.
+ Experimental results show that the variant out-performs alternative representation learning methods in the inspected RL setting.
+ A set of analysis shows that the method shows better statistical properties w.r.t. collapse and feature similarity.

Weaknesses:
- Most of the experimental results seem not related to reinforcement learning and the only RL evaluation reports inferior performance than (randomly initialized?) CNNs. I recommend the authors to consider generic video classification evaluation protocols in [A], which may be a more promising thing to pursue.
- The language is fine but the organization is pool IMHO. Too much space is spent on describing standard things like MDP, DQN or VICReg.
- Generally I am not convinced by the motivation. I think transformers are widely considered less data-efficient and I don't understand why authors expect it to behave the opposite way in reinforcement learning.
- Typos:
components that allows us -> allow
the increasing interest -> and the increasing interest
ViT presents weaker image-specific inductive biases which allow the CNNs for much sample-efficient learning; allows CNNs?? what does this mean

[A] VideoMoCo: Contrastive video representation learning with temporally adversarial examples, CVPR 2021
clarity,_quality,_novelty_and_reproducibility: Clarity: I am not very sure about the RL protocol. Is the representations trained in an offline setting that pre-stores frames?
Quality: not good enough for both technical depth or presentation quality.
Novelty: The training framework is somewhat new.
Reproducibility: Codes are provided.
summary_of_the_review: This is an empirical study yet I am not convinced by the motivation or the significance of empirical results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies if Vision Transformers (ViTs) could outperform CNNs in vision-based RL tasks. The authors compared existing self-supervised strategies and proposed a new self-supervised approach designed for the sequential observations in RL. The newly proposed method combines previous ideas from VICReg (Bardes et al, 2021) and Shuffle-and-Learn (Misra et al, 2016). The authors show that the new approach TOV-VICReg outperforms other SOTA self-supervised learning methods in the RL setting. However, from the authors experiments, CNNs still outperform ViTs trained with TOV-VICReg in eight out of the ten tested Atari games. 
strength_and_weaknesses: ### Strength

1. The proposed self-supervised learning method is well suited for RL tasks and experiment results show that it improves performance of self-supervised learning methods that does not capture the temporal relationship.

### Weaknesses

1. This paper lacks novelty in terms of the proposed self-supervised learning method. Also, the presented results are mostly well-known in the community.   
2. Temporal order verification is one type of self-supervision loss that aims to capture temporal dependency. This paper does not justify the choice of Shuffle-and-Learn against other methods [1, 2, 3]. I’d recommend the authors to add a discussion on relevant self-supervised learning literature for temporal relationship learning and provide a justification on the choice of Shuffle-and-Learn.   
3. The number of steps used for temporal order verification loss is an important hyperparameter that would affect the quality of the learned representations. This paper does not provide experimental results showing its impact. I’d encourage the authors to add an experiment varying the number of steps.

### References

[1] Lee, Hsin-Ying, et al. "Unsupervised representation learning by sorting sequences." *Proceedings of the IEEE international conference on computer vision*. 2017.

[2] Xu, Dejing, et al. "Self-supervised spatiotemporal learning via video clip order prediction." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2019.

[3] Yao, Yuan, et al. "Video playback rate perception for self-supervised spatio-temporal representation learning." *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020.
clarity,_quality,_novelty_and_reproducibility: ### Quality

This paper should be improved in terms of clairty and experiments before being accepted into ICLR. 

### Novelty:

The paper presents an incremental contribution where ViTs are trained by combining the temporal order verification loss (Misra et al, 2016) and the VICReg loss (Bardes et al, 2021). Both the methods and the results do not present significantly contributions.

### Reproducibility

The authors provided their code in the supplemental material. 

### Clarity

1. Figure 4 and 5 are confusing becuase of the different scales of the color map. 
2. The desription of the baseline methods is not clear. For the SGI ResNet Large, was it pretrained with self-supervision loss? If so, did the pretraining follow the original SGI paper? 

Minor Comments:

1. The DQN algorithm details on page 3 can be presented in an appendix and formatted with Latex algorithm packages. 
2. VICReg is a published ICLR 2022 paper. Please update the reference accordingly. 
3. For audience unfamiliar with VICReg, it would be helpful to explain how expanders work.
summary_of_the_review: I’m inclined to reject this paper mainly because: 1) the experiments do not justify the motivation of this work, i.e., use large models to solve complex RL tasks where smaller models struggle, as stated in the conclusion of this paper. Clearly, the chosen environments are not complex enough to demonstrate the potential benefits of pretrained ViTs. 2), the proposed method lack novelty and the results do not bring significant new insights to the community, 3) the paper does properly compare other self-supervised learning methods that aim to capture the temporal relations  4) the paper presentation needs further polishing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers feel negative about the current submission. The raised issues include limited main contribution and technical novelty, unclear presentation, and experimental validations. Although the authors try to address these issues, the raised novelty issue is hardly solved. After checking all the reviews and rebuttals, the AC feels the current work needs a significant revision to address the raised issues. The authors shall take these suggestions to make further improvements.
justification_for_why_not_higher_score: All reviewers feel negative on the current work, the ac feels the same.
Decision: Reject