ARVideo: Autoregressive Pretraining  for Self-Supervised Video Representation Learning
Authors: Sucheng Ren
Hongru Zhu
Chen Wei
Yijiang Li
Alan Yuille
Cihang Xie
Venue: Accepted by TMLR ()

Abstract: This paper presents a new self-supervised video representation learning framework \textbf{ARVideo}, which \textit{autoregressively} predict the next video token in a tailored sequence order.
Two key designs are included. First, we organize autoregressive video tokens into 
clusters that span both \textit{spatially} and \textit{temporally}, thereby enabling a richer aggregation of contextual information compared to the standard spatial-only or temporal-only clusters. Second, we adopt a randomized spatiotemporal prediction order to facilitate learning from multi-dimensional data, addressing the limitations of a handcrafted spatial-first or temporal-first sequence order. Extensive experiments establish ARVideo as an effective paradigm for self-supervised video representation learning. For example, 
when trained with the ViT-B backbone, ARVideo competitively attains 81.2\% on Kinetics-400 and 70.9\% on Something-Something V2, which are on par with the strong benchmark set by VideoMAE.  Importantly, ARVideo also demonstrates higher training efficiency, \ie, it trains 14\% faster and requires 58\% less GPU memory compared to VideoMAE.

Published in Transactions on Machine Learning Research (anuary/2025)
ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning
Sucheng Ren
obvvrreesuu@gmmilcomm com
Johns Hopkins University
Hongru Zhu
hongruz95@@maiil.com com
Johns Hopkins University
Chen Wei
weiccenn30128mmaai.cco
Johns Hopkins University
Yijiang Li
ligfitang30000gmaai..om com
Johns Hopkins University
Alan Yuille
syuille1@jhuz edu
Johns Hopkins Universsty
Cihang Xie
changgzie360@@maiilcoom com
UC Santa Cruz
Reviewed on OpenReview: https: // openreview. net/ forum? id- TRKuzPnXKQ
Abstract
This paper presents new self-supervised video representation learning framework ARVideo, which antoregressively predict the next video token in tailored sequence order. Two key designs are included. First, we organize autoregressive video tokens into clusters that span both spatially and temporally, thereby enabling a richer aggregation of contextual information compared the standard spalial-only or temporal-only clusters. Second, we adopt randomized spatiotemporal prediction order facilitate lcarning from multi- dimensional data, addressing the limitations of handcrafted spatial-first or temporal-firt sequence order. Extensive experiments establish ARVideo self-supervised video representation learning. For example, when trained with the ViT-B backbone, ARVideo competitively attains 81.2% on Kinetics-400 and 70.9% on Something- Something V2, which are on par with the strong benchmark set by VideoMAE.. Importantly, ARVideo also demonstrates higher training efficiency, í.e., it trains 14% faster and requires 58% less GPU memory compared to VidcoMAE.
effective paradigm for
1 Introduction
The transformer architecture, introduced in Vaswani et al. (Vaswani et al. 2017), has fundamentally transformed the field of natural language processing (NLP) through its ability to model long-range dependencies with minimal inductive bias. A crucial catalyst for its success lies self-supervised learning of robust and transferable representations from large volumes of unlabeled data. Within this paradigm, masked language modeling (MLM) (Devlin al. 2019) and autoregressive modcling (AR) (Radford al., 2018 Brown al, 2020: OpenAl 2023) stand out two leading approaches. Specifically, MLM masks random portions of input tokens and trains models to predict masked elements; whereas AR predicts subsequent words in sequence based on all preceding words. These methods have propelled state-of-the-art performance in various NL.P tasks.
In the video domain, however, the landscape is different. Previous studies have predominantly relied on supervised protraining using image datasets, often overlooking the critical aspect of temporal dynamics (Liu) et al. 2022b Bertasius al. 2021). Recently, there has been a shift towards leveraging NLP-inspired mask language modeling (Devlin et al. 2019) image-inspired mask image modeling (He et al. 2022 Bao et. al.

---

Published in Transactions on Machine Learning Research (January (lamary/2025) 20225)
Group video tokens
Tokenization
Clustering
Input video
Vidco Tokens
3
4
Random permutation
Cluster
5
4
8
3
K
Encoder
Encoder
Encoder
Encoder Block
Decoder
Block
Block
Block
C
Block
Head
Embedding
V
Figure ARVideo autoregressive predicts spatiotemporal cluster from grouping tokens span spatial and temporal dimension. Following (Tong et al., 2022), first tokenize the video into discrete tokens. These tokens are grouped into clusters, each spanning both spatial and temporal dimensions. Finally, we randomly permute the order of these clusters and adopt an autoregressive strategy to predict subsequent clusters.
to directly exploit unlabeled video datasets for pretraining, For instance, VideoMAE Tong et al. Feichtenhofer et al.) 2022) introduces mask autoencoder (He et al. 2022) for self-supervised video video representation learning: BEVT (Wang et al. 2022a) learns spatial representations from image data and joint-masked image and video modeling. Despite these advancements, autoregressive modeling another powerful sulf-super learning approach in NLP has yet to be extensively explored within the context. of video data analysis. Specifically, considering the strong temporal structure in videos where each frame depends previous frames, AR is able naturally capture those dependencies by modeling the next frame (or features of the next frame) given past frames and to learn how causal changes occur in videos, expecting to induce more robust and nuanced feature representations for facilitating downstream tasks such as action recognition, event classification, or anomaly detection.
2022
2022
Critically, in the practical instantiation, applying autoregressive pretraining to video data entails the same principle autoregressively predicting the next element in a sequential order based on predecessors. Tn natural language, these elements words are clearly defined and inherently follow a chronological order. For images, elements could be conceptualized pixels patches arranged in flattened sequence (Chen et al. 2020: El-Nouby et al. 2024 Ren et al. 2024h. The further transition to video data, however, introduces additional complexity due to its inherently multidimensional nature (i.e., including both spatial and temporal dimensions). This raises a crucial inquiry: how should we define an auloregressive 'video element' and establish a visual sequence order for self- supervisd video representation learning?
We note traditional methods, such as converting vidco into a sequence of cubes (Tong et al., 2022 Bertasius et al.] 2021 Wang al.) 2022a Liu et al., 2022b) and subsequently linearly mapping these cubes into video tokens, generally reveal critical limitations in addressing this query. Specifically, the granularity of these video tokens often fails to encapsulate the rich semantics typically represented by words in text-based models primarily because 1) these video tokens are too dimensionally limited, and 2) video inherently lacks a sequential order in its spatial dimensions, alt although retains this ure in its temporal aspects.
To address these challenges, we hereby present ARVideo, a novel autoregressive-baaed vidco representation learning paradigm with two key designs (see Figure D Firstly, we redefine 'video elements' by grouping video tokens into spatiotemporal video clusters, differentiating from conventional single-dimensional strategies
2

---

Published in Transactions on Machine Learning Research (anuary/2025)
like spatial video clusters temporal video clusters. This approach improves semantic representation by aggregating more contextually relevant multidimensionall information. Secondly, we find that, compared to well-defined yet single-dimensional spatial-first or temporal-first sequence orders, a sequence order that randomly integrates both spatial and temporal dimensions empirically yields significantly stronger results. This suggests that effectively capturing the inherent mullidimensionality of video data crucial for autoregressive modeling. Extensive experiments establish our ARVideo as an effective paradigm for video representation learning. For example, while the autoregressive video representation learning baseline only attains 74.2% Kinetics-400 and 66.4% on Something-Sometiine (+7%) and 70.9% (+4.5%), respectively. Notably, these results not only match but, in some aspects, surpass the strong benchmark set by VideoMAE, particularly with respect to training efficiency ARVideo achieves faster training speeds by 14% and reduces GPU memory consumplion by 58%.
V2, ARVideo significantly boosts the results to 81.2%
2 Related Work
2.1 Video Representation Learning
Video representation learning has witnessed significant exploration, historically driven by supervised learning
methods (Tran et al.] 2018: Wang et al.] 2019 Ren et al. 2022 Simonyan & Zisserman 2014 Ren et al.,
2021b a Bertasius et al. 2021 Ren et al., 2020 2023
Liu et al. 2022b) that pretrain backbone networks
on labeled image or video data before fine-tuning. However, such methods face challenges due to inherent discrepancy between image and video data, compounded by the scarcity of comprehensively labeled video datasets.
In the era of self-supervised learning, recent work have designed pre-tasks incorporating temporal information for self-supervised video representation learning {Xu et al.) 2019 Benaim et al.) 2020 Huang et al. 2021
Qian et al. 2021: Ranasinghe et al., 2022) and leveraging contrastive learning for effective visual representa-
tions (Qian et al. 2021 Kuang et al. 2021 Li et al. 2021 Diba el al., 2021 Han el al. 2020a by. Additional,
mask reconstruction based methods inspired by masked language modeling (Dellin al.) 2019) are introduced
into self-supervised image and vidco reprosentation learning. For example, MAE (He et al.
2022 presents
a scalable self-supervised learning method to reconstruct masked image patches while VideoMAE (Tong
et al. et al. b022b separates spatial learning from temporal dynamics, training on masked images initially before
2022 extends this approach to video data and reconstructs masked spacetime patches. BEVT Wang
jointly on masked images and videos. Christoph et al. (Feichtenhofer et al.) 2022) introduce an efficient video-based MAE extension with minimal biases and signilicant. speedups. In contrast. to prior works, our
ARVideo proposes a new path for self-supervised video representation learning via autoregressive pretraining.
2.2 Autoregressive Pretraining
As a representative approach for autoregressive pretraining, Generative Pretrained Transformer (GPT) trains language models by autoregressively predicting the next word based all preceding words in sentence of length n, denoted as {ui,.., J. The autoregressive loss minimizes the negative log- ligelioel with model parameter 0:
L = -log I (),
(1)
This modeling strategy has fundamentally changed the landscape of natural language processing, leading to the development of tremendously successful models like ChatGPT (Radford et al., 2018) and GPT-4 (OpenAll 2023). Inspired by the success of autoregressive modeling in NLP, researchers start to apply autoregressive pretraining in computer vision. ImageGPT (Chen et al.) 2020) learns effective image reprosentations by training a Transformer to autoregressively predict. image pixels without any prior knowledge of their 2D structure. Nevertheless, ImageGPT incurs significant computational overhead due to the quadratic complexity of self-attention w. the input soquence, limiting itsclf to smaller image sizes (c.g., 32 x 32) with suboptimal performance. lengthSAIM (Qi al. 2023) adopts an encoder to autoregressively learn contextual information like standard vision transformer (ViT) and decoder to predict the current. content, mutually reinforcing
3

---

Published in Transactions on Machine Learning Research (January /202))
Time
(b) Spatial Cluster
(c) Temporal Cluster
(d) Spatiotemporal Cluster
(a) Video Tokens
Figure Comparison between video token and different cluster. ARVidco groups multiple tokens into (b) Spatial cluster across the spatial domain, (c) Temporal cluster across the spatial domain, (d) Spa- tialtemporal cluster across both the spatial and temporal domain.
each other's functions. RandSAC (Hua et al. 2022) arranges image tokens into segments for parallel intra- segment and sequential inter-segment autoregressive prediction. However, applying autoregressive pretraining video data faces notable challenges due to the extra temporal dimension. ARVideo explores the design of autoregressive video elements and visual sequence orders for video representation learning.
Recently, AR modeling has been adopted for video generation works Yu et al. 2024 Kondratyuk et al. 2024
In contrast, as a video representation learning framework, ARVideo incorporates some key design differences, including (1) simpler tokenizer as opposed to a specialized tokenizer (Yu et al., 2024), (2) cluster-wist AR modeling-white operates on groups of tokens-as opposed to token-wisc AR modcling using Large Language Models (LLMs) for video generation (Kondralyuk al.) 2024), and (3) random rasterization for
spatialtemporal prediction order as opposed to predefined orders optimized for video generation tasks. These distinctions advance ARVidco upon existing works for improved self-supervised learning outcomes.
3 Method
In this section, we present ARVideo and analyze the the design of elements and the optimal prediction order as the key ingredients in ARVidco for autoregressive prediction with vidcos. Notably, throughout this paper, we use the term 'clustering' to refer to the predefined grouping of tokens along spatial and temporal dimensions, rather than the conventional meaning of unsupervised grouping based on similarity.
Illustrated in Figure 1 ARVideo autoregressively pretrains video data RT*HXWXC Note that directly extending ImageGPT to videos faces significant challenges, primarily due to the added temporal dimension, which would significantly escalate computational demands, even with low-rcsolution videos like 4 32 32. Moreover, pixels as autoregressive elements lack semantic richness compared to words in the language, further necessitating pixel grouping strategies to enhance representation learning. To better facilitate learning from multi-dimensional video data, we also explore prediction orders across spatial and temporal dimensions.
3.1 Pixel grouping
From Pixels to Video Tokens. With patch embeddings in ViT, videos can be patchified into non- overlapping cubes [Togg al. 2022 Bertasius et al., 2021 Wang et al.) 2022a Liu et al. 2022b) of size Pr x Pw Pи- Then, each cube is transformed into a vidco token through a lincar projection layer, resulting in N = P5 x Psr
H W video lokens. This tokenization significantly reduces operational elements,
x Tw
thus alleviating computational demands while ensuring that each video token encapsulates richer semantics compared individual pixels. For example, as reported in Table [ using video tokens as autoregressive elements for pretraining significantly outperforms approaches without tokenization by 3.3% while keeping pretraining resolution consistent with previous work (Tong et al. 2022 Wang et al. 2022a
4

---

Published in Transactions on Machine Learning Research (annuary/2025)
Flement
Resolution
Something- Something V2
Pixel
8 14 x 14
60.7
Token
16 x 224 x 224
64.0
Table 1: Grouping pixels into video lokens facilitates autoregressive pretraining on ligher-resolution videos and improves performance by 3.3%.
This promising transition from pixels to video tokens introduces compelling query: Can further performance gains be realized by aggregating more tokens? In pursuit of this, we examine three options: grouping video tokens into spatial, temporal, or spatiotemporal clusters. Tt important note that within each cluster, video tokens are always fully attendod to cach other. This full-attention configuration helps to enable a more effective consolidation of semantic content within each autoregressive element.
From Tokens to Spatial Clusters. As shown in Figure 2(b), we strategically group spatially neighbored tokens-thon sharing the same temporal positions but varying spatially into spatial clusters. Following the patch embedding step, video tokens within the spatial domain Pn Pw are grouped into one element, T
resulting in P.. antoregressive elements. For example, video of size 16 x 224 224 with cube embedding size of 2 x 16 x 16 (Tong et al. 2022) here will be transformed into 8 autoregressive elements, with each element comprising 14 x 14 tokens.
From Tokens to Temporal Clusters. As illustrated in Figure 2(c,, our method integrates temporal information by grouping tokens that are temporally adjacent into temporal clusters. Specifically, tokens within the temporal domain Pr are grouped into one element, resulting in H. W PH Pw autoregressive elements.
For instance, vidco of size 16 x 224 x 224 with a cube embodding size of 2 x 16 x 16 (Tong et al. 2022
here will Transformed into 14 x 14 autoregressive elements, with each element comprising 8 Lokens.
From Tokens to Spatiotemporal Clusters. Moving beyond the single-dimensional grouping strategies discussed above, now consider the inherently multidimensional nature of video data by grouping neighboring KT KH x Kw tokens into spatiotemporal clusters with no overlaps, as illustrated in Figure dd. This T strategy results in total number of Ñ = Ky X K X Pw clusters, with each containing both spatial and temporal information as an autoregressive element. ARVideo autoregressively predicts the next cluster given all preceding clusters:
H
W
Ñ L = - Il G-1,00).....1000))
(2)
Cluster captures the amount of high-level, semantically meaningful information, which is greater when multiple tokens are grouped together, as opposed to isolated low-level descriptors such as individual pixels or tokens.
3.2 SpatialTemporal Prediction Order
For the spatiotemporal cluster, we further explore its prediction order. Specifically, this strategy is expected to yield K7
T clusters at each spatial position, and H PHk#
X
Kw
W clusters at cach temporal position.
We implement two systematic strategies: a spatial-first order and a temporal first order. W The spatial-first approach prioritizes autoregressive pretraining within the Py X Pw spatiotemporal clusters along the spatial dimension, before transitioning to clusters in subsequent. temporal positions. Conversely, the temporal-fist approach prioritizes within the T Pr spatiotemporal clusters along the
Pre-defined order.
temporal dimension, then proceeds to clusters in subsequent spatial positions.
Random Rastcration. Inspired by the random sentence permutation technique used in XLNet Yangg et al. 2019) for enhancing autoregressive pretraining, our random rasterization approach scrambles the order of clusters randomly during autoregressive pretraining after adding positional embedding. By randomly
5

---

Published in Transactions on Machine Learning Research (anuary/2025)
reordering sampling frame sequences, the model encounters more diverse temporal variations and avoids overfitting to fixed frame order. Besides, random permutation makes model learn short- and long-term dependencies. Specifically, given a current frame t, the model is required to predict t+n for various n values (ranging from 1 to the final frame), thereby training it to handle both immediate and extended temporal relationships in the video. This method avoids the const of fixed sequential patterns, such as spatial-first or temporal-first, and allows ARVideo to adaptively model both long- and short-range spatial- temporal information. Such flexibility in autoregressive prediction orders not only captures the inherent multidimensionality of video data more effectively but also fosters richer, more comprehensive video representation. We adopt this random order as the default experiment setup.
3.3 Model Architecture
We adopt the ViT (Dosovitskiy et al. 2021 Tong et al. 2022) as the encoder. For the decoder, we take the Transformer decoder with cross attention but without ceff-attention. This design choice aims to simplify the decoding process, emphasizing interaction between the encoded inputs while reducing training costs. The query of the decoder is randomly initialized but includes position information to facilitate sequence generation. Our model utilizes a stralegically designed altention mask in previous work (Chen et al. 2020 Radford et al. 2018) to enable efficient autoregressive prediction in parallel computation framework. When transforring to downstream tasks, we remove the decoder and only finetune the encoder. We aggregate the encoder's 's output tokens using average pooling into a single, representative token. This pooled token then serves as the input for the final classification layer.
In our implementation, we employ a square error (MSE) loss to mcasure the discrepancy between the predicted and target cubes, utilized in MAE (He et al. 2022). Typically, predicting Ñ clusters requires multiple iterations: given Cı, we predict 62; given C1 and C2. we predict c3: and so on, outlined in Eq. 2 To
improve training efficiency, we follow the approach of (OpenAl 2023 Al 2023 Hua et al. 2022 Chen et al., 2020) by leveraging attention masks to compute all clusters in a single iteration. Specifically, we concatenate
all clusters inlo one sequence and apply attention masks to ensure that each cluster aliends only to itself and the preceding clusters. The details of the training and finctuning can be found in Appendix A
4 Experiment
4.1 Dataset and Implementation Details
We primarily evaluate ARVideo on Kinetics-400 (ay et al. 2017 and Semething-Something V2 (Goyal et al.,
2017). Specifically, Kinetics-400 contains 400 classes and 260k videos of 10s, with 240k for training and 20k for validation; Something-Somcching V2 contains 174 classes with 169k vidcos for training and 25k for validation. While Kinetics-400 provides a broad spectrum of actions with minimal context, SaanetthnnSSomehhige V2 focuses more on the interaction of actions with objects.
2022 with ARVideo, and then fine-tune the pretrained model on the target action recognition datasets. Additionally, we assess the feature transferability on AvA v2.2 (Gu et al.] 2018) and HMDB (Kuehne et al.)
For our experiments, we first pretrain vanilla video Transformer (Tong et al.
2011 AvA v2.2 is a human action localization dataset with 211k videos for training and 57k for validation; HMDB is small video dataset with 3.5k videos for training and 1.5k videos for validation.
We follow the established protocol prior work (Tong et al., 2022) to train our models. Instead of using negative log-likelihood as in GPT (Radford al.] 2018), ; we employ mean square error (MSE) loss measure the discrepancy between the predicted and target cubes, as utilized in MAE (He et al. 2022). We randomly mask 80% tokens in each element in encoder to reduce the overall training costs; note that, unlike MAE or VideoMAA, we do not reconstruct those masked regions. Please refer to Appendix for model architecture and training implementation details.
Additionally, we would like stress that, although AR modeling leveraged here, the video generation quality of our pretrained model not the focus of this insted, we focus on evaluating the finetuning performance of these pretrained models on a range of downstream video benchmarks.
6

---

Published in Transactions on Machine Learning Research (aauary/2025)
Method
Backbone
pretrain
Epoch
Frames
GFLOPs
Param
Top-1
Supervised pretraining
TANet (Liu al. 2021
15sNet152
IN-1K
100
16
242x4x3
59
79.3
TDN En (ang al. 2021
ResNet101
IN-1K
100
8+16
198x10x3
TimeSformer Bertasi iis ot al
2021
VIT-B
IN-21K
15
196x1x3
121
78.3
Motionformer Patrick et al.
2121
ViT-B
IN-21K+K400
35
16
370x 1x3
109
81.1
Video Swin (Liu
2022x
Swin-B
K400
30
321x1x3
82.7
Mask video modeling
VIMPAC (Tan et al.
2021
ViT-L
HowTo100M
100
10
N/Ax10x3
307
77.4
BEVT (Wang et al.
2022a
Swin-B
K400
150
282x1x3
76.2
VideoMAE Long et al. 2022
ViT-B
K400
800
16
180x2x3
87
80.0
VideoMAE Tong et al.
2022
ViT-B
K400
1600
16
180x2x3
87
81.5
Autoregressive pretmaining
1GPT (Chen et al. 2020
VIT-B
IN-1K
300
16
180x2x3
87
61.2
Randsac (Iua
2022
VIT-B
IN-1K
1600
16
180x2x3
70.3
To
ViT-B
IN-1K
300
180x2x3
TokenGPTi
VIT-B
K400
800
16
180x2x3
87
74.2
ARVideo
VIT-B
K400
800
16
180x2x3
87
80.1
ARVideo
VIT-B
K400
1600
180x2x3
81.2
Table 2: Comparison with the state-of-thh-art methods Kinetics-400. "N/A" indicates the numbers are not available for us. i indicates the implementation by us with the token replacing pixel in iGPT. Note that random order is adopted in sequencing clusters inside our ARVideo.
Method
Backbone
Pretrain
Epoch
Frames
GFLOPs
Param
Top-1
Supervised pretraining
TEINet
Liu et. al.
2020
Re=Nes50xa
IN-1K
50
8+-16
99x 10x3
50
66.5
TANet
2021
Re
IN-IK
50
8+116
99x2x3
66.0
TDN Km.
Wang el. al.
2021
ReaNet101x2
IN-1K
60
8+11
198x1x3
69.6
SlowFast
Feichtennooer et al.
2019
ResNet101
K400
196
8+-32
106x1x3
53
63.1
MViTvl
Fan al. 2021
MViTv1-B
K400
100
64
455x1x3
67.7
TimeSformer
Dertasius et
2021
ViT-B
IN-21K
15
196x1x3
TimeSformer [errtastus et al
2021
ViT-L
IN-21K
15
64
5549x1x3
430
62.4
VIVIT FE (Arusb et sl.
202
VIT-L
IN-21K+K
35
32
995x4x3
N/A
65.9
Motionformer Patrick
2021
VIT-B
IN-21K+K400
35
16
370x1x3
109
66.5
Video Swin (Liu et al. 2022a
Swin-B
IN-21K+K<00
30
32
321x1x3
69.6
Mask video modeling
VIMPAC (Tau et
2021
ViT-L
HowTo100M
100
N/Ax10x3
68.1
BEVT (Wang et al.
2022a
Swin-B
IN-IK K400
150
32
321x1x3
88
70.6
MaskHearp312 Wei
2122
MVTT-L
K600
1600
40
2828x 1 x3
218
75.0
VideoMAE: Long al.
2022
VIT-B
SSv2
800
180x2x3
69.6
VideoMAE Tong al.
2022
VIT-B
SSv2
2400
16
180x2x3
87
70.8
Autoregresaive prefraiming
:GPT Chen et al.
2020
VIT-B
IN-1K
300
180x2x3
Randsac Ima et al.
2022
VIT-B
IN-1K
1600
16
180x2x3
87
59.6
TokenGPT*
VIT-B
IN-1K
300
16
180x2x3
87
59.2
TokenGPTi
VIT-B
SSv2
800
16
180x2x3
66.4
ARVideo
VIT-B
SSv2
800
180x2x3
ARVideo
VTT-B
SSv2
2100
16
180x2x3
87
70.9
Table 3: Comparison with the state-of-the-art methods on Sonnethin-SSomething V2. "N/A" indicates the numbers are not available for us. t indicates the implementation by us with the token replacing pixel in iGPT. Note that random order is adopted in sequencing clusters inside our ARVideo.
4.2 Main results
Kinetics-400. We pretrain the ViT-B backbone for both 800 and 1600 epochs on Kinctics-400, and report the corresponding results Table Spatiotemporal cluster and random rasteration order are adopted. Notably, ARVideo attains 80.1% top-1 accuracy under 800 epochs and 81 31.2% top-1 accuracy under 1600 epochs,

---

Published in Transactions on Machine Learning Research (annary/2025)
Method K400 + AVA v2.2
K400 + HMDB
Contrastive Learning
MoCo
67.9
Mask video modeling
VideoMAE
26.7
73.3
Autoregressive pretraining
ARVideo
26.9
74.1
Table 4: Comparison of model transferability. We first pretrain models on Kinetics-400, and then transfer them to AVA v2.2 and HMDB.
Encoder
Decoder
Method
Training Time
GPU Memory
Key/ /aa/lalu
/Va//
VidcoMAE
160
160
1568
1568
145h
41.3G 26.1G (-36.8%)
ARVideo
300
300
1372
300
127h (-12.4%)
Table 5: The comparison of pretraining time and GPU memory.
exhibiting significant improvements over previous auloregressive methods. Specifically, taking 1600-epoch- pretrained ARVideo for comparison, it outperforms iGPT, the baseline model, by striking +20.0% and Randsac, the previous state-of-tho-art autoregressive model on images, by +10.9%. Additionally, compared to TokenGPT, which performs loken-level autoregressive prediction, ARVideo showed advancements of +12.7% when TokenGPT was pretrained on an image dataset, and +7.0% when it was pretrained on the Kinetics-400 dataset.
Moreover, we note that ARVideo performs competitively against. the strong benchmark the mask video modeling method, VideoMAF. For example, the performance difference between ARVideo and VideoMAE) is only 0.1% with 800 epochs of pretraining; this margin remains minimal at 0.3% with 1600 epoch pretraining. These results validate the effectiveness of ARVideo as a pioneering autoregressive pretraining method in self-supervised video representation learning, equalling- and in some aspects surpassing- the performance of established mask modeling methods.
Somnthhing-Something V2. We pretrain the VIT-B backbone for 800 and 2400 epochs on the Something- Something V2 dataset. Spatiotemporal cluster and random rasteration order are adopted. As reported in Table 3 ARVideo achieves top-1 accuracies of 69.8% and 70.9% for 800 and 2400 epochs, respectively, which are significantly stronger than prior autoregressive pretraining methods. For example, under 2400 epochs, ARVideo surpassed the baseline model iGPT by +16.6% and outperforms the best-performing image-based autoregressive method, Randsac, by +11.3%. also surpassed TokenGPT pre-trained on image datasets by +11.7% and on the Something-Sometiiee strong masked vidco modeling method VidcoMAE, ARVideo also performs competitively in both 800 epochs of pretraining (i.., 0.2% accuracy difference) and 2400 epochs of pretraining (f.e., 0. 0.1% 1 accuracy difference). Together with the observations in Kinetics-400, these results establish ARVideo strong alternative to masked modeling approaches for vidco analysis.
V2 dataset by +4.5%. Additionally, when compared to the
Transfer Learning. To investigate the feature transferability of ARVideo, we transfer the model trained on Kinetics-400 to AvA v2.2 and HMDB. We can observe that ARVideo demonstrate strong transferability, achieving 26.9 mAP on AvA v2.2 and 74.1% Top-1 accuracy on HMDB outperforming both VideoMAE and MoCo (sce Table For example, compared to VidcoMAE, ARVideo shows (slight) improvements of 0.2% AvA v2.2 and 0.8% on HMDB.
Computation cost. We report the training time and GPU memory usage in Table (with VIT-B trained on Kinetics-400 for 800 epochs, using 8xA6000). Compared to VideoMAE, ARVideo presents significant reductions in both GPU memory usage and training time-ARVee ARVidoo roduces training cost by 12.4% (from 145 hours to 127 hours) and GPU memory consumption by 36.8% (from 41.3G to 26.1G). This advantage stems from ARVideo's shorter sequence length we drop the last. cluster in the autoregressive modeling.
8

---

Published in Transactions on Machine Learning Research (January (lamary/2025))
VideoMAE
1400
ARVideo
1200
Attention Rank
1000
800
400
100
3
5
Layer ID)
Figure 3: The attention rank comparison between VideoMAE and ARVideo
case
KT
Kи
Kw
Token/ /Cube
1
1
1
64.0
spatial cluster
1
H
66.0
PH
spatial cluster
7
66.2
temporal cluster
T
1
1
65.2
Pr
temporal cluster
65.6
spatiotemporal cluster
4
7
7
65.5
spatiotemporal cluster (ARVideo)
2
7
7
66.8
Table 6: Ablation study the cluster shape.
Attention rank. The self-attention mechanism computes attention scores for given input sequence, forming what is known as the attention map. The rank of this matrix can serve as a measure of its ability to capture complex patterns in the data. Typically, high-rank attention matrices suggest. a model that can capture a wide variety of patterns and relationships within the data, while low-rank matrices may suggest model that does not well utilize its full capacity operates on simpler data (Wang et al. 2020). Following this instruction, plot the rank of the attention map in each layer of VideoMAE and our ARVideo in Figure 3 We can observe that, across nearly all layers except the 6t,, ARVidco maintains higher attention ranks than VideoMAE, indicating a stronger representational ability of our model's sell-attention layers.
Compatibility with recent works. We highlight that proposed ARVideo is compatible with several
recent works (Yang et al. 2022 Wang et al.
2023) that incorporate extra motion cues (Yang et al., 2022
or extra knowledge distilation (angg et al.
2023 to VideoMAE. These methods are orthogonal to both
VideoMAE and our ARVideo. As shown Table 7 ARVideo can effectively combined with these methods, yielding consistent improvements. Specifically, ARVideo with motion cues achieves a 0.6% improvement over VideoMAE with motion cues (from 71.8% 72.4%). Similarly, ARVideo with distillation outperforms VideoMAR with distillation by 0.8% (from 73.7% to 74.5%).
4.3 Ablation Study
In this part, we ablate four factors cluster shape, mask ratio, prediction order, and docoder design. Note that, unless otherwise specified, all ablations are conducted the ViT-B backbone with 200 epochs of pretraining.

---

Published in Transactions on Machine Learning Research (anuary//022)
Method
Component
Comething-Sometling V2
VideoMAE MotionMAEII Yang et al. 2022
VidcoMAE
70.8
VideoMAE-+Motion
71.8
MVD
Wang et al. 2023
TideoMAA.+Distillll
73.7
ARVideo
ARVideo
70.9
MlotionARVidoo DistilAARVides
ARVideo-Mottin Yang et al. 2022
72.4
-ediio---sssiilllllii Wang el al. 2023
74.5
Table Compatible with recent self-supervised video representation learning framework.
Mask Ratio
SSv2
Order
SSv2
75%
66.0
Spatial-First.
65.6
80%
66.8
Temporal-Firat
66.0
90%
65.6
Spatial-tempones random
66.8
95%
64.8
Table 8: Ablation study the prediction
Table Ablation study the mask ratio from 75%
order.
to 95%.
Decoder
Method
Something-Somethhng V2
Self-Atten
Cross-Atten
ARVideo
66.8
ARVideo
66.6
Table 10:
Ablation study on the decoder architecture.
Cluster shape. We group neighboring and non-overlapped Ky K KW tokens into one cluster and analyze the effect of different cluster shapes. Three situations are considered: 1) Ky Kw Kи = 1, equivalent. T to the TokenGPT, which pertains autoregressively at the token/cube level; 2) KT P. Kw Kw
Kи = 1, Pg representing a spatial cluster.
representing a temporal cluster; and 3) Kr K, Pw KH W.
We report the results in Table 6. Firstly,
observe that all clustcred configurations significantly enhance performance over the TokenGPT baseline. For example, simply grouping tokens into spa- hsssspptttmpoorrooo clusters yields 2.0%/2.2%/2..8. improvements, respectively. Then, when compar- ing different clusters, we note that our spatiotemporal cluster (ARVideo) with K7 2, Kw KH = 7 attains the best performance of 66.8%, outperforming the best-performed spatial cluster (Kr = 1, Kw K = 7) by 0.8% and the best-performed temporal clusters (Kr 2, Kw KH = 1) by 1.2%. However, it interesting note that, if poorly designed spatiotemporal cluster (Kr Kw KH 7) is used, the performance will drop to 65.5%.
tial
Decoder Width
Decoder Depth
V2
384
66.0
512
4
66.8
768
4
66.8
512
66.2
512
4
66.8
512
8
66.6
Table 11: Ablation study on the decoder depth and width.
Prediction order. In our evaluation of prediction order, which plays an important role in constructing the video sequence, first. check with the predefined spatial-first and temporal-first orders. As shown in
10

---

Published in Transactions on Machine Learning Research (anuary/2025)
Pretraining
Finetuning
Something-Somethhigg V2
Uni-direction Attention
Bi-diroction Attention
66.8
Bi-direction Attention
Bi-direction Attention
49.9
Table 12: Ablation study on the Atlention.
Case
Patch size
V2
Token
16x16 x2
64.0
Token
32x32 x4
53.2
Token
114x114 x4
48.9 (-15.1)
Spatiotemporal cluster
16x16 x2
71.0(+4.6)
Table 13: Ablation study on tokenization with larger patch size.
Table temporal-first order achieves 66.0% top-1 accuracy, which is 0.4% higher than spatial-first order. However, our randomized spatial-temporal prediction order, adept at learning both long- and short-range spatial-temporal dynamics, exhibits superior performance of 66.8%, surpassing the predefined spatial-firtt approach by 1.2% and the temporal-first approach by 0.8%.
Mask Ratio. To reduce the temporal redundancy, ARVideo randomly masks a portion of tokens in Flip (Li et al. 2023 This approach differs from VideoMAE, which relies on bidirectional information to reconstruct masked regions. Instead, ARVideo maintains its core AR paradigm by predicting the next cluster in unidirectional manner. We hereby check how the masking ratio affects the overall performance. As shown in Table our study starts from a mask ratio of 75% (i.e. same as the MAE's setup), which achieves 66.0% top-1 accuracy. Increasing the mask ratio to 80% boosted the top-1 accuracy to 66.8%, while further increases 90% and 95% lower the top-1 accuracies by 1.2% and 2.0%, respectively. We stress that, although ARVideo used a lower mask ratio than VideoMAE, it still enjoys faster training speeds and reduced GPU load (sce Section 4.2 and Table 5
Decoder Architecture. We hereby explore the effects different decoder architectures. As reported in Table 10 whether or not having sell-attention the decoder has little effect on performance (i.e., 66.6% us. 66.8%) but excluding self-attention significantly reduces computational costs. Therefore, we take the decoder without self-attention by default in ARVideo.
Decoder Width and Depth. Lastly, we systematically ablate two critical aspects designing decoders: its width and depth. We start with a four-layer decoder and follow the default setup VidcoMAE. As presented in Table 11 increasing the decoder width shows performance improvement from 66.0% width of 384 to 66.8% at width of 512. Further width increase makes the performance plateau. Meanwhile, in terms of depth, deviations from the foyr-layer standard negatively impacted performance: e.g., increasing to eight layers decreased performance by 0.2%, while reducing to two layers dropped performance by 0.6% (sce the last. three rows in Table
Bi-direction. We employ unidiroctional attention during the pretraining phase to preserve the auloregressive properly, ensuring that each cluster only altends to its preceding clusters. When we replaced this with bidirectional attention during pretraining, we observed a significant performance drop of 16.9% top-1 accuracy as shown in Table 12 This decline is attributed to information leakage, where tokens can access both past and future contexts, thereby disrupting the sequential dependencies essential for effective AR modeling. For downstream tasks, we utilize bidirectional attention leverage comprehensive video representations for classification. This ablation highlights that unidirectional attention is vital during pretraining to maintain the integrity of AR modeling,
Uni-direction
Tokenization vs. Clustering. An alternative to clustering is to use tokenization with a larger patch size, thus mapping more pixels into single token. However, as shown in Table [3 enlarging the patch size 114 114 4 results in notable 15. 15.1% performance drop, whereas our spatiotemporal clustering approach yields 4.6% improvement.
11

---

Published in Transactions on Machine Learning Research (Jamuar/
5 Conclusion
This paper introduces ARVideo for self-supervised video representation learning, inspired by the autoregressive principles of GPT in natural language processing. Diverging from conventional methods, our approach innovatively uses video token clusters as the element for autoregressive prediction, significantly reducing computational demands while still managing to capture essential spatial-tempore dynamics. This advancement improves the efficiency of video data processing and sets new paradigm for self-supervised video representation learning. The promising results obtained from ARVideo underscore its potential and advocate for further exploration and development of autoregressive pretraining methods within the video domain.
Future work. Building upon our current. framework, we believe it would be both interesting and promising to further scale our training data to billion-scale dataset and enhance compatibility with LLMs. These advancements aim to improve the scalability and versatility of our approach, potentially leading to more robust performance and broader applications in video analysis and understanding. Additionally, integrating our method with LLMs could open new avenues for multimodal learning and more sophisticated video interpretation tasks, thereby increasing the overall impact of our research in the field.
Acknowledge
This work is supported by ONR with N00014-23-1-2641.
References
Al. Introducing llama: foundational, bb-bilionparrmmeter language model, 2023.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. Tn ICLR, 2022.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Trani, and Tali Dekel. Speednet: Learning the speediness in videos. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 9922 9931, 2020.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. Tn ICML, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In NAACT, 2019.
Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Luc Van Gool. Vi2cIr: Video and image for visual contrastive learning of representation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1502-1512, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Alaacldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. TCML, 2024.
12

---

Published in Transactions on Machine Learning Research (anuaryy2025)
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenholer. Multiscale vision transformers. In ICCV, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, al. Masked autocncoders as spatiotemporal learners. Advances in neurol informalion processing systems, 35:35946 35958, 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Malerzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vi- jayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for vidco representation learning. In European conference on compuler vision, pp. 312 329. Springer, 2020a.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representation learning. Advances Newral Information Processing Systems, 33:5679-566 06979995500000 2020b.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 180000160000 2022.
Tianyu Hua, Yonglong Tian, Sucheng Ren, Michalis Raplis, Hang Zhao, and Leonid Sigal. Self-supervision through random segments with autoregressive coding (randsac). In The Eleventh International Conference on Learning Representations, 2022.
Deng Huang, Wenhao Wu, Weiwen Hu, Xu Liu, Dongliang He, Zhihua Wu, Xiangmiao Wu, Mingkui Tan, and Errui Ding. Asenet: Self-supervised video representation learning with appearance-speed consistency. In Proceedings of the IEEF/CV CCF International Conference on Computer Vision, pp. 8096-8105 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. Xi preprint Xiv: 705. (05.06950, 2017.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, al. Videopoct: A large language model for zero-shot video generation. ICML, 2024.
Haofei Kuang, Yi Zhu, Zhi Zhang, Xinyu Li, Joseph Tighe, Sören Schwertfeger, Cyrill Stachniss, and Mu Li. Video contrastive learning with global context. In Proceedings of the TEEE/CVF International Conference on Computer Vision, pp. 3195 3204, 2021.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
Rui Li, Yiheng Zhang, Zhaolan Qiu, Ting Yao, Dong Liu, and Tao Mei. Motion-focused contrastive learning of video representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 2105-2114, 2021.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23390-234 2023.
13

---

Published in Transactions on Machine Learning Research (anuary//022)
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022a.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3202-3211, 2022b.
Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Tong Lu. TEINet: Towards an efficient architecture for video recognition. In AAAT, 2020.
Zhaoyang Liu, Limin Wang. Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for vidco recognition. In ICCV, 2021.
OpenAl. Gpt-4 technical report, 2023.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao Henriques. Keeping your eye on the ball: Trajectory altention video transformers. In NeurlPS, 2021.
Yu Qi, Fan Yang, Yousong Zhn, Yufci Liu, Liwei Wu, Rui Zhao, and Wei Li. Exploring stochastic autoregressive image modeling for visual representation. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 2074-2081) 2081, 2023.
Rui Qian, Tianjian Meng, Boqing Gong, Ming Hsuan Yang, Huisheng Wang, Serge J. Belongie, and Yin Cui. Spatiotemporal contrastive vidco representation learning. In CVPR, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Michael S Ryoo. Self- supervised video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2874-2884, 2022.
Sucheng Ren, Chu Han, Xin Yang, Guoqiang Han, and Shengleng He. Tenet: Triple excitation network for video salient object detection. In Computer Eacin-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part V 16, pp. 212 228. Springer, 2020.
Sucheng Ren, Yong Du, Jianming Lv, Guoqiang Han, and Shengfeng He. Learning from the master: Distilling cross-modal advanced knowledge for lip reading. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 13325- 13333, 2021a.
Sucheng Ren, Wenxi Liu, Yongtuo Liu, Haoxin Chen, Guoqiang Han, and Shengfeng He. Reciprocal transformations for unsupervised video object segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 15455-15466 2021b.
Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted sell-attention via multi-scale token aggregation. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 10853-10862. 2022.
Sucheng Ren, Xingyi Yang, Songhua Liu, and Xinchao Wang. Sg-former: Self-guided transformer with evolving token reallocation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6003-6000 2023.
Sucheng Ren, Zeyu Wang, Hongru Zhu, Junfei Xiao, Alan Yuille, and Cihang Xie. Rejuvenating i-gpt for scalable visual representation learning. In ICML, 2024.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. NewrIPS, 2014.
14

---

Published in Transactions on Machine Learning Research (anuary/2025)
Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via. masked token prediction and contrastive learning. arv preprint ar arXiv:2106. 1 11250, 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Adeances in neural information processing systems, 35: 10078-10093, 2022.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. In NewrIPS, 2017.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE TPAMI, 2019.
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. TDN: Temporal difference networks for efficient action recognition. In CVPR, 2021.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR, 2022a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In Proceedings of the IEEE/CW conference on compuler vision and pallern recognilion, pp. 14733 14743, 2022b.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu- Gang Jiang, Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In Proceedings of the IEEE/CVF conference. on computer vision and pattern recognition, pp. 6312 6322, 2023.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with lincar complexity. ar preprint arXiv:2006. 768, 2020.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang, Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, 2019.
Haosen Yang, Deng Huang, Bin Wen, Jiannan Wu, Hongxum Yao, Yi Jiang, Xiatian Zhu, and Zehuan Yuan. Self-supervised video representation learning with motion-aware masked autoencoders. arXio preprint arXiv:2210.04154, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, al. Language model beats diffusion tokenizer is key visual generation. ICLR, 2024.
15

---

Published in Transactions on Machine Learning Research (January
A Implementation Details
All experiments exeept main results follow the same hyperparameters including using the ViT-B backbone with depth of 12 and channel of 768 with 200 epochs of pretraining TokenGPT indicates we perform token wise autoregressive modeling without any grouping or permutation. TokenGPT is a simple baseline of autoregressive modeling for video self-supervised learning.
Architecture: We utilize ViT-B as the backbone of our model. A patch embedding layer with dimensions 2 16 16 is employed to map the video input into video tokens. The encoder comprises 12 blocks, each containing attention layer and an MLP layer, with each block having channel size of 768. Conversely, the decoder consists of 4 blocks, each featuring cross-attention layer and an MLP layer, with channel size of 512.
Training Hyperparameters: We employ the AdamW optimizer with a weight decay of 0.05 and a base learning rate of 6e-4. The training schedule comprises 40-epoch warmup phase followed by a cosine decay learning rate schedule.
Finetuning For the Hyperparameters: During finetuning, remove the decoder and fine-tune the entire encoder. V2 dataset, we employ the AdamW optimizer with base learning rate of 5e-4 and a weight decay of 0.05. The batch size is set to 512, and we utilize a cosine decay learning rate schodule with warmup epochs over a total of 40 training epochs. Our data augmentation strategics include repeated augmentation (factor of 2) and Rand Aunment with parameters (9, 0.5), while flip augmentation is disabled. Additionally, we apply label smoothing (0.1), mixup (0.8), and cutmix (1.0). The drop path rate is configured 0.1, and no dropout applied for Sonnehhing-Smethhing V2. The layer-wise learning rate decay factor is set to 0.75.
For the Kinetics-400 dataset, most settings remain unchanged except for the following adjustments: the base learning rate is increased to le-3, flip augmentation is enabled, and the total number of training epochs is extended to 75.
16


Reviews:
[{"review_id": "zzybLGSumk", "reviewer": "Reviewer_pp1D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- Paper is clearly written and well-organized\n- Tackles the less-studied problem of autoregressive self-supervised learning in videos\n- Experimental results support the following main claims of the paper - grouping spatiotemporal video tokens into video element is better than spatial-only and temporal-only & random sequence for AR modeling outperforms fixed sequences\n- Comparable performance with respect to the baseline\n\n**Weaknesses**:\n- AR video modeling is used in recent (self-supervised) video generation frameworks [1, 2]. The relation between this paper and these methods is not studied.\n  - The key problems mentioned in the paper i.e. how to construct a video element & how to construct a video sequence are already studied in these works (e.g. they use high-level quantized tokens as video elements). Thus, the comparison of this paper to existing works is not up-to-date. Consequently, the AR baselines used in the main tables (Table 2 & 3) are quite old. \n- AR modeling in both language & video are typically motivated from generative use cases. It's unclear whether we need AR modeling for video classification use cases. On the contrary, sequence models based on state-space approaches [3] that are inherently auto-regressive in nature actually include bidirectional blocks for vision understanding tasks to achieve better results. This raises an important question of why AR modeling is needed here. The motivation is not discussed in the paper.\n  -  The results also slightly favor masked modeling baseline compared with proposed AR modeling method\n- The proposed method also utilizes masking, so it is not a pure AR model.  As shown in the ablation study (Table 8), masking appears to significantly affect the proposed method's performance. However, the paper does not discuss this aspect in the introduction and motivation.\n- Video element construction\n  - Nearby spatiotemporal video tokens are grouped into a video element.\n    - \"Clustering\" is used here in a somewhat unconventional sense, as the reader might mistake the use of a more sophisticated algorithm such as K-means\n    - Based on the neighborhood, pixels are first grouped to tokens and then the tokens are grouped into a video element. Is this additional step necessary? Can we simply increase the patch size of video tokens?\n    - This is not a novel grouping, and is very similar to how the video tokens are constructed from pixels.\n  - The authors suggest that video elements need to have rich semantic content similar to \"words in language\". However, mere grouping of nearby tokens may not achieve this effect. For example, [1, 2] use tokens from a codebook that has high compression ratios.\n- Sequence construction - random rasterization is used\n  - What are the implications of a random sequence order for AR modeling? Does this just treat the video as a bag of words instead of a meaningful sentence, if we use the analogy with text modeling.\n- All the baselines used in the main tables (Table 2 & 3) are quite old, with latest being from 2022. \n- The presented results are lower than current SOTA.\n- Limitations, future work and impact are not discussed.\n- It's unclear how the masking is done. Is it in the decoder or encoder or both?  \n\nBased on the weaknesses presented, it's unclear if the proposed autoregressive modeling framework satisfies the claim that it is an effective self-supervised learning strategy.\n\n[1] Language Model Beats Diffusion: Tokenizer is key to visual generation \\\n[2] VideoPoet: A Large Language Model for Zero-Shot Video Generation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an autoregressive video modeling framework. To develop this framework, the authors first construct a video element by grouping tokens spatiotemporally, and then propose to use a random sequence order of these elements for autoregressive prediction. Such spatio-temporal video grouping outperforms spatial-only or temporal-only grouping, and similarly, the random sequence ordering outperforms other fixed sequence orders. Competitive results against a masked modeling framework, i.e. VideoMAE, are presented on Kinetics400 and SomethingSomethingV2 dataset while achieving a better training efficiency.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Update motivation of why AR modeling is needed for video understanding, especially since the proposed approach doesn't show any video generation results.\n- Add an ablation study where unidirectional attention is replaced with bidirectional attention, while keeping everything else the same, to bring forth the impact of AR modeling.\n- Since masking appears crucial for the approach to work, it should be explicitly included in the main proposal, and the introduction should highlight its importance.\n- Discuss the relation of this work with other AR video modeling works in generative space. Contrast with their strategy of selecting video elements and video sequence, and why the author's proposal is better (or worse).\n- Add a baseline using high-level tokens from [1] or other similar works that used quantized codebooks as video elements instead of the proposed token groupings.\n- I would suggest to replace \"clustering\" with \"grouping\" or some other term to avoid any confusion to the readers\n- Add important baselines from 2023 and 2024 to table 2 & 3.\n- Add limitations, future work and impact sections.\n- Expand on masking section to make the procedure clear.\n\n[1] Language Model Beats Diffusion: Tokenizer is key to visual generation", "broader_impact_concerns": "There is no impact statement, and adding one can help the readers understand the impact. \nI don't have any ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j0J5yzwbGv", "reviewer": "Reviewer_m1qG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea is simple and easy to follow\n- The idea of using spatiotemporal cluster and tokens is sound\n- The paper evaluates the cluster shape, mask ratio, prediction order, and decoder design factors from the proposal\n\nWeaknesses:\n- The presentation of the method could be improved significantly.  In this version, the method is a short presentation of the autoregressive losses from GPT and ImageGPT, followed by different tokenization options and a brief description of the orders and the architectures.  These descriptions will benefit from analysis of the design choices as well as more details to fully reproduce the proposal.\n- The ablation studies could include the reasons for the values that were studied and why not other were selected.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a self-supervised video representation method that is trained using autoregressive prediction of next video tokens.  The main claimed contribution is the use of spatiotemporal cluster of spatiotemporal tokens as the main elements for the autoregressive prediction.  The paper evaluates a pre-defined and random order for the autoregressive prediction.  The paper shows results on action recognition datasets and shows improvements over the compared methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n- Eqs. 1 and 2 corresponds to the loss, and not to the probability of the word.  To fix the equations the left hand side should be changed. \n- What is the relevance of introducing GPT and ImageGPT in Sections 3.1 and 3.2 when only the autoregressive loss is stated.  I recommend to streamline the presentation by removing these sections and present the autoregressive loss and their previous use at the begining of Section 3.3.\n- Section 3.3 is missing the loss used for training.  While the authors have a line at the end of Section 4.1, it will be better to have a cohesive description of the method in a single section that the reader can follow to reproduce the work.\n- Overall, as mentioned in the weaknesses, the whole Section 3 could be reviewed and improved.  In its current form it doesn't introduce clearly what the proposal is now how it is implemented.  While it seems that the authors are relaying on existing architectures and losses, a brief and clear description of what is done in a single place could improve the proposal.\n\nMinor:\n- In Tables 2 and 3, the text '\"Ex. labels ✗” means only unlabelled data is used during the pretraining phase' seems to be a mistake since there are no \"labels ✗\" in the table.\n- What are the spatiotemporal order used in the results in Tables 2 and 3?\n- What is the difference between the last two rows of Table 6?\n- Can you convert the training time in Table 5 into FLOPS?", "broader_impact_concerns": "There are no ethical implications of the work that would require an impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wxgyqRK0AX", "reviewer": "Reviewer_GWgB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Simple and intuitive.\n* Performance gains over autoregressive baselines.\n* More sample efficient than most baselines.\n\n\n**Weaknesses**:\n* The authors extensively use the term “semantic richness” throughout this paper (Introduction, Section 3.3), yet do not explain or demonstrate its meaning (specifically, in the context of videos). To my understanding, the authors mean “a description of entities”, such as objects (e.g., words mean whole-entities such as “dog”, but pixels only describe a small part of an entity which might not be informative enough as a unit). I highly recommend spending a few words to explain it as it is one of the driving forces of the motivation for their approach.\n* Section 3.3.1: the authors refer to Table 1 for an example where tokenization outperforms non-tokenization, but they never mention what is the experiment in question, the performance of what is reported? In addition, what do they mean by “without tokenization”? Is it just pixel values? This requires more details (which should be added to the caption of Table 1 as well).\n* Model architecture: only explained in words and would benefit from an illustrative figure. Missing details such as the queries dimensions and an explanation why this approach of utilizing only cross-attention reduces the training cost (I understand that it is related to the number of entities in the attention mechanism, but it would be better to put it explicitly in front of the reader's eyes).\n* For the downstream action prediction tasks (specifically, “Main results” - section 4.2), how exactly is the pre-trained model utilized (what is the token-aggregation mechanism employed, if any). That is, do you use all of the tokens output embeddings, take one specific token or perform some pooling operation over the output? Do you fine-tune the pre-trained model or is it completely frozen?\n* Results (and ablation study) are not entirely convincing (it is also very hard to establish significance without some variance) in terms of performance. It seems the main contribution comes from the method being more efficient.\n* “Mask Ratio” ablation: I don’t understand this ablation and the writing in this paragraph should be improved (“To reduce the temporal redundancy, ARVideo randomly mask a portion of tokens”). Are you replacing the autoregressive mechanism with masking or adding masking? Given the answer to this, the results should also be further explained, as it seems that there is not much difference from the main results (which might suggest the gains come from the architecture rather than the training paradigm?).\n* Missing a lot of architecture, implementation and training details, details of hyper-parameters, details about baselines (e.g., TokenGPT is first mentioned in the results section, but there is no explanation what this model is and how it is different from your approach, except for the “token-level autoregressive prediction”, which does not provide a lot of information), evaluation protocol. This information is usually found in the appendix, but this paper has **no appendix at all**.\n\n**Minor**:\n* Figure 1: I believe the caption of this figure requires a bit more detail, such that the reader can look at the figure and understand the gist of the paper (which is not complex). Similarly for Figure 2, also maybe add notations for the tokenization dimension (e.g., $T$, $H \\times W$).\n* You provided useful concrete examples for the spatial and temporal tokenization process, but not for the “From Tokens to Spatiotemporal Clusters” part, which is the main contribution.\n* Table 2 and 3: the captions mention something about “”Ex. labels X”. There is no such marking in any of these tables? Table 3 - the “MaskFeat” line is gray, is it on purpose? What is the meaning of that?\n* Table 6 - missing upper cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an autoregressive self-supervised video representation learning framework based on clustered tokens. The authors propose to build tokens by performing spatial and temporal clustering, where the next-token prediction training paradigm is augmented with permuting the tokens order. The main implementation is based on ViT-B and the method is benchmarked on 2 datasets – Kinetics 400 and Something-Something. While the method does not provide significant performance gains, it is more (sample-) efficient compared to the VideoMAE baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Introduction, page 2: the authors claim that video tokens don’t encapsulate rich dimensions “primarily because 1) these video tokens are too dimensionally limited, and 2) video inherently lacks a sequential order in its spatial dimensions, although it retains this feature in its temporal aspects.”. I don’t understand what the authors mean in 1 by “video tokens are too dimensionally limited”, is that referring to the compression usually employed in tokenization or something else? In addition, in 2, if the authors refer to the order of patches in image for the spatial part, I’d suggest making it clearer.\n* In general, the writing could be improved.\n* Limitations and future work: please add a discussion of the method’s limitations (also directions for future work would be appreciated).\n* Open-source code: do the authors plan to provide an open-source code and pre-trained checkpoints?\n* See “Weaknesses” and “Minor”.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised numerous questions regarding this work, including clarity, contribution, and experimental validation. The authors have participated actively in addressing these issues with the authors. Ultimately, the authors successfully address the issues, and all reviewers lean toward accepting this paper. Overall, this paper is recommended for acceptance based on well-supported claims and sufficient audience.

I suggest the authors make the source code and trained models available to the public for reproducible research and impact.
Decision: Accept (poster)