UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer
Authors: Kunchang Li
Yali Wang
Yinan He
Yizhuo Li
Yi Wang
Limin Wang
Yu Qiao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Vision Transformer
Action Recognition
Video Learning

Abstract: Learning discriminative spatiotemporal representation is the key problem of video understanding. Recently, Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. Unfortunately, they exhibit limitations in tackling local video redundancy, due to the blind global comparison among tokens. UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format. However, this model has to require a tiresome and complicated image-pretraining phrase, before being finetuned on videos. This blocks its wide usage in practice. On the contrary, open-sourced ViTs are readily available and well-pretrained with rich image supervision. Based on these observations, we propose a generic paradigm to build a powerful family of video networks, by arming the pretrained ViTs with efficient UniFormer designs. We call this family UniFormerV2, since it inherits the concise style of the UniFormer block. But it contains brand-new local and global relation aggregators, which allow for preferable accuracy-computation balance by seamlessly integrating advantages from both ViTs and UniFormer. Without any bells and whistles, our UniFormerV2 gets the state-of-the-art recognition performance on 8 popular video benchmarks, including scene-related Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. The models will be released afterward.

Under review as a conference paper at ICLR 2023
UNIFORMERV2: SPATIOTEMPORAL LEARNING BY ARMING IMAGE VITS WITH VIDEO UNIFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
Learning discriminative spatiotemporal representation is the key problem of video understanding. Recently, Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. Unfortunately, they ex- hibit limitations in tackling local video redundancy, due to the blind global com- parison among tokens. UniFormer has successfully alleviated this issue, by uni- fying convolution and self-attention as a relation aggregator in the transformer format. However, this model has to require a tiresome and complicated image- pretraining phrase, before being finetuned on videos. This blocks its wide usage in practice. On the contrary, open-sourced ViTs are readily available and well- pretrained with rich image supervision. Based on these observations, we propose generic paradigm to build a powerful family of video networks, by arming the pre- trained ViTs with efficient UniFormer designs. We call this family UniFormerV2, since it inherits the concise style of the UniFormer block. But it contains brand- new local and global relation aggregators, which allow for preferable accuracy- computation balance by seamlessly integrating advantages from both ViTs and UniFormer. Without any bells and whistles, our UniFormerV2 gets the state-of- the-art recognition performance on 8 popular video benchmarks, including scene- related Kinetics-400/000/000 and Moments Time, temporal-related Something- Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. The models will be released afterward.
INTRODUCTION
Spatiotemporal representation learning is a fundamental task in video understanding. Recently, Vision Transformers (ViTs) have achieved remarkable successes in the image domain (Dosovitskiy et al. 2021 Wang et al. 202 lb Liu et al., 2021 Li al. (0022a). Therefore, researchers make great effort to transfer image-based ViT' for video modeling (Bertasius et al. 2021] Arnab et al.) 202T Yan et al. 2022). by extending Multi-Hear Self- Self-Attention (MHSA) along the temporal dimension. However, the spatiotemporal attention mechanism in these approaches mainly focuses on capturing global video dependency, while lacking the capacity of tackling local video redundancy. As a result, these models bear a large computational burden to encode local video representations in the shallow layers, leading to unsatisfactory accuracy-efffciencc balance in spatiotemporal learning.
To tackle these problems, researchers introduce a concise UniFormer (Li et al. 2022a), which unifies convolution and self-attention Multi-Head Relation Aggregator (MHRA) a transformer fashion. By modeling local and global relations respectively in shallow and deep layers, it can not only learn discriminative spatiotemporal representation but also largely reduce computation burden. However, as new architecture for video modeling, UniFormer does not have any image-based pretraining as a start. To obtain robust visual representation, it has to go through a tedious supervised pretraining phase by learning images from scratch, before finetuning on videos. Alternatively, we notice that there are various open-sourced image ViTs (Wightman) 2019 Touvron et al. (021). which have been well-pretrained on huge web datasets under rich supervision such image-text contrastive learning [Radford et al. 2021) and mask image modeling (He et al. 2022 Bao et al. 2021). These models exhibit great generalization capacity a range of vision tasks (Luo et al. 2022 Chen et al. 2022
Shen et al.] 202T). Hence, we are motivated by a natural question: Can we integrate advantages from both ViT's and UniFormer for video modeling?

---

Under review as a conference paper at ICLR 2023
(a) Unified Transformer (UniFormer)
Image Pretraining
Vision Transformers (ViTs)
huge
CL
MIM
stngle
Supervision
lnflatngg
Video Finetuning
Inserting
good
Acc-FLO1S
bad
Temporal
strong
Modellngg
(c) UnioormeV22: Arming Pretrained VITs with Efficient UniFormes Designs
900%
Ours-L
K400 Top Accuracy 1%6
86-
Curs
arming
84-
Use Untomment prua
sourced
02
WTT
VIT
Supervised Learning
T: Temporal MISSA
CL: Contrastive Learning
-8- 1/s (ilnninn
1: Loral MHRA
MIM:
Mask Modeling
Lnsforment y'aa ar
G: Global MIRA
MISA:: Multi-llead Self Aef-Attention
F: Multi-stage Fusion
MHRA: Multi-Head Relation Aggregator
10c
FLOPs/Videos
10a
10a
Figure 1: Comparison video modeling paradigm. UniFormerVI requires costly image pre- training. while directly inserting temporal MHSA into ViTs struggles for Eccuracy-FLOP balance. UniFormerV2 can effectively and efficiently arm well-pretrained ViTs with concise UniFormer de- signs, thus integrating ad advantages from both models for spatiotemporal representation learning. To our best knowledge, it is the first model that achieves 90.0% top-1 accuracy on Kinetics-400.
this paper, we propose a generic paradigm to construct a powerful family video networks, by arming the image-preee preereppet ViTs with efficient video designs of UniFormer. We called the resulting model UniFormerV2 (Fig. [ since it inherits the concise style of UniFormer but equips local flexibly insert local temporal
and global UniBlocks with new MHRA. In the local UniBlock, MHRA hefore the spatial VIT block. In this case, we can largely reduce temporal redundancy well as leverage the well-pretrained ViT block, for learning local spatiotemporal representation effectively. In the global UniBlock, we introduce a query-based cross MHRA. Unlike the costly global MHRA the original UniFormer, our cross MHRA can summarize all the spatiotemporal tokens into a video token, for learning global spatiotemporal representation efficiently. Finally, we re-onganize local and global UniBlocks as multi-stage fusion architecture. can adaptively integrate multi-scale spatiotemporal representation to capture complex dynamics in videos.
We deploy our paradigm on ViTs that are pretrained on three popular supervision, including super- vised learning, contrastive learning, and mask image modeling. All the enhanced models have great performance on video classification, showing the generic property of our UniFormerV2. Moreover, we develop a compact Kinetics-710 benchmark, where we integrate action categories of Kinetics- 400/600/700, and remove the repeated and/or leaked videos in the training sets these benchmarks for fairnsss (i.e., the total number of training videos is reduced from 1.14M to 0.66M). After training on K710, our model can simply achieve higher accuracy on K400/600/700 via only 5-epoch fine- tuning. Finally, extensive experiments show that, our UniFormerV2 achieves state-of-the-art perfor- mance on popular video benchmarks, including scene-related datasets (i.e., Kinetics-400/600/700 (Carreira & Zisserman 2017, Carreira al.) 2018; 2019) and Moments in Time (Monfort al. 2020)), temporal-related datasets (i.c., Somethiinn V1/V2 (Goyal et al., (017b]), and untrimmed datasets (i.e., ActivityNet (Heilbron al. 2015) and HACS (Zhao et al. 2019)). To our best knowledge, it is the first model to achieve 90.0% top-I accuracy on Kinetics-400.
2 RELATED WORK
Vision Transformer Following Transformer in NLP (Vaswani et al. 2017). Vision Transformer
(ViT) (Dosovitskiy et al. 2021) l) has made great successes in various vision tasks, including object
detection Carion al. 2020 Zhu al. (2021) semantic segmentation Xie et al. (202T): Cheng
2

---

Under review as a conference paper at ICLR 2023
et al. (2021). low-level image processing Liang al.
(2021): Cui et al. (2022) action recognition [Brttasuus et al. 2021 Arna et al. 2021), temporal localization (Zhang et al. 2022) and multi-
modality learning (Radford al.) 2021 Wang et al. 2022). To make VIT more efficient and ef-
fective, researchers introduce scale and locality modeling in different ways, such as multi-scale architectures (Wang al.] 202 Fan et al., (02T), local window (Liu al.) 202T), early convo- lution embedding (Xiao et al. 2021] Yuan et al. 202 and convolutional position encoding (Chu Dong et al.) 2022). Alternatively, UniFormer (Li al. 2022a) unifies convolution and self- self-attention as relation aggregator in a transformer manner, thus reducing large local redundancy.
al. 2021;
Video Learning. 3D Convolutional Neural Networks (CNNs) once played a dominant role in video understanding (Tran et al., 2015] Carreira & Zisserman 2017). Due to the difficult optimization problem of 3D CNNs, great efforts have been made to factorize 3D convolution in the spatiotempo- ral dimension (Tran et al. 2018 Qiu et 2017 Feichtenhofer et al., 2019) or channel dimension (Tran al. 2019 Feichtenhofer | Kondratyuk et al., (002)) However, the local receptive field limits 3D convolution to capture long-range dependency. The global attention motivates researchers to transfer aaaage-pretrainee ViTs to video tasks (Bertasius et al.) 202T; Neimark et al. 2021 Zhang al. 2021b; Arnab et al., 2021 Bulat et al., 2021; Patrick al., (2021). To make the video trans- former more efficient, prior works introduce hierarchical structure with pooling self-attention (Fan al. 2021), local self-attention (Liu al.) 2002) unified attention (Li al.] 2022a). Though these novel models are adept at temporal modeling, they rely on tiresome image pretraining. In contrast, various well-pretraiin ViTs with rich supervision are open-sourced (Wightman 2019). In this paper, we aim to extend efficient UniFormer designs to VIT, arming it as a strong video learner.
3 METHOD
Overall Framework. We propose to arm an image ViT with video designs of UniFormer (Li et al. 2022a), leading to UniFormerV2. On one hand, spatial interactions in well-pretrained ViT can be fully leveraged and preserved to enhance spatial modeling, On the other hand, hierarchical temporal interactions in efficient UniFormer can be flexibly adopted to enhance temporal modeling. Our overall architecture is shown in Fig. 2 It firstly projects input videos into tokens, then conducts local and global modeling by the corresponding UniBlocks. Finally, a multi-stage fusion block will adaptively integrate global tokens of different stages to further enhance video representation.
Specifically, we first use 3D convolution (i.e., 3x16x 16) to project the input video as L spatiotem- poral tokens X E _xC, where L=T*xxX (T', H. and W respectively denote temporal, height, and width). Following the original ViT design (Dosovitskiy et al., 202I), we perform spatial downsampling by factor of 16. For better temporal modeling, we conduct temporal downsampling by a factor of 2. Next, we construct the local and global UniBlocks. For our local block, we re- formulate the image-preee paaggeprettaieed ViT block, by inserting the local temporal MHRA (Li et al. 2022a) before it. In this case, we can effectively leverage the robust spatial representation of ViT as well as efficiently reduce local temporal redundancy. Moreover, we introduce global UniBlock of each local UniBlock, which can capture full spatiotemporal dependency. For computational effi- ciency, we design query-based cross MHRA aggregate all the spatiotemporal tokens as a global video token. All these tokens with different-level global semantics from multiple stages are further fused for discriminative video representation.
top
3.1 LOCAL UNIBLoCK
To efficiently model temporal dependency upon the well-learned spatial representation, propose a new local UniBlock, by inserting a local temporal MHRA before the standard ViT block,
x -TTMHRA (Norm (X')) X
X = GS MHRA (Norm (x*) + X
(2)
X L = FFN (Norm (x*) + x
LT MHRA and MHRA refer to MHRA with local temporal affinity and global spatial affinity
respectively. FFN consists of two linear projections separated by GeLU (Hendrycks & Gimpel 2016). Additionally, following the normalization in UniFormer (Li et al. 2022a). we adopt Batch Norm (BN) (loffe & Szegedy, 2015) before local MHRA, and Layer Norm (LNN (Ba et al., 2016)
3

---

Under review as a conference paper at ICLR 2023
Local UniBlock V2
Block
MIIRA Temporal
Globai MHRA
Spazial
X
FPN
91 auyie 892 91x9
Local UntBlack,
Local UniBlackg
Local UniBlocky
Multi-stage Fuston
Global UniBlack,
Global UniBllckk
Global UniBlacky
Temporal Downsampling
Global UniBlock v2
K
Global Cross MHRA Spatiotemporad
FFN
X-S
X
Figure 2: Overall framework of our UniFormerV2. There are three key blocks, i.e., local and global UniBlocks, and multi-stage fusion block. All these designs are efficient and effective.
before global MHRA and FFN. Note that GS_MHR and FFN come from the aaage-pretrained ViT block. In general, MHRA (Li et al. 2022a) learn token relation via multi-head fusion:
R₂ (X) A,, (X),
MHRA(X) = Concat (2);R((X); ;Rx(X)UU
(5)
where (-) refers the relation aggregator the n-th head. A, is affinity matrix that describes token relation and V, (-) is linear projection, while U E RCxC is learnable fusion matrix. For our local UniBlock, we insert LT MHRA to reduce local temporal redundancy, which shares a similar design insight with the original UniFormer (Li et al. 2022a). Hence, the affinity in LT MHRA is local with a learnable parameter matrix an E XIIIII in the temporal tube x x
(6) This allows to efficiently learn the local temporal relation between one token X; and other tokens X in the tube. Alternatively, GS MHRA belongs the original ViT block. Therefore, the affinity in GS MHRA refers global spatial self-attention in the single frame 1 x H x W,
A XX) = where j E 22
exp( Qn (X,) (K,,,, (X,))
GS X X,) Σ EjeER exp Q. (X,) ) (X,))
where (-) and (-) E R are different linear projections in the n--hh head.
Discussion. (I) Note the spatiotemporal affinity in our local Uni Blockock is decomposed as local tem- poral one ALT in Eq. 6 and global spatial one AGS in Eq. 77. In this case, we not only leverage the efficient video processing design of UniFormer but also inherit the effective image pre- training of VIT. Alternatively, such local affinity in the original UniFormer eLi al. 2022a) is jointly spatiotemporal, i.e., loeat (X,, X,) a where j belongs to 3D tube 22 xxxxxx The parameter matrix has to learn from scratch, which inevitably increases the training cost. (II) Compared with
UniFormer, we abandon its Dynamic Position Encoding (DPE) in the local UnilBlock, since the po- sition encoding in the ViT block has characterized token locations. Table [9b) also reveals an extra DPE in the local UniBlock does not help. (III) Instead of applying global temporal modeling as in TimeSformer (Bertasius et al. 2021), we use local affinity for temporal characterization, largely reducing the computation burden by tackling temporal redundancy in the UniFormer style.
3.2 GL.BBAL UNIBLOCK
To explicitly conduct long-range dependency modeling on the spatiotemporal scale, we introduce a global UniBlock in our UniFormerV2. Specifically, this global UniBlock consists of three basic components including DPE, MHRA, and FFN as follows.
X DPF (X*) + X
ST = C.MHRA (Norm (q), Norm (x)),
(9)
X = FFN (Norm (X) + ST.
(10)
4

---

Under review as a conference paper at ICLR 2023
X
X
X
0.-1
4-11
9:
9141
G
Q1-4
4-1
Q1+1
G1-1
u
x
X
X--1
X/1x
(a) Sequentiol
(b) Parallet
X
[X] N.
[X] Xi.1l
X
X
X
G
G
[XL
G
Giat
X
X
X
Xi+1
cc) Hierarchical KV
(d) Hierarchical Q
Figure 3: Multi-Stage Fusion Block.
The DPE is instantiated as depth- wise spatiotemporal convolution (Li et al, 2022a). We design the global C. MHRA in a cross-attention style to efficiently construct video representation,
(q. X) = A. X),,.)
(11)
C.MHRA(q.X) = Concat(Ry (q, X); Rq ..D......RRR R& X.)U.
(12)
(g. the cross relation aggregator, which can convert a learnable query q E 1IxC into video representation, via modeling dependency between this query q and all the spatiotemporal tokens X. First, it computes the cross affinity matrix Ag X) to learn relation between q and X,
exp(Qn(q)" (,,(),)),
A (q. Xj) EjeArHHx
(13)
Then, it uses the linear projection to transform X spatiotemporal context V (x..).. Subsequently. it aggregates such context V,,(X (X) into the learnable query, with guidance of their affinity (q X). Finally, the enhanced query tokens from the heads are further fused as a final video representation, by linear projection U E C'C, Note the query token is zero-initialized for stable training.
Discussion. We further discuss the distinct design of our global UniBlock, compared to the one in the original UniFormer (Li et al., 2022a). (I) We add the global UniBlock top of the local UniBlock, extracting multi-scale spatiotemporal representations in token form. Such design helps strengthen the discriminative video representation without compromising the pretrained architec- ture. (II) The typical global spatiotemporal attention is computationally heavy, due to its quadratic complexity. To pursue better accuracy-compputttio balance, we introduce cross-attention style of global MHRA in UniFormerV2, thus largely reducing the computation complexity from O(L²) to OL), where is the number of tokens. More importantly, since the query q is learnable, it can adaptively integrate the spatiotemporal context from all L tokens to boost video recognition. (III) The global UniBlock inherits DPE design from UniFormer, and we find also helps in Table 9c
3.3 MULTI-STAGE FUSION BLOCK
We propose multi-stage fusion block to integrate all video tokens from each global UniBlock as Fig. 3 For simplicity, we denote the i-th global block as X = G. X²). Given the tokens X from the local UniBlock, the global block transforms the learnable query q into video token
X In this paper, explore four fusion strategies to integrate the video tokens from all the global blocks ( fusion regarding efficacy and efficiency.
fi=l into final video representation F, and employ the sequential way to conduct
The studied fusion methods given below. (a) Sequential: We sequentially use the video token from the previous global block X. as the query token in the current global block qi where X Gx(XE,,, Xf). (b) Parallel: We concatenate all the global tokens {X@) in parallel, and use a linear projection UF E NxC to obtain the final token, where F = Concat(XC
X (c) Hierarchical KV: We use the video token from the previous global block X as a part of contextual tokens in the current global block, where X = Go(qi, [ x(1). (d) Hierarchical
Q: We use the video token from the previous global block x part of query tokens in the current global block, i.e. X. (x 4.), X/).
5

---

Under review as a conference paper at ICLR 2023
Image Pretrain
Video Preirain
Param
FLOPs
K400
Method
Ready Data
Epoch
Cropx
(
(M)
(T)
Top-1
Tup-5
Slowfast NL (T Teichtenholer et al. 2019
N/A
Nome
196
80x3x
7.0
79.8
93,9
X3D-XXI. 312 Feichtenhiieer 2020)
N/A
Nome
256
24x3x
20
5.8
80.4
94.6
UniFormerV
eL 20224
IN-1K
110
32x3x4
50
3.1
83.0
Swin-L 3841 [Liu
2024
IN-2IK
32x5x10
105.4
84,9
96.7
MViTv2-L 3127
et 202]
IN-21K
105
40x3x5
218
42.2
86.1
97.0
TimeSformer-L.
[Brraasus el 2021
IN-21K
15
96x3x1
121
7.1
80.7
94.7
Mformer-HR 3367
[Patrick el
2021
IN-21K
x6x3x
109
81.1
UniFormerV--B/16
IN-2IK
5S
16x3x4
113
3.6
83.4
96.2
UniFormerV2-L16
IN-22K
55
x2x3x2
355
12.2
85.4 97.0
Methods with wel-scole data. W7S contains 60M uxpublishdd video-text pairs.
ALIGN contains
7.8B
image-text
pairs.
VIVIT-H/14x2 (Amab et 2021
JFT-300M
30
32
6654
47.88
84.9
95.8
TokenLeamee--//U et al. 2021
x 3FT-300M
x4
450
48.9
85.4
96.3
MTV-H Yan al. 2022
JFT-300M
32x3x4
1000+
44.5
85.8
Florence 384 Yuan ct 2021b
FLD-900M
30
32x3x4
647
N/A
86.5 97.3
CoCa et al. 2022
x
JFT-3B+ALIGH
N/A
N/A
1000+
N/A
88.9
CoVeR 448
[Zhang Cral.
2021a
x
3FTM
20f
16x3xl
431
17.6
86.3
CoVeR 4481
[Zhang ct 2021a
JFT-3B
201
16x3xI
17.6
87.1
MTV-H Yan ar. 2022
IN-21K
WTS-60M
30
32x3x4
1000+
44.5
89.1 98.2
MTV-H 2807
Yan cr 2022
x
IN-21K
WTS-60M
30
32x3x4
1000+
73.6
89.9 98.3
EVL-L/14 (froze1)) 3300
LLllt
2022
CLIP-40OM
32x3x1
67
19.1
87.7
X-CLIP-L/14 336 (Ni
2022
CLIP-400M
16x3x4
453
37.0
87.7
UniFormerV2-Ll14 (Trozen]
CLIP-400M
K710-0.66M
8x1x3
51
4.7
87.8
98.0
UniForerr2--1114 (froven) 3361
CLJP-400M
K710-0.66M
32x3x1
51
18.8
88.8
98.1
UniFoomerV2-B/16
CLIP-400M
K7700-066MM
8x1x3
115
0.4
85.2
96.7
UniFoomerV2-B/16
CLIP-400M
K710-0.666
8x3x4
115
1.8
85.6
97.0
UniFormerV2-L114
CLIP-400M
K<700-0.66
8x3x4
354
8.0
88.8
98.1
Unil VmiFoo1erV2-LL14
CLIP-400M
K710-0.66M
32x3x2
354
16.0
89.3
98.2
UmiFormerV2-LL144 3361
CLIP-400M
K7100-066M
32x3x2
354
37.6
89.7
98.3
UniFormerV2-L/14:33 3361
CLIP-40OM
K710-0.66M
64x3x2
354
75.3
90.0 98.4
Table 1: Comparison with the state-of-the-art on Kinetics-400, FT indicates the video finetuning.
i marks ro-finetuning with K400-SSV2+MiT+IN.
Our UniFormerV2 outperforms most of the
current methods in terms of accuracy and/or efficiency. It firstly achieves 90.0% top-1 accuracy on
K400. More explanations of model comparison be found in the
K600
K700
Method
Framee x Cropx
Param, (M)
FLOPs
Top-1
Top-5
Top-1
Top-5
FlowFa
Feichteenoofr et al. 2019
80
60
7.0
81.8
93.1
71.0
89.6
MoViNet-A6
Kondralyuk el al. 2021
120x1x1
31
0.4
83.5
96.2
72.3
MViTv2-L 312
eL al. 2021
40x3x3
218
42.2
97.8
79.4
94.9
CoVeR 448 (Zhang al.
16x3xI
431
431
87.9
79.8
MTV-H Yan et
32x3x4
1000+
44.5
89.6
98.3
82.2
95.7
CoCa 5767 Yu al. 2022
N/A
1000+
N/A
89.4
82.7
UniFormer V2-1 V2-1/14
(frozen) 3361
32x3x1
31
18.8
89.1
98.2
80.6
95.2
UniForme V2-1/14
32x3x2
354
16.0
89.5
98.3
81.5
95.7
UniFormer V2-L/14 3361
32x3x2
354
37.6
98.5
82.1
UniFormerV2-L/14 3361
64x3x2
354
75.3
90.1
98.5
82.7
96.2
Table 2: Comparison with the state-of-the-art Kinetics-600/700
Finally, we dynamically integrate the final tokens from both local and global blocks, effectively promoting recognition performance in empirical studies (Table 12 Specifically, we extract the class token FC from the final local UniBlock, and add it with the video token F by weighted sum, i... Z oF + (1 aCF where is learnable parameter processed by the Sigmoid function.
EXPERIMENTS
Datasets. To verify the learning capacity our UniFormerV2, we conduct experiments on 8 popular video benchmarks, including the trimmed videos less than 10 seconds, and the untrimmed videos more than min. For the trimmed video benchmarks, we divide them into two categories. (a) Scene- related datasets: Kinetics family (Kay et al. (017)) (i.c., Kinetics-400, 600 and 700) and Moments in Time V1 (Monfort et al. 2020). (b) Temporal related datasets: Something-SSmeething V 1/V2 (Goyal al. 2017b). For the untrimmed video recognition, choose ActivityNer (Heilbron et al. 2015)
and HACS (Zhao et al. 2019). More dataset details can be found in Appendix A
Kinetics-710 for Post-Pretraining We propose unified video benchmark for post-pretraine pest--rrtriinng Uni- Former V2. Different from (Yan et al. 2022) that exploits web-scale video dataset (i.e., 60M video-text pairs), we build much smaller video benchmark based the Kinetics-400/600/0000

---

Under review as a conference paper at ICLR 2023
Image Pretrain
Framex
Param
FLOPs
MIT VI
Method
Modality
ViT
Ready
Duta
CrupxClip
(M)
(T)
Top-1
Tup-5
AssembleNet- 101 (Ryoo el al
2020
RGB+Flow
X
N/A
None
NIA
53
0.8
34,3
62.7
MoViNet-A6 (Kondratykk ct 2021
RGB
x
N/A
None
120x
31
0.3
40.2
VIVIT-L/I6x FE [Arnab et al. 2021
RGB
IN-2IK
32x3x1
612
11.9
38.5
64.2
MTV-H Yan et
2022
RGB
IN-21K
32x3x4
1000+
44.5
45.6
74.7
MTV-H 2807 Yan oanelaa 2022
RGB
"
IN-21K
32x3x4
1000+
73.6
47.2
75.7
CoVeR 4481 (Zhang at 2021a
RGB
'
x
JFT-3B
16x3x1
431
17.6
46.1
Umil'ormerV2-BB/1
RGB "
CLIP-40OM
8x3x4
115
1.8
42.7
71.5
MiiFormerV2-L/11
RGB <
CLIP-400M
8x3x4
354
47.0
76.1
UniFormerV2-L//k 3361
RGB
4LIP-400M
8x3x4
354
18.8
47.8
76.9
Table 3: Comparison with the state-of-the-art on Moments in Time V1.
Image Pretrain
Video Pretrain
FT
Framex
Param. FLOPs
SSV2
Method
Ready Dara
Data
Epoch
Epoch
Crop Clip
(M)
(T)
Top-1
Top-5
VidcoMAE-B Tong al
2022
N/A
None
SSV2
2400
40
16x3x2
87
1.1
70.3
92.7
VideoMAR-I. Tong at 2022
N/A
None
SSV2
2400
32x3x1
305
75.3
MViTv1-B Fan
2021
N/A
None
K400
200
100
64x3x1 1
36.6
1.4
67.7
90.9
MaskFeat-1.
312 WC clat al:
2022
N/A
None
K400
905
40
40x3x4
218
28.3
74.4
94.6
MVITv2-B Li ct al. 2021
IN-21K
K400
100
100
32x3x1
51.1
0.7
72.1
93.4
UniFormer (L 2022a
IN-IK
K4000
110
50
32x3x1
50
0.8
71.2
92.8
Swin-B (Liu al
IN-21K
K400
30
32x3x1
89
1.0
69.6
92.7
CoVeR 448
Zhang et al. 2021a
JFT-3B
None
0
20
16x3x1
431
17.6
70.8
ViVi-- FE (Amahh el al
2021
IN-2TK
K400
30
35
32x3x14
612
47.6
65.4
89.8
MTV-B 3201 Yan cl al 2022
IN-21K
K400
30
100
32x3x4
310
11.2
68.5
90.4
TimeSformer-Fit (Herlasiil
2021
IN-21K
Vone
16x3x1
5.1
62.5
EVL-L/14 (Lin et ar.
CLIP-400M
None
46
32x3x1
67
9.6
66.7
V2-FormerV--B/10
CLIP-10OM
None 0
22
16x3x1
163
0.6
69.5
92.3
UniFormerV--B//
CLIP-400M
None 0
22
32x3x1
163
1.1
70.7
93.2
UniFormerV2-L/1e
CLIP-100M
None
16x3x1
574
2.6
72.1
93.6
UniFomer\2-LL/1
CLIP-100M
None
32x3x1
574
5.2
73.0
94.5
Table 4: Comparison with the state-of-the-a on Something-Someethin V2. The methods with-
out image pretraining are marked in gray. marks co-finctuning with --000--SS++MMTIN--LUL
Concretely, we merge the training set of these Kinetics datasets, and then delete the repeated videos according to Youtube IDs. Note we also remove testing videos from different Kinetics datasets leaked in our combined training set for correctness. As a result, the total number of training videos is reduced from 1.14M to 0.66M. Additionally, we merge the action categories in these three Kinet- ics datasets, which leads to 710 classes in total. Hence, we call this video benchmark Kinetics-710. More detailed descriptions can be found in Appendix [ In our experiments, we empirically show the effectiveness of our Kinetics-710. For post-pretraining. we simply use 8 input frames and adopt the same hyperparameters as training on the individual Kinetics dataset. After that, no matter how many frames are input (16. 32, or even 64), we only need 5-epoch finetuning for more than 1% top-1 accuracy improvement on Kinetics-400/600/000) as shown in Table De
Implement Details. Unless stated otherwise, we follow most of the training recipes in UniFormer (Li et al., 2022a), and the detailed training hyperparameters can be found in Appendix [A] We build UniFormerV2 based on ViTs pretrained with various supervisions (see Table 8). showing the gener- ality of our design. For the best result, we adopt CLT (Radford et al. 2021) as the backbone by default, due to its robust representation pretrained by vision-language contrastive learning. For most datasets, we insert the global UniBlocks in the last layers of V/T-B/L to perform the multi-stage fusion. But for Sth-Sth V1/V2, we insert the global UniBlocks in the last 8/16 layers of ViT-B/L for better temporal modeling. The corresponding ablation studies are shown in Table D Finally, we adopt sparse sampling (Wang al. 2016) with the resolution of 224 for all the datasets.
4.1 COMPARISON STATE-OF-THE-
Kinetics. Table _ presents the state-of-the-ee comparison on Kinetics-400, (1) The first part lists the models pretrained on open-source datasets like ImageNet (Deng et al. 2022). On one hand, compared with UniFormerV 1-B (Li et al.) 2022a), our UniFormerV2-B only uses 50% fine-tuning epochs but achieves a better accuracy, showing the importance of inheriting the pretrained weights. On the other hand, compared with TimeSformer-L (Bertasius et al. 2021), our model achieves 2.7% performance gain with 50% FLOPs. showing the importance of adopting the UniFormer designs. Besides, compared with Swin-L (Liu et al. 2002), our MiiFoomerVV-L. based on BeiT (Bao et al.) 202I) that pretrained on ImageNet-22K, achieves comparable results but with 12% FLOPs. (2) The second part shows the methods using web-scale data. On one hand, compared with MTV-H
7

---

Under review as a conference paper at ICLR 2023
Method
Frame
Top-1
Top-5
Method
Frame
Top-1
TSN-R5O
2016
47.3
CSN-R152 (Tran al. 2019
32
91.5
[SM-R50 al. 2019
77.1
TimeSformer (Bertasiis ot al.
2021
8
91.8
TEA-R50
2020b
51.9
80.3
ViVIT-B (Arnab al. 2021
32
91.9
C5-Net-R50 al. 2020a
52.5
80.9
SuiloomerV2--I11
16
95.5
TDN-R101 Wang et al. 2172
55.3
88.3
UniFormer V2-1/14
32
95.4
UniFommer Cf 20224
57.1
84.9
Table 7: Results on HACS.
UniFomerVI-- [T 2022a
61.0
87.6
56.8
84.2
Type
Method
Data
K400
SSV2
Vm-FomerV2-B/166
59.4
86.2
TimeSrormer
IN-2TK
78.7
59.5
V2-LL/14
60.5
86.5
ViI
IN-21K
81.6
67.5
V2-Fo14
32
62.7
88.0
SL
DeiT III
IN-21K
82.7
66.5
Table 5: Results on Comething-Somettine V1.
DINO
IN-IK
78.7
65.8
CL
CLIP
CLIP-400M
84.4
69.5
Method
Frame
Top-I
MAE
IN-1K
78.8
65.1
DSN-R34 (Zhene et al. 2020
32
MIM
BeiT
IN-22K
82.2
67.7
MARL-R15Z et 2019
32
NSNet-Swin- L
al. 2022
32
90.2
Table 8: Different pretrained ViTs. Our
UniFormer V2-L714
16
94.3
UniFormerV2 based on different open- sourced ViTs beat TimeSformer, especially for Something-SSmethiin
UniFormer V2-L/14
32
94.7
Table 6: Results on ActivityNet.
(ensembling 4 models) (Yan al., 2022). our single model only requires 1 1% video post- post-pretraining, 16% finetuning epochs and 35% model parameters to achieve competitive accuracy. On the other hand, under same CLIP-400M pretraining, our UniFormerV22-L (frozen) only uses 25% FLOPs to achieve the competitive accuracy compared with EVL-L (frozen) (Lin et al. 2022). and obtains 1.1% accuracy improvement with similar FLOPs. Finally, our UniFormer V2 Is the first model to achieve 90.0% top-1 accuracy K400, to best knowledge. For Kinetics-60N and 700, UniFormer V2 also obtains the state-of-the-ar performance (90. % and 82.7%, see Table
Moments in Time. Due to complex inter-class and intra-class variation, MiT is more challenging than Kinetics. As shown in Table our model beats most of the recent methods, i.e., compared with ViViT-L (Amab al. 2021). UniFormerV2-B obtains 4.2% performance gain but only with 19% model parameters and 15% FLOPs. Compared with MTV-H (Yan et al.) (022), UniFormerV2-L only uses 35% model parameters and 25% FLOPs to achieve 1.2% top-5 accuracy improvement.
Something-Somethhe In Table we show the results on Sth-Sth First, our model outperforms those standard models based on the well-pretrained image ViT hand. For example, under the same CLIP-400M pretraining and the same number of sampled frames, our UniFormerV2-B obtains 4% higher accuracy with only 11% FLOPs, compared with EVL-L (Lin al. (022). Second, we compare our model with those models whose backbone is specially designed. Since the pretraining unavailable for these models, they have to perform a tedious training phrase, consisting image- pretraining, vidco pretraining and video finetuning. Alternatively, our UniFormerV2 can work well with only video finetuning, c.g., our model only uses 22 epochs to achieve the performance of UniFormerV1 (Li al. 2022a), which requires 10+50=160 video epochs to obtain results. Finally, we compare UniFormer V2 with those models which do not apply image pretraining. Such models require a huge number of training epochs, e.g., VideoMAE-B (Tong at. 2022) contains 2400 video pretraining epochs and 40 video finetuning epochs, much longer than our UniFormerV2-B with similar accuracy (only 22 video finetuning epochs, i.e., 0.9 % training epochs of VideoMAE-B). For Sth-Sth V1 in Table 5 we reach the new state-of -oe---fftthttr- performance (62.7 %... The above results reveal the effectiveness and efficiency of our UniFormerV2 for temporal modeling.
ActivityNet and HACS. For the untrimmed videos, it is essential to capture long-range temporal in- formation, since the action may occur multiple times at arbitrary moments. As shown Table and [ our UniFormerV2 V2 significantly outperforms the previous best results on the large-scale untrimmed benchmark ActivityNet and HACS by 4.5% % and 3.6 %, respectively. These results demonstrate the strong long-term modeling capacity of our UniFomrerV2.
4.2 ABLATION STUDIES
To evaluate the effectiveness of UniFormerV2, investigate each key structure design, shown in Table Table 9 All the models are directly finetumed from CLIP- ViT-B/16 by default. We utilize xxxx3'' and '16x 1 xx3 testing strategies for Kinetics and Something-Something respectively.
8

---

Under review as a conference paper at ICLR 2023
Global
Local
T-Down
K400
SSV2
Design
SSV2
Layers
Reduction SSV2
83.1
45.1
Temporal MHSA
65.2
1-12
4.0
68,9
84.4
63.3
Temporal Convolution
67.5
1-12
2.0
69.1
x
83.6
67.7
ST-Adapter
68.0
1.5
69.5
x
84,4
68.7
Local MHRA
69.1
1-12
1.0
69.5
84.4
69,5
Local MHRA DPE
69.1
1-8
1.5
67.9
(a) Components of UniFormerV22
Local MHRA 2
69.5
1-4
1.5
67.6
(b) Local UniBlock.
Layers
DPE
K400
SSV2
Design
SSV2
Pretraining
Finetuning
K400K6600K700
9-12
x
84.2
68.1
Sequential
69.5
None
Individual
84.4
85.0
75.8
9-12
84.4
68.5
Parallel
69.1
K400/600/700
K400/600/700
85.6
86.0
75.6
5-12
84.4
69.5
Hierarchical KV
68.9
K710
K400/600/700
85.6
86.3 76.1
1-12
84.4
69.4
Hicrarchical Q
69.5
K710
Individual
85.6
86.3
76.3
(c) Global UniBlock.
(d) Multi-Stage Fusion.
Different Training Scripts.
Table 9: Ablation studies. T-Down means temporal downsampling, and we double the frames to
maintain similar GFLOPs. ST-Adapter is proposed in Pan et al. (2022). Compared with simple co-training, our K710 pretraining saves 33% cost with consistent improvement (see Appendix
Pretraining Sources. To demonstrate the generality of our UniFormerV2 design, we apply it on the ViTs with different pertaining methods, including supervised learning (Dosovitskiy et al. 2021 al. 2021 Radford al. [021] and mask image modeling (He et al. 22022) Bao et al.] (021). Table 8 shows that all the models beat TimeSformer (Bertasius et al.J 2021), especially for Something-Sometiie
Touvron al. 2022). contrastive learning(Carrnn
that relies on strong temporal model- ing. It also reflects that better-pretrained VIT is helpful for stronger video performance.
Different Components. In Table 9a note the global UniBlock is crucial for the scene-related benchmark (c.g.. K400), since this block can effectively provide holistic video representation for classification. Alternatively, the local UniBlock critical for the temporal-related benchmark (c.g.. SSV2), since this block can effectively describe detailed video representation for classification. Be- sides, using temporal downsampling with double input frames (similar FLOPs) is also helpful for distinguishing fine-grained videos like SSV2, due the larger temporal receptive field.
Local UniBlock. To explore the structure of local UniBlock, we conduct experiments in Table 9b] reveals that convolution is better than self-attention for temporal modeling. and our local MHRA is more powerful than both of them in SSV2. Following ST--dapterr (Pan et al.) (002),, we add another local MHRA after the spatial MHRA for better performance. Besides, add local MHRA in all the layers and reduce the channel by 1.5 times for the best accuracy-flops trade-off.
Global UniBlock and Multi-stage Fusion. In Table Dc. we find that the features in the deep layers are critical for capturing long-term dependency, while the DPE and the middle information are necessary for identifying the motion difference. For the fusion strategy, Table 9d shows that the simplest sequential fusion is adequate for integrating multi-stage features.
Training Recipes. We compare different training and finetuning methods in Table Dc Note that when co-training with K400, K600 and K700, we remove the leaked videos in the validation set and introduce three classification heads. K710 maintains only about 60% of the total training videos (0.66M vs. 1.14M for K400+K600+K3 K700), but it improves classification performance significantly for Kinetics. Meanwhile saves about 33% training cost (see Appendix (A). Besides, direct training on it works better than Kinetics co-training, especially for K600 (+1.3% vs. +1.0%) and K700 (+0.5 vs. -0.2%). Though co-fin ro-finetuning shared the backbone and saved parameters, we adopt indi- vidual finetuning for each dataset considering the best performance.
5 CONCLUSION
In this paper, we propose powerful video model, namely UniFormerV2. It arms image-pretrained ViTs with efficient UniFormer designs for video learning. By novel local and global video relation aggregators, it is capable of conducting effective spatiotemporal modeling with a tractable complex- ity. Besides of seamlessly integrating advantages from both ViTs and UniFormer, we also introduce multi-scale token fusion for further enhancing video representation. Our UniFormerV2 achieves state-of-the-a performance on 8 popular video benchmarks, and firstly reaches 90% top-1 accu- racy Kinetics-400, to best knowledge.
9

---

Under review as a conference paper at ICLR 2023
Reproducibility. To ensure all the results can be reproduced, give the details of the datasets, model and training hyperparameters in our experiments (see Table I0 and Table For Kinetics- 710, provide its label list in Table 20 for reproduction. All the codes are based on the UniFormer Li al. 2022b) repository.
REFERENCES
Jean-Bas Sun-Baptitt Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022.
Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In /EEE/CV International Conference on Computer Vision, 2021.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. abb 1607.06500 2016.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit:
Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, 2021.
Adrian Bulat, Juan-Manuel Perez-Rua, Swathikiran Sudhakaran, Brais Martinez, and Georgios Tz- imiropoulos. Space-time mixing attention for video transformer. In Neural Information Process- ing Systems, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' l'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF Inter- national Conference on Computer Vision, 2021.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
João Carreira, Eric Noland, Andras Banki- Horaath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. ArXiv, abs/1808. 018.01134, 340, 2018.
João Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. ArXiv, 0abs/1997.06999 2019.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Y. Qiao. Vision transformer adapter for dense predictions. ArXiv, abs/2205. 08534, 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is all you need for semantic segmentation. In Neural Information Processing Systems, 2021.
Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? ArXiv, abs/2102.10882, 2021.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical auto- mated data augmentation with a reduced search space. In IEFF/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
Ziteng Cui, Kunchang Li, Lin Gu, Sheng Su, Peng Gao, Zhengkai Jiang, Yu Jiao Qiao, and Tatsuya Harada. You only need 90k parameters to adapt light: A light weight transformer for image enhancement and exposure correction. ArXiv, abs/2205.1 14871, 2022.
10

---

Under review as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In /EEE/CVE Conference on Computer Vision and Pattern Recognition, 2022.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022.
A. Dosovitskiy, L. Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, G. Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In IEEE/CVF International Confer- ence on Computer Vision, 2021.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In JEEE/CVF International Conference Computer Vision, 2019.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training ima- 2017a.
genet hour. ArXiv,
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag. Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In IEEE International Conference on Computer Vision, 2017b.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll' ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In IEEE Conference Computer Vision and Pattern Recognition, 2015.
Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: Learning. 2016.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, 2015.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and An- drew Zisserman. The kinetics human action video dataset. ArXiv, abs/17055000 770.06950, 2017.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Bo- qing Gong. Movinets: Mobile video networks for efficient video recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.

---

Under review as a conference paper at ICLR 2023
Kunchang Li, Xianhang Li, Yali Wang. Jun Wang, and Yu Qiao. Ct-net: Channel tensorization net- work for video classification. In International Conference on Learning Representations, 2020a.
Kunchang Li, Yali Wang, Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Con- ference on Learning Representations, 2022a.
Kunchang Li, Yali Wang, Junhao Zhang. Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Jiao Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. ArXiv, abs/2201 000//201.09450, 2022b.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In JEEE/CVF conference on computer vision and pattern recognition, 2020b.
Yanghao Li, Chaoxia Wu, Haoqi Fan, Kartrikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detec- tion. ArXiv, abs/2112.01552 2021.
Jingyun Liang, Jie Cao, Guolei Sun, Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In JEEE/CVE International Conference on Computer Vision Workshops, 2021.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In IEEE International Conference on Computer Vision, 2019.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. arXiv preprint arXiv:2208. 03550, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In JEEE/CVF Interna- tional Conference on Computer Vision, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
I. Loshchilov and F. Hutter. Fixing weight decay regularization in adam. ArXiv, abs/1711.05100 2017a.
Iyy Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter- national Conference on Learning Representations, 2017b.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. ArXiv, 0tb//2104008880 2022.
Mathew Monfort, Bolei Zhou, Sarah Adel Bargal, Alex Andonian, Tom Yan, Kandan Ramakrishnan, Lisa M. Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, and Aude Oliva. Moments time dataset: One million videos for event understanding. IEEE Transactions Pattern Analysis and Machine Intelligence, -0,,8, 2020.
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In IEEE/CVF International Conference on Computer Vision, 2021.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recog- nition. ArXiv, 0ts/2208.02216. 2022.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. Parameter-effiient image-to- video transfer learning. arXiv, abs/2206. 13559, 2022.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze. Christoph Feichten- hofer, Andrea Vedaldi, and João F Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In Neural Information Processing Systems, 2021.
12

---

Under review as a conference paper at ICLR 2023
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In IEEE International Conference on Computer Vision, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- lional Conference on Machine Learning, 2021.
Michael S. Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Search- ing for multi-stream neural connectivity in video architectures. In International Conference on Learning Representations, 2020.
Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. To- kenlearner: Adaptive space-time tokenization for videos. In Advances in Neural Information Processing Systems, 2021.
Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor image worth 16x16 words, what is video worth? ArXiv, 01122103.19915. 2021.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-lange and-language tasks? In International Conference on Learning Representations, 2021.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky. Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15: 1929-1958, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shiens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAF: Masked autoencoders are data- efficient learners for self-supervised video pre-training. In Neural Information Processing Sys- tems, 2022.
Hugo Touvron, M. Cord, M. Douze, Francisco Massa, Alexandre Sablayrolles, and Herv's e J'egou. Training data-cfficient image transformers & distillation through attention. In International Con- ference on Machine Learning, 2021.
Hugo Touvron, Matthieu Cord, and Herv'e J' J'egou. Deit ii:: Revenge of the vit. ArXiv, abs/2204.071 18, 2022.
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In IEEE International Conference on Computer Vision, 2015.
Du Tran, Hong xiu Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Du Tran, Heng Wang, L.. Torresani, and Matt Feiszli. Video classification with channel-separated convolutional networks. In IEEE/CVF International Conference on Computer Vision, 2019.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Neural Information Processing Systems, 2017.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. European conference on computer vision, 2016.
13

---

Under review as a conference paper at ICLR 2023
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. TDN: Temporal difference networks for effi- cient action recognition. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2021a.
Wenhai Wang. Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE/CVE International Conference on Computer Vision, 2021b.
Wenhui Wang, Hangbo Bao, Li Dong. Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: Beit pretraining for all vision and vision-language tasks. ArXiv, abs/2208. 10442, 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Ross
Wightman. Pytorch image models.
https://olthub..omm/vw..gg
monge-modelln 2019.
Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In /EEE/CVF Interna- fional Conference on Computer Vision, 2019.
Boyang Xia, Wenhao Wu, Haoran Wang, Rui Su, Dongliang He, Haosen Yang, Xiaoran Fan, and Wanli Ouyang. Nsnet: Non-saliency suppression sampler for efficient video recognition. ArXiv, abs/2207.101 10388, 2022.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross B. Girshick. Early convolutions help transformers see better. Neural Information Processing Systems, 2021.
Enze Xic, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems, 2021.
Shen Yan, Xuehan Xiong, Anurag Arna,, Zhichao Lu. Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learn- ing Research, 2022.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating con- volution designs into visual transformers. In IEEE/CVF International Conference on Computer Vision, 2021a.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. ArXiv, abs/21 1. 1432, 2021b.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEF/CVF International Conference on Computer Vision, 2019.
Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. ArXiv, abs/2112.07175.. 2021a.
Chen-Lin Zhang, Jian Zhai Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. ArXiv, abs/2202.079255. 2022.
14

---

Under review as a conference paper at ICLR 2023
Zhagg Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. In IEEE/CVF International Conference on Computer Vision, 2021b.
Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In IEEE/CVF International Confer- ence on Computer Vision, 2019.
Yin-Dong Zheng, Zhaoyang Liu, Tong Lu, and Limin Wang. Dynamic sampling networks for efficient action recognition in videos. IEEE Transactions on Image Processing, 29:7970-7983, 2020.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In AAAI conference on artificial intelligence, 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ArXiv, abs/2010.04159, 2021.
15

---

Under review as a conference paper at ICLR 2023
Training #Samples
Validation #Samples
Average Length
Dataset
#Actions
Kinetics-710 (ours)
658,340
66,803
10s
710
240,436
19,787
Kinetics-400 Carreira Zisserman
2017
10s
246,245*
20,000*
366,006
27,935
Kinetics-600 Carreira al.
2018
10s
600
392,622*
30,000
529,573
33,861
Kinetics-700 Carreira et al. 2019
10s
700
545,317*
35,000*
802,244
33,899
Moments in Time Monfort ct al. 2020
3s
339
802,264*
33,900
Something-SSmeethingg VI (Goyal et al.
2017b
86,017
11,522
4.0s
174
Something -Soertiing V2 (Goyal et al.
20176
168,913
24.777
4.0s
174
Activity (Heilbron ct al. 2013)
10,024
4,926
117s
200
HACS Zhhä et al. 2019)
37,452
5,953
149s
200
Table 10: Dataset descriptions.
indicates the original video number.
K710
K400/600/700
MiT
ANet&HACS SSVI/V2
Optimization
Optimizer
AdamW (Loshchilov & Hutter 2017a
Momentum
31,32
U.9.0.995 0.999
Weight decay
0.05
Learning rate schedule
cosine decay Loshchilov & Hutter 2017b
Start learning rate
1e-6
End learning rate
1c-6
Batch size
256
512
64
Learning rate (Base)
2e-5
2e-6
2e-5
4e-5
Learning rate (Large)
le-5
1.5e-6
le-5
5c-6
2c-5
Warmup epuchs (Goyal et al. 2017a
5
5
Total epochs (Base)
55
5
24
Total epochs (Large)
40
5
18
20
15
Data augmentation
Inception-style cropping Szegedy al. 2015
Scale
[0.08, 1.00]
Jitter aspect ratio
[0.75. 1.33]
Color jitter
Rand augment (Cubuk et al.
2020
rand-m7 1001-m7-n44--eee fonddm7-n4-mstd.555
Repeated sampling (Hoffer et al 2020
2
Regularisation
Dropout (Srivastava et al. 2014
Backbone
0.5
Global branch
0.5
Drop path (Huang et al. 2016
Backbone
0.2
Global branch
0.4
Table 11: Training heprraarameters for our experiments. "."" indicates that the related method not used. Values constant in all the datasets are listed once. Datasets are denoted as follows: K (Kinetics), MiT (Moments in Time), ANet (ActivityNet), SS (Something-Somethig Something)).
A ADDITIONAL IMPLEMENTATION DETAILS
Datasets. In Table we give more details of our datasets. Kinetics family (Kay al. 2017) is the most widely-used benchmark, includes Kinetics-400, 600 and 700. Since some videos are unavailable on YouTube, the Kinetics datasets are gradually shrinking over time. We report the video number of our version for more fair comparison. Moments in Time V1 (Monfort et al. 2020) contains 0.8M 3-second video clips annotated with 339 classes, which suggests capturing the gist of dynamic scene. Something-Something V 1//V Goyal et al. (201 consist of 174 actions interacted with everyday objects. They require strong temporal model lig to distinguish confusing actions such opening/closing something. ActiviryNer (Heilbron al. 2015) and HACS (Zhao al. 2019) two large-scale untrimmed video benchmark. They respectively contain about 20K and 50K videos in 200 human daily living actions. For these two datasets, sample those video clips containing
16

---

Under review as a conference paper at ICLR 2023
CAN Foz. Fia, Foz, Video
Foa. Faa, Fett,
Pete Bea. Fea.,
GAR
(a) TimeSformer: push (x)
(b) UniFormerV2: yoga (v)
(c) UniFormerV1: yoga (V)
Figure 4: More visualizations. Frames are sampled from Kinetics according different sampling
strategies in different methods. For UniFormerV1, it samples double frames and downsamples the
temporal resolution the patch embedding.
action for training, thus we do not add another background class. While for testing, we sample the frames sparsely from the whole untrimmed videos.
Implementation Details. For the scene-related datasets, we only insert the global UniBlocks in the last layers of VIT-B/L. to perform multi-stage fusion, since the local UniBlocks and temporal downsampling do not further improve the results in Table 9a| But for Something-Sometthie adopt all the designs and insert the global UniBlocks in the last 8/16 layers of VIT-B/L for better temporal modeling. Besides, when finetuning those models with large-scale dataset pretraining, is necessary initialize the parameters properly. For stable training, we initialize some of the layers, including the last point-wise convolutions in the local temporal MHRA, the query tokens and output projection layers in the query-based cross MHRA, the last linear layers in the FFN of the global UniBlock, and the learnable fusion weights. What's more, we provide the detailed hyperparameters in Table IT Most of the training scripts follow UniFormer (Li al., 2022a)), but differently. we do not apply Mixup (Zhang et al. 2018). CutMix Yunn et al. al. 2019)). Label Smoothing (Szegedy et al. 2016) and Random Erasing (Zhong et al., 2020). When finetuning the full models on Kinetics directly from image pretraining, we adopt the same hyperparameters as in K710 pretraining. the backbone is frozen, we use larger learning rate (4e-4) without warmup.
VI/V2,
Training Cost. In table 9e we compare different training scripts. When finetuning Kinetics-400, 600 and 700 individually, we train the models for 55 epochs, and the total training data is about 0.24 + 0.366 + 0.529 1.14M. When pretraining with Kinetics-7 (0.66M), we only finetune the models for 5 epochs. Thus the percentage of saving cost is as follows,
1 0.66 x 55 + x 0.33
(14)
1.14 55
Thus we save almost 33% the training cost. More importantly, for the models with more frames (16, 32, even 64), only need to finetune the K710 pretrained models with 8 frames. Our training scripts are very efficient while effective for the Kinetics family.
B VISUALIZATIONS
In Figure we compared UniFormerV2 with the typical ViT-based model, i.e., TimeSformer (Berta- sius al. 202T), and UniFormerV1 (Li et al., 2022a) through visualization. Since UniFormer VI is multi-scal architecture, we show its features at the bottom of 4 stages. For TimeSformer and UniFormerV2, they are based on ViTs with fixed resolution, thus show their features every layers. We use CAM (Zhou et al. 2016) to show the most discriminative features that the network locates. The red parts indicate where the models focus more on, while the blue parts are ignored.
It reveals that both UniFormerV1 and UniFormerV2 are good at capturing local details, but Uni- 1 may lose information deeper layers due to the shrinking resolution, thus it fails to activate the discriminative parts. In contrast, TimeSformer only learns local features in the shallow layers, thus struggling to focus on meaningful areas. As for UniFormerV2, it surprisingly maintains local details even in the deep layers. More importantly, it observe the whole video and leam to concentrate more on the woman's 's leg, which helps recognize the action. These results demonstrate that our UniFormerV2 is effective capture local details and long-term dependency.
17

---

Under review as a conference paper at ICLR 2023
Method
#FFrmme
K400
Pretrain
#Frame
SSVI
SSV2
Only Global
8x3x4
81.8
CLIP-400M
16x3x1 1
56.8
69.5
Globll
8x3x4
84.4
400
16x3x1
55.8
68.4
Table 12: Output token combination.
Table 13: K400 pretraining.
#Query
#Frame
SSV2
Method
Param. (M)
FLOPs (G)
SSV2
16x3x1
69.5
Mean Pooling
422
4
16x3x1
69.1
Divided Space- Time MHSA
555
63.4
16
16x3x1
68.6
Joint Space-Time MHSA
86
539
65.8
Temporal Convolution
86
422
65.6
Table 14: Query Number.
Temporal Shift
86
422
6.577
Temporal Transformer
128
423
61.5
Local MHRA (Ours)
105
511
67.7
Table 15: Different modules.
Frame Cropx
Param.
FLOPs
K400
K600
K700
Method
Pretrain
(M)
(T)
Top-1
Top-5
Top-1
Top-5
Top-1
Top-
UmiFormerV2--/166
8x1x3
115
0.4
84.0
96.3
84.8
96.8
75.4
92.6
UnIHormerV2-B/16
8x3x4
115
1.6
84.4
96.3
85.0
97.0
75.8
92.8
CLIP-40DM
UnIFormerV2-1/14
8x1x3
354
2.0
87.3
97.7
87.8
97.6
80.0
95.0
V2-Lormer2--L/14
8x3x4
354
8.0
87.7
98.3
88.0
97.7
80.3
95.2
UniFormerV2-B/16
8x1x3
115
0.4
85.2
96.7
85.6
75.8
92.4
UmiForeerV-BB/16
8x3x4
115
1.8
85.6
97.0
86.1
97.2
76.3 92.7
UniFormerV2-LJ144
8x1x3
354
2.0
88.4
97.9
88.6
98.1
80.4 95.2
UnFooreeV2-LL14
8x3x4
354
8.0
88.8
98.1
89.0
98.2
80.8 95.4
UnFooreeV2-LLL14
16x3x1
354
4.0
88.9
98.0
89.2 98.2
80.9 95.4
U Fommer/2LL/14
16x3x4
354
16.0
89.1
98.2
89.4 98.3
81.2 95.6
CLIP-400M
Uni UniFomerr22-L14
32x3x1
354
16.0
89.2
98.2
89.3 98.2
81.3 95.6
+K710
Uni Formerr22-L114
32x3x2
354
16.0
89.3
98.2
89.5 98.3
81.5 95.7
UnLFormerV2-L/14
32x3x4
354
32.0
89.5
98.2
89.5
98.3
81.4
95.8
UnIFormerV2--L114 336
32x3x2
354
37.6
89.7
98.3
89.9
98.5
82.1
96.1
UniFormerV2-L/14433
32x3x4
354
75.3
89.7
98.3
89.9
98.5
82.2
96.1
UmFFormerV2-L114 336"
64x3x2
354
75.3
90.0
98.4
90.1
98.5
82.7
96.2
UniFormerV2-L/14433 336"
64x3x4
354
150.6
90.0
98.4
90.1
98.5
82.7
96.3
UniFormerV2-L/14 (frozen) 336 CLIP-400M
8x xx3
51
4.7
86.7
93.4
87.4
97.7
79.6
94.6
Jonooeer/--- (Irozen) 336T
8x1x3
15
4.7
87.8
98.0
88.2
98.0
79.7
94.7
CLIP-400M
UniFormerV2-L/14 (frozen) 336
32x3x1
51
18.8
88.8
98.1 1
89.1
98.2
80.6
95.2
+K710
UnFFomerV2-L/14 (frozen) 336
32x3x4
51
75.3
88.9
98.2
89.2
98.2
80.8
95.4
Table 16: More results on Kinetics-400, 600 and 700.
C MORE ABLATION STUDIES
We conduct more ablation studies based on CLIP-ViT-BIA (Radford al. 2021
Output token combination. When only using global token for classification. the top-1 accuracy drops from 84.4% to 81.8% in Table 12 shows that both local and global output tokens are essential for maintaining performance.
Kinetics pretraining for Something-Sometting Different from the prior works (Li et al. 2022a Fan et al. 2021). in Table 13. we find that extra Kinetics pretraining harms the representation inher- ited from CLIP, leading to lower performance.
Query number. In Table 14 try to increase the query number. However, more queries lead to severe overfitting, thus the performance drops.
compare our local MHRA with popular temporal modules, including simple mean pooling (Wang et al. 2016)), divided and joint space-time MHSA (Bertasius) al. 202I), temporal convolution (Tran al. 20188), temporal shift (Lin al. 2019) and temporal transformer (Sharir et al. 2021). All the modules are inserted before all the spatial MHSA, except that the 6-layer temporal transformer is added after the backbone. The results shows that our local MHRA beats the previous methods, achieving 2.0% to 22.6% higher top-1 accuracy. It demonstrate the effectiveness of our local MHRA for temporal modeling.
Different modules. In Table 15
18

---

Under review as a conference paper at ICLR 2023
Framex
Param
FLOPs
MIT
Method
Pretrain
C
(M)
(T)
Top-1
Top-5
UniFomerV2-B//16
CLIP-400M
8x3x4
115
1.8
42.2
71.3
UniFomerV2-B/16
CLIP-40OM
32x3x4
115
7.2
42.2
71.5
UmiFomerV2-B//16
CIIP-400M+K710
8x3x4
115
1.8
42.6
71.6
UmFFomeer2--B/16
LLIP-400M-K710-KKA4
8x3x4
115
1.8
42.6
UniFomerV2-B/16
CLIP-40M-KK710-K7000
8x3x4
115
1.8
42.4
71.2
UmfFomerr22LL11
CLIP-400M
8x3x4
354
8.0
46.2
76.0
UniFomerrV2-L/14
CLIP-400M
x6x3x4
354
16.0
46.2
76.2
UmiFomeerV2-L/14
CLIP-400M
32x3x4
354
32.0
46.4
76.2
UniFormerV2-L/14
CLIP-400M+K710
8x3x4
354
8.0
46.7
UniFomerV2-LL114
-00-0000M-K00004000
8x3x4
354
8.0
47.0
UniFoomerV2-IJ14 33367
CLIP-40DM
8x3x4
354
18.8
47.2
76.5
UniFomerV2-1J144 3367
SLJP-400M+K710
8x3x4
354
18.8
47.6
76.7
UniFomerV2-LL1443 3367
177-4000+KK00010
8x3x4
354
18.8
47.8
76.9
Table 17: More results on Moments in Time VI.
Framex
Param.
FLOPs
SSVI
SSV2
Method
Cropx Clip
(M)
(T)
Top-1
Top-5
Top-1
Top-5
UniFormerV--B/16
16x3x1
163
0.6
56.8
84.2
69.5
92.3
UniFormerV2-B/16
16x3x2
163
57.2
84.3
69.7
92.5
UniFormerV2-BB16
32x3x1
163
59.4
86.2
70.7
93.2
UniFormerV--B116
32x3x2
163
2.2
59.5
86.2
71.0
93.2
UnfFormerV2-L114
16x3x1
574
2,6
60.5
86.5
72.1
93.6
UniFormerV2-L/14
16x3x2
574
5.2
60.9
86.8
72.2
93.7
UniFormerV--L114
32x3x1
574
5,2
62.7
88.0
73.0
94.5
UniFormerV2-L14
32x3x2
574
62.9
88.3
Table 18: More results on Somihiinng-Smttiig.. All models are directly finetuned from CLIP.
3x4
10
Dataset
Pretrain
Frame
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
400
8
92.8
99.0
92.8
99.1
93.0
99.1
CL1P--400M+K4000
16
93.5
99.4
93.5
99.5
93.6
99.5
ActivityNet
-LLP-400M-K710--4400
93.9
99.4
99.5
CLIP-40MM-K<10-KX700
94.0
99.4
94.2
99.5
94.3
99.6
CLIP-400M-K710-KKAA
32
94.3
99.6
94.5
99.6
94.7
99.5
CLIP-400M-K400
16
94.7
99.8
94.7
99.8
94.9
99,9
HACS
BLIP-400M-K710+KKO
16
95.3
99.9
95.2
99.8
95.5
99.8
CLIP-400M-K7700-X7700
94.7
99.7
94.7
99.8
94.9
32
95.2
99.8
95.3
99.8
95.4
99.8
Table 19: More results ActivityNet and HACS. All models are based on UniFormerV2-LLe
D ADDITIONAL RESULTS
In Table 16 Table 17 Table 18 and Table 19 we give more results on the 8 video benchmarks, i.e.,
Kinetics-4000/000 700, Moments in Time, Something-Somethhng V1/V2, ActivityNet and HACS.
E
MORE DISCUSSIONS
Local UniBlock vs. ST-Adapter (Pan et al. 2022). Our Local UniBlock is motivated by the style of UniForme rıLi et 2022a), i.e. we treat temporal depth- convolution as local temporal relation aggregator. Hence, like UniFormer, we introduce extra BatchNorm (Toffe & Szegedy 2015) before the first linear projection V(.). Alternatively, ST-adapter does not have this design, since it simply treats temporal depth-wise convolution adaptation. With such motivation, further introduces extra activation function for enhancing such adaptation, while our local UniBlock does not need it. In fact, have also made comparisons in Table 9b It shows that local MHRA beats ST-Adapter (69.15 % vs. 68.0%).
Perceiver (Jaegle et al. 2021), DETR (Carion et al., 2020 and Flamingo (lamingo(Alayrac al., (202)). Our Glocal UniBlock also motivated by the style of UniFormer (Li et al. 2022a). But differently, decrease the global computation in UniFormer, change
Global UniBlock
19

---

Under review as a conference paper at ICLR 2023
self-attention MHRA cross-attention MHRA in our UniFormerV2. Hence, Global UniBlock consists of Dynamic Position Embedding (DPE), cross MHRA and FFN. On the contrary, none of those works belong such operation combination, without insight of UniFormer in video learning. fact, these methods often use the standard cross-style transformer block including self MHRA, cross MHRA and FFN.
Limitations. UniFormerV2, we propose the effective designs to arm pretrained ViT as spatiotem- poral learners. Although its training more efficient compared non-trivial video backbones, its performance tends to depend on the scale of pretraining data, as shown in Table g Hence, it would be interesting to explore our UniFormerV2 on huge image foundation models pretrained by massive datasets, for further evaluating its scalability and generalization capacity.
F LABEL LIST OF KINETICS-7I0
To generate our Kinetics-7Hl 710, we align labels in different Kinetics datasets by filtering symbols and replacing synonyms. The final label list shown Table20 Compared with Kinetics-700. there are 8 and unique labels in Kinetics-400 and Kinetics-600 respectively. When finetuning the models pretrained on Kinetics-710, it is vital to load the pretrained weight of the classification layer, thus map the weight according to the label list.
20

---

Under review as a conference paper at ICLR 2023
Table 20: Labels of Kinetics-710,
Label
K4
K6
K7
Label
K4
K6
K7 Label
K4
K6
K7
luge
x
krumping
skiing mono ski jumping driving car tap dancing hockey stop tobogganing
yoga
slapping
vault
v
x
decoupage
X
X
4
squat
arresting
lunge
surveying
zumba
fly tying
X
situp
V
capsizing
X
cooking egg
sewing
X
tiptoeing
slacklining
cumbia
using
pushing
crying
waking up fidgeting
ice skating
dining
ice fishing celebrating
digging
tie dying wrestling whistling high kick abseiling high jump trapezing
chasing
windsurfing
sieving
riding mule
staring
waxing legs deadlifting bee keeping pumping gas tapping pen headbanging bookbinding flying kite fixing hair
karaoke
burping
packing
licking
skydiving
winking
bandaging side kick jetsküng
arguing
ironing
drawing
long jump
archery
hopscotch
egg hunting mowing lawn
jogging
dodgeball
singing
crocheting
triple jump
yawning
ski ballet
milking
writing
geocaching
doing nails dyeing hair eating cake paragliding headbutting
push up
bulldozing
tai chi
cosplaying spelunking
sailing
welding
jaywalking
smoking
head stand
bobsledding
parkour
contorting
kitesurfing
texting
plastering
petting cat waxing back making slime steering car rolling eyes moving child pouring milk grooming cat doing sudoku closing door pouring wine cutting cake milking goat playing oboe filling cake sanding wood jumping sofa taking photo silent disco ironing hair
bowling
bartending beatboxing applauding pole vault barbequing snowkiting making tea auctioning snorkeling testifying high fiving moving baby shoot dance pirouetting
kissing
busking
gargling
spraying
coughing
saluting
shouting
sleeping
smashing
tackling
shopping
pinching
X
huddling
bottling
coloring in sawing wood
drooling
tickling
calculating
X
X
knitting
waving hand
X
X
unboxing
watching
shot put
calligraphy
planing wood
marching
carving ice bodysurfing
gold panning
capoeira
X
pillow fight
X
pull ups
lifting hat
combing hair
X
laughing
bathing dog
laying stone
hurdling
chewing gum parasailing
photobombing
sncezing
playing lute
clapping
sipping cup
land sailing
21

---

Under review as a conference paper at ICLR 2023
Label
K4
K6
K7
Label
K4
K6
K7
Label
K4
K6
K7
scrapbooking
X
washing feet diving cliff golf putting motorcycling
ripping paper crawling baby
tasting wine
X
V
v
V
docking boat
X
V
V
cleaning pool brushing hair
4
photocopying
X
V
V
V
V
clam digging
V
breakdancing
V
V
sanding floor
V
ice swimming
X
drinking heer
V
X
belly dancing
roasting pig
X
V
swinging legs
V
X
feeding goats shaking hands swing dancing carrying baby bending metal playing poker grinding meat shining shoes folding paper blasting sand arm wrestling rock climhing catching fish playing drums cracking neck tying necktic juggling fire golf chipping javelin throw
pouring beer
X
bull fighting
X
V
smoking pipe
X
tossing salad
X
lock picking
playing cards
steer roping
X
slicing onion
x
hugging baby
X
stacking diee
x
embroidering
X
helmet diving
longhoarding
dealing cards
laying tiles
treating wood
playing gong
X
eating nachos
X
base jumping
X
being excited
X
X
playing polo moon walking
X
vacuuriing car
X
x
petting horse
X
opening door tasting food shaving legs pumping fist making sushi snowmobiling tasting beer golf driving waxing chest
stacking cups
poaching eggs
yarn spinning
X
card stacking
X
rope pushdown
smelling feet card throwing playing darts chopping meat making cheese
skateboarding
laying bricks playing piano playing flute salsa dancing cating burger skipping rope climbing tree washing hands playing chess tango dancing using computer cleaning floor exercising arm baby waking up waxing armpits mixing colours carving marble
X
faceplanting
cating chips
crossing eyes
playing harp
cracking back
spianing poi front raises
building lego
using inhaler
X
reading book
jumping jacks
shaking head
using puppets sucking lolly
snowboarding
scuba diving
cutting apple lighting fire surfing water playing organ hoverboarding
bending back
drop kicking
using segway
ice elimbing
tosiing coin
feeding birds
cheerleading
blowing glass
blowing nose
building shed
peeling banana breaking glass
pushing cart water skiing making pizza punching bag
setting table doing laundry braiding hair mopping floor tying bow tie cutting nails skiing slalom making cake chopping wood
laying decking brushing floor
herding cattle blending fruit seasoning food checking watch
X
feeding fish
X
riding camel shaving head
X
throwing axe
massaging neck leatherworking
grooming dog
X
curling hair
somersaulting riding bike
acting in play
X
air drumming
chiseling wood
X
training dog
surfing crowd
square dancing
disc golfing
holding snake
sausage making
hula hooping
water sliding
using a wrench
washing hair
V
playing cello
weaving fabric
X
cartwheeling
V
throwing ball
breathing fire
changing oil
cating hotdog
rolling pastry
hammer throw
rohot dancing
cutting orange
22

---

Under review as a conference paper at ICLR 2023
Label
K4
K6
K7
Label
K4
K6
K7
Label
K4
K6
K7
needle telting
Ripping bottle
lagging grafithi raising eyebrows
x
skipping stone
X
V
splashing water
X
X
x
scrubbing face
X
V
V
carrying weight
X
X
threading needle
X
flint knapping
V
V
spinning plates fencing (sport)
popping balloons
shuffling feet
V
cooking scallops
X
throwing knife
X
V
curling (sport)
X
backflip (human) falling off bike playing scrabble visiting the zoo mosh pit dancing shucking oysters looking at phone throwing tantrum tying shoe laces dancing macarena playing bagpipes eating ice cream playing monopoly flipping pancake getting tattoo building cubinet playing clarinet eating spaghelti drumming fingers cating doughnuts playing trombone moving furniture contact juggling playing recorder wrapping present
X
fixing bicycle
X
separating eggs
X
X
making bubbles
<
V
playing ocarina
X
counting money applying cream blowing leaves shoveling snow brush painting making the bed playing tennis playing violin tapping guitar picking apples
playing netball
polishing metal
jumping bicycle
X
trimiiig shrubs
X
X
playing marbles
X
blowdrying hair
dyeing eyebrows
laying concrete
playing pinball
X
dumpster diving putting on sari playing maracas delivering mail proparing salad vacuuming floor
X
doing aerobics
drinking shots
bungee jumping
shearing shoop juggling balls stretching arm news anchoring smoking hookah massaging back weaving basket
X
X
chiseling stone
breaking boards
elimbing ladder hurling (sport) throwing discus recording music playing trumpet sled dog racing stomping grapes carving pumpkin unloading truck watering plants playing ukulele cleaning toilet folding napkins playing cymbals riding unicycle playing cricket climbing rope scrambling eggs opening present folding clothes waiting line finger snapping riding elephant waxing eyebrows shuffling cards walking the dog driving tractor strumming guitar filling eyebrows playing rounders squeezing orange making latte art opening coconuts
making snowman
checking tires
planting trees
hitting baseball playing kickball cleaning gutters cleaning windows peeling potatoes playing keyboard looking in mirror walking on stilts playing billiards curling eyelashes playing beer pong directing traffic twiddling fingers
spray painting
stretching leg
clean and jerk
peeling apples dancing ballet making jewelry grooming horse
X
X
playing guitar
X X
sword fighting washing dishes roller skating massaging foet clcaning shoes bench pressing riding scooter sweeping floor brushing teeth trimming trees baking cookies massaging legs crossing river cating carrots taking a shower cooking chicken shredding paper metal dctecting
X
X
marriage proposal
X
making horseshoes
X
cracking knuckles
adjusting glasses
X
tightrope walking playing laser tag X
X
installing carpet lawn mower racing standing on hands playing pan pipes playing ping pong falling off chair playing blackjack
X
X
x
x
X
X
x
X
mushroom foraging
lighting candle
X
X
playing harmonica
using megaphone
X
X
playing checkers
X X
cutting pineapple
playing piccolo
X
X
sword swallowing
sharpening knives
entering church
playing dominoes
playing badminton
playing mahjong
putting on shoes
getting haircut
23

---

Under review as a conference paper at ICLR 2023
Label
K4
K6
K7
Label
K4
K6
K7
Label
K4 K6
K7
playing saxophone
V
swimminng backstroke sking crosscountry answering questions assembling computer sticking tongue out biking through snow playing bass guitar shooting basketball blowing out candles rock scissors paper riding mountain bike playing slot machine swimming with sharks playing shufflehour using paint roller home roasting coffee battle rope training changing gear in car swinming front crawl wading through water walking through snow attending conference casting fishing line opening refrigerator hand washing clothes
putting wallpuper on wall playing american football carving wood with knife bouncing on bouncy casile putting in contact lenses archaeological excavation swimming butterfly stroke tying knot (not on a tic) person collecting garhage trimming or shaving beard giving or recetving award breading or breadrrumbing opening bottle (not wine) sign language interpreting mountain climber (exeroive) playing hand clapping games presenting w weather forecust bouncing ball
making sandwich
v
V
V
V
playing xylophone
7
<
V
V
reading newspaper
V
<
jumping into pool arannging flowers
V
V
<
frying vegetables
V
<
<
sharpening pencil
<
playing accordion
eating watermelon
jumpstyle dancing playing paintball playing nose flute getting piercing wading through mud wood burning (art) using circular saw assembling hicycle blowing bubble gum repairing puncture poking bellybutton
x
x
x
x
X
X
X
x
fnot juggling)
X
changing wheel fnot on bike)
x
x
catching throwing frisbee
putting on mascara throwing snowballs riding snow blower shining flashlight using microscope kicking field goal playing ice hockey playing controller cutting watermelon dancing charleston hugging (not baby)
X
x
riding walking with horse
X
X
catching
playing field hockey juggling soccer ball dribbling baskethall country line dancing canocing or kayaking running on treadmill walking with crutches pulling espresso shot letting go of balloon being in zero gruvity roasting manshmallows using bagging machine
X
throwing softhall
playing squash or racquelbull
<
V
-
decorating the christmas tree
V
V
catching or throwing baschall
x
x
springboard diving
x
x
exercising with exercise ball passing American foothall game)
playing basketball
x
dunking basketball playing volleyhall playing didgeridoo inflating balloons extinguishing fire
x
X
passing American foothall fnot in game)
X
talking on cell phone putting on foundation using sledge hammer
X
X
pushing wheelchair chopping vegetables pulling rope (game) picking blucherrics
x
swinging baseball bat making balloon shapes dancing gangnam style cooking sausages snatch weight lifting swinging on something swimming with dolphins shooting off fireworks throwing water balloon historical reenactment swinming breast stroke bouncing on trampoline shooting goal (soccer) riding mechanical bull making paper aeroplanes using remote controller massaging person's 's head
X
V
X
V
playing road hockey uncorking champagne polishing furniture playing with trains
V
x X
pushing whoelbarow shaping hread dough
x
x X
x
x V
alligator wrestling building sandcastle
x <
V
doing jigsaw puzzic opening wine bottle putting on eyeliner passing soccer ball playing rubiks eube using power drill
x
X
V 4
V
putting on lipstick kicking soccer ball
gospel singing church
X
punching person (boxing)
V V
cooking on camplire gymnastics tumbling clay pottery making
petting animal (not cat)
pretending to he statue
x
listening with headphones
x
24


Reviews:
[{"review_id": "thdtNte9zZA", "reviewer": "Reviewer_AXo7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new video Transformer architecture, termed UniformerV2, that extends the pre-trained ViTs (on images) to the video action recognition tasks. Specifically, UniformerV2 introduces local temporal aggregation, global temporal modeling and multi-stage fusion to enhance the original image-based ViTs to capture temporal information. Although some of the modules are inspired by the design in Uniformer, the paper proposes better and more efficient design choices for local / global temporal aggregation, and presents extensive ablation studies to verify their effectiveness. The paper reports experiment results on multiple video recognition benchmarks and UniformerV2 achieves state-of-the-art results.", "summary_of_the_review": "This paper proposes a good design choice for extending pretrained ViTs to video recognition task. The proposed method achieves superior results on multiple video recognition benchmarks and the effectiveness of the model is verified by extensive ablation study. In summary, I suggest \"accept\" to this paper.\n\n*After reviewers' discussion*: I agree with the other reviewers that the novelty of the proposed local & global block is not as significant as claimed in the paper. However, I do feel that this paper provides some good practice and insights for making use of pre-trained image-based ViTs for spatiotemporal modeling in video recognition. Although the individual modules are not novel, the whole system presented in this paper still differs from prior work on extending image ViTs to video modeling (e.g., ST-Adapter, X-CLIP) and achieves better results (or comparable results with better efficiency). By reconsidering the strength and weakness of this paper, I'd update my rating to \"weak accept\".", "strength_and_weaknesses": "Strengths\n1. The paper presents strong experimental results on multiple video recognition datasets, including those requires stronger capacity on temporal modeling (e.g., Something-something) and long-range modeling (e.g., ActivityNet). The paper also provides extensive ablation study to present the impact of each component of the model and different design choices. These design choices are good practices for video understanding research and could be valuable for the community.\n\n\nMinor questions:\n1. How is the 3D convolution for patch embedding initialized? Is it inflated from patch embedding weights in the ViTs? \n2. The four multi-stage fusion strategies share similar results in the ablation study. Which one would the author prefer to use in this case? Other than recognition accuracy, how about the comparison of computational cost / running time among them?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mkTZGFQJkX", "reviewer": "Reviewer_GtFh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes UniFormerV2, which could arm the readily available and well-pretrained image ViT with efficient Uniformer designs. Extensive experiments demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper proposes an effective way of using well-pretrained image ViTs to facilitate the performance of video ViTs. The proposed method achieves state-of-the-art results on 8 popular video benchmarks. \n\n**After rebuttal**: I carefully read the discussion between Reviwer ikef and the authors. I agree that some contributions are overclaimed since they are not entirely new and bear some similarities with other architectures. As mentioned by AC, this may need a re-writing of the manuscript and a new round of review.  But I think this paper gives us some good insights and practices on designing competitive video backbones. I lean to weekly accept this paper. ", "strength_and_weaknesses": "Pros:\n1. The idea of arming rich well-pretrained image ViTs with efficient Uniformer design for video performance improvement is novel and useful. \n2. The proposed method achieves state-of-the-art results on many popular video benchmarks.\n\nCons:\n1. Lack of analysis of limitations of the proposed method\n2. What is the video pretrain method used in Table 1? Is it supervised or self-supervised? And how many epochs does it take?\n3. Why not use video pretrain in Table 4 for SSV2? Is there any special consideration?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EUOvpBRpqN", "reviewer": "Reviewer_ikef", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a new improved architecture for video recognition based on Uniformer (ICLR 2022), dubbed Uniformer-V2. To this end some of the techniques proposed in Uniformer are re-used and extended to create a video architecture that can benefit from existing pre-trained ViT models (CLIP in particular). The architecture is  evaluated on a number of video/action recognition benchmarks where the method is shown to be among the best reported method.", "summary_of_the_review": "The paper has some merits but it's confusingly written. Moreover, the contributions in my opinion are not so strong as claimed. Some of the experimental results appear strong, but a closer look reveals that, in many cases, they are not so impressive as might have been thought from a first glance.", "strength_and_weaknesses": "Strengths:\n+ CLIP adaptation methods is a reasonable direction for improving video recognition although the best models are really huge.\n+ The architecture is evaluated on a large number of benchmarks where the method is shown to be among the most accurate ones. \n\nWeaknesses:\n- The method proposed in this paper is primarily an adaptation of CLIP for video recognition and it should have been presented as such. On the contrary the authors have tried to present Uniformer-V2 as a \"generic paradigm to build a powerful family of video networks\". \n- The novelty of the proposed local and global uni-blocks seems thin: The local uni-block resembles a temporal depthwise convolution (can the authors make this clear as in the uniformer paper?) followed by a spatial transformer block as in the image encoder of CLIP. Hence, this block is basically a simple temporal adapter that is put on top of CLIP's ViT (something similar I believe has been proposed in Pan 2022). The global block resembles a perceiver block (ICML 2021) and/or DETR decoder (ECCV 2020) and in a similar fashion has been used in Flamingo (Neurips 2022). I think the authors need to discuss the similarities with these works.  \n- The experimental results are the paper's strongest point but they are not so impressive as they look from a first glance: (1) by using CLIP, there's a huge advantage over all other methods (which dont use CLIP), but that comes to no surprise. (2) When CLIP is not used, as in Table 1 (upper part), Uniformer-V2 is not better than Uniformer (which is trained on IN-1K). (3) When CLIP is used, the best results are reported by finetuning on their K710 (can you please clarify why this is needed? and explain in relation to table 9e) (4) On Something-Something V2, Uniformer outperforms Uniformer-V2 for the same comp. budget.\n\n**After rebuttal:**\nUnfortunately, the rebuttal does not address my concerns as also explained in my responses to the authors' responses. I still believe that the proposed blocks (local and global) must be described in a more clear manner to fully understand the differences to previous work. As it stands the proposed contributions seem thin. Moreover, as demonstrated in my review, the results are not so significant as claimed. For example on SS-V2, their method is outperformed by previous work and on K400 to surpass EVL their method needs additional pre-training.  Overall the paper shows that if CLIP is used one can achieve very high results on video recognition which is not a surprising result at all. Hence, I would like to keep my original score.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DUrnpPr-e9", "reviewer": "Reviewer_DTkU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new video transformer backbone, which is named UniFormerV2. Compared to the previous UniFormer, this paper improves the original local Multi-Head Relation Aggregator (MHRA) with one local MHRA temporal and one global MHRA spatial blocks. In which, the global MHRA spatial block can leverage the pretrained ViT model. Extensive experimental results demonstrate the new state-of-the-art performance in various video benchmarks.\n", "summary_of_the_review": "Overall, I would vote weakly accept for this paper. The main reasons are: 1. The new design of UniFormer is well introduced and verified. 2. Extensive experimental results in 8 popular video benchmarks demonstrate its effectiveness and will potentially contribute to the community by releasing the pre-trained model.", "strength_and_weaknesses": "Strength:\n\nThis paper is well-written. The discussion of new architecture design and empirical study are mostly well done. The motivation is well justified and the proposed algorithm is easy to follow.\n \nThe major contribution of this paper can be summarized as:\n\n1. it decouple the previous local uniblock with one local MHRA temporal and one global MHRA spatial blocks, which not only reduce the local temporal redundancy, but also inherit the effective image pretraining of ViT architecture.\n\n2. This work provides extensive empirical study of multi-stage fusion methods, pre training pipelines, training cost and intermediate feature locations of UniFormer. This provides insights of how each component works in the UniFormer.\n\n3. This paper demonstrates strong performance in various video benchmarks, outperforming previous state-of-the-art performance. I believe the pre-trained models of this work are able to contribute to the community in various downstream video-based tasks.\n \nFrom Figure one, it can be seen that the UniFormerV2 achieves good accuracy-FLOPs balance. Compared to other methods, UniFormerV2 achieves better performance with lower cost.\n \nThe implementation details and ablation set-up are well introduced, which provides much convenience for other researchers to re-implement the proposed algorithm in this paper.\n\nWeakness & Questions:\n\nCompared to the Local UniBlock in the UniFormerV1, the current version looks like a hybrid block of MHRA and MHSA. It would be interesting to see the performance of swapping local MHRA temporal with the temporal MHSA. In this case, the Local UniBlock will be like a divided Space-time attention introduced in the TimsFormer[1]\n \nWhen leveraging the MHSA from image pretrained ViT, how to deal with the gap between ViT features and UniBlock features. The input to Global MHRA Spatial in UniFormer and the one to ViT MHSA should be very different.\n \nThanks authors for doing lots of experiments in various datasets. However, from the limited number of apple-to-apple comparisons, the improvement of UniFormerV2 is not significant.(for example compared to the MViTv2)\n\n[1] Is Space-Time Attention All You Need for Video Understanding? ICML, 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject