UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning
Authors: Kunchang Li
Yali Wang
Gao Peng
Guanglu Song
Yu Liu
Hongsheng Li
Yu Qiao
Venue: ICLR 2022 Poster (2022)
Keywords: Spatial-Temporal Representation Learning
3D Convolution
Transformer

Abstract: It is a challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due to large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. Although 3D convolution can efficiently aggregate local context to suppress local redundancy from a small 3D neighborhood, it lacks the capability to capture global dependency because of the limited receptive field. Alternatively, vision transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation on reducing local redundancy with blind similarity comparison among all the tokens in each layer. Based on these observations, we propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computation and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, our UniFormer achieves 82.9%/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods. For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy respectively. Code is available at https://github.com/Sense-X/UniFormer.

Published as a conference paper at ICLR 2022
UNIFORMER: UNIFIED TRANSFORMER FOR EFFICIENT SPATIOTEMPORAL REPRESENTATION LEARNING
Kunchang Li Yali Wang"; Peng Gaoi, Guanglu Song*
Yu Liu', Hongsheng Li , Yu Qiao*3i
"ShenZhen Key Lab of Computer Vision and Pattern Recognition, STAT-SenseTime Joint Lab, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
*University of Chinese Academy of Sciences. 'Shanghai AI Laboratory, Shanghai, China "SenseTTmme Research, 'The Chinese University of Hong Kong (kc. i, yl wang @ aaiittt. ac en, {gaopeng, qiaoyu @p j j1 aal.bb.. .rrg.or..o songguanglu@senee sengeetttsteeeteeeeeeeee ime. com, liuyuis saaariiianai@@maaaaaaa com hslißee cuhk. edu. hk
ABSTRACT
It is challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transform- ers. Although 3D convolution can efficiently aggregate local context suppress local redundancy from small 3D neighborhood, it lacks the capability to cap- ture global dependency because of the limited receptive field. Alternatively, vi- sion transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation similarity comparison among all the tokens in each layer. Based on these ob- servations, we propose novel Unified transFormer (UniFormer) which seam- lessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computa- tion and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct exten- sive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics- Someetinn VI&V2. With only ImageNet-IK pretraining,
reducing local redundancy with blind
600, and our UniFormer achieves 82.9% 8% top-1 accuracy on Kinetics-400/Kineee 600, while requiring 10x fewer GFLOPs than other state-of-the-an methods. For Something-Somen
V1 and V2, our UniFormer achieves new state-of-the-aud performances of 60.9% and 71.2% top-1 accuracy respectively. Code is available at htnps://gituubccoo/eeneee///iiii
INTRODUCTION
Learning spatiotemporal representations fundamental task for video understanding. Basically, there two distinct challenges. On the hand, videos contain large spatiotemporal redundancy, where target motions across local neighboring frames are subtle. On the other hand, videos contain complex spatiotemporal dependency, since target relations across long-range frames dynamic.
The advances in video classification have mostly driven by 3D convolutional neural networks (Tran al. 22155; Carreira & Zisserman, 220 7b Feichtenhofer et al., (0119) and spatiotemporal transform- ers (Bertasius et al. 2021 Amab et al. (021). Unfortunately, each of these two frameworks focuses on one of the aforementioned challenges. 3D convolution can capture detailed and local spatiotem- poral features, by processing each pixel with context from small 3D neighborhood (e.g.. 3x3x3).
first authors ((kc.li, yl.wang) @ @siat.ac.cn) Conresponding author (qiaoyu @pjlah.org.c))

---

Published as a conference paper at ICLR 2022
Input Frames
Layer3 Output Feature
Layer3
Spatial
Attention
Layer3
Temporal
Attention
Time
Figure 1: Some visualizations of TimeSformer. We respectively show the feature, spatial and temporal attention from the 3rd layer of TimeSformer (Bertasius et al. 2021). We find that, such transformer learns local representations with redundant global attention. For anchor token (green box), spatial/temporal attention compares it with all the contextual tokens for aggregation, while only its neighboring tokens (boxes filled with red color) actually work. Hence, it wastes large computation to encode only very local spatiotemporal representations.
BL9131
K400 Top-1 Accuracy (%6
B15(4)
Top Accuracy (96
70-
516(4)
68
532/11
66
X3D
TENet IN JK
S1613)
Molilee:
Slowfast
64
TON INIK
CTi-eel IN-LK
MVT
5th
76
lineSformer Durs H--3
TimeSformer I8-21K
1000
3000
5000
FLOPs/Vide (G)
FLOPs//dde (G)
Figure 2: Accuracy vs. per-video GFLOPs on Kinetics-400 and Sonething-SSmethiinn V2. B- 32(4) means we test UniFormer- (ooomerr-Bgg// with 4 clips and S-16(3) means we UniFormer-S -1nmer--Sigfaa with 3 crops (more testing details can be found in Section 4.3) Our UniFormer achieves the best balance between accuracy and computation on both datasets.
Hence, can reduce spatiotemporal redundancy across adjacent frames. However, due to the lim- ited receptive field, 3D convolution suffers from difficulty in learning long-range dependency (Wang ctt al.] [2018) Li et al. (020a). Alternatively, vision transformers are good at capturing global depen- dency, with the help of self-attention among visual tokens (Dosovitskiy al. 2021). Recently, this design has been introduced in video classification via spatiotemporal attention mechanism (Berta- sius al. (021). However, we observe that, video transformers are often inefficient encode local spatiotemporal features in the shallow layers. We take the well-known and typical TimeSformer (Bertasius al. (02T)) for illustration. As shown in Figure [] TimeSformer indeed learns detailed video representations in the early layers, but with very redundant spatial and temporal attention. Specifically. spatial attention mainly focuses on the neighbor tokens (mostly in 3x3 3 local regions), while learning nothing from the rest tokens in the same frame. Similarly, temporal attention mostly only aggregates tokens the adjacent frames, while ignoring the rest the distant frames. More importantly, such local representations are learned from global token-to-token similarity comparison all layers, requiring large computation cost. This fact clearly deteriorates computation-acout balance of such video transformer (Figure?) 2
To tackle these difficulties, we propose to effectively unify 3D convolution and spatiotemporal self- attention in concise transformer format, thus name the network Unified transFormer (Uni- Former), which can achieve preferable balance between efficiency and effectiveness. More specif- ically, UniFormer consists of three core modules, i.e., Dynamic Position Embedding (DPE),
2

---

Published as a conference paper at ICLR 2022
Multi-Head Relation Aggregator (MHRA), and Feed-Forward Network (FFN). The key difference between our UniFormer and traditional video transformers is the distinct design of our relation ag- gregator. First, instead of utilizing self-attention mechanism in all layers, our proposed relation aggregator tackles video redundancy and dependency respectively. In the shallow layers, our ag- gregator learns local relation with small learnable parameter matrix, which can largely reduce computation burden by aggregating context from adjacent tokens in a small 3D neighborhood. In the deep layers, our aggregator learns global relation with similarity comparison, which can flexibly build long-range token dependencies from distant frames in the video. Second, different from spa- tial and temporal attention separation in the traditional transformers (Bertasius et al. 2021] Arnab et al. )02T), our relation aggregator jointly encodes spatiotemporal context in all the layers, which can further boost video representations in a joint learning manner. Finally, we build up our model by progressively integrating UniFormer blocks in a hierarchical manner. In this case, we enlarge the cooperative power of local and global UniFormer blocks for efficient spatiotemporal representa- tion learning in videos. We conduct extensive experiments the popular video benchmarks, c.g., Kineticss-400 (Carreira & Zisserman 2017a), Kinetics-600 (Carreira et al.) 2018) and Something- Something V1&V2 (Goyal et al.] 2017b). With only ImageNet- IK pretraining,
UniFormer achieves 82.9% top-1 accuracy on Kinetics-400KKinetiiss -600, while requiring 10x fewer GFLOPs than other comparable methods (e.g.. 16.7x fewer GFLOPs than ViViT (Arnab et al.) 2021 with JFT-300M pre- pre-training). For Something-Something VI and V2, our UniFormer achieves 60.9% and 71.2% top-1 accuracy respectively, which are new state-of- -----ht-artt performances.
RELATED WORK
Convolution- Video Networks. 3D Convolution Neural Networks (CNNs) have been dom- inant in video understanding (Tran et al. 2015 Feichtenhofer et al. 2019). However, they suffer from the difficult optimization problem and large computation cost. To resolve this issue, I3D (Car-
reira & Zisserman 20176) inflates the pre-trained 2D convolution kernels for better optimization.
Other prior works Tran al., 2018 Qiu al. [017]] Tran et al., 2019 Feichtenhofer 2020 Wang
et al. (020a) try to factorize 3D convolution kernel in different dimensions to reduce complexity.
Recent methods propose well-designed modules to enhance the temporal modeling ability for 2D
Luo CNNs (Wang et al. 2016 Lin et al. 2019 Luo & Yuille 2019; Jiang et al., 2019 Liu et al. 2020aa
Li ct al.) 2020b Kwon et al., 2020 Li et al., [020al] 202 Wang et al. 2020b). However, 3D
convolution struggles to capture long-rauge range dependency, due to the limited receptive field.
Transformer-based Video Networks. Vision Transformers (Dosovitskiy et al. 2021; Touvron et al. 2021a|b b; Liu al. 2021a) have been popular for vision tasks and outperform many CNNs.
Based on VIT, several prior works (Bertasius et al. 2021 Neimark et al. 2021 [hharir et al. 2021
Li 20216; Arnab et al. 2021 Bulat al.] Patrick al. [021]] Zha et
(221)) 21.21 propose
different variants for spatiotemporal learning, verifying the outstanding ability of the transformer to capture long-term dependencies. To reduce high dot-product computation, MVIT (Fan et al.) 2021)
introduces the hicrarchical structure and pooling self-attention, while Video Swin (Liu et al. 2021b) advocates an inductive bias of locality for video. Nevertheless, the self-attention mechanism inef- ficient to encode low-level features, hindering their high potential. To tackle this challenge, different from Video Swin that applies self-attention in a local 3D window, we adopt 3D convolution in a concise transformer format to encode local features. Besides, we follow their hierarchical designs and propose our UniFormer, achieving powerful performance for video understanding.
METHOD
In this section, we describe our UniFormer in detail. First, we introduce the overall architecture of the UniFormer block. Then, we explain the vital designs of our UniFormer for spatiotemporal modeling, i.e., multi-head relation aggregator and dynamic position embedding. Finally, we hierar- chically stack UniFormer blocks to build up our video network.
3.1 OVERVIEW OF UNIFORMER BLOCK
To overcome problems of spatiotemporal redundancy and dependency, we propose a novel and con- cise Unified transFormer (UniFormer) shown in Figure 3 We utilize basic transformer format
3

---

Published as a conference paper at ICLR 2022
3xTxXX
1 7
128K
320x 2" 16 11
11 W
512x 2M 33
P
EX24T SEUES
Ex Sndde
1
FaBens
EoBles
*a
215
Stage,
XL1
DPE
MHRA
FFN
VE
DWCcomv
Norm
Concat
Norm
Linear
Linear
An
Figure 3: Overall architecture our Unified transFormer (UniFormer). UniFormer block consists of three key modules, i.e., Dynamic Position Embedding (DPE), Multi-Head Relation Aggregrator (MHRA), and Feed Forward Network (FFN). Detailed explanations can be found in Section 3
Vaswani et al. 2017) but specially design it for efficient and effective spatiotemporal representa- tion learning. Specifically, our UniFormer block consists of three key modules: Dynamic Position Embedding (DPE), Multi-Head Relation Aggregator (MHRA), and Feed-Forward Network (FFN);
X DPE (X.n) + Xin:
(1)
Y MHRA (Norm (X)) +X,
(2)
Z FFN (Norm (Y)) +Y
Considering the input token tensor (frame volumes) X E RCXTxHXW we first introduce DPE to dynamically integrate 3D position information into all the tokens (Eq. D. which effectively makes use of spatiotemporal order of the tokens for video modeling. Then, we leverage MHRA to aggregate each token with its contextual tokens (Eq. 2. Different from the regular Multi-Head Self-Attentin (MHSA), our MHRA smartly tackles local video redundancy and global video dependency, by flex- ible designs of token affinity learning the shallow and deep layers. Finally, we add FFN with two linear layers for pointwise enhancement of each token (Eq.
3.2 MULTI-HEAD RELATION AGGREGATOR
As discussed above, we should solve large local redundancy and complex global dependency, for efficient and effective spatiotemporal representation learning. Unfortunately, the popular 3D CNNs and spatiotemporal transformers only focus on one of these two challenges. For this reason, we design an alternative Relation Aggregator (RA), which can flexibly unify 3D convolution and spa- tiotemporal self-attention in a concise transformer format, solving vidco redundancy and depen- dency in the shallow layers and deep layers respectively. Specifically, our MHRA conducts token relation learning via multi-head fusion:
R2 (X) A, Vn (X),
MHRA(X) = Concat Rx(); RR
(X))U.
(5)
Given the input tensor X [C we first reshape it to a sequence of tokens E L L-Tx H xW. R.(-) (-) is the relation aggregator (RA) in the n-thh head, and U E RCXC is a learnable parameter matrix integrate N heads. Moreover, each RA consists of token context encoding and token affinity learning. Via linear transformation, one can transform the original token into context (X) e L & Subsequently, the relation aggregator of the token affinity A, E L The key in our RA is how to learn A, in videos.
summarize context with the guidance
Local MHRA. In the shallow layers, aim at learning detailed video representation from the local spatiotemporal context in small 3D neighborhoods. This coincidentally shares a similar insight with the design of a 3D convolution filter. As result, we design the token affinity parameter matrix operated in the local 3D neighborhood, i.e., given one anchor token X,, RA learns local spatiotemporal affinity between this token and other tokens in the small tube 2 xhxw,
learnable
A locol X,) a where j E x2xhxx
4

---

Published as a conference paper at ICLR 2022
Basic
Tackle Local
Capture Global
Efficiency
Method
Operation
Redundancy
Dependency
GFI.Oss
Top-1
X3D Feeichtenhofer 2020
WWCon--DCCon--.e Conv
X
5823
80.4
TimeSformer [Bertasius et al., 2021
Divided MHSA
7140
80.7
Our UniFormer
Joint MHRA
168
80.8
Table I: Comparison to different methods. 'Local Redundancy' means the redundant computation
for capturing local features. 'Global Dependency" means the long-range dependency among frames.
where On E xxxx and Xj refers any neighbor token in 22x (i - j) means the relative token index that determines the aggregating weight (Appendix shows more details). In the shal- low layers, video contents between adjacent tokens vary subtly, is significant to encode detailed features with local operator to reduce redundancy. Hence, the token affinity is designed as local learnable parameter matrix, whose values only depend on the relative 3D position between tokens.
Comparison to 3D Convolution Block. Interestingly, we find that our local MHRA can be inter- preted as spatiotemporal extension of MobileNet block (Sandler et al., 2018 Tran et al. 2019 Fe- ichtenhofer 2020). Specifically, the linear transformation V(.) can be instantiated as pointwise con- volution (PWConv). Furthermore, the local token affinity Alocal is a spatiotemporal matrix that oper- ated on each output channel (or head) Vn (X), thus the relation aggregator R., (X) Afocal V, (X) can be explained as a depthwise convolution (DWConv). Finally, all heads are concatenated and fused by a linear matrix U, which also be instantiated pointwise convolution (PWConv). As a result, this local MHRA can be reformulated with a manner of PWConv- WWCCon--DCCone DWConv-PWCoom in the MobileNet block. In our experiments, we flexibly instantiate our local MHRA such channel- separated spatiotemporal convolution, so that our UniFormer can inherit computation efficiency for Light-weihtt video classification. Different from the MobileNet block, our UniFormer block is de- signed as generic transformer format, thus an extra FFN is inserted after MHRA, which can further mix token context at each spatiotemporal position to boost classification accuracy.
Global MHRA. In the deep layers, we focus on capturing long-term token dependency in the global video clip. This naturally shares a similar insight with the design of self-attention Hence, we design the token affinity via comparing content similarity among all the tokens in the global view:
(X.) (K()))
A global (X,, X2) EyEsrraww K.
where X, can be any token in the global 3D tube with size of T H x xW, while (.) and (-(()) are two different linear transformations. Most video transformers apply self-attention in all stages, intro- ducing a large amount of calculation. To reduce the dot- prouprr computation, the prior works tend to divide spatial and temporal attention (Bertasius et al.) [022 Arnab al.] (02T), but it deteriorates MHRA performs local relation aggrega-
the spatiotemporal relation among tokens. In contrast, tion in the early layers, which largely saves the computation of token comparison. Hence, instead of factorizing spatiotemporal attention, we jointly encode spatiotemporal relation in our MHRA for all balance.
the stages, in order to achieve a preferable computation
Comparison to Transformer Block. In the deep layers, our UniFormer block is equipped with global MHRA giobbl (Eq. 7) It can be instantiated as a spatiotemporal self attention, where (.),), (.(()) and (-) become Query, Key and Value in the transformer (Dosovitskiy et al., (021]. Hence, it can effectively learn long-term dependency. Instead of spatial and temporal factorization in the previous video transformers (Bertasius al. [021 Arnab al. (02T), our global MHRA is based on joint spatiotemporal learning to generate more discriminative video representation. Moreover, we adopt dynamic position embedding (DPE, see Section 3..) to overcome permutation-innaaiiane which can maintain translationiinvanie and is friendly to different input clip lengths.
3.3 DYNAMIC POSITION EMBEDDING
Since videos are both spatial and temporal variant, it is necessary to encode spatiotemporal position information for token representations. The previous methods mainly adapt the absolute relative position embedding of image tasks to tackle this problem (Bertasius et al., 2021 Arnab et al.) 2021). However, when testing with longer input clips, the absolute one should be interpolated to target input size with fine-tuning. Besides, the relative version modifies the self-attention and performs worse due to lack of absolute position information (Islam al. 2020). To overcome the above problems,
5

---

Published as a conference paper at ICLR 2022
K400
K600
Method
Pretrain
#Frame
GFLOPs
Top-1
Top-5
Top-1
Top-5
LGD Qiu
2019
IN-IK
N/8xN/A
N/A
94.4
81.5
95.6
SlowFasteNL Frichtcnhooer
2019
16x3x10
7020
79.8
93.9
81.8
95.1
ip-CSN Tran al. 2019
1
32x3x10
3270
79.2
93.8
CorNNet Wang ef al. 2020a
I M
32x3x 10
6720
81.0
X3D-M Feichtenhooorrr
20200
16x3x 10
186
76.0
92.3
78.8
94.5
(F 2020
16x3x
1452
79.1
93.9
81.9
95.5
MoViNet-AS ( Kondratlyk el 2021
120x1x1
281
80.9
94.9
82.7
95.7
MoViNet-A6 Kondratyuk ef al.
202T
120x x
386
81.5
95.3
83.5
96.2
ViT-B-VIN (Neimark ct al. 2021
IN-21K
250x1x1
3992
78.6
93.7
TimeSformer- (eeeterrr-BBBaanuutee el
2021
IN-21K
96x3x1
7140
80.7
94.7
82.2
95.5
STAM (Sharir of 202
IN-2IK
64x1x1
1040
79.2
X- Bulat et al.
202T
IN-21K
16x3x1
850
80.2
94.7
84.5
96.3
Mformer- HK[Patri et al.
2021
IN-2KK
16x3x10
28764
81.1
95.2
82.7
96.1
MVTT-B, 16x 1 Fan
2021
16x1x5
353
78.4
93.5
82,1
95.7
MVIT-B.32x33 Fan
2021
32x1x5
94.4
83.4
96.3
ViViI-L(AAmab
2021
IN-21K
16x3x4
17352
94.7
95.6
ViVIT-I Arnah al.
JFT-300M
16x3x4
17352
82.8
95.3
84.3
96.2
Swin-T et
2021b
IN-IK
32x3x4
1056
78.8
93.6
Swin-B
20211
IN-IK
32x3x4
3384
80.6
94.6
Swin-B Liu al.
20216
IN-21K
32x3x4
3384
82.7
95.5
84.0
96.5
Our UniFormer-S
IN-IK
16x1x4
80.8
94.7
82.8
95.8
Our UniFormer-B
IN-1K
16x1x4
389
82.0
95.1
84.0
96.4
Our UniFormer-B
IN-IK
32x1x4
1036
82.9
95.4
84.8
96.7
Our UniFormer-B
IN-IK
32x3x4
3108
83.0
95.4
84.9
96.7
Table 2: Comparison with the state-of-the-an on Kinetics-400&6600 Our UniFormer outper-
forms most of the current methods with much fewer computation cost.
we extend the conditional position encoding (CPE) (Chu et al. 2021 to design our DPE:
DPE(X;,.) DW..n
where DWConv means simple 3D depthwise convolution with zero paddings. Thanks to the shared parameters and locality of convolution, DPE overcome sermutation-invariinne and is friendly to arbitrary input lengths. Moreover, it has been proven in CPE that zero paddings help the tokens on the borders be aware of their absolute positions, thus all tokens can progressively encode their absolute spatiotemporal position information via querying their neighbor.
3.4 MODEL ARCHITECTRRE
We hierarchically stack UniFormer blocks to build up our network for spatiotemporal learning. As shown in Figure 3 our network consists of four stages, the channel numbers of which are 64, 128, 320 and 512 respectively. We provide two model variants depending on the number of UniFormer blocks in these stages: {3, 4, 8, 3} for UniFormer-S and {5, 8, 20, 7} for UniForrme-BB. In the first two stages, we utilize MHRA with local token affinity (Eq, (6) to reduce the short-term spatiotempo- ral redundancy. The tube size is set 5x5 5 and the head number N is equal to the corresponding channel number. In the last two stages, we apply MHRA with global token affinity (Eq. (7) to capture long-term dependency, the head dimension of which is 64. We utilize BN (loffe & Szegedy 2015) for local MHRA and LN (Ba et al.) 2016) for global MHRA. The kernel size of DPE 3x3x3
(TxHx W) and the expand ratios of FFN all layers are 4. We adopt a 3x4x4 convolution with stride 2x before the first stage, which means the spatial and temporal dimensions are both downsampled, Before other stages, we apply 1x2x2 convolutions with stride xx2x2. Finally, the spatiotemporal average pooling and fully connected layer are utilized to output the final predictions.
Comparison Convolution+ Donvolutionn-Trnsoormmee Network. The prior works have demonstrate that self- attention can perform convolution (Ramachandran et al. 2019 Cordonnier et al. 2020), but they propose to replace convolution instead of combining them. Recent works have attempted to intro- duce convolution to vision transformers (Wu et al. 2021 Dai et al.) 2021 Gao et al.) 2021 Srinivas) ctt al. 202I). but they mainly focus on image recognition, without any spatiotemporal consideration for video understanding. Moreover, the combination is almost straightforward in the prior video transformers, e.g., using transformer as global attention (Wang et al. 2018) or using convolution as patch stem (Liu et al. 2020b). In contrast, our UniFormer tackles both video redundancy and depen- dency with an insightful unified framework (Table [].. Via local and global token affinity learning, achieve preferable computation-accuraa balance for video classification.

---

Published as a conference paper at ICLR 2022
SSVI
SSV2
Method
Pretrain
#Frame
GFLOPs
Top-1
Top-5
Top-1
Top-5
TSNe 6
IN-IK
x6
19.9
47.3
30.0
60.5
TSM 2019
IN-IK
16x1x1
47.2
77.1
GST Yunlle 2019
IN-IK
16x
48.6
77.9
62.6
87.9
MSNet
IN-IK
16x1x1
101
52.1
82.3
64.7
89.4
CT-Net ef
IN-IK
16x1x1
75
52.5
80.9
64.5
89.3
CT-Netg
IN-IK
8+12+16+24
280
56.6
83.9
67.8
91.1
TDN Wang et al. 2020p
IN-IK
16x1x1
72
53.9
82.1
65.3
89.5
TDN Ex Wang al. 2020b
IN- N-1K
8+16
198
56.8
84.1
68.2
91.6
et
2021
IN-2IK
16x3x1
5109
62.5
X X-ViT Bular et al. 2021
IN-2IK
32x3x1
1270
65.4
90.7
Mformer-L Patrick et 2021
K400
32x3x1
3555
68.1
91.2
ViViT-L el al 2021
K400
16x3x4
11892
65.4
89.8
MVIT-B, 123,
Fan et al. 2021
K400
64
1365
67.7
90.9
MVIT-B-24,32:5 32x3 Fan et al. 2021
K600
x2
708
68.7
91.5
Swin-B Liu ct al. 2021b)
K400
32x3x1
963
69.6
92.7
Our Uni Former-S
K400
16XIXI
42
53.8
81,9
63.5
88.3
Our UniFormer-S
K600
16x1xI
42
54.4
81.8
65.0
89.3
Our UniFormer-S
K400
16x3x1
125
57.2
84.9
67.7
91.4
Our UniFormer-S
K600
16x3x1
125
57.6
84.9
69.4
92.1
Our UniFormer-B
K400
16x3x1
290
59.1
86.2
70.4
92.8
Our UniFormer-B
K600
16x3x1
290
58.8
86.5
70.2
93.0
Our UniForrmr-BB
K400
32x3x1
777
60.9
87.3
71.2
92.8
Our UniFormer-B
K600
32x3x1
777
61.0
87.6
71.2
92.8
Table 3: Comparison with the state-of-the-art on Sonething-Somethhin
V1&V2. Our Uni-
Former achieves new state-of-the-art performances on both datasets.
4
EXPERIMENTS
4.1 DATASETS AND EXPERIMENTAL SETUP
We conduct experiments widely-used Kinetics-400 (Carreira & Zisserman 201 and larger benchmark Kinetics-600 (Carreira et al. 2018). We further verify the transfer learning performance on temporal-related datasets Something- Nomething-Sometting
Something &I& V2 (Goyal al. 20176). For training, utilize the dense sampling strategy (Wang et al.) 2018) for Kinetics and uniform sampling strategy [Wang et al. 2016) for Something-Something et a 2021) by default, but the random horizontal flip is not applied for
We adopt the same training recipe as MVIT (Fan
To reduce the total training cost, inflate the 2D convolution kernels pre-trained on ImageNet for Ki- netics (Carreira & Zisserman, 2017b). More implementation specifics are shown in Appendix C For testing, we explore the sampling strategies in our experiments. To obtain preferable computation- accuracy balance, we adopt multi-clip testing for Kinetics and multi-crop testing for Something- Something. All scores are averaged for the final prediction.
4.2 COMPARISON STATE-OF-THE-ARE
Kinetics-400&6600. Table presents comparisons the state-of-the-art methods Kinetics-400 and Kinetics-600. The first part shows the prior works using CNN. Compared with SlowFast (Fe- ichtenhofer et al.) 2019), our UniFormer-S requires 42x fewer GFLOPs but obtains 1.0% per- Tormance gain on both datasets. Even compared with MoViNet (Kondratyuk et al.) 2021). which designed through extensive neural architecture search, our model achieves slightly better results with fewer input frames (16f x4 vs. 120f). The second part lists the recent works based on vi- sion transformers. With only ImageNet- 1K pre-training UniFormer-Bio surpasses most of the other backbones with large dataset pre-training. For example, compared with ViViT-L pre-trained from JFT-300M and Swin-B pre-trained from ImageNet-21K UniFormer-Baa/ obtains comparable performance with 16.7 and 3.3x fewer computation both Kinetics-400 and Kinetics-600.
Sereehing-SSmethinng V2& Results on Something- Something V1&V2 are shown Table Since these datasets depend on temporal relation modeling, it is difficult for the CNN-based methods to capture long-term dependencies, which leads to their worse results. contrast, transformer- based backbones good processing long sequential data and demonstrate better transfer learning capabilities (Zhou et al, 2021). Our UniFormer pre- trained from Kinetis 600 outperforms all the current methods under the same settings. In fact, our best model achieves the mate-of-thheart

---

Published as a conference paper at ICLR 2022
ImageNet
K400 1x4
Unified
Joint
DPE
Type
GFLOPs
#Puram
Top-1
Top-5
GFLOPs
#Param
Top-1
Top-5
LLGG
3.6
21.5
82,9
96.2
41.8
21.4
79.3
94.3
LLGG
3.3
21.3
82.6
96.1
41.0
21.3
78.6
93.6
LLGG
3.6
21.5
82.9
96.2
36.8
27.7
78.7
94.1
x
LLGG
3.6
21.5
82,4
96.0
41.4
21.3
77.6
93.5
LLLL
3.7
23.3
81.9
95.9
31.6
23.7
77.2
92.9
LLLG
3.7
22.2
82.5
96.1
31.6
22.4
78.4
93.3
LGGG
3.6
21.6
82.7
96.1
39.0
21.4
94.1
GGGG
3.7
20.1
82.1
95.9
72.0
19.8
75.3
92.4
(a) Structure design. All models are trained for 50 epochs on Kinetics-4000 To guarantee the parameters and
computation of all the models are similar, when modifying the stage types. we modify the stage numbers and reduce the computation of self-attention as MViT (Fan al. 2021 for I.GGG and GGGG.
K400 x4
Type
Joint
GFLOPs
Pretruin
SSVI Top-1
Model
Sampling Method
K400
T4p01Top-1
GFLOPss
Top1
LLLL
V
26.1
ImageNet
49.2
1x1
1x4
41.0
K400
49.2(+0.0)
Small
16x4
76.2
80.8
43.6
41.8
79.3
79.1
LLGG
X
36.8
ImageNet
K400
51.8 51.8(-0.1)
51.9
16x8
16x4
78.4
78.1
80.7
82.0
46.6
78.9
LLGG
41.8
ImageNet
Base
16x8
79.3
81.7
(b) Tube size. Our net- work is basically robust the tube size.
K400
+3.8(+18.)
Small
32x2
77.3
81.2
(c) Transfer learning. Jointly manner performs
32x4
79.8
82.0
better when pre-training from larger dataset.
(d) Sampling method.
Table 4: Ablation studies. 'Unified' means whether use our local MHRA (X means use Mo-
bileNet block). 'Joint' means whether to use joint attention. 'L'"G" refers to local/global MHRA.
results: 61.0% top-1 accuracy on Something-Something V1 (4.2% higher than TDNEN) (Wang cet al. (020bb and 71.2% top-1 accuracy on Sannething-Somethineg V2 (1.6% % higher than Swin-B (Liu et al. 2021b)). Such results verify the capability of spatiotemporal learning for UniFormer.
4.3 ABLATION STUDIES
Does transformer-sty/ FFN help? As mentioned in Section 3.2 our UniFormer block in the shallow layers can be interpreted as a transformer-styll spatiotemporal MobileNet block (Tran et al.) 2019) with extra FFN. Hence, we first investigate its effectiveness by replacing our UniFormer blocks in shallow layers with MobileNet blocks (the expand ratios are set to 3 for similar parameters). As expected, our default UniFormer outperforms such spatiotemporal MobileNet block Table shows that, FFN in our UniFormer can further mix token context at each spatiotemporal position boost classification accuracy.
UniFormer vs. Convolution:
UniFormer vs. Transformer: Is joint or divided spatiotemporal attention better? As discussed in Section 3 our UniFormer block in the deep layers can be interpreted transformer block, but our attention is jointly learned a spatiotemporal manner, instead of dividing spatial and temporal attention (Bertasius al., (021aAAmab et al.) 2021). As shown in Table 4a the joint version is more powerful than the separate one, showing that joint spatiotemporal attention can learn more discrim- inative video representations. What's more, the joint attention is friendly to transfer learning with pre-training. As shown in Table 4c when the model is gradually pre-trained from ImageNet to Kinetics-400, the performance of our UniFormer becomes better. Such distinct characteristic is not observed in the pure local MHRA structure (LLLL) and the splitting version. It demonstrates that the joint learning manner is preferable for video representation learning.
Does dymamic position embedding matter to UniF o With dynamic position embedding, our UniFormer improve the top-1 accuracy by 0.5% and 1.7% on ImageNet and Kinetics-400. It shows that via encoding the position information, our DPE can maintain spatiotemporal order, contributing to better spatiotemporal representation learning.
How much does local MHRA help? Since our UniFormer is equipped with local and global token affinity respectively in the shallow and deep layers, we investigate the configuration of our network stage by stage. As shown in Table 4al when we only use local MHRA (LLLL), the computation cost will be light. However, the accuracy is largely dropped, since the network lacks the capacity of learning long-term dependency without global MHRA. When we gradually replace local MHRA with global MHRA, the accuracy becomes better as expected. However, the accuracy dramatically dropped with heavy computation load when all the layers apply global MHRA (GGGG). It is
8

---

Published as a conference paper at ICLR 2022
Top-1 Accuracy (%)
83
81
70
58
82
80
69
57
81-
79
68
SthV2-3crop
-tho1-3crop
78
67
SUNV2-1crop
56
SthV1-1crop
80
K600-3crop
77
K400-3crop
66
55
K600-1crop
K400-1crop
79
76
90
2
5
8
#Number of Clips
#Number of Clips
#Number of Clips
# #Nmberr of Clips
Figure 4: Multi-clip/erop testing comparison different datasets. Multi-clip testing is better
for Kinetics and multi-crop testing is better for Sameehing-SSmmthiing
mainly because that, without local MHRA, the network lacks the capacity of extracting detailed video representations, leading to severe model overfitting with redundant spatiotemporal attention. In our experiments, we choose local MHRA and global MHRA in the first two stages and the last two stages respectively, in order to achieve a preferable computation-accuran
balance.
Is our UniFormer more transferable? We further verify the transfer learning ability of our Uni- Former in Table 4c| All models share the same stage numbers but the stage types are different. Compared with pre-training from ImgeNet, pre-training from Kinetics-400 will further improve the top-1 accuracy by 1.8%. However, such distinct characteristic is not observed in the pure local MHRA structure and UniFormer with divided spatiotemporal attention. It demonstrates that the joint learning manner is preferable for transfer learning.
Empirical investigation on model parameters. We further evaluate the robustness of our Uni- Former network several important model parameters. (1) size of local tube: In our local token affinity (Eq. ( we aggregate spatiotemporal context from a small local tube. Hence, we inves- tigate the influence of this tube by changing its 3D size (Table [bb.. Our network is robust to the tube size. We choose 5 5x5x5 5 for better accuracy. (2) sampling method: We explore the vital sampling method shown in Table 4d] For training, 16x4 means that we sample 16 frames with frame stride 4. For testing, 4x1 means four-clip testing. As expected, sparser sampling method achieves a higher single-clip result. For multi-clip testing, dense sampling is slightly better when sampling few frames. However, when sampling more frames, sparse sampling is obviously better. (3) testing strategy: We evaluate our network with different numbers of clips and crops for the validation videos. As shown in Figure 4 since Kinetics is scenc-related dataset and trained with dense sampling, multi-clip testing is prefcrable to cover more frames for boosting performance. Al- ternatively, Someeting-SSmmetiee so multi-crop testing is better for capturing the discriminative motion for boosting performance.
temporal-related dataset and trained with uniform sampling,
4.4 VISUALIZATION
To further verify the effectiveness of UniFormer, we conduct some visualizations of different struc- tures (see Appendix D). In Figure 5 We apply Grad-CAM (Selvaraju et al. 2019) to show the areas of the greatest concern in the last layer. reveals that GGGG struggles focus on the key object, i.e, the skateboard and the football, as it blindly compares the similarity of all tokens in all layers. Alternatively, LLLL only performs local aggregation. Hence, its attention tends to be coarse and inaccurate without a global view. Different from both cases, our UniFormer with LLGG can coop- eratively learn local and global contexts in a joint manner. As result, it can effectively capture the most discriminative information, by paying precise attention to the skatcboard and the football. In Figure 66, we present the accuracies of different structures on Kinetics-400 (Carreira & Zisserman 2017a). It shows that LLGG outperforms other structures in most categories, which demonstrates that our UniFormer takes advantage of both 3D convolution and spatiotemporal self-attention.
5 CONCLUSION
In this paper, we propose a novel UniFormer, which can effectively unify 3D convolution and spa- tiotemporal self-attention in concise transformer format overcome video redundancy and depen- dency. We adopt local MHRA in shallow layers to largely reduce computation burden and global MHRA in deep layers to learn global token relation. Extensive experiments demonstrate that our UniFormer achieves a preferable balance between accuracy and efficiency on popular video bench- marks, Kinetics-400/600 and Something-Someetinn V1/V2.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is partially supported bythe National Natural Science Foundation of China (1876176,011812200 Guangdong NSF Project (No. 1020B151512005 the Shenzhen Research (oggram(RII2222001111111 14555770777 the Shanghai Committee of Science and Technology, China (Grant No. 21D21100100).
REFERENCES
A. Arna,, M. Dehghani, G. Heigold, Chen Sun, Mario Lucic, and C. Schmid. Vivit: video vision transformer. ArXiv, abs/2103. 15691, 2021. 2021.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/160.0044555. 2016.
Gedas Bertasius, Heng Wang, and L. Torresani. Is space-time attention all you need for video understanding? ArXiv, abs/2102.05095, 2021.
Andrew Brock, Soham De, Samuel L. Smith, and K. Simonyan. high-performance large-scale image recognition without normalization. ArXiv, abs/2102.06171, 2021.
Adrian Bulat, Juan-Manuel Pérez-Rúa, Swathikiran Sudhakaran, Brais Martínez, and Georgios Tz- imiropoulos. Space-time mixing attention for video transformer. ArXiv, abs/2106.05968 2021.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724- 4733, 2017a.
João Carreira, Eric Noland, Andras Banki- Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. ArXiv, abs/ 018.808114400 014.01144,,4 2018.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724 2017,.
2017b.
Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? ArXiv, abs/2102. 2021.
Jean- Cean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. ArXiv, abs/1911 1121911..3330 .03584, 2020.
Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. ArXiv, abs/2100.04803. 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255-255. leee, 2009.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and B. Guo. Cswin transformer: A general vision transformer backbone with cross-shaped win- dows. ArXiv, 0bs/2100.00652. 2021.
A. Dosovitskiy, L. Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, G. Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11924 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, J. Malik, and Christoph Feichtenhofer. Multiscale vision transformers. ArXiv, abs/2104.11227 2021.
Christoph Feichtenhofer. X3d:
Expanding architectures for efficient video recognition. 2020 IFFE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 200-210, 2020.
10

---

Published as a conference paper at ICLR 2022
Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6201 6210, 2019.
Peng Gao, Jiasen Lu, Hongsheng Li, R. Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. ArXiv, abs/2106.01401 2021.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training ima- genet in hour. ArXiv, 0bss/706.02677 2017a.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag. Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic, The "something something" video database for learning and evaluating visual common sense. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5843-5851, 2017b.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, abs/ 0tob//552.03166 2015.
Md. Amirul Islam, Sen Jia, and Neil D. B. Bruce. How much position information do convolutional neural networks encode? ArXiv, his/20001.02245 2020.
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. 2019 IEEE International Conference Computer Vision (ICCV), pp. 2000-2009, 2019.
Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. arXiv preprint arXiv:2104. 10858, 2021.
D. Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew A. Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. ArXiv, abs/2103. 11511, 2021.
Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. In ECCV, 2020.
Kunchang Li, Xianhang Li, Yali Wang, Jun Wang, and Y. Qiao. Ct-net: Channel tensorization network for video classification. ArXiv, abs/2106.01603, 2021a.
X. Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contextual views for video classification. 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1089-1098, 2020a.
Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. ArXiv, abs/2104.11746 2021b.
Yinong Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. ArXiv, abs/2004.01.33 0013999 2020b.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. 2019 IEEE International Conference on Computer Vision (ICCV), pp. 7082-7092, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ArXiv, abs/2103.14030) 2021a.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, S. Lin, and Han Hu, Video swin transformer. ArXiv, abs/2106 22..0.11 13230, 2021b.
11

---

Published as a conference paper at ICLR 2022
Zhaoyang Liu, D. Luo, Yabiao Wang, L. Wang. Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang. and Tong Lu. Teinet: Towards an efficient architecture for video recognition. ArXiv, abs/1911.09435, 2020a.
Zhouyong Liu, Shun Luo, Wubin Li, Jingben Lu, Yufan Wu, Chunguo Li, and Luxi Yang. Convtransformer:: A convolutional transformer network for video frame synthesis. ArXiv, abs/2011.10185,, 2020b.
Loshchilov and Hutter. Fixing weight decay regularization in adam. ArXiv, abs/1771.05101, 2017a.
Iya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv: Learning, 2017b.
Chenxu Luo and Alan L. Yuille. Grouped tion. 2019 IEEE International Conference on Computer Vision (ICCV), pp. 5511-5520, 2019.
aggregation for efficient action recogni-
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. ArXiv, abs/2102.00719, 2021.
Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichten- hofer, A. Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. ArXiv, abs/2106.05392, 2021.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5534- 5542, 2017.
Zhaofan Qiu, Ting Yao, C. Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12048-12057, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10425-10433, 2020.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In NeurlPS, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 45200 2018
Ramprasaath R. Selvaraju, Abhishck Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. International Journal of Computer Vision, 128:336-350 3593, 2019.
Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor. An image is worth 16x 6 words, what is video worth? ArXiv, abs/2103. 13915, 2021.
A. Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, P. Abbeel, and Ashish Vaswani. Bottle- neck transformers for visual recognition. ArXiv, abs/2101. 1605, 2021.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ArXiv, abs/1905.1 1946, 2019.
Mingxing Tan and Quoc V. Le. abs/2104.00298, 2021.
Efficientnetv2: Smaller models and faster training. ArXiv,
Hugo Touvron, M. Cord, M. Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 202
Hugo Touvron, M. Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herv'e J J'egou. Going deeper with image transformers. ArXiv, 17239,, 2021b.
12

---

Published as a conference paper at ICLR 2022
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. 2015 IEEE International Conference on Computer Vision (ICCV), pp.
Du Tran, Hong xiu Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6450-6459, 2018.
Du Tran, Heng Wang, L. Torresani, and Matt Feiszli. Video classification with channel-separated convolutional networks. 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 5551-5560, 2019.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
Heng Wang, Du Tran, L. Torresani, and Matt Feiszli. Video modeling with correlation networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 349-358, 2020a.
Wang, Yuanjun Xiong, Zhe Wang. Y. Qiao, D. Lin, X. Tang, and I.. Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. ArXiv, abs/2012. 10071, 2020b.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In ACL, 2019.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. ArXiv, abs/2102. 12122, 2021.
X. Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7794-7803, 2018.
Haiping Wu, Bin Xiao. N. Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. ArXiv, abs/2 100.210331800 15808, 2021.
Li Yuan, Y. Chen, Tao Wang. Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokennsto-token vit: Training vision transformers from scratch on imagenet. ArXiv, abs/2101.11986, 2021.
Xuefan Zha, Wentao Zhu, Tingxun Lv, Sen Yang, and Ji Liu. Shifted chunk transformer for spatio- temporal representational learning. ArXiv, abs/2108. 1575,, 2021.
Hong- Yu Zhou, Chixiang Lu, Sibei Yang, and Yizhou Yu. Convnets vs. transformers: Whose visual representations are more transferable? ArXiv, abs/2108.05305, 2021.
13

---

Published as a conference paper at ICLR 2022
A MORE DETAILS ABOUT LOCAL MHRA
For local MHRA, it is vital to determine the neighbor tokens. Considering any token X₃ (k E [0, 1J, can calculate its index (ty, hk, wk) as follows:
k
L =
(9)
H W
h& k Lx
x
W
W |,
(10)
Wk (k: tk H W) mod W.
(11)
Therefore, for an anchor token x,, any of its neighbor tokens Xj in x should satisfy
t
14 1 2
(12)
h [hg hy1 2 -
(13)
wws wyl
(14)
Thus the local spatiotemporal affinity in Eq. can be calculated as follows:
A Local
[X X, a²
(15)
= alt; - tj h1s - hgg ws wwyl
(16)
For other tokens not in xxxxx Alocal (X,,
(X,,, X,) O.
B MORE DETAILS ABOUT FFN.
We adopt the standard FFN (Eq. 3) in vision transformers (Dosovitskiy et al. 2021
z" Lineary (GELU (Linear, (Z'))),
(17)
where GELU is a non-linear function. The channel number will be first expanded by ratio 4 and then reduced. All token representations will be enhanced after performing FFN.
C ADDITIONAL IMPLEMENTATION DETAILS
Architecture details. As in ViT (Dosovitskiy et 2021). adopt the necooamallzzaionn configu- ration Wang et al., 2019) that applies norm layer at the beginning of the residual function (He et al. 2016). Differently, we utilize BN (loffe & Szegedy 2015) for local MHRA and LN (Ba et al.) 2016) for global MHRA. Moreover, we add an extra layer normalization in the downsampling layers.
Training details. We adopt AdamW (1 (Loshchilov & Hutter 2017a) optimizer with cosine learning rate schedule (Loshchilov & Hutter 2017b) to train the entire network. The first or 10 epochs are used for warm-up (Goyal al.) [Q]]]01]] 2017a) overcome early optimization difficulty. For UniFormer- S. the warmup epoch, total epoch, stochastic depth rate, weight decay are set to 10, 110, 0.1 and 0.05 respectively for Kineties and 5, 50, 0.3 and 0.05 respectively for Something Something-Somethhng UniFormer-B B, all the hyper-parameten are the same unless the stochastic depth rates are doubled.
For
We linearly scale the base learning rates according to the batch size, which are le hatchsize and
32
2e x batchsize for Kinetics and Something-Something 32
D VISUALIZATION
We choose three structures used in our experiments (Table 4a) to make comparisons: LLGG, LLLL and GGGG. The stage numbers of them are {3, 4, 3,, (3, 5. 10, 4} and {2, 7,3) respectively.
In Figure 5 we conduct attention visualization of different structures. Partl shows the input videos Grad-CAM (Selvaraju 2017a). In part2,
selected from Kinetics-400 (Carreira & Zisserman
14

---

Published as a conference paper at ICLR 2022
Sampling stride
K400
K600
Method
#Frrme
GFLOPs
#PParm
Top-1
Top-5
Top-1
Top-5
16x1xI
41.8
21.4
76.2
92.2
79.0
93.6
16x1x4
167.2
21.4
80.8
94.7
82.8
95.8
1
41.8
21.4
78.4
92.9
80.8
94.7
16x1x4
167.2
21.4
80.8
94.4
82.7
95.7
UniFormer-S
32
109.6
21.4
77.3
92.4
32x1x4
438.4
21.4
81.2
94.7
x1
109.6
21.4
79.8
93.4
32x1x4
438.4
21.4
82.0
95.1
16x1x1
96.7
49.8
78.1
92.8
80.3
94.5
16x1x4
386.8
49.8
82.0
95.1
84.0
96.4
16x1x1
96.7
49.8
79.3
93.4
81.7
95.0
UniFormer-B
16x1x4
386.8
49.8
81.7
94.8
83.4
96.0
32x1x1
259
49.8
80.9
94.0
82.7
95.7
32x1x4
1036
49.8
82.9
95.4
84.8
96.7
Table 5: More results Kinetics-400&&600.
et al. 2019) to generate the corresponding attention in the last layer. Since GGGG blindly com- pares the similarity of all tokens in all, it struggles to focus the key object, i.e., the skateboard and the football. Alternatively, LLLII only performs local aggregation without global view, lead- ing coarse and inaccurate attention. Different from both cases, UniFormer with LLGG can cooperatively learn local and global contexts in joint manner. As result, it can effectively capture the most discriminative information, by paying precise attention to the skateboard and the football.
Additionally, in Figure 6, we show the top-1 accuracies of different structures on Kinetics-400 (Car- reira Zisserman 2017a) It demonstrates that GGGG surpasses the other two structures in categories. Furthermore, we analyze the prediction results of several categories in Figure 7 It shows that gargling is often misjudged as brushing teeth, while swing dancing is often misjudged other types of dancing. We argue that these categories are easier to be discriminated against based on the spatial details. For example, toothbrush often exists brushing teeth not in gargling, and the people's 's poses are different in different dancing. Therefore, LLLL performs better than GGGG in these categories thanks to the capacity encoding detailed spatiotemporal features. What's more, playing guitar and strummning guitar are difficult to be classified, since their spatial contents almost the same. They require the long-range dependency between objects, e.g., the interaction be- tween the people's hand and the guitar, thus GGGG does better. More importantly, UniFormer with LLGG competitive with the other two methods these categories, which means it takes advantage of both 3D convolution and spatiotemporal self-attention.
E ADDITIONAL RESULTS
E. I MORE RESULTS ON KINETICS
Table 5 shows more results on Kinetics-400 (Carreira & Zisserman 2017a) and Kinetics-600 (Car- reira et al. 2018). The trends of the results on both datasets are similar. When sampling with large frame stride, the corresponding single-clip testing result will be better. It is mainly because sparser sampling covers larger time range. For multi-clip testing, sampling with frame stride always performs better, thus adopt frame stride by default.
E.2 MORE RESULTS ON SOMETHING-SOMETTING
Table 6 presents more results Something -oaiiinnn-Somhiinggg V1&V2 (Goyal et al. 2017b). For UniFormer-S, pre-training with Kinetics- 600 better than pre-training with Kinetics 400, improv- ing the t accuracy approximately 1.5%. However, for UniFormer-B. the improvement is not obvious. We claim that the small model is difficult to fit, thus larger dataset pre-training can help fit better. Besides, UniFormer-B with 16 frames performs better than UniFormer-S with 32 frames.
15

---

Published as a conference paper at ICLR 2022
Input
Attention LLGG
Attention LLLL
Attention GGGG
(a) hoverboarding.
Input
Attention LLGG
Attention LLLL
Attention GGGG
(b) passing American football (not game).
Figure 5: Attention visualization of different structures. Videos chosen from Kinetics-400 (Car- reira & Zisserman 2017a).
16

---

Published as a conference paper at ICLR 2022
100
Top-1 Accuracy Differnnce (59
60
20
LLLL
LLGG
GOGG
50
100
150
200
250
300
350
400
Category Index
Figure 6: Accuracy of different structures on Kinetics-400.
gargling
swing dancing
playing gultar
strumming guitar
Predictions
pargling
garplig
Saiin dinccing
swiig denclng
play/ig giater
playlig guater
strumiing
strumising guale"
brstting
eeting chips
dhinkingg
dencing chartimttoon
deocing thartestirr
Buskking
busking
singlog
supping guste
arttknng beer
tap denctog
MAp dencing
taoping glittr
strummingg gutter
reconolsg muric
paying bass giatar
LLGG
GGGG
dancing
marching
40
Top-1 Accuracy(%6)
Top Accuracy(9%)
Top-1 Accuracy(%)
Top-1 Accuracy(%)
Figure 7: Prediction comparisons of different structures.
SSVI
SSV2
Method
Pretrain
#Frame
GFLOPs
#Param
Top-1
Top-5
Top-1
Top-5
16x1x1
41.8
21.3
53.8
81.9
63.5
88.5
K400
16x3x1
125.4
21.3
57.2
84.9
67.7
91.4
16x3x2
250.8
21.3
57.3
85.1
68.1
91.7
16x1x1
41.8
21.3
54.4
81.8
65.0
89.3
K600
16x3x1
125.4
21.3
57.6
84.9
69.4
92.1
16x3x2
250.8
21.3
57.8
84.9
69.5
92.2
UniFormer-S
32x1x1
109.6
21.3
55.8
83.6
64.9
89.2
K400
32x3x1
328.8
21.3
58.8
86.4
69.0
91.7
32x3x2
657.6
21.3
58.9
86.6
69.2
91.8
32x1x1
109.6
21.3
56.9
83.8
66.4
90.2
K600
32x3x1
328.8
21.3
59.9
86.2
70.4
93.1
32x3x2
657.6
21.3
59.9
86.3
70.5
92.9
16x
96.7
49.7
55.4
82.9
65.8
89,9
K400
16x3x1
290.1
49.7
59.1
86.2
70.4
92.8
16x3x2
580.2
49.7
59.3
86.4
70.7
92.9
16x1x1
96.7
49.7
55.7
83.3
66.
90.0
K600
16x3x1
290.1
49.7
58.8
86.5
70.2
93.0
16x3x2
580.2
49.7
59.1
86.5
70.7
92.9
UniFormer-B
32x1x1
259
49.7
58.1
84.9
67.2
90.2
K400
32x3x1
777
49.7
60.9
87.3
71.2
92.8
32x3x2
1554
49.7
61.0
87.3
71.4
92.8
32x1x1
259
49.7
58.0
84.9
67.5
90.2
K600
32x3x1
777
49.7
61.0
87.6
71.2
92.8
32x3x2
1554
49.7
61.2
87.6
71.3
92.8
Table 6: More results on Something-Somethin V1&V2.
17

---

Published as a conference paper at ICLR 2022
E.3 COMPARSION TO state-of-the-art ImageNet (Deng et al. 2009). We design four
ON IMAGENET
Table compares our method with the model variants follows:
UniFormer-S: channel numbers=(64, 128, 320, 512), stage numbers=(3, (3,4,
UniFormer-SPy channel numbers=(64, 128, 320, 512), stage 53 5,,
UniFormer-B B: channel numbers=(64, 128, 320, 512), stage numbers=(5,8,, 20, 7}
UniFormer-L: channel numbers=(128, 192, 448, 640), stage (5mbers=(5,10, 24,
All the other model parameters the same we mention in Section 3.4 For UniFormer-SSt we adopt overlapped convolutional patch embedding. All the training hyper-parameters are the same as DeiT (Touvron et al., 2021a) by defaults. When training our models with Token Labeling, we follow the settings used in LV-ViT (Jiang et al., (021). It shows that our models outperform other methods with similar parameters/FiLOP. on ImageNet, especially when training with Token Labeling. Moreover, our model surpasses those models combining CNN with Transformer, e.g., CvT (Wu et al., 202T) and CoAtNet (Dai et af.) 202T), which reflects our UniFormer can unify convolution and self-attention better for preferable accuracy-computation balance.
18

---

Published as a conference paper at ICLR 2022
Train Size
Test Size
ImageNet
Method
Architecture
#Param
GFLOPs
Top-1
RegNetY-4CC Radosavovic al.
2020
CNN
224
224
EffcientNet-BB B3 ( Tan ( & Le Le|
2019
CNN
30
9.9
456
456
83.6
EfficienNNat2--5 Tan & 2021
CNN
22
8.5
384
384
83.9
DeiT-S Touvron et a1. 2021a
Trans
22
4.6
224
224
79.9
PVT-S Wang et 2021
Trans
25
3.8
224
224
79.8
T2T-14 Yuan al.
2021
Trans
22
5.2
224
224
80.7
Swin-T Lm
2212a
Trans
4.5
224
224
81.3
CSwin-T
384 Dong
2021
Trans
224
384
84.3
LV-VIT-S Jiang ct al. 2021
Trans
6.6
224
224
83.3
LV-VIT-S
384 UJang et al.
2021
Trans
22.2
224
384
84,4
CVT-13
Wu et al. 2021
CNN+Trans
20
4.5
224
224
81.6
CvT-13
[384 WW
2021
CNN+Trans
20
16.3
224
384
83.0
CoAtNet-O Dai et 2021
CNN+Trans
25
4.2
224
224
81.6
CoAtNet-O
384 (Da
2021
CNN+Trans
13.4
224
384
83.9
Container Gao
CNa+Trans
224
82.7
UniFormer-S
CNN+Trans
3.6
224
224
82.9
UniFormer-S+TT
CNN+Trans
22
3.6
224
224
83.4
UniFormer-S+77 1384
CNN+Trans
22
11.9
224
384
84,6
UniFormer-St
CNN+Trans
24
4.2
224
224
83.4
UniFormer-Sitee
CNN+Trans
24
4.2
224
224
83.9
UniFormer-Si+411. 1384
CNN+Trans
13.7
224
384
84.9
NegNerY-KGi (Radosavovic
2020
CNN
224
224
81.7
EflcienNeet-B7 [am Le 2019
CNN
66
39.2
600
600
84.3
EffccienNNetV2-11 TTn &
2021
CNN
54
25.0
480
480
85.1
PVT-L WWng al.
2021
Trans
61
9.8
224
224
81.7
T2T-24 Yuan
2021
Trans
13.2
224
224
82.2
Swin-S Lau
2021a
Trans
8.7
224
224
83.0
CSwin-S 7384 (Dong al.
2021
Trans
22.0
224
384
85,0
IV-VIT-M (Jinn et al. 2021
Trans
224
224
84.1
LV-VIT-M 1384 (Jaan et al.
2021
Trans
56
42.2
224
384
85.4
CvT-21 Wu et al. 2022
CNN+Trans
32
7.1
224
224
82.5
CvT-21 1384 Wu et al. 2021
CNN+Traas
32
24.9
224
384
83.3
CoAtNet-1 (Dai et
CNN+Trans
42
8.4
224
224
83.3
CoAtNet-1 384 DDr
2021
CNN+Trans
27.4
224
384
85.1
UnilFormer-B
CNN+Trans
224
224
83.9
UniFormer -BTTL
CNN+Trans
50
8.3
224
224
85.1
UniFormer-B-11 Bhoommer-B+L... 1384
CNN+Trans
50
27.2
224
384
86.0
Reg NoGY (Radosavovic et 2020)
CNN
84
16.0
224
224
82.9
EficientNe(V2-11 (an & Le 2021
CNN
121
53
480
480
85.7
N4 (Brock
CNN
316
215.3
384
512
85.9
NFNet-FF BBrock et al.
2021
CNN
289.8
416
544
86.0
DeiT-B Touvron cl al. 20
Trans
17.5
224
224
81.8
Swin-B Lau el al. 202
Trans
15.4
224
224
83.3
CSwin 384 (Dong et al.
2021
Trans
78
47.0
224
384
85.4
LV-ViT-I. Jiing et 2021
Trans
150
59.0
288
288
85.3
LV-VIT-L 448 (Jiag al.
2021
Trans
150
157.2
288
448
85.9
CaiT--488 1384 TTuvron et al. 2021h
Trans
63.8
224
384
85.1
CaiT-M36
4488
[Louvron el al. 2021b
Trans
247.8
224
448
86.3
BoTNet--77 (Srinivas et 2021
CNN+Trans
19.3
256
256
84,2
BoTNet-T7 338 (Smiivas el al.
2021
CNN+Trans
79
45.8
256
384
84.7
CoAtNet-3 Dai al 2021
CNN+Trans
168
34.7
224
224
84.5
CoAtNet-3 1384 (Dar et 2021
CNN+Trans
168
107.4
224
384
85.8
UniFForerr-LH11
CNN+Trans
100
12.6
224
224
85.6
UniFormer- L+TL 1384
CNN+Trans
39.2
224
384
86.3
Table 7: Comparison with the state-of-the-art ImageNet. "Train Size' and "Test Size" refer to resolutions used in training and fine-tuning respectively. 'TL' means token labeling proposed in LV- Jiang et al. 2021). We group the models based their parameters.
19


Reviews:
[{"review_id": "rRFoml0o2CL", "reviewer": "Reviewer_Na5h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper makes an insightful connection between X3d's conv block and transformer block, and build a unified computational block. Compared to current methods focus on adding good properties of conv blocks to new transformer blocks (e.g., swin-transformer), the proposed method is more elegant conceptually.\n- On the standard video action recognition benchmark, the proposed method achieves SOTA result, which is impressive.\n- The paper provides comprehensive ablation studies on the design choices.\n\n\nWeakness\n- Lack of reference of previous works on the relationship between conv layer and transformer block.\n- Table 2 and Table 3. The bold highlights are inconsistent. Table 3 highlights the best accuracy results among all methods, while Table 2 highlights results are not so. In addition, highlights in Table 2 may make readers ignore the fact that MoViNet-A6 results are comparable in both efficiency and accuracy.\n- Lack of explanation of DPE. Currently, DPE is briefly introduced as an extension of CPE in the subsection of MHRA. However, DPE is not part of MHRA and it'll be better to have it as its own subsection.\n- Lack of comparison with MoViNet-A6+transformer. Despite the bigger design space of L and G, the ablation studies show that the CNN+Transformer architecture works the best. Thus, it is unclear if the proposed method can outperform MoViNet-A5+Transformer blocks with similar GLFOPs. This is only a minor comment, as the proposed method already achieves significant enough performance for publication.", "summary_of_the_paper": "This paper proposes a new computation block that unifies 3D convolution and transformer block for video action recognition. Starting from the transformer block, the core of the proposed block is its attention module which is equivalent to 3D CNN or transformer block depending on the design of the learnable parameters. On the standard action recognition benchmark, the proposed method outperforms prior art in both accuracy and computation efficiency. The comprehensive ablation studies provide insights on the design choices.", "summary_of_the_review": "This paper proposes a novel computational module that is elegant conceptually and effective empirically. Conceptually, the proposed block bridges the design choices between the 3D CNN and transformer block, which can be influential to future works on designing better structures in the attention module. Empirically, this paper achieves SOTA results in both accuracy and efficiency. Thus, I r", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "inpnLsbnHX", "reviewer": "Reviewer_axKi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Well-written \n+ Thorough ablation study, including clear visualization of the attention\n+ Clean formulation for the local and global affinity token and how they imitate 3Dconv and the standard transformer model.\n+SOTA accuracy with 10x more efficiency compared to strong baselines.\n\n-It would be nice to understand how the method apply to fine grained tasks such as human keypoints or segmentation as transformers are usually not very good in such task (vs. convolution). \n\n", "summary_of_the_paper": "This paper proposes a new transformer model for video understanding task. The proposal algorithm has the benefit of both 3Dconv to efficient capture local context and transformer for global reasoning. The other innovation is the dynamic position encoding. The proposed method is validated on Kinetics and Something something where it set new accuracy record.", "summary_of_the_review": "Solid paper with clean formulation, experimental analysis, and well written. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cc8LzyPSDr", "reviewer": "Reviewer_rXsf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The authors tackle an important problem of interest to many researchers in the video community.\n+ Very strong results on multiple action recognition benchmarks.\n+ Impressive efficiency considering how good the action recognition accuracy is.\n+ Extensive ablation experiments.\n\nWeaknesses:\n- The paper is poorly written and is quite difficult to understand.\n- The technical novelty of the paper is limited.\n\nIn my view, these two weaknesses are connected as I will discuss next. As discussed in Section 3.4, the proposed local UniFormer block is very similar to depthwise 3D convolutions. I appreciate that the authors included this section in their draft, as it makes it easier to understand their approach. However, before this section, the authors spend several paragraphs \"selling\" this local UniFormer block as a novel transformer block design. I think that's a bit misleading to the readers, and it also makes the paper much more difficult to read (in my view). At this point, most readers are probably familiar with depthwise 3D convolutions. Therefore, in my view, from the very beginning, the authors should describe their local block design by connecting it closely to the concept of depthwise 3D convolutions. Instead, section 3.2 is presented as a novel transformer block with many important details only included in the Appendix. This makes the draft convoluted and harder to read than it should be. Furthermore, the missing technical details wouldn't even be necessary to include if the proposed block was formulated as a depthwise 3D convolution extension. \n\nConnected to my point above, I don't know if the authors are using this strange formulation of their local block to increase the technical novelty of their proposed approach. However, considering the similarities with a 3D MobileNet block, I think the technical novelty of the proposed approach is quite limited. The proposed model is essentially a combination of 3D CNNs and Vision Transformers (with some bells and whistles). Personally, I wouldn't even have an issue with this because, in my view, understanding how to combine 3D convolutions and spatiotemporal self-attention is an important research question. However, I have a problem that the paper is \"being sold\" as a novel transformer-based architecture, as opposed to an extensive and valuable empirical study of combining 3D convolutional, and self-attention blocks. In a way, this paper reminds me a lot of the R(2+1)D paper \"A Closer Look at Spatiotemporal Convolutions for Action Recognition\", and I think it should be written in a similar style as this paper (i.e., empirical paper as opposed to claiming a novel architecture).  The authors might argue that the proposed local block is different than a depthwise 3D MobileNet block by pointing to the results in Table 4. However, in my view, those results largely support my claims, i.e., the gain of using the proposed block over the standard MobileNet block is marginal. Furthermore, considering my points above, I'm also not sure if UniFormer is the best name for the proposed approach. It's vague and it doesn't indicate in any way that the unification is between 3D convolutions and self-attention (as opposed to different modalities, different video tasks, or something else that's much broader).\n\nLastly, I also wanted to mention that the section on DPE is not very informative. The authors simply mention that they \"extend the conditional position encoding (CPE) (Chu et al., 2021), utilizing a simple 3D depthwise convolution with zero paddings as dynamic position embedding (DPE)\". It would be useful to add more details on this for those readers who are not familiar with CPE. \n\nThere are also typos, and grammar mistakes in the draft (a few of them listed below):\n- \"Sprots\" -> \"Sports\" in Table 2\n- Wrong ViViT citation in Table 3\n- \"Spatial-Temporal\" -> \"Spatiotemporal\"\n\nI'd be interested to hear how the authors plan to address my concerns in the camera-ready version of the draft. ", "summary_of_the_paper": "The paper proposes a UniFormer architecture, which combines 3D convolutions and spatiotemporal self-attention for efficient and effective video classification performance. Specifically, the authors show that using local 3D convolution-like layers in the early stages of the network and global self-attention layers in the later stages of the network provides a good tradeoff between efficiency and accuracy. The proposed method reports state-of-the-art results on several major action recognition benchmarks.", "summary_of_the_review": "Despite the shortcomings listed above, I'm still largely in favor of accepting this paper. It provides a very valuable empirical study of combining 3D convolution-based operators with self-attention. Furthermore, the authors obtain very impressive results at a relatively small computational cost (especially compared to prior video transformer papers). The presented ablation studies would also be highly beneficial for the research community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5o-wz3BJVoK", "reviewer": "Reviewer_4ga6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The empirical study is sufficient, both the state-of-the-art comparison and ablation study.\n- The general idea of taking the best from 3D CNN and vision transformer is interesting. \n\nWeakness:\n- The related work is pretty brief, which does well explain the relation w.r.t. to the literature in similar direction. E.g. (Liu et al., 2021b).\n- The phrase \"local redundancy\" appears quite vague to me. According to the paper, the redundancy seems to refer to the pixels within a local patch, which can be reduced by CNN with 3D convolution. For me, what carried by a local patch is more of \"local structure\" than \"redundancy\", which should be capture instead of \"suppress\". \n- The paper introduces \"DPE\" module and the description can barely be found. It is not clear why DPE is able to \"dynamically integrate 3D position information\".\n- Also, the \"FFN\" module is introduced with only one sentence.\n- In shallow layers, the affinity map A_n becomes a learnable parameter matrix a_n. In this case, A_n is fixed for any test video. This sound counterintuitive. Please correct me if I mistook it.\n", "summary_of_the_paper": "In this paper, a new architecture, Uniformer, is proposed to learn spatial-temporal pattern in videos. \nThe new architecture is claimed to effectively aggregate both local information and global information.\nIn contrast, the previous two main approaches, 3D CNN and vision transformer, can only take care of one aspect.\nThe main building block for Uniformer is a Multi-Head Relation Aggregator, which behaves differently at shallow layers and deep layers, to aggregate local and global information respectively.\nExtensive empirical studies demonstrate the strength of the proposed method.  ", "summary_of_the_review": "This paper explores an interesting direction on how to enable transformer-like architecture aware of both local and global information.\nThe experiments are carefully designed and results are encouraging. The reviewer would appreciate it if the authors address the above concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach for spatio-temporal representation learning using Transformers. It introduces a particular architecture design, which shows an impressive computational efficiency. The reviewers agree that the experimental results are strong, and unanimously recommend the paper for acceptance. The reviewers find their concerns regarding the details of the approach/setting address after the authors' response.

We recommend accepting the paper.
Decision: Accept (Poster)