Shifted Chunk Transformer for Spatio-Temporal Representational Learning
Authors: Xuefan Zha
Wentao Zhu
Lv Ting Xun
Sen Yang
Ji Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representational learning
action recognition
transformer
spatio-temporal
shifted self-attention

Abstract: Spatio-temporal representational learning has been widely adopted in various fields such as action recognition, video object segmentation, and action anticipation.Previous spatio-temporal representational learning approaches primarily employ ConvNets or sequential models, e.g., LSTM, to learn the intra-frame and inter-frame features.  Recently, Transformer models have successfully dominated the study of natural language processing (NLP), image classification, etc. However, the pure-Transformer based spatio-temporal learning can be prohibitively costly on memory and computation to extract fine-grained features from a tiny patch. To tackle the training difficulty and enhance the spatio-temporal learning, we construct a shifted chunk Transformer with pure self-attention blocks. Leveraging the recent efficient Transformer design in NLP, this shifted chunk Transformer can learn hierarchical spatio-temporal features from a local tiny patch to a global videoclip. Our shifted self-attention can also effectively model complicated inter-frame variances. Furthermore, we build a clip encoder based on Transformer to model long-term temporal dependencies. We conduct thorough ablation studies to validate each component and hyper-parameters in our shifted chunk Transformer, and it outperforms previous state-of-the-art approaches on Kinetics-400, Kinetics-600,UCF101, and HMDB51.

Shifted Chunk Transformer for Spatio-Temporal Representational Learning
Xuefan Zha
Wentao Zhu Kuaishou Technology ventaozhn@@kuaishou..com
Kuaishou Technology
a com
Tingxun Lv Kuaishou Technology Ivtingxun@kuaishoor com
Sen Yang
Ji Liu
Kuaishou Technology
Kuaishou Technology
senyangCkuaishou com
ji. 1iu. uwisc8Gmail.com
Abstract
Spatio-temporal representational learning has been widely adopted in various fields such as action recognition, video object segmentation, and action anticipation. Previous spatio-temporal representational learning approaches primarily employ ConvNets or sequential models, e.g., LSTM, to learn the intra-frame and inter- frame features. Recently, Transformer models have successfully dominated the study of natural language processing (NLP), image classification, etc. However, the pure-Trane Transformer based spatio-temporal learning can be prohibitively costly memory and computation to extract fite fine- features from tiny patch. To tackle the training difficulty and enhance the spatio-temporal learning, we construct shifted chunk Transformer with pure self-attention blocks. Leveraging the recent efficient Transformer design in NLP, this shifted chunk Transformer can learn hierarchical apatio-temporal features from local tiny patch to global video clip. Our shifted self-attention can also effectively model complicated inter-frame variances. Furthermore, we build clip encoder based Transformer to model long-term temporal dependencies. We conduct thorough ablation studies to validate each component and typer-parameter in shifted chunk Transformer, and it outperforms previous state-of-the-ee approaches on Kinetics-400, Kinetics-600, UCF101, and HMDB51.
Introduction
Spatio-temporal representational learning tries to model complicated intra-frame and inter-frame re-
lationships, and it is critical to various tasks such as action recognition [20), action detection [55] 52)
object tracking 23 and action anticipation (2 Deep learning based spatio-temporal representation learning approaches have been largely explored since the success of AlexNet on image classifica-
tion [1] Previous deep temporl learning can be mainly divided into two aspects: deep
ConvNets based methods [35
[5 16] and deep sequential learning based methods [55]
28 29
Deep ConvNets based methods primarily integrated various factorization techniques [5] 34]
or a priooi [16] for efficient spatio-temporal learning [15). Some works focus on extracting effec-
tive spatio-temporal features [4] g or capturing complicated long-range dependencies [49]. Deep
sequential learning based methods try to formulate the spatial and temporal relationships through
advanced deep sequential models 28 the attention mechanism 29
On the other hand, the Transformer has become the de-facto standard for sequential learning tasks 54 19]. The great success of Transformer
such speech and language processing 44 natural language processing (NLP) has inspired computer vision community to explore self-attention
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Clip
Image Chunk Self-Attention
Frame Classificattin Embedding
MLP Head
Image Paich Ershyolinng
Shidted MSA
Shifted MSA
Stufted MSA
Shifted MSA
Chenk achells un'ees
Visual Local Transfornce (MLLT) M blocks
Image LSit Anention
Image Chunk
Image Chunk Atiention
Image Chunk Atlention
Image Chunk Attention
Local
Imsage Patch
1. X
Video Clip
Figure 1: The framework of the proposed shifted chunk Transformer which involves two main components, frame encoder (dark grey) and clip encoder. The frame encoder consists of N alternative blocks of image chunk self-attention (left) and shifted multi-head self-attention (MSA).
structures for several vision tasks, egg., image classification super-resolution [33]. The main difficulty in pure-Transformee models for vision is that Transformers lack the inductive biases of convolutions, such as translation equivariance, and they require more data [K] or stronger regularisation [40 in the training. It is only very recently that, vision Transform (VIT), a pure Transformer architecture, has outperformed its convolutional counterparts in image classification when pre-trained on large amounts of data [13. However, the hurdle is aggravated when the Tare-Transfomer design is applied to
13
36
object detection 6 and
representational learning.
Recently, a few attempts have been made design Tare-Transformer structures for spatio-temporal representation learning [
5 14 2] Simply applying Transformer to 3D video domain is compu- tationally intensive [4] The Transformer based spatio-temporal learning methods primarily focus on designing efficient variants by factorization along spatial- and temporal-dimension 4 [5] or employing multi-scale pyramid structure for trade-off between the resolution and channel capacity while reducing the memory and computational cost 14] The spatio-temporal learning capacity can be further improved by extracting more effective fine-grained features through advanced and efficient intra-frame and inter-frame representational learning.
In this work, propose novel spatio-tempont learning framework based pure- mure-Transformer called shifted chunk Transformer as illustrated Fig. which extracts effective fine- fine-grained intra- frame features with low computational complexity leveraging the recent advance of Transformer in NLP [22]. Specially, we divide each frame into several local windows called image chunks, and construct hierarchical image chunk Transformer, which employs locality -ensitiveee hashing (LSH) to enhance the dot-product attention in each chunk and reduces the memory and computation consumption significantly. To fully consider the motion effect of object, design robust self- attention module, shifted self-attention, which explicitly extracts correlations from nearby frames. We further design pure- Transformer based frame-wiis attention module, clip encoder, model the complicated inter-frame relationships with a minimal extra computational cost. Our contributions can be summarized follows:
We construct image chunk self-attention to mine fine-grained intra-frame features leverag- ing the recent advance of Transformer. The hierarchical image chunk Transformer employs becality-sesiitvve hashing (LSH) 3 to reduce the memory and computation consumption significantly, which enables an effective spatio-temporal learning directly from a tiny patch.
We build a shifted self-attention to fully consider the motion effect objects, which yields effective modeling of complicated inter-frame variances in the spatio-temporal representa-

---

tional learning. Furthermore, clip encoder with a Trrn-Tranffrmer structure is employed for frame-wis wise attention, which models complicated and long-term inter-frame relationships at a minimal extra cost.
The shifted chunk Transformer with pure- Transformer outperforms previous statc-of-the art approaches on several action recognition benchmarks, including Kinectics-400 [20).
Kinetics-600 77. UCF101 (39 and HMDB51 [25
Related Work
Conventional deep learning based action recognition Conventional deep spatio-temporal repre-
sentational learning mainly involves two aspects: deep sequential learning based methods [55] 28 29]
and deep ConvNet based methods [35 T5
The recurrent networks can be extended to 3D
spatio-temporal domain for action recognition 28
In deep ConvNet based methods, two-stream
ConvNet employs two branches of 2D ConvNets and explicitly models motion by optical flow [38
The C3D 41 and I3D [8] directly extend 2D ConvNets to 3D ConvNets, which is natural for 3D
spatio-temporal representational learning [9]. However, the 3D ConyNet requires significantly more computation and more training data achieve desired accuracy [5I]. Thus, P3D (34] and S3D [5]] attempt to factorize the 3D convolution into a 2D spatial convolution and a 1D temporal convolution. SlowFast network [16] and X3D 15 conduct trade-off's among resolution, temporal frame rate and the number of channels for the efficient video recognition. Non-local network [49] proposes to add non-local operations in deep network and captures long-range dependencies. The recent Ture-Transformer based spatio-temporal learning enables longer dependency relationship modeling and further increases the accuracy of action recognition
Vision Transformers NLP community has witnessed the great success of pre-training by Trans- former [4 [2). and it has been emerging for image classification [13 40 36), object detection (6), and image super-resolution [3). CPVT [0] employs pre-defined and independent input tokens to increase the generalization for image classification. Tran-Tranffommer network has no inductive bias or prior as ConyNets. ViT [3 pre-trains on large amounts of data and attains excellent results on image classification. CvT [50 introduces convolutions into ViT to yield better performance and
efficiency. DeiT [40] and MVIT [4] instead employ distillation and multi-scale to cope with the training difficulty. PVT [47] and segmentation Transformer (33) further extend Transformer to dense prediction tasks, e.g., object detection and semantic segmentation. Simply applying Transformer to 3D video spatio-temporal representational learning aggravates the training difficulty significantly, and it requires advanced model design for pure- Tare-Tranffomer based spatio-temporal learning.
Transformer based action recognition Recently, only a few works have been conducted using Most of the efforts focus on designing pure-Transformer for spatio-temporal learning 4 5
efficient Transformer models to reduce the computation and memory consumption. ViVIT and TimeSformer [5] study various factorization methods along spatial- and MVIT 14] conducts a trade-off between resolution and the number of channels, and constructs multi-scale Transformer to learn a hierarchy from simple dense resolution and fine-grained features complex coarse features. VATT ( conducts unsupervised multi-modality self-supervised learning with a pure-Trranfforer structure. In this work, we extract fine-grained intra-frame features from each tiny patch and model complicated inter-frame relationship through efficient and advanced self-attention blocks.
3 Method
In this section, we describe each component of our shifted chunk Transformer for spatio-temporal representation learning in video based action recognition.
3.1 Overview
Let X x2,,,,,,,"' X E RTxxxWxx be one input clip of T RGB frames sampled from a video, where X; E kRll X3 is the i-th frame in the clip, and H and W are the frame size. To design an efficient pure-Transformer based spatio-temporal learning, we construct a shifted chunk Transformer, including image chunk self-attention blocks, shifted multi-head self-attention blocks,
3

---

and a clip encoder, as illustrated in Fig. 1 We first construct an image chunk self- attention block leveraging the advanced efficient Transformer design in NLP [22. which is illustrated in the left of Fig. [ The locality-sensitiy hashing (LSH) 3 in the image chunk self-attention enables a relatively small patch as a token, thus it is capable of extracting fine-grained intra-frame features. A linear pooling layer is designed to adaptively reduce the resolution after LSH attention. After that, shifted self-attention is designed to extract motion related inter-frame features. Our shifted self-attention considers the motion of objects across nearby frames and explicitly models the temporal relationship into self-attention. The frame encoder shown in dark grey color can be an effective feature extractor which can be stacked for several times. The hierarchical frame encoder and image chunk self-attention further learns an effective multi-level feature from local to global abstraction. Lastly, we employ a Tune-Transfomer to learn complicated inter-frame relationships and frame- wise attention along the temporal dimension. We use multi-head self-attention (MSA) in all the blocks.
3.2 Image Chunk Self-Attention
Transformer can learn complicated long range dependencies which can be computed through the high efficient matrix product :4 Differ-
Table I: ViT-B [3]
with a smaller patch size yields higher accuracy (%).
ent from convolution which has inherent induc- tive bias [3), Transformer learns the entire fea- tures from data. The main challenge for pure- Transformer based vision model mainly involves two aspects: 1) how to design an efficient model to learn effective features from the entire image, because simply treating each pixel as a token is computationally intensive, 2) how to train this power- ful model and learn various effective features from data. ViT [13] treats each patch of size 16 x 16 as one token and pre-trains the model with large amounts of data. We argue that Transformer with smaller patch as a token can extract fine-grained features which improves spatio-temporal learning for action recognition. For ViT-B-16 [13 of crop size 224 x 224 as frame encoder followed by a shifted MSA and clip encoder, a smaller patch size of 8 x 8 yields better accuracy on Kinectics-400 and UCF101 as shown in Table
Crop size
Patch size
K400
UCF101
224
16
75.3
95.3
224
8
78.4
97.0
The Transformer computes each pairwise correlation through a dot product, thus it has a high computational complexity of O(L)). where L is the totally number of tokens. In natural language processing (NLP), LSH attention [22] employs locality-sensitiv hashing (LSH) bucketing [3 and chunk sorting for queries and keys approximate the attention matrix computation. Leveraging the efficient LSH approximation, the LSH attention reduce the computation complexity to OL log L).
To preserve locality property and learn transition and rotation invariant low-level features from images, we firstly design a visual local transformer (ViLT) with shared parameters along different image local windows, or image chunks, Each image chunk consists of multiple tiny patches as illustrated in the left bottom block of Fig. T We intend to employ patches of a small size in the ViLT which is the first level abstraction of the input, that the model extracts a fine- traiind representation which enhances the entire spatial-temm temporal learning. Employing the small patch yields large number of tokens in the following self-attention, and we construct image locality- sensitive hashing (LSH) attention leveraging advanced design of Transformer in NLP [22]. The image LSH attention can efficiently extract higher-level and plentiful intra-frame features ranging from a tiny patch to the entire image.
The framework of image chunk self-attention is illustrated in the left part of Fig. 0
Visual local transformer (ViLT) In the shifted chunk Transformer, which slides one self-attention for each tiny patch along the whole image. The ViLT is illus- trated in the bottom block of Fig. [ Let h, w be the height and width of the tiny image patch E mhxwx3 Following the success of ViT [33, we treat each patch as one dimensional vector of length h x X 3. Suppose each chunk consists of m (P1.1 P1,2, ... PI.n' P2.1, PP.ni Pmm.1 Pm,u)- )- After flattening the chunk into a list of tiny patches, we denote the chunk as (Pi, P2, Pr.) without loss of generality, where L -= m
firstly construct a ViLT
tiny patches, denoted as
In ViLT, we use a learnable 1D position embedding Eposs E LxD to retain position information
2g E. :
Punx E E Epos:
E E (1xxx33)
(1)

---

where E is the linear patch embedding matrix, and D is the embedding dimension. Then we can construct alternating layers of multi-head self-attention (MSA), MLP with GELU [8] non- con-linerrity, Layerorm (LN) and residual connections [44) for the chunk as
(2) where M is the number of blocks. We conduct the ViLT sliding the entire image without over- lapping. The parameters of the ViLT are shared among all the image chunks, which forces the chunk self-attention to learn translation and rotation invariant, and fine-grained features. The tiny patch-wise feature extraction preserves the locality property. which is a strong prior for nat- ural images. After the ViLT, we obtain the extracted features for the entire image denoted as y
zm (S))(L) + Zm-1, Zm = (L z/m. m = 1, 1,... M,
12; :YL; yYLxI/) where the entire image can be split into I' = m' x n' = chunks, and we conduct zero padding for the last chunks in each row and column.
W 6xn
1xm
The ViLT forces to learn image locality features which is a desired property for a low-level feature extractor [26]. For pure- urr--Tranffomer based vision system, it also reduces the memory consumption significantly because it restricts the correlation of one tiny patch within the local chunk. Therefore, the memory and computational complexity of dot-product attention in ViLT can be reduced to ( compared to the complexity of conventional self-attention O(LL' x L)
Image locality-sensitive hashing (L.SH) attention After ViLT blocks, we obtain local fine-grained features of length I' x L. Since the patch size is tiny, the total number of patches can be large, which leads to more difficult training than other vision Transformers 4 [.). On the other hand, the problem of finding nearest neighbors quickly in high-dimensional spaces can be solved by locality-sensitive hashing (LSH), which hashes similar input items into the same "buckets" with high probability. In NLP, LSH attention [22 is proposed to handle quite long sequence data, which employs locality- sensitive hashing (LSH) bucketing approximation and bucket sorting to reduce the computational complexity of matrix product between query and key in self-attention.
In dot-product attention, the softmax activation function pushes the attention weights close to I or 0, which means the attention matrix is typically sparse. The query and key can be approximated by locality-sensitive hashing (LSH)) 3 to reduce the computational complexity. Furthermore, through bucketing sort, the attention matrix product can be accelerated by chunk triangular matrix product, which has been validated by LSH attention [22].
The used multi-head image LSH attention can be constructed as MLP(LN(y')) +y',
y' MSA(LSSAA((LN())) +y,
where LSH attention I (SH(t(()) employs angular distance to conduct LSH hashing 3 The image LSH attention reduces the memory and time complexity to O x L. log [.' x I.)), compared with that of conventional dot-product attention O( x 2))). The image LSH attention reduces the complexity significantly because the patch size is tiny and the number of tiny patches L' x L are large. The image level LSH attention in the second level learns relatively global features from the first level's 's local fine-grained features.
The hierarchical feature learning from local to global has been validated as an effective principle for Inspired by hierarchical abstraction in ConvNets (26), we construct a linear pooling layer which firstly conducts squeeze then employs linear projection for feature dimension reduction. The linear pooling adaptively squeezes the sequence length by 1.
vision system design 26 451
Reshape: s [$1,1, 81,axn"; 38mxm',,," Smxm' mxx'),
Squeeze: s' = |s S 1nnx' /2inn/2i--1 /2:--- (ixxm//211" S' mmxm /2,nxn'/22 /2/.
(4)
Linear: t |s Ex, $1 Eg m /2, E,, Smxm' /2,n////n E.).
where s'., in s' concatenates 5 S 12x-1.25-1-- S2i- 1.2j. S21,2 1 and S2.2, from S. Eg is the linear projec- tion matrix to adaptively reduce the number of dimensions after squeeze by half. The reshape is to retain the spatial relationship of each patch, and the squeeze reduces the number of patches by 1 and enlarges the feature dimensions by four times. The linear pooling layer forces the model to learn high-level global features in the following layers.
3.3 Shifted Multi-Head Self-Attention
Considering the motion effect of objects, we explicitly construct shifted multi-head self-attention (MSA) for spatio-temporal learning after image chunk self-attention illustrated in Fig. | and Fig. 2
5

---

For video classification, special classification token ([CLS)) [2] can be prepended into the feature sequence. To learn frame-wise spatio-temporal representations, we prepend the [CLS] token each frame. Without loss of generality, we denote the image chunk self-attention feature to be a list of [ta,1; t,,21-' : [ for the i-th frame. Then, we obtain the input of T frames for shifted MSA as
= [
(5)
Cici t1,1 it,,1x///4" dl tT, tt,,LxI/44.
The shifted multi- self-atte eeffaattention explicitly considers the inter-frame motion of objects, which computes the correlation between the current frame and the previous frame in the attention matrix, which can be formulated as
Pi.p (N = (. WK. K: Vi,p = (N((t),)))) W
where W W k and Wi are projection matrices for head and employ cyclic way caleulate the key the first frame by defining to,p p. By concatenating q ki vi.p into matrices Q Ki, Vi along patch location the shifted MSA for frame l can be calculated as
a, Concat(Attention(() Q K! Vi),... Attention (Q/, K2 Vi Wo:
a MLP(LN((.)) ax.
where / is the number of heads in multi-head self-attention. The shifted self-attention com- pensates object motion and spatial variances. We explicitly integrate motion shift into self- attention, which extracts robust features for spatio-temporal learning. The block with al- ternating layers of image chunk self-attention and shifted MSA can be stacked for multiple times fully extract effective hierarchical fine- grained features from tiny local patches to the whole clip in our shifted chunk Transformer.
Chunk 1-1
Re-1
3.4 Clip Encoder for Global Clip Attention
Chunkt
To learn complicated inter-frame relationship from the extracted frame-level features, we de- sign a clip encoder based on pure-Transformer structure adaptively learn frame-wise atten- tion. To facilitate the video classification. we prepend a global special classification token ((CLS)) into the frame-level feature sequence. In this module, we employ the classification feature aL corresponding to Zt,cls as the frame-level feature for frame t. To consider frame position, we also employ standard learnable 1D position embedding as each frame position embedding. The clip encoder can formulated as
Pt
Figure 2: Illustration of shifted MSA, which ex- plicitly extracts fine-grained motion information along two frames.
bo [beisi a1 E': ET] E poss Epas E (IT++))
by MSA(LN(( (M bm-1, b MLP(LN((b,.)) +b b m 11 M', C = MLP(LN((br1))
(8)
where E' is the linear frame embedding matrix, D' is the clip encoder embedding size, M' is the number of blocks, and ba the clip-level classification feature for the classification token bate is the video classification logit for softmax. We use dropout (24] for the second last layer and cross-entropy loss with label smoothing [32 for training. The clip encoder can be efficient with minimal computational cost in Appendix to achieve powerful inter-frame representation learning.
4 Experiment
We evaluate our shifted chunk Transformer, denoted as SCT, on five commonly used action recognition datasets: Kinetics-400 (20). Kinetics-600 (7), Moment-in-TT Time [31] (Appendix), UCF101 (39 and

---

Table 2: Hyper-paramelers of data processing and optimization.
K400
K600
U101
H51
MMT
Frame Rate
5
5
10
10
8
Frame Stride
10
10
8
8
10
#Warmup epochs
2
2
3
4
2
Learning rate
0.3
0.3
0.25
0.25
0.3
Label smoothing
0.1
0.1
0
0
0.3
Dropout
0.2
0.2
0
0
0.2
Table 3: The model structure of three shifted chunk Transformers.
MLP size
Clip MLP size
Model
M
#Heads
#Param
GFLOPs
SCT-$
384
4
192
768
[4668
18.72M
88.18 18
SCT-M
128
512
6
192
768
[4888]]
33.48M
162.90
SCT-L
192
768
4
192
768
[4
59.89M
342.58
We adopt ImageNet-21K for the pre-training [7 11 because large model capacity of SCT. The default patch size for each image token is 4 x 4. In the training, we use synchronous stochastic gradient descent with momentum of 0.9, cosine annealing schedule [30], and the number of epochs of 50. We use batch size of 32, 16 and 8 for SCT-S, SCT-M and SCT-L, respectively. And frame crop size set to be 224 x 224. For data augmentation, we randomly select the start frame to generate the input clip. In the inference, the final prediction by averaging the softmax probabilistic scores from these multi-view predictions. The details of initial learning rate, optimization and data processing are shown in Table 2 All the experiments are run on NVIDIA Tesla V100 32 GB GPU cards.
HMDB51 25
extract multiple views from each video and obtain
We construct three shifted chunk Transformers, SCT-S, SCT-M, and SCT-L,, in terms of various model sizes and computation complexities. We employ four consecutive blocks with alternating one image chunk self-attention and one shifted MSA. The patch embedding size D, MLP dimension of these self-attentions, the number of ViLT layers M, the clip encoder embedding size D', MLP dimension of clip encoder, the numbers of heads in ViLT, LSH attention, shifted MSA and clip encoder are shown Table Each image chunk self-attention consists of M layers ViLT followed by image LSH attention and linear pooling layer to reduce the number of spatial dimensions. We use four-layer clip encoders to obtain the video classification results as validated in the Appendix.
Validating frame feature extractor We compare the frame encoder our shifted chunk Trans- former (SCT) with ViT [13] in Table For fair comparison, we only replace the ViLT with VIT, and remain all other components the same. From Table we observe that 1) large models, VIT-L-16 and SCT-I., yield higher accuracy than base models, V6T-B-16 and SCT-S; 2) ViT with small patch size achieves better accuracy than ViT with large patch; 3) SCT-L improves the accuracy of ViT-I. by 4% while using much less number of parameters and FLOPs. The tiny patch and enforced locality prior in ViLT are validated to be effective for spatio-temporal learning.
Table 4: The effect of frame extractor and the number of tokens Top-1 accuracy (%).
Method
Patch size
Chunk size
#Tokens
#Param
GFLOPs
Kinetics-400
ViT-B-16
16x 16
14x 14
114.25M
405,06
75,33
ViT-B-16
12x 12
18x 18
114.25M
665.78
77.12
VYT-B-16
8x8
28x28
114.25M
1603.67
78.95
VIT-L-16
16x16
14x
328.63M
1413.45
79.15
SCT-S
4x4
7x7
8x8
18.72M
88.18
78.41
SCT-M
4x4
7x7
8x8
33.48M
162.90
81.26
SCT-L
4x4
7x7
8x8
59.89M
342.58
83.02

---

Table 5: The effect of the number of shifted MSA layers and shifted frames on Top-1 accuracy (%).
#Shifted
#Shifted frame
Method
K400
U101
MSA
SCT-S
0
76.91
97.01
SCT-S
1
1
78.41
98.02
SCT-S
5
77.02
97.15
SCT-S
2
1
77.45
97.33
Figure 4: Visualization of the patch and frame attention maps (the second and fourth rows).
Effect of shifted MSA We conduct experiments on Kinectis-400 and UCF101 validate the hyper- parameters of the shifted MSA layer, including the number of shifted MSA layers, and the number of shifted frames used in the calculation of key equation (6). All other network configurations follow the Table The #Shifted MSA of O and #Shifted frame of O in Table that one standard MSA is used instead of shifted MSA. From Table 5 we observe that 1) shifted MSA improves the accuracy up to 1.5% compared with the conventional MSA: 2) one layer shifted MSA with the shifted number of frames one yields the best accuracy. The shifted MSA explicitly formulates the motion effect of object by considering the nearby frames, which improves the accuracy for video classification. We use one layer shifted MSA with the number of shifted frames one in our experiment.
Varying the number of input frames and temporal views In our experiments so far, we have kept the number of input frames fixed to 24 across different datasets. To discuss the effect of the number of input frames video level inference accuracy, we validate the number of input frames of 24, 48, 96, and the number of temporal views from 1 to 8. Fig. 3 shows that as increase the number of frames, the accuracy using single clip increases, since the network is incorporated longer temporal information. How-
Top-1 Video Accuruyyt%
27.3
77.9
ever, as the number of used views increases, the accuracy difference is reduced. We use the num- ber of frames of 24 and the number of temporal views of in our experiment.
Dumme 24
15.5
trama-40
Views
Figure 3: The effect varying the number of input Patch and frame attention Our shifted frames and temporal views.
chunk Transformer (SCT) can detect fine-grained discriminative regions for each frame in the entire clip in Fig. 4 Specifically, we average attention weights of the shifted MSA across all heads and then recursively multiply the weight matrices of all layers (I. which accounts for the attentions through all layers. The designed framework of SCT leads an easy diagnosis and explanation for

---

Table 6: Top-1 and Top-5 accuracy (%) comparisons to state-of-the-ar approaches on Kinectics-400.
Method
TFLOPsX Views
#Param
Runtime (s)
Top1
Top5
TEA 127
0003
76.1
92.5
I3D NL
54M
77.7
93.3
CorrNet- 101
46
0.224x 10x3
79.2
ip-CSN- 152
42
0.109x 10x3
33M
79.2
93.8
SlowFast [16
0.234 10x3
60M
79.8
93.9
X3D-XXL [1]
0.194 10x3
20M
0.176
80.4
94.6
TimeSformer [5]
2.38x1x3
121M
0.475
80.7
94.7
MVVT-B x3
14
x3x35x3x3
37M
0.153
81.2
95.1
ViVIT-L
399.2x4x3
89M
81.3
94.7
SCT-S
0.088
19M
0.051
78.4
93.8
SCT-M
0.163x4x3
33M
0.072
81.3
94.5
SCT-L
0.343x4x3
60M
0.106
83.0
95.4
Table 7: Classification accuracy (%) comparisons to state-of-the-art approaches on Kinectics-600,
UCF101 (3 splits) and HMDB51 splits). 'K400' denotes pretraining on Kinetics-400 and ImageNet. 'K600' denotes pretraining on Kinetics-600 and ImageNet.
Method
Pretrain
U101
H51
Method
Views
#Para
Top1
Top5
I3D g
K400
95.4
74.5
AutenNAS
48
79.8
94.4
ResNeXt
17
K400
94.5
70.2
LGD-3D
35
81.5
95.6
R(2+1)D
43
K400
96.8
74.5
SlowFast
16
10x3
60M
81.8
95.1
S3D-G [51
K400
96.8
75.9
X3D-XL
[15
10x3
11M
81.9
95.5
LGD-3D
135
K600
97.0
75.7
TimeSformer
5
1 1x3
121M
82.4
96.0
ViViT-I. 4
4x3
89M
83.0
95.7
SCT-S
ImageNet ImageNet
98.0
76.5
SCT-L
97.7
81.4
SCT-S
4x3
19M
77.5
93.1
SCT-M
4x3
33M
81.7
95.5
SCT-S
K400
98.3
81.5
SCT-L
4x3
60M
84.3
96.3
SCT-M
K400
98.5
83.2
SCT-L
K400
98.7
84.6
the prediction, which potentially makes SCT applicable to various critical fields, egg., healthcare and autonomous driving.
Comparison to -het--rrt- approaches We compare shifted chunk Transformer (SCT) to the current state-of-the-ar approaches based on the best aaperrparameters validated in the previous ablation studies. We obtain the results of previous state-of- the-art approaches from their papers. We obtain the actual runtime (s) in one single NVIDIA V100 16GB GPU by averaging 50 inferences with batch size of one. In Kinetics-400 and Kinetics-600. we initialize our ViLT and LSH attention trained on Im.
Our shifted chunk Transformers (SCT) surpass previous state-of-the-art approaches including both recent Transformer based video classification and previous deep ConvNets based methods by 2.7%, 1. 1.3%, 1.7% and 8.9% on Kinectis-400, Kinectic-600, UCF101 and HMDB51 in Table based on RGB frames, respectively. The local scheme and LSH approximation in image chunk self-attention enables to use patches of a small size. Because of the efficient model design, SCT achieves the best accuracy even only using pretraining on the ImageNet on UCF101 and HMDB51. Besides the higher action recognition accuracy, the SCT employs less number of parameters and FLOPs than ViVIT because we employ less number of channels and our SCT effective for spatio-temporal learning.
5 Conclusion
In this work, we proposed new spatio-temporal learning called shifted chunk Transformer inspired by the recent success vision Transformer image classification. However, the current pure- Transformer based spatio-temporal learning is limited by computational efficiency and feature

---

robustness. To address these challenges, we propose several efficient and powerful components for spatio-temporal Transformer, which is able learn fine-grained features from a tiny image patch and model complicated spatio-temporal dependencies. We construct an image chunk self-attention which leverages locality-sensitie hashing to efficiently capture fine-grained local representation with relatively low computation cost. Our shifted self-attention can effectively model complicated inter- frame variances. Furthermore, we build a clip encoder based on pure- Transforoorr for frame-wise attention and long-term inter-frame dependency modeling. We conduct thorough ablation studies to validate each component and previous state-of-ther approaches including both pure- Transforrerr architectures and deep 3D convolutional networks on various datasets in terms of accuracy and efficiency.
in our shifted chunk Transformer. It outperforms
Acknowledgement
This work is supported by Kuaishou Technology. No external funding was received for this work. Moreover, we would like to thank Hang Shang for insightful discussions.
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-41977 2020.
[2] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104 / 2021.
[3] Alexandr Andoni, Piotr Indyk, TMM Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. Practical and optimal Ish for angular distance. In Advances in Neural Information Processing Systems (NIPS 2015), pages 1225-1233. Curran Associates, 2015.
[4] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint 1...::22033166)) 2021.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095. 2021.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213 13-229. Springer, 2020.
[7] Joao Carreira, Eric Noland, Andras Manki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 0008001340, 2018.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299 6308, 2017.
[9] Christoph and Feichtenhofer Axel Pinz. Spatiotemporal residual networks for video action recognition. Advances in Neural Information Processing Systems, pages 3468-3476, 2016.
[10] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv preprint arXiv:2 102. 10882, 2021.
[11] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
10

---

[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[14] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104. 2021.
[15] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203-213, 2020.
[16] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202-6211, 2019.
[17] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6546-6555, 2018.
[18] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1606. 05 2016.
[19] Yufang Huang, Wentao Zhu, Deyi Xiong, Yiye Zhang, Changjian Hu, and Feiyu Xu. Cycle- consistent adversarial autoencoders for unsupervised text style transfer. In Proceedings of the 28th International Conference Computational Linguistics, pages 2213-2223, 2020.
[20] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint IrXivv:7055 079500, 2017.
[21] Qiuhong Ke, Mario Fritz, and Bernt Schiele. Time-conditioned action anticipation in one shot. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9925-9934, 2019.
[22] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Representations, 2020.
[23] Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, Roman Pflugfelder, Koni-Kristiaan Kamarainen, Luka Cehovin Zajc, Ondrej Drbohlav, Alan Lukezic, Amanda Berg, et al. The seventh visual object tracking vot2019 challenge results. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pages 0-0, 2019.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25:1097- 100,, 2012.
[25) Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. International Conference on Computer Vision, pages 2556-2563. IEEF, 2011.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEF, 86(1 1):2278--2224, 1998.
[27] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 909-9188 2020.
[28] Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal lstm with trust gates for 3d human action recognition. European Conference on Computer Vision, pages 816-833. Springer, 2016.
11

---

[29] Jun Liu, Gang Wang, Ping Hu, Ling- Yu Duan, and Alex C Kot. Global context-aware attention Istm networks for 3d action recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1647-1656, 2017.
[30] Iya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
[31] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):502-500 2019.
[32] Rafael Müller, Simon Korbblihh, and Geoffrey Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, 2019.
[33] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
[34] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning gpatio-tempont representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533-5541 2017.
[35] Zhaofan Qiu, Ting Yao. Chong-War Wah Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12056-12065, 2019.
[36] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems, 2019.
[37] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104. 10072 10972, 2021.
[38] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recog- nition in videos. In Proceedings of the 27th International Conference on Neural Information Processing Eastems-Volume 1, pages 568-576, 2014.
[39] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv: 12 12.04022 2012.
[40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 2877, 2020.
[41] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 4489-4497 2015.
[42] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5552-5561, 2019.
[43] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, volume 30, pages 1998-6008, 2017.
[45] Andrea Vedaldi and Brian Fulkerson. Vifeat: An open and portable library of computer vision algorithms. In Proceedings of the 18th ACM International Conference on Multimedia, pages 1469-1472, 2010.
12

---

[46] Heng Wang, Du Tran, Lorenzo Torresani. and Matt Feiszli. Video modeling with correlation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 352-361, 2020.
[47] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXi::2102.12122 2021.
[48] Xiaofang Wang, Xuehan Xiong. Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua. Attentionnas: Spatiotemporal attention cell search for video classification. In European Conference on Computer Vision, pages 449-465. Springer, 2020.
[49] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794-7803, 2018.
[50] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.1 15808, 2021.
[51] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning: Speed-accurace trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 305-321, 2018.
[52] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detection from frame glimpses in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2678-2687, 2016.
[53] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a Conference on Computer Vision and Pattern Recognition, 2021.
perspective with transformers. In Proceedings of the IEEF/CVF
[54] Wentao Zhu, Tianlong Kong, Shun Lu, Jixiang Li, Dawei Zhang, Feng Deng, Xiaorui Wang, Sen Yang, and Ji Liu. Speechnas: Towards better trade-off between latency and accuracy for large-scale speaker verification. In ASRU, 2021.
[55] Wentao Zhu, Cuiling Lan, Junliang Xing, Wenjun Zeng, Yanghao Li, Li Shen, and Xiaohui Xie. Co-occurrence feature learning for skeleton based action recognition using regularized deep Istm networks. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 30, 2016.
13


Reviews:
[{"review_id": "y2qdeIe0aCW", "reviewer": "Reviewer_yF1i", "summary": "The paper proposes a novel transformer-based architecture for video processing. The input tokens are small patches extracted for each frame. The model consists of 4 modules: First, the tokens are processed locally, using a transformer with restricted, local connectivity (each image is split into multiple chunks and only the patches from the same chunk exchange messages using the self-attention layer). Second, a global processing is conducted using an efficient transformer (that use LSH attention approximation) to capture long-range interactions. In order to take into account the motion effect, a shifted self-attention is introduced, modifying the classical self-attention layer to use keys computed from the previous time step. Similar to language modelling, a special token ([CLS]) is used to extract frame-level representations that are fed into a final transformer block to compute the final video representation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pros:*\n- Transformers have been already successfully applied in image processing. However, fewer methods managed to extend them to the video domain.\n- The proposed method is more efficient (in terms of FLOPS and params) than other transformers used for video processing, as shown in experiments.\n- The experiments show improvement over existing methods, proving that the architecture is effective.\n\n*Cons:*\n - Even if efficient transformers for spatio-temporal data represents a useful direction for the computer-vision community, my main concern regards the novelty brought by this work and the relevance for the NeurIPS community. The model shows how to combine/modify different existing models but fails to clearly validate the usefulness of each individual one for the final performance. The paper contains a set of ablations but, in my opinion, the central contributions are still not empirically validated as I will explained in the following:\n\n- The frame-level processing consisting of an “Image Local Chunk Attention” and an “Image LSH Attention” is only compared against a ViT model. Some questions regarding this approach are still open: Is it necessary to have a local processing (the chunk-level one) besides the global one? Is this local processing better than a convolutional block, since both approaches aim to process local information? Is the global processing (LSH Attention) necessary? I would suggest to include more ablation studies by a)dropping the Image Local Chunk Attention; b) replacing it with a convolutional network; c) dropping the LSH Attention\n\n- The “Image Local Chunk Attention” was designed to process the information locally. The same motivation and a very similar solution is proposed in [34]. A discussion regarding the similarities/differences but more importantly the possible advantages would be appreciated.\n\n- The idea of shifting the temporal dimension to capture the objects’ motion efficiently is interesting (and as far as I know novel in transformer literature) and seems to be effective. However the paper lacks an intuitive motivation for this. What does it mean to aggregate global information based on the similarities computed w.r.t previous time steps?\n\n- To fully validate the usefulness of the proposed model in video datasets, results on more dynamic datasets, where the temporal dimension plays a crucial role such as Smt-Smt V2 would be appreciated. The authors  provide some initial results on the Moments in Time datasets, but the comparison doesn’t seem to be beneficial. Moreover, important comparisons are missing from Table 3 in supplementary (e.g. VATT method that obtain better results)\n\n\nOverall I believe that, with proper validations in terms of additional ablation studies, this work could be relevant for the  CV community. However, I don’t think that the novelty brought by this work is sufficient for the acceptance at NeurIPS.\n\n######## POST-REBUTTAL #########\n\nI appreciate the additional experiments included in the rebuttal. In my opinion, they validate each component and gives me more confidence in the proposed method. I am still not fully convinced by the explanation for how the shifted module models temporal information, but the quantitative results are quite impressive, especially on heavy temporal datasets (such as SSv2 and Diving-48 included in the rebuttal).\n\nI will increase my score to 6: Marginally above the acceptance threshold, but I strongly encourage the authors to include all the ablation studies in the manuscript, as neither those experiments nor similar ones exist in the current version of the paper, and as other reviewers suggested, it is very important to understand the benefits of using each component of the model.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not addressed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "qQjWw2aEY99", "reviewer": "Reviewer_EmhS", "summary": "This paper proposes a new variant of Transformer, called Shifted Chunk Transformer (SCT), for video understanding. Compared with ViT, SCT has three major contributions: 1) SCT uses a smaller patch size ($4\\times 4$ instead of $16\\times 16$) to extract more fine-grained features; 2) A combination of local chunk attention (i.e., self-attention within a local window) and global LSH attention is used to efficiently process the large number of tokens at each frame; 3) A frame-wise attention between adjacent frames and a clip encoder beyond the frame-level features are used for modeling temporal information. Experiments on multiple video benchmark datasets show the effectiveness and efficiency of the proposed SCT model.\n\n\nPost rebuttal:\nI've read the author's response and reviews from other reviewers. I think most of our questions are well addressed in the rebuttal, and I will keep my original rating \"6: Marginally above the acceptance threshold\".", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a new Transformer model that achieves the state-of-the-art results with less number of parameters and FLOPs. One of my main concerns is its novelty because most components of the model, if viewed individually, are not novel and have been explored in prior work. However, the overall technical contribution of the work is still valuable, especially the idea of using small patches for fine-grained features and the methods for modeling temporal information.\n\nOverall, the paper is well organized and clear, but some technical details are missing:\n1) In L257, what is the majority voting used for obtaining final result? Is it different from the standard averaging of multiple views?\n\n2) In Table 4, how are the chunk size and #Tokens defined? Is the chunk size 49 comes from $7\\times 7$ patches within a chunk? How about the #Tokens then?\n\n3) How is the visualization generated in Figure 3? Where is the query position? More details should be described.\n\nI also suggest the authors to provide more ablations and analysis:\n1) One important claim of the paper is that using smaller patch size can improve the performance, and the claim is verified by ablations on ViT-B. However, the change of patch size may have different impacts for different models. I'm curious about the results of using different patch size for SCT as well, to further confirm the necessarily of using really small patch size.\n\n2) SCT use both local and global attention at each frame. An ablation on the impact of these two types of attention is suggested to better understand the model design.\n\n3) Why using more shift MSA block or more shifted frames will lead to worse results (Table5)? The authors should provide more discussion about the negative effect of the shifted MSA block.\n\n4) Why using more views leads to worse results (Figure 2)? This is a bit counter-intuitive as more views provide more complete information and less noise, especially for trimmed datasets like Kinetics. More analysis on this observation is required.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Q_sY62SeJ_T", "reviewer": "Reviewer_zDkv", "summary": "This paper proposes a new Transformer architecture for video modeling. There are following several components with their respective design motivation:\n- First, each frame is split into a large number of tiny patches (e.g. 4x4 pixels) which are then linearly projected into token vectors. Then, the authors propose Image Local Chunk Attention to enforce locality in cross attention to ensure attention is only computed among neighboring patches in local neighborhoods. The authors argue that it's necessary to have small patches for good accuracy. And the Image Local Chunk Attention is therefore there to reduce computation costs brought by small patches and also enforce locality. \n- With the features computed from Image Local Chunk Attention, the authors then apply a locality sensitive hashing (LSH) attention to approximate the true cross attention to increase efficiency. \n- Now tokens are mixed with attention for each frame, the authors then propose Shifted Multi-Head Self-Attention that computes cross-attention across frames to model object motion and spatial variances. \n- Finally, a special CLS token is extracted from each frame and passed into another clip encoder to produce a clip-level summarization, which is used for final classification tasks.\nOverall, the authors proposed a set of methods to model both intra and inter frame relationships in videos using transformers and achieved SOTA results on several video classification tasks like Kinetics, UCF101, HMDB51 and Moment-in-Time. The authors also conducted thorough ablation to dissect the contribution of each components.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Given its impressive performance on images, it's now an impactful problem to study how to best utilize transformers for video modeling. \n+ The paper has addressed several painpoints of video spatio-temporal modeling using transformers. First, it has proposed specific modules for intra (Image Local Chunk Attention) and inter (Shifted Multi-Head Self-Attention) frame modeling. Also, it has addressed the computation issue of transformers with the local chunk attention and locality sensitive hashing. I feel the authors have done a good job integrating these different techniques into a highly performant system with decent computation footprint. \n+ Due to its high accuracy and relative low FLOPs, the proposed transformer architecture could be potentially very useful for the community, upon which more explorations can be carried out on video modeling. \n+ The paper is well-written and with good illustration. \n+ The authors have conducted thorough ablation to dissect their approach. \n\n- As the authors have put a lot efforts to reduce computation of transformers. It would be nice to have the actual runtime (in seconds) for the proposed model compared to previous methods in Table 6. \n- The authors designed Image Local Chunk Attention to induce locality in attention and reduce computation. It resembles the utility of convolutions. How does this compare to directly employing convolutions? \n- What's the difference between \"Frame Rate\" and \"Frame Stride\" in Table 2? \n- L286-7, \"including the number of shifted MSA layers used in each image chunk self-attention\" I'm confused about what this line means. \n\n######### POST REBUTTAL #########\n\nI thank the authors for their feedback. The response has clarified most of my questions, thus I will keep my rating to accept the paper. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern spotted. \n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "EgxuUu4eJgk", "reviewer": "Reviewer_DdYC", "summary": "The paper proposes a new transformer-based architecture for video classification. The proposed architecture builds on the recently introduced vision transformer model (ViT), which first decomposes a given image/video frame into a set of non-overlapping patches. However, unlike in prior work,  instead of applying self-attention directly on patches, the authors group locally adjacent patches into local \"chunks\", and apply self-attention on top of these chunks. To ensure that the model can propagate local chunk information globally across the entire image/video frame, the authors introduce  shifted self-attention. Lastly, a separate clip encoder is used to incorporate temporal video information. The proposed approach achieves state-of-the-art on several action recognition benchmarks such as Kinetics-400, and Kinetics-600.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Video transformers are growing in popularity. Therefore, the proposed approach that addresses several limitations of prior video transformer based methods is timely and relevant.\n- The proposed approach achieves impressive results on Kinetics-400, and Kinetics-600.\n- The proposed method is also more efficient than prior transformer based techniques.\n\nWeaknesses:\n- Personally, I found the draft quite difficult to understand due to the writing related issues (i.e., incoherent or grammatically incorrect sentences), missing technical details, and the technical presentation style. Below I discuss some of these issues in more detail.\n- The paragraph on \"Image locality-sensitive hashing (LSH) attention\" is confusing and difficult to understand both due to numerous writing issues, and also due to the lack of details related to LSH. The authors talk about \"bucketing\" but it would be useful to include a brief technical overview so that the readers who are less familiar with LSH would still understand the approach without having to look up other papers.\n- The description of the shifted attention could be significantly improved. Could the authors make a figure illustrating this concept and include it in the draft? Unlike written in the draft, Figure 1 doesn't illustrate how shifted attention mechanism works in practice. To be honest, I'm still confused about it. From Equation 6, it seems that the key features are computed from time t-1 and that there is no temporal overlap between query and value features, both of which are computed at time t. Therefore, I don't see how the temporal information is propagated into the feature representation y' (as in Eq.3 ), since the key features are only used to compute similarity matrix, but not the new feature representation y' . Isn't that a limiting factor in the model design? In my view, a new feature representation y' that incorporates information from both time t-1, and also time t would be more powerful. What is the motivation of such model design? Am I missing something? Having said this, the results in Table 5 indicate that the proposed scheme leads to ~1.5% improvement, which is quite substantial.\n- The authors make claims about their approach learning inter-frame motion of objects. However, Kinetics, HMDB, and UCF are known to be heavily biased towards spatial scene dynamics. To verify these claims, one would need to conduct experiments on temporally heavy datasets such as Something-Something-V2 and Diving-48. I would urge the authors to conduct these experiments for the rebuttal.\n- Many relevant experimental details are missing or they are inconsistent with the previously published work. For example, in Table 1, the authors list that ViT-B achieves 75.3 % accuracy on Kinetics-400. How's this baseline implemented? For example, in the TimeSformer paper [4] , the accuracy of a 8-frame spatial-only model is listed as 76.9 % on Kinetics-400. Therefore, I was wondering what's the reason behind this gap. Furthermore, in Table 4, the authors list ViT-B to have 196 tokens, and ~405 GFLOPs. How are these numbers computed? The 8-frame spatial-only ViT-B has the cost of ~141 GFLOPs. How many frames are all of these models using? These details should be included both in the paper, and in the tables.\n- The term \"image local chunk self-attention\", which is used many times throughout the paper is cumbersome. I would urge the authors to rename their model to something more elegant. \n- The paper often feels like code/implementation documentation. The authors tend to focus on very low-level implementation details too much (e.g., Sections 3.1-3.4), making the paper more difficult to read. \n\nQuestions / Comments:\n- For pretraining, do the authors pretrain their model on ImageNet-21k themselves, or do they use publicly available ImageNet-21k pretrained models? If it's the former, it would be good to include pretraining related details. If it's the latter, for reproducibility, it would be good to know, which public models the authors are using, as the models displayed in Table 3 have a custom design (compared to standard ViT).\n- Could the authors also include the total number of TFLOPs in their tables so that it would be easier to compare computational complexity of each method?\n- In Table 5, the performance of using two layers of shifted MSA drops quite substantially, which is a bit counterintuitive. Do the authors have an explanation for this?\n- In Table 4, what exactly does the chunk size (i.e., 49) mean? Does it mean that it encompasses 49 patches on each axis? It seems that this should be 7 instead of 49.\n- All the tables list the # of tokens for a single frame. However, I assume the model operates on multiple frames. Therefore, it would be useful to include the number of used frames in the tables.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the proposed approach are not discussed, i.e., Section 5 only contains limitations for previous work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance of this submission. The final ratings are: 6, 6, 6, 8.

All four reviewers acknowledge the strong empirical results, both in terms of accuracy as well as efficiency. The inclusion of results on motion-heavy datasets in the rebuttal period was deemed valuable and informative. It is recommended to add these results to the main paper. 

Reviewers comment on the poor presentation, especially the section concerning the shifted attention. The authors should use the detailed feedback given by the reviewers in order to improve the technical discussion as well as the motivation for the approach. 

The ACS agree on the recommendation of acceptance.
Decision: Accept (Poster)