Shifted Chunk Transformer for Spatio-Temporal Representational Learning
Year: 2021

Abstract:
Spatio-temporal representational learning has been widely adopted in various fields such as action recognition, video object segmentation, and action anticipation. Previous spatio-temporal representational learning approaches primarily employ ConvNets or sequential models, e.g., LSTM, to learn the intra-frame and interframe features. Recently, Transformer models have successfully dominated the study of natural language processing (NLP), image classification, etc. However, the pure-Transformer based spatio-temporal learning can be prohibitively costly on memory and computation to extract fine-grained features from a tiny patch. To tackle the training difficulty and enhance the spatio-temporal learning, we construct a shifted chunk Transformer with pure self-attention blocks. Leveraging the recent efficient Transformer design in NLP, this shifted chunk Transformer can learn hierarchical spatio-temporal features from a local tiny patch to a global video clip. Our shifted self-attention can also effectively model complicated inter-frame variances. Furthermore, we build a clip encoder based on Transformer to model long-term temporal dependencies. We conduct thorough ablation studies to validate each component and hyper-parameters in our shifted chunk Transformer, and it outperforms previous state-of-the-art approaches on Kinetics-400, Kinetics-600, UCF101, and HMDB51.

heading: 1 Introduction
text: Spatio-temporal representational learning tries to model complicated intra-frame and inter-frame relationships, and it is critical to various tasks such as action recognition [20], action detection [55, 52], object tracking [23], and action anticipation [21]. Deep learning based spatio-temporal representation learning approaches have been largely explored since the success of AlexNet on image classification [24, 11]. Previous deep spatio-temporal learning can be mainly divided into two aspects: deep ConvNets based methods [35, 15, 16] and deep sequential learning based methods [55, 28, 29]. Deep ConvNets based methods are primarily integrated various factorization techniques [51, 34], or a priori [16] for efficient spatio-temporal learning [15]. Some works focus on extracting effective spatio-temporal features [41, 8] or capturing complicated long-range dependencies [49]. Deep sequential learning based methods try to formulate the spatial and temporal relationships through advanced deep sequential models [28] or the attention mechanism [29].
On the other hand, the Transformer has become the de-facto standard for sequential learning tasks such as speech and language processing [44, 12, 54, 19]. The great success of Transformer on natural language processing (NLP) has inspired computer vision community to explore self-attention
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
structures for several vision tasks, e.g., image classification [13, 40, 36], object detection [6], and super-resolution [33]. The main difficulty in pure-Transformer models for vision is that Transformers lack the inductive biases of convolutions, such as translation equivariance, and they require more data [13] or stronger regularisation [40] in the training. It is only very recently that, vision Transform (ViT), a pure Transformer architecture, has outperformed its convolutional counterparts in image classification when pre-trained on large amounts of data [13]. However, the hurdle is aggravated when the pure-Transformer design is applied to spatio-temporal representational learning.
Recently, a few attempts have been made to design pure-Transformer structures for spatio-temporal representation learning [4, 5, 14, 2]. Simply applying Transformer to 3D video domain is computationally intensive [4]. The Transformer based spatio-temporal learning methods primarily focus on designing efficient variants by factorization along spatial- and temporal-dimensions [4, 5], or employing a multi-scale pyramid structure for a trade-off between the resolution and channel capacity while reducing the memory and computational cost [14]. The spatio-temporal learning capacity can be further improved by extracting more effective fine-grained features through advanced and efficient intra-frame and inter-frame representational learning.
In this work, we propose a novel spatio-temporal learning framework based on pure-Transformer, called shifted chunk Transformer as illustrated in Fig. 1, which extracts effective fine-grained intraframe features with a low computational complexity leveraging the recent advance of Transformer in NLP [22]. Specially, we divide each frame into several local windows called image chunks, and construct a hierarchical image chunk Transformer, which employs locality-sensitive hashing (LSH) to enhance the dot-product attention in each chunk and reduces the memory and computation consumption significantly. To fully consider the motion effect of object, we design a robust selfattention module, shifted self-attention, which explicitly extracts correlations from nearby frames. We further design a pure-Transformer based frame-wise attention module, clip encoder, to model the complicated inter-frame relationships with a minimal extra computational cost. Our contributions can be summarized as follows:
• We construct an image chunk self-attention to mine fine-grained intra-frame features leveraging the recent advance of Transformer. The hierarchical image chunk Transformer employs locality-sensitive hashing (LSH) [3] to reduce the memory and computation consumption significantly, which enables an effective spatio-temporal learning directly from a tiny patch.
• We build a shifted self-attention to fully consider the motion effect of objects, which yields effective modeling of complicated inter-frame variances in the spatio-temporal representa-
tional learning. Furthermore, a clip encoder with a pure-Transformer structure is employed for frame-wise attention, which models complicated and long-term inter-frame relationships at a minimal extra cost.
• The shifted chunk Transformer with pure-Transformer outperforms previous state-of-theart approaches on several action recognition benchmarks, including Kinectics-400 [20], Kinetics-600 [7], UCF101 [39] and HMDB51 [25].

heading: 2 Related Work
text: Conventional deep learning based action recognition Conventional deep spatio-temporal representational learning mainly involves two aspects: deep sequential learning based methods [55, 28, 29] and deep ConvNet based methods [35, 15, 16]. The recurrent networks can be extended to 3D spatio-temporal domain for action recognition [28]. In deep ConvNet based methods, two-stream ConvNet employs two branches of 2D ConvNets and explicitly models motion by optical flow [38]. The C3D [41] and I3D [8] directly extend 2D ConvNets to 3D ConvNets, which is natural for 3D spatio-temporal representational learning [9]. However, the 3D ConvNet requires significantly more computation and more training data to achieve a desired accuracy [51]. Thus, P3D [34] and S3D [51] attempt to factorize the 3D convolution into a 2D spatial convolution and a 1D temporal convolution. SlowFast network [16] and X3D [15] conduct trade-offs among resolution, temporal frame rate and the number of channels for the efficient video recognition. Non-local network [49] proposes to add non-local operations in deep network and captures long-range dependencies. The recent pure-Transformer based spatio-temporal learning enables longer dependency relationship modeling and further increases the accuracy of action recognition [4].
Vision Transformers NLP community has witnessed the great success of pre-training by Transformer [44, 12], and it has been emerging for image classification [13, 40, 36], object detection [6], and image super-resolution [33]. CPVT [10] employs pre-defined and independent input tokens to increase the generalization for image classification. Pure-Transformer network has no inductive bias or prior as ConvNets. ViT [13] pre-trains on large amounts of data and attains excellent results on image classification. CvT [50] introduces convolutions into ViT to yield better performance and efficiency. DeiT [40] and MViT [14] instead employ distillation and multi-scale to cope with the training difficulty. PVT [47] and segmentation Transformer [53] further extend Transformer to dense prediction tasks, e.g., object detection and semantic segmentation. Simply applying Transformer to 3D video spatio-temporal representational learning aggravates the training difficulty significantly, and it requires advanced model design for pure-Transformer based spatio-temporal learning.
Transformer based action recognition Recently, only a few works have been conducted using pure-Transformer for spatio-temporal learning [4, 14, 5, 2]. Most of the efforts focus on designing efficient Transformer models to reduce the computation and memory consumption. ViViT [4] and TimeSformer [5] study various factorization methods along spatial- and temporal-dimensions. MViT [14] conducts a trade-off between resolution and the number of channels, and constructs a multi-scale Transformer to learn a hierarchy from simple dense resolution and fine-grained features to complex coarse features. VATT [2] conducts unsupervised multi-modality self-supervised learning with a pure-Transformer structure. In this work, we extract fine-grained intra-frame features from each tiny patch and model complicated inter-frame relationship through efficient and advanced self-attention blocks.

heading: 3 Method
text: In this section, we describe each component of our shifted chunk Transformer for spatio-temporal representation learning in video based action recognition.

heading: 3.1 Overview
text: Let X = [X1,X2, · · · ,XT ] ∈ RT×H×W×3 be one input clip of T RGB frames sampled from a video, where Xi ∈ RH×W×3 is the i-th frame in the clip, and H and W are the frame size. To design an efficient pure-Transformer based spatio-temporal learning, we construct a shifted chunk Transformer, including image chunk self-attention blocks, shifted multi-head self-attention blocks,
and a clip encoder, as illustrated in Fig. 1. We first construct an image chunk self-attention block leveraging the advanced efficient Transformer design in NLP [22], which is illustrated in the left of Fig. 1. The locality-sensitive hashing (LSH) [3] in the image chunk self-attention enables a relatively small patch as a token, thus it is capable of extracting fine-grained intra-frame features. A linear pooling layer is designed to adaptively reduce the resolution after LSH attention. After that, a shifted self-attention is designed to extract motion related inter-frame features. Our shifted self-attention considers the motion of objects across nearby frames and explicitly models the temporal relationship into self-attention. The frame encoder shown in dark grey color can be an effective feature extractor which can be stacked for several times. The hierarchical frame encoder and image chunk self-attention further learns an effective multi-level feature from local to global abstraction. Lastly, we employ a pure-Transformer to learn complicated inter-frame relationships and frame-wise attention along the temporal dimension. We use multi-head self-attention (MSA) in all the blocks.
3.2 Image Chunk Self-Attention
Transformer can learn complicated long range dependencies which can be computed through the high efficient matrix product [44]. Different from convolution which has inherent inductive bias [13], Transformer learns the entire features from data. The main challenge for a pureTransformer based vision model mainly involves two aspects: 1) how to design an efficient model to learn effective features from the entire image,
because simply treating each pixel as a token is computationally intensive, 2) how to train this powerful model and learn various effective features from data. ViT [13] treats each patch of size 16× 16 as one token and pre-trains the model with large amounts of data. We argue that Transformer with a smaller patch as a token can extract fine-grained features which improves spatio-temporal learning for action recognition. For ViT-B-16 [13] of crop size 224× 224 as a frame encoder followed by a shifted MSA and a clip encoder, a smaller patch size of 8× 8 yields better accuracy on Kinectics-400 and UCF101 as shown in Table 1.
The Transformer computes each pairwise correlation through a dot product, thus it has a high computational complexity of O(L2), where L is the totally number of tokens. In natural language processing (NLP), LSH attention [22] employs locality-sensitive hashing (LSH) bucketing [3] and chunk sorting for queries and keys to approximate the attention matrix computation. Leveraging the efficient LSH approximation, the LSH attention reduce the computation complexity to O(L logL).
To preserve locality property and learn transition and rotation invariant low-level features from images, we firstly design a visual local transformer (ViLT) with shared parameters along different image local windows, or image chunks. Each image chunk consists of multiple tiny patches as illustrated in the left bottom block of Fig. 1. We intend to employ patches of a small size in the ViLT which is the first level abstraction of the input, so that the model extracts a fine-grained representation which enhances the entire spatial-temporal learning. Employing the small patch yields large number of tokens in the following self-attention, and we construct an image locality-sensitive hashing (LSH) attention leveraging advanced design of Transformer in NLP [22]. The image LSH attention can efficiently extract higher-level and plentiful intra-frame features ranging from a tiny patch to the entire image. The framework of image chunk self-attention is illustrated in the left part of Fig. 1.
Visual local transformer (ViLT) In the shifted chunk Transformer, we firstly construct a ViLT which slides one self-attention for each tiny patch along the whole image. The ViLT is illustrated in the bottom block of Fig. 1. Let h, w be the height and width of the tiny image patch p ∈ Rh×w×3. Following the success of ViT [13], we treat each patch as one dimensional vector of length h × w × 3. Suppose each chunk consists of m × n tiny patches, denoted as {p1,1,p1,2, · · · ,p1,n;p2,1, · · · ,p2,n; · · · ;pm,1, · · · ,pm,n}. After flattening the chunk into a list of tiny patches, we denote the chunk as {p1,p2, · · · ,pL} without loss of generality, where L = m× n. In ViLT, we use a learnable 1D position embedding Epos ∈ RL×D to retain position information
z0 = [p1E;p2E; · · · ;pm×nE] +Epos, E ∈ R(h×w×3)×D, (1)
where E is the linear patch embedding matrix, and D is the embedding dimension. Then we can construct alternating layers of multi-head self-attention (MSA), MLP with GELU [18] non-linearity, Layernorm (LN) and residual connections [44] for the chunk as
z′m = MSA(LN(zm−1)) + zm−1, zm = MLP(LN(z ′ m)) + z ′ m, m = 1, · · · ,M, (2)
where M is the number of blocks. We conduct the ViLT sliding the entire image without overlapping. The parameters of the ViLT are shared among all the image chunks, which forces the chunk self-attention to learn translation and rotation invariant, and fine-grained features. The tiny patch-wise feature extraction preserves the locality property, which is a strong prior for natural images. After the ViLT, we obtain the extracted features for the entire image denoted as y = [y1;y2; · · · ;yL; · · · ;yL×L′ ], where the entire image can be split into L′ = m′ × n′ = d Hh×me × d W w×ne chunks, and we conduct zero padding for the last chunks in each row and column.
The ViLT forces to learn image locality features which is a desired property for a low-level feature extractor [26]. For pure-Transformer based vision system, it also reduces the memory consumption significantly because it restricts the correlation of one tiny patch within the local chunk. Therefore, the memory and computational complexity of dot-product attention in ViLT can be reduced to O(L2) compared to the complexity of conventional self-attention O((L′ × L)2). Image locality-sensitive hashing (LSH) attention After ViLT blocks, we obtain local fine-grained features of length L′ ×L. Since the patch size is tiny, the total number of patches can be large, which leads to more difficult training than other vision Transformers [4, 13]. On the other hand, the problem of finding nearest neighbors quickly in high-dimensional spaces can be solved by locality-sensitive hashing (LSH), which hashes similar input items into the same “buckets” with high probability. In NLP, LSH attention [22] is proposed to handle quite long sequence data, which employs localitysensitive hashing (LSH) bucketing approximation and bucket sorting to reduce the computational complexity of matrix product between query and key in self-attention.
In dot-product attention, the softmax activation function pushes the attention weights close to 1 or 0, which means the attention matrix is typically sparse. The query and key can be approximated by locality-sensitive hashing (LSH) [3] to reduce the computational complexity. Furthermore, through bucketing sort, the attention matrix product can be accelerated by a chunk triangular matrix product, which has been validated by LSH attention [22].
The used multi-head image LSH attention can be constructed as y′ = MSA(LSHAtt(LN(y))) + y, s = MLP(LN(y′)) + y′, (3)
where LSH attention LSHAtt(·) employs angular distance to conduct LSH hashing [3]. The image LSH attention reduces the memory and time complexity to O(L′ × L log(L′ × L)), compared with that of conventional dot-product attention O((L′ × L)2). The image LSH attention reduces the complexity significantly because the patch size is tiny and the number of tiny patches L′ × L are large. The image level LSH attention in the second level learns relatively global features from the first level’s local fine-grained features.
The hierarchical feature learning from local to global has been validated as an effective principle for vision system design [26, 45]. Inspired by hierarchical abstraction in ConvNets [26], we construct a linear pooling layer which firstly conducts squeeze then employs linear projection for feature dimension reduction. The linear pooling adaptively squeezes the sequence length by 14 .
Reshape: s = [s1,1, · · · , s1,n×n′ ; · · · ; sm×m′,1, · · · , sm×m′,n×n′ ], Squeeze: s′ = [s′1,1, · · · , s′1,n×n′/2; · · · ; s ′ m×m′/2,1, · · · , s ′ m×m′/2,n×n′/2], Linear: t = [s′1,1Et, · · · , s′1,n×n′/2Et; · · · ; s ′ m×m′/2,1Et, · · · , s ′ m×m′/2,n×n′/2Et],
(4)
where s′i,j in s ′ concatenates s2i−1,2j−1, s2i−1,2j , s2i,2j−1 and s2i,2j from s, Et is the linear projection matrix to adaptively reduce the number of dimensions after squeeze by half. The reshape is to retain the spatial relationship of each patch, and the squeeze reduces the number of patches by 14 and enlarges the feature dimensions by four times. The linear pooling layer forces the model to learn high-level global features in the following layers.

heading: 3.3 Shifted Multi-Head Self-Attention
text: Considering the motion effect of objects, we explicitly construct a shifted multi-head self-attention (MSA) for spatio-temporal learning after image chunk self-attention as illustrated in Fig. 1 and Fig. 2.
For video classification, a special classification token ([CLS]) [12] can be prepended into the feature sequence. To learn frame-wise spatio-temporal representations, we prepend the [CLS] token to each frame. Without loss of generality, we denote the image chunk self-attention feature to be a list of [ti,1; ti,2; · · · ; ti,L×L′/4] for the i-th frame. Then, we obtain the input of T frames for shifted MSA as
t′ = [t′1,1; · · · ; t′1,1+L×L′/4; · · · ; t ′ T,1; · · · ; t′T,1+L×L′/4]
= [z1,cls; t1,1; · · · ; t1,L×L′/4; · · · ; zT,cls; tT,1; · · · ; tT,L×L′/4]. (5)
The shifted multi-head self-attention explicitly considers the inter-frame motion of objects, which computes the correlation between the current frame and the previous frame in the attention matrix, which can be formulated as
qit,p = LN(t ′ t,p)W i Q, k i t,p = LN(t ′ t−1,p)W i K , v i t,p = LN(t ′ t,p)W i V , (6)
where WiQ, W i K and W i V are projection matrices for head i, and we employ a cyclic way to calculate the key of the first frame t = 1 by defining t′0,p = t ′ T,p. By concatenating q i t,p, k i t,p v i t,p into matrices Qit, K i t, V i t along patch location p, the shifted MSA for frame t can be calculated as
at = Concat(Attention(Q1t ,K 1 t ,V 1 t ), · · · ,Attention(Qlt,Klt,Vlt))WO, a′t = MLP(LN(at)) + at, (7)
where l is the number of heads in multi-head self-attention. The shifted self-attention compensates object motion and spatial variances. We explicitly integrate motion shift into selfattention, which extracts robust features for spatio-temporal learning. The block with alternating layers of image chunk self-attention and shifted MSA can be stacked for multiple times to fully extract effective hierarchical finegrained features from tiny local patches to the whole clip in our shifted chunk Transformer.

heading: 3.4 Clip Encoder for Global Clip Attention
text: To learn complicated inter-frame relationship from the extracted frame-level features, we design a clip encoder based on a pure-Transformer structure to adaptively learn frame-wise attention. To facilitate the video classification, we prepend a global special classification token ([CLS]) into the frame-level feature sequence.
In this module, we employ the classification feature a′t1 corresponding to zt,cls as the frame-level feature for frame t. To consider frame position, we also employ a standard learnable 1D position embedding as each frame position embedding. The clip encoder can be formulated as
b0 = [bcls;a ′ 11E ′; · · · ;a′T 1E ′] +E′pos, E ′ pos ∈ R(T+1)×D ′ ,
b′m = MSA(LN(bm−1)) + bm−1, bm = MLP(LN(b ′ m)) + b ′ m, m = 1, · · · ,M ′,
c = MLP(LN(bM ′1))
(8)
where E′ is the linear frame embedding matrix, D′ is the clip encoder embedding size, M ′ is the number of blocks, and bM ′1 is the clip-level classification feature for the classification token bcls, c is the video classification logit for softmax. We use dropout [24] for the second last layer and cross-entropy loss with label smoothing [32] for training. The clip encoder can be efficient with a minimal computational cost in Appendix to achieve a powerful inter-frame representation learning.

heading: 4 Experiment
text: We evaluate our shifted chunk Transformer, denoted as SCT, on five commonly used action recognition datasets: Kinetics-400 [20], Kinetics-600 [7], Moment-in-Time [31] (Appendix), UCF101 [39] and
HMDB51 [25]. We adopt ImageNet-21K for the pre-training [37, 11] because of large model capacity of SCT. The default patch size for each image token is 4× 4. In the training, we use a synchronous stochastic gradient descent with momentum of 0.9, a cosine annealing schedule [30], and the number of epochs of 50. We use batch size of 32, 16 and 8 for SCT-S, SCT-M and SCT-L, respectively. And the frame crop size is set to be 224× 224. For data augmentation, we randomly select the start frame to generate the input clip. In the inference, we extract multiple views from each video and obtain the final prediction by averaging the softmax probabilistic scores from these multi-view predictions. The details of initial learning rate, optimization and data processing are shown in Table 2. All the experiments are run on 8 NVIDIA Tesla V100 32 GB GPU cards.
We construct three shifted chunk Transformers, SCT-S, SCT-M, and SCT-L, in terms of various model sizes and computation complexities. We employ four consecutive blocks with alternating one image chunk self-attention and one shifted MSA. The patch embedding size D, MLP dimension of these self-attentions, the number of ViLT layers M , the clip encoder embedding size D′, MLP dimension of clip encoder, the numbers of heads in ViLT, LSH attention, shifted MSA and clip encoder are shown in Table 3. Each image chunk self-attention consists of M layers ViLT followed by an image LSH attention and a linear pooling layer to reduce the number of spatial dimensions. We use four-layer clip encoders to obtain the video classification results as validated in the Appendix.
Validating frame feature extractor We compare the frame encoder of our shifted chunk Transformer (SCT) with ViT [13] in Table 4. For a fair comparison, we only replace the ViLT with ViT, and remain all other components the same. From Table 4, we observe that 1) large models, ViT-L-16 and SCT-L, yield higher accuracy than base models, ViT-B-16 and SCT-S; 2) ViT with a small patch size achieves better accuracy than ViT with a large patch; 3) SCT-L improves the accuracy of ViT-L by 4% while using much less number of parameters and FLOPs. The tiny patch and enforced locality prior in ViLT are validated to be effective for spatio-temporal learning.
Effect of shifted MSA We conduct experiments on Kinectis-400 and UCF101 to validate the hyperparameters of the shifted MSA layer, including the number of shifted MSA layers, and the number of shifted frames used in the calculation of key in equation (6). All other network configurations follow the Table 3. The #Shifted MSA of 0 and #Shifted frame of 0 in Table 5 mean that one standard MSA is used instead of shifted MSA. From Table 5, we observe that 1) a shifted MSA improves the accuracy up to 1.5% compared with the conventional MSA; 2) one layer shifted MSA with the shifted number of frames of one yields the best accuracy. The shifted MSA explicitly formulates the motion effect of object by considering the nearby frames, which improves the accuracy for video classification. We use one layer shifted MSA with the number of shifted frames of one in our experiment.
Varying the number of input frames and temporal views In our experiments so far, we have kept the number of input frames fixed to 24 across different datasets. To discuss the effect of the number of input frames on video level inference accuracy, we validate the number of input frames of 24, 48, 96, and the number of temporal views from 1 to 8. Fig. 3 shows that as we increase the number of frames, the accuracy using a single clip increases, since the network is incorporated longer temporal information. However, as the number of used views increases, the accuracy difference is reduced. We use the number of frames of 24 and the number of temporal views of 4 in our experiment.
Patch and frame attention Our shifted chunk Transformer (SCT) can detect fine-grained discriminative regions for each frame in the entire clip in Fig. 4. Specifically, we average attention weights of the shifted MSA across all heads and then recursively multiply the weight matrices of all layers [1], which accounts for the attentions through all layers. The designed framework of SCT leads to an easy diagnosis and explanation for
the prediction, which potentially makes SCT applicable to various critical fields, e.g., healthcare and autonomous driving.
Comparison to state-of-the-art approaches We compare our shifted chunk Transformer (SCT) to the current state-of-the-art approaches based on the best hyper-parameters validated in the previous ablation studies. We obtain the results of previous state-of-the-art approaches from their papers. We obtain the actual runtime (s) in one single NVIDIA V100 16GB GPU by averaging 50 inferences with batch size of one. In Kinetics-400 and Kinetics-600, we initialize our ViLT and LSH attention trained on ImageNet-21K.
Our shifted chunk Transformers (SCT) surpass previous state-of-the-art approaches including both recent Transformer based video classification and previous deep ConvNets based methods by 2.7%, 1.3%, 1.7% and 8.9% on Kinectis-400, Kinectic-600, UCF101 and HMDB51 in Table 6-7 based on RGB frames, respectively. The local scheme and LSH approximation in image chunk self-attention enables to use patches of a small size. Because of the efficient model design, SCT achieves the best accuracy even only using pretraining on the ImageNet on UCF101 and HMDB51. Besides the higher action recognition accuracy, the SCT employs less number of parameters and FLOPs than ViViT because we employ less number of channels and our SCT is effective for spatio-temporal learning.

heading: 5 Conclusion
text: In this work, we proposed a new spatio-temporal learning called shifted chunk Transformer inspired by the recent success of vision Transformer in image classification. However, the current pureTransformer based spatio-temporal learning is limited by computational efficiency and feature
robustness. To address these challenges, we propose several efficient and powerful components for spatio-temporal Transformer, which is able to learn fine-grained features from a tiny image patch and model complicated spatio-temporal dependencies. We construct an image chunk self-attention which leverages locality-sensitive hashing to efficiently capture fine-grained local representation with a relatively low computation cost. Our shifted self-attention can effectively model complicated interframe variances. Furthermore, we build a clip encoder based on pure-Transformer for frame-wise attention and long-term inter-frame dependency modeling. We conduct thorough ablation studies to validate each component and hyper-parameters in our shifted chunk Transformer. It outperforms previous state-of-the-art approaches including both pure-Transformer architectures and deep 3D convolutional networks on various datasets in terms of accuracy and efficiency.

heading: 6 Acknowledgement
text: This work is supported by Kuaishou Technology. No external funding was received for this work. Moreover, we would like to thank Hang Shang for insightful discussions.

Review summaries:
The paper proposes a novel transformer-based architecture for video processing. The input tokens are small patches extracted for each frame. The model consists of 4 modules: First, the tokens are processed locally, using a transformer with restricted, local connectivity (each image is split into multiple chunks and only the patches from the same chunk exchange messages using the self-attention layer). Second, a global processing is conducted using an efficient transformer (that use LSH attention approximation) to capture long-range interactions. In order to take into account the motion effect, a shifted self-attention is introduced, modifying the classical self-attention layer to use keys computed from the previous time step. Similar to language modelling, a special token ([CLS]) is used to extract frame-level representations that are fed into a final transformer block to compute the final video representation.

This paper proposes a new variant of Transformer, called Shifted Chunk Transformer (SCT), for video understanding. Compared with ViT, SCT has three major contributions: 1) SCT uses a smaller patch size ($4\times 4$ instead of $16\times 16$) to extract more fine-grained features; 2) A combination of local chunk attention (i.e., self-attention within a local window) and global LSH attention is used to efficiently process the large number of tokens at each frame; 3) A frame-wise attention between adjacent frames and a clip encoder beyond the frame-level features are used for modeling temporal information. Experiments on multiple video benchmark datasets show the effectiveness and efficiency of the proposed SCT model.   Post rebuttal: I've read the author's response and reviews from other reviewers. I think most of our questions are well addressed in the rebuttal, and I will keep my original rating "6: Marginally above the acceptance threshold".

This paper proposes a new Transformer architecture for video modeling. There are following several components with their respective design motivation: - First, each frame is split into a large number of tiny patches (e.g. 4x4 pixels) which are then linearly projected into token vectors. Then, the authors propose Image Local Chunk Attention to enforce locality in cross attention to ensure attention is only computed among neighboring patches in local neighborhoods. The authors argue that it's necessary to have small patches for good accuracy. And the Image Local Chunk Attention is therefore there to reduce computation costs brought by small patches and also enforce locality.  - With the features computed from Image Local Chunk Attention, the authors then apply a locality sensitive hashing (LSH) attention to approximate the true cross attention to increase efficiency.  - Now tokens are mixed with attention for each frame, the authors then propose Shifted Multi-Head Self-Attention that computes cross-attention across frames to model object motion and spatial variances.  - Finally, a special CLS token is extracted from each frame and passed into another clip encoder to produce a clip-level summarization, which is used for final classification tasks. Overall, the authors proposed a set of methods to model both intra and inter frame relationships in videos using transformers and achieved SOTA results on several video classification tasks like Kinetics, UCF101, HMDB51 and Moment-in-Time. The authors also conducted thorough ablation to dissect the contribution of each components.  

The paper proposes a new transformer-based architecture for video classification. The proposed architecture builds on the recently introduced vision transformer model (ViT), which first decomposes a given image/video frame into a set of non-overlapping patches. However, unlike in prior work,  instead of applying self-attention directly on patches, the authors group locally adjacent patches into local "chunks", and apply self-attention on top of these chunks. To ensure that the model can propagate local chunk information globally across the entire image/video frame, the authors introduce  shifted self-attention. Lastly, a separate clip encoder is used to incorporate temporal video information. The proposed approach achieves state-of-the-art on several action recognition benchmarks such as Kinetics-400, and Kinetics-600.