Space-time Mixing Attention for Video Transformer
Year: 2022

Abstract:
This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer’s depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code for our method is made available here.

text: This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer’s depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code for our method is made available here.

heading: 1 Introduction
text: Video recognition – in analogy to image recognition – refers to the problem of recognizing events of interest in video sequences such as human activities. Following the tremendous success of Transformers in sequential data, specifically in Natural Language Processing (NLP) [39, 5], Vision Transformers were very recently shown to outperform CNNs for image recognition too [48, 13, 35], signaling a paradigm shift on how visual understanding models should be constructed. In light of this, in this paper, we propose a Video Transformer model as an appealing and promising solution for improving the accuracy of video recognition models.
A direct, natural extension of Vision Transformers to the spatio-temporal domain is to perform the self-attention jointly across all S spatial locations and T temporal locations. Full space-time attention though has complexityO(T 2S2) making such a model computationally heavy and, hence, impractical even when compared with the 3D-based convolutional models. As such, our aim is to exploit the temporal information present in video streams while minimizing the computational burden within the Transformer framework for efficient video recognition.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A baseline solution to this problem is to consider spatial-only attention followed by temporal averaging, which has complexity O(TS2). Similar attempts to reduce the cost of full space-time attention have been recently proposed in [3, 1]. These methods have demonstrated promising results in terms of video recognition accuracy, yet they have been also shown to induce, in most of the cases, significant computational overheads compared to the baseline (spatial-only) method due to the additional modelling of the temporal information.
Our main contribution in this paper is a Video Transformer model that has complexity O(TS2) and, hence, is as efficient as the baseline model, yet, as our results show, it outperforms recently/concurrently proposed work [3, 1] in terms of efficiency (i.e. accuracy/FLOP) by significant margins. To achieve this our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer’s depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. Fig. 1 shows the proposed approximation to space-time attention. We also show how to integrate two very lightweight mechanisms for global temporal-only attention, which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model is surprisingly effective in terms of capturing long-term dependencies and producing very high recognition accuracy on the most popular video recognition datasets, including Something-Something-v2 [17], Kinetics [4] and Epic Kitchens [9], while at the same time being significantly more efficient than other Video Transformer models.

heading: 2 Related work
text: Video recognition: Standard solutions are based on CNNs and can be broadly classified into two categories: 2D- and 3D-based approaches. 2D-based approaches process each frame independently to extract frame-based features which are then aggregated temporally with some sort of temporal modeling (e.g. temporal averaging) performed at the end of the network [42, 26, 27]. The works of [26, 27] use the “shift trick” [45] to have some temporal modeling at a layer level. 3D-based approaches [4, 16, 36] are considered the current state-of-the-art as they can typically learn stronger temporal models via 3D convolutions. However, they also incur higher computational and memory costs. To alleviate this, a large body of works attempt to improve their efficiency via spatial and/or temporal factorization [38, 37, 15].
CNN vs ViT: Historically, video recognition approaches tend to mimic the architectures used for image classification (e.g. from AlexNet [23] to [20] or from ResNet [18] and ResNeXt [47] to [16]). After revolutionizing NLP [39, 32], very recently, Transformer-based architectures showed promising results on large scale image classification too [13]. While self-attention and attention were previously used in conjunction with CNNs at a layer or block level [6, 50, 33], the Vision Transformer (ViT)
of Dosovitskiy et al. [13] is the first convolution-free, Transformer-based architecture that achieves state-of-the-art on ImageNet [11].
Video Transformer: Recently/concurrently with our work, vision transformer architectures, derived from [13], were used for video recognition [3, 1], too. Because performing full space-time attention is computationally prohibitive (i.e. O(T 2S2)), their main focus is on reducing this via temporal and spatial factorization. In TimeSformer [3], the authors propose applying spatial and temporal attention in an alternating manner reducing the complexity to O(T 2S + TS2). In a similar fashion, ViViT [1] explores several avenues for space-time factorization. In addition, they also proposed to adapt the patch embedding process from [13] to 3D (i.e. video) data. Our work proposes a completely different approximation to full space-time attention that is also efficient. To this end, we firstly restrict full space-time attention to a local temporal window which is reminiscent of [2] but applied here to space-time attention and video recognition 1. Secondly, we define a local joint space-time attention which we show that can be implemented efficiently via the “shift trick” [45].

heading: 3 Method
text: Video Transformer: We are given a video clip X ∈ RT×H×W×C (C = 3). Following ViT [13], each frame is divided into K ×K non-overlapping patches which are then mapped into visual tokens using a linear embedding layer E ∈ R3K2×d. Since self-attention is permutation invariant, in order to preserve the information regarding the location of each patch within space and time we also learn two positional embeddings, one for space: ps ∈ R1×S×d and one for time: pt ∈ RT×1×d. These are then added to the initial visual tokens. Finally, the token sequence is processed by L Transformer layers.
The visual token at layer l, spatial location s and temporal location t is denoted as: zls,t ∈ Rd, l = 0, . . . , L− 1, s = 0, . . . , S − 1, t = 0, . . . , T − 1. (1)
In addition to the ST visual tokens extracted from the video, a special classification token zlcls ∈ Rd is prepended to the token sequence [12]. The l−th Transformer layer processes the visual tokens Zl ∈ R(ST+1)×d of the previous layer using a series of Multi-head Self-Attention (MSA), Layer Normalization (LN), and MLP (Rd → R4d → Rd) layers as follows:
Yl = MSA(LN(Zl−1)) + Zl−1, (2)
Zl = MLP(LN(Yl)) +Yl. (3)
The main computation of a single full space-time Self-Attention (SA) head boils down to calculating:
yls,t = T−1∑ t′=0 S−1∑ s′=0 Softmax{(qls,t · kls′,t′)/ √ dh}vls′,t′ , { s=0,...,S−1 t=0,...,T−1 } (4)
where qls,t,k l s,t,v l s,t ∈ Rdh are the query, key, and value vectors computed from zls,t (after LN) using embedding matrices Wq,Wk,Wv ∈ Rd×dh . Finally, the output of the h heads is concatenated and projected using embedding matrix Wh ∈ Rhdh×d. The complexity of the full model is: O(3hTSddh) (qkv projections) +O(2hT 2S2dh) (MSA for h attention heads) +O(TS(hdh)d) (multi-head projection) +O(4TSd2) (MLP) 2. From these terms, our goal is to reduce the costO(2T 2S2dh) (for a single attention head) of the full space-time attention which is the dominant term 3. For clarity, from now on, we will drop constant terms and dh to report complexity unless necessary. Hence, the complexity of the full space-time attention is O(T 2S2).
Our baseline is a model that performs a simple approximation to the full space-time attention by applying, at each Transformer layer, spatial-only attention:
yls,t = S−1∑ s′=0 Softmax{(qls,t · kls′,t)/ √ dh}vls′,t, { s=0,...,S−1 t=0,...,T−1 } (5)
1Other attempts of exploiting local attention can be found in [29, 7, 49], however they are also different in scope, task/domain and implementation.
2For this work, we used S = 196, T = {8, 16, 32} and d = 768 (for a ViT-B backbone). 3The MLP complexity is by no means negligible, however the focus of this work (similarly to [3, 1]) is on
reducing the complexity of the self-attention component.
the complexity of which is O(TS2). Notably, the complexity of the proposed space-time mixing attention is also O(TS2). Following spatial-only attention, simple temporal averaging is performed on the class tokens zfinal = 1T ∑ t zL−1t,cls to obtain a single feature that is fed to the linear classifier.
Recent work by [3, 1] has focused on reducing the cost O(T 2S2) of the full space-time attention of Eq. 4. Bertasius et al. [3] proposed the factorised attention:
ỹls,t = T−1∑ t′=0 Softmax{(qls,t · kls,t′)/ √ dh}vls,t′ ,
yls,t = S−1∑ s′=0 Softmax{q̃ls,t · k̃ls′,t)/ √ dh}ṽls′,t,
{ s = 0, . . . , S − 1 t = 0, . . . , T − 1 } , (6)
where q̃ls,t, k̃ l s′,tṽ l s′,t are new query, key and value vectors calculated from ỹ l s,t 4. The above model reduces complexity to O(T 2S + TS2). However, temporal attention is performed for a fixed spatial location which is ineffective when there is camera or object motion and there is spatial misalignment between frames.
The work of [1] is concurrent to ours and proposes the following approximation: Ls Transformer layers perform spatial-only attention as in Eq. 5 (each with complexity O(S2)). Following this, there are Lt Transformer layers performing temporal-only attention on the class tokens zLst . The complexity of the temporal-only attention is, in general, O(T 2).
Our model aims to better approximate the full space-time self-attention (SA) of Eq. 4 while keeping complexity to O(TS2), i.e. inducing no further complexity to a spatial-only model.
To achieve this, we make a first approximation to perform full space-time attention but restricted to a local temporal window [−tw, tw]:
yls,t = t+tw∑ t′=t−tw S−1∑ s′=0 Softmax{(qls,t · kls′,t′)/ √ dh}vls′,t′ = t+tw∑ t′=t−tw Vlt′a l t′ , { s=0,...,S−1 t=0,...,T−1 } (7)
where Vlt′ = [v l 0,t′ ;v l 1,t′ ; . . . ;v l S−1,t′ ] ∈ Rdh×S and alt′ = [al0,t′ , al1,t′ , . . . , alS−1,t′ ] ∈ RS is the vector with the corresponding attention weights. Eq. 7 shows that, for a single Transformer layer, yls,t is a spatio-temporal combination of the visual tokens in the local window [−tw, tw]. It follows that, after k Transformer layers, yl+ks,t will be a spatio-temporal combination of the visual tokens in the local window [−ktw, ktw] which in turn conveniently allows to perform spatio-temporal attention over the whole clip. For example, for tw = 1 and k = 4, the local window becomes [−4, 4] which spans the whole video clip for the typical case T = 8.
The complexity of the local self-attention of Eq. 7 is O(T (2tw + 1)2S2). To reduce this even further, we make a second approximation on top of the first one as follows: the attention between spatial locations s and s′ according to the model of Eq. 7 is:
t+tw∑ t′=t−tw Softmax{(qls,t · kls′,t′)/ √ dh}vls′,t′ , (8)
i.e. it requires the calculation of 2tw +1 attentions, one per temporal location over [−tw, tw]. Instead, we propose to calculate a single attention over [−tw, tw] which can be achieved by qls,t attending kls′,−tw:tw , [k l s′,t−tw ; . . . ;k l s′,t+tw
] ∈ R(2tw+1)dh . Note that to match the dimensions of qls,t and kls′,−tw:tw a further projection of k l s′,−tw:tw to R
dh is normally required which has complexity O((2tw +1)d 2 h) and hence compromises the goal of an efficient implementation. To alleviate this we use the “shift trick” [45, 26] which allows to perform both zero-cost dimensionality reduction, spacetime mixing and attention (between qls,t and k l s′,−tw:tw ) in O(dh). In particular, each t
′ ∈ [−tw, tw] is assigned dt ′ h channels from dh (i.e. ∑ t′ d t′ h = dh). Let k l s′,t′(d t′ h ) ∈ Rd t′ h denote the operator for
4More precisely, Eq. 6 holds for h = 1 heads. For h > 1, the different heads ỹl,hs,t are concatenated and projected to produce ỹls,t.
indexing the dt ′
h channels from k l s′,t′ . Then, a new key vector is constructed as:
k̃ls′,−tw:tw , [k l s′,t−tw(d t−tw h ), . . . ,k l s′,t+tw(d t+tw h )] ∈ R dh . (9)
Fig. 2 shows how the key vector k̃ls′,−tw:tw is constructed. In a similar way, we also construct a new value vector ṽls′,−tw:tw . Finally, the proposed approximation to the full space-time attention is given by:
ylss,t = S−1∑ s′=0 Softmax{(qlss,t · k̃ls′,−tw:tw/ √ dh}ṽls′,−tw:tw , { s=0,...,S−1 t=0,...,T−1 } . (10)
This has the complexity of a spatial-only attention (O(TS2)) and hence it is more efficient than previously proposed video transformers [3, 1]. Our model also provides a better approximation to the full space-time attention and as shown by our results it significantly outperforms [3, 1].
Temporal Attention aggregation: The final set of the class tokens zL−1t,cls , 0 ≤ t ≤ L− 1 are used to generate the predictions. To this end, we propose to consider the following options: (a) simple temporal averaging zfinal = 1T ∑ t z L−1 t,cls as in the case of our baseline. (b) An obvious limitation of temporal averaging is that the output is treated purely as an ensemble of per-frame features and, hence, completely ignores the temporal ordering between them. To address this, we propose to use a lightweight Temporal Attention (TA) mechanism that will attend to the T classification tokens. In particular a zfinal token attends the sequence [zL−10,cls, . . . , z L−1 T−1,cls] using a temporal Transformer layer and then fed as input to the classifier. This is akin to the (concurrent) work of [1] with the difference being that in our model we found that a single TA layer suffices whereas [1] uses Lt. A consequence of this is that the complexity of our layer is O(T ) vs O(2(Lt − 1)T 2 + T ) of [1]. Summary token: As an alternative to TA, herein, we also propose a simple lightweight mechanism for information exchange between different frames at intermediate layers of the network. Given the set of tokens for each frame t, Zl−1t ∈ R(S+1)×dh (constructed by concatenating all tokens zl−1s,t , s = 0, . . . , S), we compute a new set of R tokens Z l r,t = φ(Z l−1 t ) ∈ RR×dh which summarize the frame information and hence are named “Summary” tokens. These are then, appended to the visual tokens of all frames to calculate the keys and values so that the query vectors attend the original keys plus the Summary tokens. Herein, we explore the case that φ(.) performs simple spatial averaging zl0,t = 1 S ∑ s z l s,t over the tokens of each frame (R = 1 for this case). Note that, forR = 1, the extra cost that the Summary token induces is O(TS).
X-ViT: We call the Video Transformer based on the proposed (a) space-time mixing attention and (b) lightweight global temporal attention (or summary token) as X-ViT.

heading: 4 Results

heading: 4.1 Experimental setup
text: Datasets: We train and evaluate the proposed models on the following datasets (all datasets are publicly available for research purposes):
Kinetics-400 and 600: The Kinetics [21] dataset consists of short clips (typically 10 sec long sampled from YouTube) labeled using 400 and 600 classes, respectively. Due to the removal of some videos from YouTube, the version of the dataset used in this paper consists of approximately 261K clips for Kinetics-400. Note, that these amounts are lower than the original version of the datasets and thus might represent a negative performance bias when compared with prior works.
Something-Something-v2 (SSv2): The SSv2 [17] dataset consists of 220,487 short videos (of duration between 2 and 6 sec) that depict humans performing pre-defined basic actions with everyday objects. Because the objects and backgrounds in the videos are consistent across different action classes, this dataset tends to require stronger temporal modeling. Due to this, we conducted most of our ablation studies on SSv2 to better analyze the importance of the proposed components.
Epic Kitchens-100 (Epic-100): is an egocentric large scale action recognition dataset consisting of more than 90,000 action segments spanning 100 hours of recordings in home environments, capturing daily activities [10]. The dataset is labeled using 97 verb classes and 300 noun classes. The evaluation results are reported using the standard action recognition protocol: the network predicts the “verb” and the “noun” using two heads. The predictions are then merged to construct an “action” which is used to report the accuracy.
Training details: All models, unless otherwise stated, were trained using the following scheduler and training procedure: specifically, our models were trained using SGD with momentum (0.9) and a cosine scheduler [28] (with linear warmup) for 35 epochs on SSv2, 50 on Epic-100 and 30 on Kinetics. The base learning rate, set at a batch size of 128, was 0.05 (0.03 for Kinetics). To prevent over-fitting we made use of the following augmentation techniques: random scaling (0.9× to 1.3×) and cropping, random flipping (with probability of 0.5; not for SSv2) and autoaugment [8]. In addition, for SSv2 and Epic-100, we also applied random erasing (probability=0.5, min. area=0.02, max. area=1/3, min. aspect=0.3) [52] and label smoothing (λ = 0.3) [34] while, for Kinetics, we used mixup [51] (α = 0.4).
The backbone models follow closely the ViT architecture of Dosovitskiy et al. [13]. Most experiments were performed using the ViT-B/16 variant (L = 12, h = 12, d = 768, K = 16), where L represents the number of transformer layers, h the number of heads, d the embedding dimension and K the patch size. We initialized our models from a pretrained ImageNet-21k [11] ViT model. The spatial positional encoding ps was initialized from the pretrained 2D model and the temporal one, pt, with zeros so that it does not have a great impact on the tokens early on during training. The models were trained on 8 V100 GPUs using PyTorch [30].
Testing details: Unless otherwise stated, we used ViT-B/16 and T = 8 frames. We mostly used Temporal Attention (TA) for temporal aggregation. We report accuracy results for 1 × 3 views (1 temporal clip and 3 spatial crops) departing from the common approach of using up to 10 × 3 views [26, 16]. The 1× 3 views setting was also used in Bertasius et al. [3]. To measure the variation between runs, we trained one of the 8–frame models 5 times. The results varied by ±0.4%.

heading: 4.2 Ablation studies
text: Throughout this section, we study the effect of varying certain design choices and different components of our method. Because SSv2 tends to require a more fine-grained temporal modeling, unless otherwise specified, all results reported, in this section, are on the SSv2.
Table 2: Effect of: (a) proposed SA position, (b) temporal aggregation and number of Temporal Attention (TA) layers, (c) space-time mixing qkv vectors and (d) amount of mixed channels on SSv2.
(a) Effect of applying the proposed SA to certain layers.
Transform. layers Top-1 Top-5
1st half 61.7 86.5 2nd half 61.6 86.3 Half (odd. pos) 61.2 86.4 All 62.6 87.8
(b) Effect of number of TA layers. 0 corresponds to temporal averaging.
#. TA layers Top-1 Top-5
0 (temp. avg.) 62.4 87.8 1 64.4 89.3 2 64.5 89.3 3 64.5 89.3
(c) Effect of space-time mixing. x denotes the input token before qkv projection. Query produces equivalent results with key and thus omitted.
x key value Top-1 Top-5
7 7 7 56.6 83.5 X 7 7 63.1 88.8 7 X 7 63.1 88.8 7 7 X 62.5 88.6 7 X X 64.4 89.3
(d) Effect of amount of mixed channels. * uses temp. avg. aggregation.
0%* 0% 25% 50% 100%
45.2 56.6 64.3 64.4 62.5
Effect of local window size: Table 1 shows the accuracy of our model by varying the local window size [−tw, tw] used in the proposed space-time mixing attention. Firstly, we observe that the proposed model is significantly superior to our baseline (tw = 0) which uses spatial-only attention. Secondly, a window of tw = 1 produces the best results. This shows that more gradual increase of the effective window size that is attended is more beneficial compared to more aggressive ones, i.e. the case where tw = 2. A performance degradation for the case tw = 2 could be attributed to boundary effects (handled by filling with zeros) which are aggravated as tw increases. Based on these results, we chose to use tw = 1 for the models reported hereafter. For short to medium long videos, it seems that tw = 1 suffices as the temporal receptive field size increases as we advance in depth in the model allowing it to capture a larger effective temporal window. For the datasets used, as explained earlier, after a few transformer layers the whole clip is effectively covered. However, for significantly longer video sequences, larger window sizes may perform better.
Effect of SA position: We explored which layers should the proposed space-time mixing attention be applied to within the network. Specifically, we explored the following variants: Applying it to the first L/2 layers, to the last L/2 layers, to every odd indexed layer and, finally, to all layers. As the results from Table 2a show, the exact layers within the network that self-attention is applied to do not matter; what matters is the number of layers it is applied to. We attribute this result to the increased temporal receptive field and cross-frame interactions.
Effect of temporal aggregation: Herein, we compare the two methods used for temporal aggregation: simple temporal averaging [41] and the proposed Temporal Attention (TA) mechanism. Given that our model already incorporates temporal information through the proposed space-time attention, we also explored how many TA layers are needed. As shown in Table 2b, replacing temporal averaging with one TA layer improves the Top-1 accuracy from 62.5% to 64.4%. Increasing the number of layers further yields no additional benefits. In Table 2d, we also report the accuracy of spatial-only attention (0% mixing) plus TA aggregation. In the absence of the pro-
posed space-time mixing attention, the TA layer alone is unable to compensate, scoring only 56.6%. In the same table, 45.2% is the accuracy of a model trained without the proposed local attention and TA layer (i.e. using a temporal pooling for aggregation). Overall, the results highlight the need of having both components in our final model. For the next two ablation studies, we used 1 TA layer.
Effect of space-time mixing qkv vectors: Paramount to our work is the proposed space-time mixing attention of Eq. 10 which is implemented by constructing k̃ls′,−tw:tw and ṽ l s′,−tw:tw efficiently via channel indexing (see Eq. 9). Space-time mixing though can be applied in several different ways in
the model. For completeness, herein, we study the effect of applying space-time mixing to various combinations for the key, value and to the input token prior to qkv projection. As shown in Table 2c, the combination corresponding to our model (i.e. space-time mixing applied to the key and value) significantly outperforms all other variants by up to 2%. This result is important as it confirms that our model, derived from the proposed approximation to the local space-time attention, gives the best results when compared to other non-well motivated variants.
Effect of amount of space-time mixing: We define as ρdh the total number of channels coming from the adjacent frames in the local temporal window [−tw, tw] (i.e. ∑tw t′=−tw,t6=0 d t′
h = ρdh) when constructing k̃ls′,−tw:tw (see Section 3). Herein, we study the effect of ρ on the model’s accuracy. As the results from Table 2d show, the optimal ρ is between 25% and 50%. Increasing ρ to 100% (i.e. all channels are coming from adjacent frames) unsurprisingly degrades the performance as it excludes the case t′ = t when performing the self-attention.
Effect of Summary token: Herein, we compare Temporal Attention with Summary token on SSv2 and Kinetics-400. We used both datasets for this case as they require different type of understanding: fine-grained temporal (SSv2) and spatial content (Kinetics-400). From Table 4, we conclude that the Summary token compares favorable on Kinetics-400 but not on SSv2 showing that it is more useful in terms of capturing spatial information. Since the improvement is small, we conclude that 1 TA layer is the best global attention-based mechanism for improving the accuracy of our method adding also negligible computational cost.
Effect of number of input frames: Herein we evaluate the impact of increasing the number of input frames T from 8 to 16 and 32. We note that, for our method, this change results in a linear increase in complexity. As the results from Table 7 show, increasing the number of frames from 8 to 16 offers a 1.8% boost in Top-1 accuracy on SSv2. Moreover, increasing the number of frames to 32 improves the performance by a further 0.2%, offering diminishing returns. Similar behavior can be observed on Kinetics and Epic-100 in Tables 5 and 8.
Effect of number of tokens and different model sizes: Herein, we vary the number of input tokens by changing the patch size K. As the results from Table 3 show, even when the number of tokens decreases significantly (e.g. ViT-B/32 or ViT-S/32) our approach is still able to produce results of satisfactory accuracy. The benefit of that is having a model which is significantly more efficient. Similar concusions can be observed when the model size (in terms of parameters and FLOPs) is varied. Our approach provides consistent results in all cases, showcasing its ability to scale well from tiny (XViT-T) to large (XViT-L) models.
Latency and throughput considerations: While the channel shifting operation used by the proposed space-time mixing attention is zero-FLOP, there is still a small cost associated with memory movement operations. In order to ascertain that the induced cost does not introduce noticeable performance degradation, we benchmarked a Vit-B/16 (8× frames) model using spatial-only attention and the proposed space-time mixing attention on 8 V100 GPUs and a batch size of 128. A model with spatial-only attention has a throughput of 312 fps while our model has 304 fps.

heading: 4.3 Comparison to state-of-the-art
text: Our best model uses the proposed space-time mixing attention in all the Transformer layers and performs temporal aggregation using a single lightweight temporal transformer layer as described in Section 3. Unless otherwise specified, we report the results using the 1× 3 configuration for the views (1 temporal and 3 spatial) for all datasets. Regarding related work on transformer-based video recognition [1, 3], we included their very best models trained on the same data as our models. For TimeSformer, this is typically the TimeSformer-L version. For ViVit, we used the 16x2 configuration, with factorized-encoding for Epic-100 and SS-v2 (as reported in Tables 6d and 6e in [1]) and the full version for Kinetics (as reported in Table 6a in [1]).
On Kinetics-400, we match the current state-of-the-art while having significantly lower computational complexity than the next two best recently proposed methods that also use Transformer-based architectures: 20× fewer FLOPs than ViVit [1] and 8× fewer than TimeSformer-L [3]. Note that both models from [1, 3] and ours were initialized from a ViT model pretrained on ImageNet-21k [11] and take as input frames at a resolution of 224 × 224px. Similar conclusions can be drawn from Table 6 which reports our results on Kinetics-600.
On SSv2, we match and surpass the current state-of-the-art, especially in terms of Top-5 accuracy (ours: 90.7% vs ViViT: 89.8% [1]) using models that are 14× (16 frames) and 9× (32 frames) faster. Finally, we observe similar outcomes on Epic-100 where we set a new state-of-the-art, showing large improvements especially for “Verb” accuracy, while again being more efficient.

heading: 5 Ethical considerations and broader impact
text: Current high-performing video recognition models tend to have high computational demands for both training and testing and, by extension, significant environmental costs. This is especially true for the transformer-based architectures. Our research introduces a novel approach that matches and surpasses the current state-ofthe-art while being significantly more efficient thanks to the linear scaling of the complexity with respect to the number of frames. We hope such models will offer noticeable reduction in power consumption while setting at the same time a solid base for future research. We will release code and models to facilitate this. Moreover, and similarly to most data-driven systems,
bias from the training data can potentially affect the fairness of the model. As such, we suggest to take this aspect into consideration when deploying the models into real-world scenarios.

heading: 6 Conclusions
text: We presented a novel approximation to the full space-time attention that is amenable to an efficient implementation and applied it to video recognition. Our approximation has the same computational cost as spatial-only attention yet the resulting video Transformer model was shown to be significantly more efficient than recently proposed Video Transformers [3, 1]. By no means this paper proposes a complete solution to video recognition using video Transformers. Future efforts could include combining our approaches with other architectures than the standard ViT, removing the dependency on pre-trained models and applying the model to other video-related tasks like detection and segmentation. Finally, further research is required for deploying our models on low power/resource devices.

Review summaries:
The authors propose a more efficient self-attention operation, specifically for video models, such that the compute requirements increase linearly with the length of the video. The author's contributions consist of two main things: 1) Self-attention along the temporal axis is limited to a smaller window instead of the whole sequence (with multiple transformer layers, the temporal "receptive field" grows to cover the whole video) and 2) The authors efficiently perform "space-time mixing" by constructing the keys and values for attention by concatenating features from different frames. This operation requires 0 floating point operations, but is still able to combine temporal information from adjacent frames effectively.  The authors show good experimental results. They achieve similar results to ViViT [1] and TimeSFormer [3] whilst using far fewer GFLOPs, on Kinetics, SSv2 and Epic Kitchens. The paper is also well written in general.

This paper presents a video Transformer model for video recognition. It limits the temporal window to the local area and proposes efficient space-time mixing to reduce computational costs. On Kinetics, Something-Something v2, and Epic Kitchens datasets,  the proposed video transformer model is more effective than other transformer-based models yet has the same computational cost as spatial-only attention models.

In this paper, the authors proposed a Video Transformer model the complexity of which scales linearly (instead of quadratic) with the number of frames in the video sequence. The key ideas are 1) restricting time attention to a local temporal window, and 2) using efficient space-time mixing to attend jointly spatial and temporal locations. They empirically verified that the proposed method outperforms existing state-of-the-art models in popular action recognition datasets.

In this paper, the authors focus on reducing the computation cost caused by the space-time attention modeling in the Transformer framework for video recognition. Specifically, the proposal includes 1) a space-time mixing attention module that mixes channels from neighboring frames within a local window as key-value tokens, and 2) a temporal attention layer at the end of the model to aggregate global temporal information. The authors provide promising results on Kinetics-400 and SSv2 in the paper.