Space-time Mixing Attention for Video Transformer
Authors: Adrian Bulat
Juan-Manuel Perez-Rua
Swathikiran Sudhakaran
Brais Martinez
Georgios Tzimiropoulos
Venue: NeurIPS 2021 Poster (2021)
Keywords: video recognition
transformers
efficient networks

Abstract: This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer's depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models.

Space-time Mixing Attention for Video Transformer
Adrian Bulat
Juan-Manuel Perez-Rua
Samsung Cambridge
Samsung Al Cambridge
perez-rua@samsung. com
Swathikiran Sudhakaran Samsung Al Cambridge swathikir s@samsung.
Brais Martinez
Georgios Tzimiropoulos
Samsung Al Cambridge
Samsung AI Cambridge
wtt.hhiiirrssssmuunnn... com
brais a@samsung. com
Queen Mary University of London g. tzzimiropoollos@mmm... ac uk
Abstract
This paper is on video recognition using Transformers. Very recent attempts this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations
the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes the Transformer's depth obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code for our method is made available here
Introduction
Video recognition in analogy to image recognition refers to the problem of recognizing events of interest video sequences such as human activities. Following the tremendous success of Transformers in sequential data, specifically in Natural Language Processing (NLP) [39 5), Vision
Transformers were very recently shown to outperform CNNs for image recognition too 48 13 35),
signaling paradigm shift how visual understanding models should be constructed. In light of this, in this paper, we propose a Video Transformer model as an appealing and promising solution for improving the accuracy of video recognition models.
direct, natural extension of Vision Transformers to the spatio-temporal domain is to perform the self-attention jointly across all S spatial locations and T temporal locations. Full space-time attention though has complexity ( making such a model computationally heavy and, hence, impractical even when compared with the 3D-based convolutional models. As such, our aim is to exploit the temporal information present in video streams while minimizing the computational burden within Transformer framework for efficient video recognition.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Teee
Teee
(a) Full space-time atten- (b) Spatial-only attention: tion: 5²)
(c) TimeSformer [3] and 3) M:
(d) Ours: O(T'²)
O(T5))
ViVIT (Model
O T
Figure 1: Different approaches space-time self-attention for video recognition. In all cases, the key locations that the query vector, located at the center of the grid in red, attends are shown in orange. Unlike prior work, our key vector is constructed by mixing information from tokens located at the same spatial location within local temporal window. Our method then performs self-attention with these tokens. Note that our mechanism allows for an efficient approximation local space-time attention at no extra cost.
baseline solution to this problem is to consider spatial-only attention followed by temporal averaging, which has complexity ) Similar attempts to reduce the cost of full space-time attention have been recently proposed in 3 [. These methods have demonstrated promising results in terms of video recognition accuracy, yet they have been also shown to induce, in most of the cases, significant computational overheads compared to the baseline (spatial-only) method due the additional modelling of the temporal information.
Our main contribution in this paper is Video Transformer model that has complexity O(TS²) and, hence, as efficient as the baseline model, yet, as our results show, outperforms re- cently/concurrently proposed work
in terms of efficiency (i.e. accuracy/FLOPP by significant margins. To achieve this our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to local temporal window and capitalizes on the Transformer's depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing attend jointly spatial and temporal locations without inducing any additional cost on top of spatial-only attention model. Fig. [ shows the proposed approximation to space- space-time attention. We also show how to integrate two very lightweight mechanisms for global temporal-only attention. which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model surprisingly effective in terms of capturing long-term dependencies and producing very high recognition accuracy on the most popular video recognition datasets, including Something-Some Something-v2 [17, Kinetics 4 and Epic Kitchens (9), while the same time being significantly more efficient than other Video Transformer models.
Related work
Video recognition: Standard solutions are based on CNNs and can be broadly classified into two categories: 2D- and 3D-based approaches. 2D-based approaches process each frame independently to extract frame-based features which are then aggregated temporally with some sort of temporal
modeling (e.g. temporal averaging) performed at the end of the network 42 26 27 The works of [26 [2] use the "shift trick" [45] to have some temporal modeling at a layer level. 3D-based approaches 4 16, 36) are considered the current state-of-th -ohe------h-rrt-- art as they typically learn stronger temporal models via 3D convolutions. However, they also incur higher computational and memory costs. To alleviate this, large body of works attempt to improve their efficiency via spatial and/or temporal factorization
38
CNN vs ViT: Historically, video recognition approaches tend to mimic the architectures used for image classification (e.g. from AlexNet (33 to (20) or from ResNet [18] and ResNeXt [47 to [16).. After revolutionizing NL.P results on large scale image classification too [13]. While self-attention and attention were previously
39 32
very recently. Transformer-
architectures showed promising
used in conjunction with CNNs at layer or block level 66, [5 33), the Vision Transformer (ViT)

---

of Dosovitskiy et al. [3 is the first convolution- free, free, Transformer- architecture that achieves state-of-the-art on ImageNet [T].
Video Transformer: Recently/concureenly with our work, vision transformer architectures, derived from [3], were used for video recognition [3 I. too. Because performing full space-time attention is computationally prohibitive (i.e. O ))),, their main focus is on reducing this via temporal and spatial factorization. TimeSformer (3] the authors propose applying spatial and temporal attention in an alternating manner reducing the complexity to O + +T In a similar fashion, ViViT M explores several avenues for space-time sace-timee factorization. In addition, they also proposed to adapt the patch embedding process from [3] to 3D (i.e. video) data. Our work proposes completely different approximation to full space- space-time attention that is also efficient. To this end. we firstly restrict full space- timee-ttme&eae attention to local temporal window which is reminiscent of [ but applied here to space-time attention and video recognition] [ Secondly, we define a local joint space-time attention which we show that can be implemented efficiently via the "shift trick"
45
3 Method
Video Transformer: We are given a video clip X RTXHXWXG (C
(C=3) 3). Following ViT [3]. each frame is divided into K x K non-overlapping patches which are then mapped into visual tokens using a linear embedding layer E E xg to preserve the information regarding the location of each patch within space and time we also learn two positional embeddings, one for space: P, E IR xxSxx and one for time: Ps R xxxxx 1x These are then added to the initial visual tokens. Finally, the token sequence is processed by L Transformer layers.
xd Since self-attention is permutation invariant, in order
The visual token at layer 1, spatial location and temporal location / is denoted as:
z E R2d 1 0. = S 1, t o
(1)
In addition to the ST visual tokens extracted from the video, special classification token 2/4 E Bd prepended to the token sequence [12. The 1-th Transformer layer processes the visual tokens Z' E (ST==))xd of the previous layer using a series of Multi-head Self-Attention (MSA), Layer Normalization (LN), and MLP (R + R.dd > IRd) layers as follows:
Y = MSA(LN(Z
+ Z
(2)
z' = MLP(LNN(Y')) + Y
(3)
The main computation of a single full space-time Self-Attention (SA) head boils down to calculating:
T-15-1
ΣΣ
Softmax((q',, k/y,/// dd Vn.vv's se
1'=0===0
where Lh,,, k Vs, E TR are the query, key, and value vectors computed from z!, (after LN) using embedding matrices Wq: Wk. W. E Rddd, Finally, the output of the h heads is concatenated and projected using embedding matrix W. E xd
The complexity of the full model is: Od3hTSdd,) (gkv projections) SO (MSA for h attention heads) ((dd, )d) (multi-head projection) +O(4TSd²) (MLP) From these terms, (for single attention head) of the full space-time attention which is the dominant term 3] For clarity, from now on, we will drop constant terms and dp to report complexity unless necessary. Hence, the complexity of the full space-time attention is 52)
our goal is reduce the cost 222T2
Our baseline is a model that performs a simple approximation to the full space-time attention by applying, at each Transformer layer, spatial-only attention:
S-1
Softmax
VnnVvl. (
S-1
(5)
='=0
'ther attempts of exploiting local attention can be found in 29 scope. task/domain and implementation.
49
however they are also different in
For this work, we used $ 196, = (8, 16, 32} and 768 (for a VIT-B backbone). 'The MLP complexity is by no means negligible, however the focus of this work (similarly to 3 is on reducing the complexity of the self-attentin component.
3

---

the complexity of which is O(TS²). Notably, the complexity of the proposed space-time mixing attention also OTT5²). Following spatial-only attention, simple temporal averaging is performed on the class tokens final r' E 1 to obtain single feature that is fed to the linear classifier.
Recent work by 3 has focused reducing the cost 52) of the full space-time attention of Eq. Bertasius et al. 3 proposed the factorised attention:
foftmax((qbe. K.,,)) V/nn)//dn dn
(6)
0, T 7-11
y's Softmax (ā k//l dn V
where L',,, K V a are new query, key and value vectors calculated from y The above model reduces complexity to O(T'S + ) However, temporal attention is performed for fixed spatial location which is ineffective when there is camera or object motion and there is spatial misalignment between frames.
The work of [1] is concurrent ours and proposes the following approximation: La Transformer layers perform spatial-only attention as in Eq. 5 (each with complexity O(5²)). Following this, there are Lç Transformer layers performing temporal-only attention on the class tokens zf The complexity of the temporal-only attention is, in general, (
Our model aims to better approximate the full space-time self eeff-attention (SA) of Eq. 4 while keeping complexity to (T))), i.e. inducing no further complexity spatial-only model.
To achieve this, we make a first approximation to perform full space-time attention but restricted to a local temporal window [-tw,tui
L+ Σ
4+ Σ f'=--tw
Σ
Softmax((q'.. k/,e0// dn
Viraa, { (1-0...--11
(=0.....-))
Yat.
1'=tt-aas' s'=0
:.......v''' veeeee E RR x and a,, = lab.er -........... 1--1, E TRS is the vector with the corresponding attention weights. Eq. shows that, for a single Tränsformer layer, y',, is a spatio-temporal combination of the visual tokens in the local window [ tw]. It follows that, after Transformer layers, ys will be a spatio-temporal combination of the visual tokens in the local window [-ktwey kt which in over the whole clip. For example, for byr = and A: = 4, the local window becomes [-4, 4] which spans the whole video clip for the typical case T = 8.
where V = [[vl,r: V
conveniently allows to perform spatio-temporal attention
The complexity of the local seff-attention of Eq. is O(T + 1)? To reduce this even further, make second approximation top of the first one follows: the attention between spatial locations and s' according to the model of Eq. is:
Softmax ( ((''. k//)\/ dr.v'
(8)
i.c it requires the calculation of 2t + attentions, one per temporal location over [-tw, tw]. Instead, we propose to calculate a single attention over [-tw; tw] which can be achieved by L. attending k', -twit A [K. (ks..ttei....... kkr,ttt,.)) E TR. +1)ds Note that match the dimensions of q and k-, tt. a further projection of k to Rd is normally required which has complexity O((2t + 1)d2)) ) and hence compromises the goal of an efficient implementation. To alleviate this we use the "shift trick" [45] 26 which allows to perform both zero-cost dimensionality reduction, space- time mixing and attention (between q',, and K,, -tuit in (dz,). J. In particular, each t' E [-tw, twe
assigned dy channels from dy (i.e. En df dr). Let ky, (dk, E R denote the operator for
'More precisely, Eq. 6 holds for h heads. For h 1. the different heads y concatenated and projected to produce y

---

MottAul
Mathuu
SontMox
Scale
Scale
MotMu
MatMull
S/T
S+T
(a) Full space-time attention.
(b) Proposed space-time mixing attention.
Figure 2: Detailed self-attention computation graph for (a) full space-time attention and (b) the only S tokens participate instead
proposed space-time time mixing approximation. Notice that in of ST. The temporal information is aggregated by indexing channels from adjacent frames. Tokens of identical colors share the same temporal index.
indexing the di channels from k.... Then, a new key vector is constructed as:
K I (d Kg
se+t Kerttr (d)tt==] E Rds
Fig. 2 shows how the key vector - constructed. a similar way, we also construct a new value vector v -ttttti Finally, the proposed approximation to the full space-time attention is given by:
S-1 Σ
Ve S Ky V "te
(10)
This has the complexity of a spatial-only attention (O(T52) and hence it is more efficient than previously proposed video transformers [. Our model also provides better approximation the full space-time attention and as shown by our results it significantly outperforms []
Temporal Attention aggregation: The final set of the class tokens Z 0 2 5
used to generate the predictions. To this end, propose consider the following options: (a) simple temporal averaging final $ Σ, 2 in the case of our baseline. (b) An obvious limitation of temporal averaging is that the output is treated purely hence, completely ignores the temporal ordering between them. To address this, we propose to use a lightweight Temporal Attention (TA) mechanism that will attend to the T classification tokens. In particular final token attends the sequence [2) 27-1,o using a temporal Transformer layer and then fed as input to the classifier. This is akin to the (concurrent) work of [ with the difference being that our model we found that single TA layer suffices whereas uses L,. consequence of this is that the complexity of our layer is O(T) vs O(2(I. 1)7 T) of [..
ensemble of per-frame features and,
Summary token: As alternative TA, herein, also propose simple lightweight mechanism for information exchange between different frames at intermediate layers of the network. Given the set of tokens for each frame t, z E (S+1)xxd, (constructed by concatenating all tokens 2/ 8 .....,,),, S), we compute a new set of R tokens Z = (Z!-11) E RRx,, which summarize the frame information and hence are named "Summary" tokens. These are then, appended to the visual tokens of all frames to calculate the keys and values so that the query vectors attend the original keys plus the Summary tokens. Herein, we explore the case that Φ(.) performs simple spatial averaging zh. 1
E. z'_, over the tokens of each frame (R for this case). Note that, for R the extra cost that the Summary token induces is OTTS).
X-V:T: We call the Video Transformer based on the proposed (a) space-time mixing attention and (b) lightweight global temporal attention (or summary token) as X-V.T.
5

---

4 Results
4.1 Experimental setup
Datasets: We train and evaluate the proposed models on the following datasets (all datasets are publicly available for research purposes):
Kinetics-400 and 600: The Kinetics (21] dataset consists of short clips (typically 10 see long sampled from YouTube) labeled using 400 and 600 classes, respectively. Due to the removal of some videos from YouTube. the version of the dataset used in this paper consists of approximately 26 clips for Kinetics-400. Note, that these amounts are lower than the original version of the datasets and thus might represent a negative performance bias when compared with prior works.
Something-Someen Something--2 v2 (SSv2): The SSv2 17 dataset consists of 220,487 short videos (of duration between and sec) that depict humans performing pre-defined basic actions with everyday objects. Because the objects and backgrounds in the videos are consistent across different action classes, this dataset tends to require stronger temporal modeling. Due to this, we conducted most of our ablation studies on SSv2 to better analyze the importance of the proposed components.
Epic Kitchens- 100 (Epic-100): an egocentric large scale action recognition dataset consisting of more than 90,000 action segments spanning 100 hours of recordings in home environments, capturing daily activities [10]. The dataset is labeled using 97 verb classes and 300 noun classes. The evaluation results are reported using the standard action recognition protocol: the network predicts the "verb" and the "noun" using two heads. The predictions are then merged to construct an "action" which is used to report the accuracy.
Training details: All models, unless otherwise stated, were trained using the following scheduler and training pro-
Table I: Effect of local window size. To isolate its effect from that of temporal aggregation, the models were trained using temporal averaging. Note, that (Bo.) indicates that only features from the boundaries of the local window were used, ignoring the intermediate ones.
cedure:
specifically, our models were trained using SGD
with momentum (0.9) and a cosine scheduler [28] (with linear warmup) for 35 epochs on SSv2, 50 on Epic-100 100 and 30 on Kinetics. The base learning rate, at batch size of 128, was 0.05 (0.03 for Kinetics). To prevent over- werr-fiting made of the following augmentation techniques: random scaling (0.9x to 1.3x) and cropping, random flip- ping (with probability of 0.5; not for SSv2) and autoaug- ment [8]. In addition, for SSv2 and Epic-100, we also applied random erasing (probabilit=0.55, min. area=0.02,
Variant
Top-1
Top-5
tu
45.2
71.4
tw
62.5
87.8
tw
60.5
86.4
max. area=1/3, min. aspect=0.3) (52) and label smoothing
tu = 2 (Bo.)
60.4
86.2
(A 0.3) [4 while, for Kinetics, we used mixup [5T] (a = 0.4).
The backbone models follow closely the ViT architecture of Dosovitskiy et al. [13 Most experiments were performed using the ViT-B/16 variant (L = 12, h h 12, d 768, K = 16), where L represents the number of transformer layers, h the number of heads, d the embedding dimension and K the patch size. We initialized our models from a pretrained ImageNet-21k [1] ViT model. The spatial positional encoding P, was initialized from the pretrained 2D model and the temporal one, P., with zeros so that it does not have a great impact on the tokens early on during training. The models were trained on 8 V100 GPUs using PyTorch (30]
Testing details: Unless otherwise stated, we used ViT-B/16 and T = 8 frames. We mostly used Temporal Attention (TA) for temporal aggregation. We report accuracy results for 1 x 3 views (I temporal clip and 3 spatial crops) departing from the common approach of using up to 10 x 3 views 26 16]. The 1 3 views setting also used in Bertasius et al. [3). To measure the variation between runs, we trained one of the 8-frame models times. The results varied by #0.4%.
4.2 Ablation studies
Throughout this section, we study the effect of varying certain design choices and different compo- nents of our method. Because SSv2 tends to require a more fine-grained temporal modeling, unless otherwise specified, all results reported, in this section, are on the SSv2.

---

Table 2: Effect of: (a) proposed SA position, (b) temporal aggregation and number of Temporal Attention (TA) layers, (c) space-time mixing gkv vectors and (d) amount of mixed channels on SSv2.
(a) Effect of applying the proposed SA to certain layers.
(c) Effect of space-time mixing. x denotes the input token before okv projection. Query produces equivalent results with key and thus omitted.
Transform. layers
Top-1
Top-5
1st half
61.7
86.5
X
key
value
Top-1 1
Top-5
2nd half
61.6
86.3
Half (odd. pos)
61.2
86.4
X
X
X
56.6
83.5
All
62.6
87.8
V
X
X
63.1
88.8
X
V
X
63.1
88.8
(b) Effect of number of TA layers. 0 corresponds to temporal averaging.
X
X
V
62.5
88.6
X
V
V
64.4
89.3
#. TA layers
Top-1
Top-5
(d) Effect of amount of mixed channels. * uses temp. avg. aggregation.
(temp. avg.)
62.4
87.8
64.4
89.3
0% *
0%
25%
50%
100%
2
64.5
89.3
3
64.5
89.3
45.2
56.6
64.3
64.4
62.5
Effeet of local window size: Table I shows the accuracy of our model by varying the local window size| [-tw, tw] used in the proposed space-time mixing attention. Firstly, we observe that the proposed model is significantly superior to our baseline (f., 0) which uses spatial-only attention. Secondly, a window of ly produces the best results. This shows that more gradual increase of the effective window size that is attended is more beneficial compared to more aggressive ones, i.e. the case where to = 2. A performance degradation for the case tw = 2 could be attributed to boundary effects (handled by filling with zeros) which are aggravated as fu increases. Based on these results, we chose to use by
for the models reported hereafter. For short to medium long videos, seems that tu = 1 suffices as the temporal receptive field size increases as we advance in depth in the model allowing it to capture a larger effective temporal window. For the datasets used, as explained carlier, after few transformer layers the whole clip is effectively covered. However, for significantly longer vidco sequences, larger window sizes may perform better.
We explored which layers should the proposed space-time mixing attention be applied to within the network. Specifically, we explored the following variants: Applying first L/2 layers, to the last L/2 layers, to every odd indexed layer and, finally, to all layers. As the results from Table show the exact layers within the network that self-attention is applied to do not matter; what matters is the number of layers it is applied to. We attribute this result to the increased temporal receptive field and cross-frame interactions.
Effect of SA position:
the
Effect of temporal aggregation: Herein, we compare the two methods used for temporal aggregation: simple tempo- ral averaging 44] and the proposed Temporal Attention (TA) mechanism. Given that our model already incorporates tem- poral information through the proposed space-time attention, we also explored how many TA layers are needed. As shown in Table 2b] replacing temporal averaging with one TA layer improves the Top-1 I accuracy from 62.5% to 64.4%. Increasing the number of layers further yields no additional benefits. In Table 2d] we also report the accuracy of spatial-only attention (0% mixing) plus TA aggregation. In the absence of the pro- posed space-time time mixing attention, the TA layer alone is unable to compensate, scoring only 56.6%. In the same table, 45.2% is the accuracy of model trained without the proposed local attention and TA layer (i.e. using a temporal pooling for aggregation). Overall, the results highlight the need of having both components in final model. For the next two ablation studies, we used TA layer.
Table Effect of number of tokens on SSv2.
Variant
Top-1
Top-5
XVIT-T/16
54.7
82.8
XVIT-S/32
57.0
84.6
XVTT-S/16
61.1
88.0
XVTT-B/32
60.5
87.4
XVIT-L/32
61.8
88.3
XVIT-B/16
64.4
89.3
Effeet of space-time mixing gke vectors: Paramount to our work the proposed space-time mixing attention of Eq. 10 which implemented by constructing K - and v/ -tust efficiently via A channel indexing (see Eq. 9). Space-time mixing though can be applied in several different ways in
7

---

Table Comparison between TA and Summary token on SSv2 (left) and Kinetics senettcss4000 (right).
Summary
TA
Top-1
Top-5
Summary
TA
Top-1
Top-5
X
62.4
87.8
X
X
77.8
93.7
x
63.7
88.9
X
78.7
93.7
v
63.4
88.9
v
78.0
93.2
X
v
64,4
89.3
X
V
78.5
93.7
Table 5: Comparison with state-of-the-art on the Kinetics-400.
Method
Top-1
Top-5
# Frames
Views
Params
FLOPs (x10")
bLVNet [4
73.5
91.2
24 x 2
x 3
25M
840
STM 19
73.7
91.6
16
24M
TEA 25
76.1
92.5
16
10 x 3
25.6M
2,100
TSM R50 26
74.7
16
10 3
25.6M
650
I3D NL 44
77.7
93.3
128
10 X 3
10,800
CorrNet-101
40
79.2
32
10 3
6,700
ip- Jp-CSN-152
[3
79.2
93.8
8
10 X 3
3,270
LGD-3D R101
[31
79.4
94.4
SlowFast 8 R101+NL
16
78.7
93.5
8
10 X
3,480
SlowFast 16x8 R101+NL
79.8
93.9
16
10
7,020
X3D-XXL 15
80.4
94.6
10 X 3
20.3M
5,823
TimeSformer-L
80.7
94.7
96
X
121M
7,140
ViViT-L/16x2
80.6
94.7
32
4
312M
17,352
X-VIT (Ours)
78.5
93.7
8
1
92M
425
X-VIT (Ours)
79.4
93.9
8
2 3
92M
850
X-VIT (Ours)
80.2
94.7
16
1x 3
92M
850
X-VIT (Ours)
80.7
94.7
16
2 3
92M
1700
the model. For completeness, herein, we study the effect of applying space-time mixing to various combinations for the key, value and to the input token prior to gkv projection. As shown in Table 22c the combination corresponding to our model (i.e. space-time time mixing applied the key and value) significantly outperforms all other variants by up to 2%. This result is important it confirms that our model, derived from the proposed approximation to the local space-time attention, gives best results when compared to other non-well motivated variants.
Effect of amount of space-time mixing: We define as pdh the total number of channels coming from the adjacent frames in the local temporal window [-twe tw] (i.e. E EV= d/ pdl) when
constructing kg .-tuff (see Section 3). Herein, study the effect of p on the model's accuracy. As the results from Table Table show, the optimal is between 25% and 50%. Increasing to 100% (i.e. all channels coming from adjacent frames) unsurprisingly degrades the performance it excludes the case = when performing the self-attention.
Herein, we compare Temporal Attention with Summary token on SSv2 and Kinetics-400. We used both datasets for this case as they require different type of understanding: fine-grained temporal (SSv2) and spatial content (Kinetics-400), From Table 4 we conclude that the Summary token compares favorable on Kinetics-400 but not on SSv2 showing that is more useful in terms of capturing spatial information. Since the improvement is small, conclude that I TA layer is the best global attention-base mechanism for improving the accuracy of our method adding also negligible computational cost.
Effeet of Summary token:
Effect of number of input frames: Herein we evaluate the impact of increasing the number of input frames T from 8 to 16 and 32. We note that, for our method, this change results in linear increase in complexity. As the results from Table show,, increasing the number frames from 8 to offers 1.8% boost in Top-1 accuracy on SSv2. Moreover, increasing the number of frames to 32 improves the performance by a further 0.2%, offering diminishing returns. Similar behavior can observed on Kinetics and Epic- 100 in Tables and

---

Table Comparison with state-of- aef-the-an on the Kinetics-600 dataset. 7' is the number of frames used by our method.
Method
Top-1
Top-5
Views
FLOPs (x109)
AttentionNAS
45
79.8
94.4
1,034
LGD-3D RIO1
33
81.5
95.6
x
SlowFast R R101+NL
[16
81.8
95.1
10 x
3,480
X3D-XL [5]
81.9
95.5
x
1,452
TimeSformer-HIt
82.4
96.0
1x3
5,110
ViViT-L/16x2
82.5
95.6
4
17,352
X-ViT (8x) (Ours)
82.5
95.4
1 1
425
X-VIT (16x) (Ours)
84.5
96.3
1 1
850
Effeet of number of tokens and different model sizes: Herein, we vary the number of input tokens by changing the patch size K. As the results from Sbble]ssow,, even when number of tokens decreases significantly (e.g. ViT-B/32 or ViT-S/32) our approach is still able to produce results of satisfactory accuracy. The benefit that is having model which significantly more efficient. Similar concusions can be observed when the model size (in terms of parameters and FLOPs) is varied. Our approach provides consistent results in all cases, showcasing its ability to scale well from tiny (XViT-T) to large (XViT-L) models.
Latency and throughput considerations. While the channel shifting operation used by the proposed space- time mixing attention is zero-FLOP, there is still small cost associated with memory movement operations. order to ascertain that the induced cost does not introduce noticeable performance degradation, we benchmarked Vit-B/16 (8x x frames) model using spatial-only attention and the proposed space-time mixing attention on 8 V100 GPUs and batch size of 128. A model with spatial-only attention has throughput of 312 fps while our model has 304 fps.
Table Comparison with state-of art art on SSv2. pretrained on Kinetics 600
Method
Top-1
Top-5
# Frames
Views
FLOPs (x109)
TRN 53
48.8
77.6
SlowFast-multigria
46
61.7
1 x 3
TimeSformer-L.
3
62.4
96
1 x
7,140
TSM R50 226
63.3
88.5
16
2
STM 19]
64.2
89.8
MSNet 24
64.7
89.4
16
TEA 25
65.1
89.9
16
ViViT-L/16x2
65.4
89.8
32
4
11,892
X-VIT (Ours)
64.4
89.3
1x3
425
X-ViT (Ours)
66.2
90.6
16
1 1
850
X-ViT* (Ours)
67.2
90.8
16
1 X
850
X-VIT (Ours)
66.4
90.7
32
1
1,270
4.3 Comparison to state-of-the-art
Our best model uses the proposed space-time mixing attention in all the Transformer layers and performs temporal aggregation using single lightweight temporal transformer layer described Section 3 Unless otherwise specified, we report the results using the 3 configuration for the views (1 temporal and spatial) for all datasets. Regarding related work on transformer-bar we included their very best models trained on the same data as our models. For TimeSformer, this is typically the TimeSformer-L. version. For ViVit, we used the 16x2 configuration, with factorized-encoding for Epic-100 100 and SS-v2 (as reported in Tables and in 1 and the full version for Kinetics (as reported in Table 6a in (T).
video
recognition [ 3

---

On Kinetics-400, we match the current ofate-of-ie oftte--f-thh-arre art while having significantly lower computational complexity than the next two best recently proposed methods that also use Transformer- architectures: 20x fewer FLOPs than ViVit [1] and 8x fewer than TimeSformer-I. 3 Note that both models from 1 3 and ours were initialized from a ViT model pretrained on ImageNet-21k [[]]
and take as input frames at resolution of 224 x 224px. Similar conclusions can be drawn from Table@/whice reports our results on Kinetics-600.
On SSv2, we match and surpass the current state-of-the-ar,, especially in terms of Top-5 accuracy (ours: 90.7% vs ViVIT: 89.8% [T] using models that are 14x (16 frames) and 9x (32 frames) faster.
Finally, we observe similar outcomes on Epic-100 where we set a new state-of saate-of-the-ane the-art, showing large improvements especially for "Verb" accuracy, while again being more efficient.
5 Ethical considerations and broader impact
Current nigh--pefforing video recognition mod- els tend to have high computational demands for both training and testing and. by extension, significant environmental costs. This espe- cially true for the transformer- based architec- tures. Our research introduces a novel approach that matches and surpasses the current state-of- the-art while being significantly more efficient thanks to the linear scaling of the complexity with respect to the number of frames. We hope such models will offer noticeable reduction in power consumption while setting at the same time a solid base for future research. We will release code and models to facilitate this. More- over, and similarly most data-driven systems, bias from the training data can potentially affect the fairness of the model. As such, we suggest to take this aspect into consideration when deploying the models into real-world scenarios.
Table 8: Comparison with state-of-the-art on Epic- 100. T' is the #frames used by our method. Re-
sults for other methods are taken from M.
Method
Action
Verb
Noun
TSN
41
33.2
60.2
46.0
TRN
53
35.3
65.9
45.4
TBN
22
36.7
66.0
47.2
TSM
22
38.3
67.9
49.0
SlowFast [16
38.5
65.6
50.0
ViVIT-L/16x2 [I]
44.0
66.4
56.8
X-VIT (8x) (Ours)
41.5
66.7
53.3
X-VIT (16x) (Ours)
44.3
68.7
56.4
6 Conclusions
We presented a novel approximation to the full space-time attention that is amenable to an efficient implementation and applied it to video recognition. Our approximation has the same computational cost as spalial-only attention yet the resulting video Transformer model was shown to be significantly more efficient than recently proposed Video Transformers I. By no means this paper proposes complete solution to video recognition using video Transformers. Future efforts could include combining our approaches with other architectures than the standard ViT, removing the dependency on pre-trained models and applying the model to other video-related tasks like detection and seg- mentation. Finally, further research required for deploying our models on low power/resource devices.
10

---

References
[1] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: video vision transformer. arXiv preprint ar3iv:2 15691, 2021.
[2] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- doc-meetnnnt transformer. arXiv preprint axX1::2004055700, 2020.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05099 2021.
[4] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308,
2017.
[5] Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster. Zhifeng Chen, et al. The best of both worlds: Combining recent advances in neural machine translation. arXiv preprint M.Xiv:880000949 2018.
[6] Yunpeng Chen, Yannis Kalantidis. Jianshu Li, Shuicheng Yan, and Jiashi Feng. A2-nets: Double attention networks. arXiv preprint arXiv:1810. 11579, 2018
[7] Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint 1.XXivI904..100000 2019.
[8] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quac V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.009 0805,0950,, 2018.
[9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari. Evangelos Kazakos. Davide Moltisanti, Jonathan Munro. Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Rescaling egocentric vision. arXiv preprint arXiv:2006. 13256. 2020.
[11] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, lcce, 2009.
[12] Jacob Devlin. Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04900 04,00, 2018.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2:Xxi22000.11222 2020.
[14] Quanfu Fan, Chun-Fu Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation. arXiv preprint arXiv: 1911v:1912.000666 2019.
[15] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 203-213, 2020.
[16] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202-622 6211,11222211 2019.
[17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Hacnel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pages 5842 5850, 2017.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Boyuan Jiang, Meng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2000-2009, 2019.

---

[20] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings the IEEE conference on Computer Vixion and Puttern Recognition, pages 1725-1732, 2014.
[21] Will Kay. Joao Carreira. Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjjaynnarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint urXiv:/705.06950 2017.
[22] Evangelos Kazakos. Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5492-5500 2019.
[23] Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[24] Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. In European Conference on Computer Vision, pages 345-362. Springer, 2020.
[25] Yan Li, Bin Ji, Xintian Shi. Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 909-99999999 918, 2020.
[26] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In P Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083-7093, 2019. 2019.
[27] Zhaoyang Liu, Limin Wang. Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. arXiv preprint (:x.ivv005006000 2020.
[28] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.00 03983,.0333.....03 2016.
[29] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning, pages 4055-4064. PMLR, 2018
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perffrmance deep learning library. arXiv preprint drXiv:1912.01703, 2019.
[31] Zhaofan Qiu, Ting Yao. Chong Wah Ngo. Xinmei Tian, and Tao Mei. Learning pattootemppraa representa- tion with local and global diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12056-12065. 2019.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910.1063: 2019.
[33] Aravind Srinivas. Tsung Yi Lin, Niki Parmar, Jonathon Shlens. Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101.11605, 2021.
[34] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 2826, 2016.
[35] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 1rXiv:2012.12877, 2020.
[36] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 4489-44 4497, 2015.
[37] Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray. Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018
[38] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel-separated convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5552-556 126155611220 2019.
12

---

[39] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 1rXiv:1706.03622. 2017.
[40] Heng Wang, Du Tran, Lorenzo Torresani, and Mait Feiszli. Video modeling with correlation networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 352-361, 2020.
[41] Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool, Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[42] Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks for action recognition videos. IEEE transactions on pattern analysis and machine intelligence, 41(11):2740-275 2018.
[43] Xiaofang Wang. Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua. Attentionnas: Spatiotemporal attention cell search for video classification, In European Conference on Computer Vision, pages 449-465. Springer, 2020.
[44] Xiaolong Wang, Ross Girshick. Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[45] Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative to spatial convolutions. In CVPR, 2018.
[46] Chao- Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 133-162, 2020.
[47] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-15000 1500, 2017.
[48] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. tokens-to-tokee vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101. 1986, 2021.
[49] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
[50] Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pages 7364-7363. PMLR, 2019.
[51] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 17.00997 2017.
[52] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13001- 13008, 2020,
[53] Bolci Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 803-818, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section[5]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
13

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material however all implementation details required to reproduce our work. We will make the
URL)? [No] We include
code and the models available. The datasets used already publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Due to prohibitive costs, we reported the variations between multiple training runs for single 8 frames model in Section 4. This should be representative for the rest of the trained models.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? (Yss)
(b) Did you mention the license of the assets? [Yes] See Section 4.1
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] No data was collected, publicly available datasets were used (e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] To our knowledge the datasets used do contain such info
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "vHC1u7bIRn8", "reviewer": "Reviewer_C6rH", "summary": "This paper presents a video Transformer model for video recognition. It limits the temporal window to the local area and proposes efficient space-time mixing to reduce computational costs. On Kinetics, Something-Something v2, and Epic Kitchens datasets,  the proposed video transformer model is more effective than other transformer-based models yet has the same computational cost as spatial-only attention models.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For originality, this paper extends the mixing technique in TSM to the Transformer based backbone architecture, to obtain a space-time mixing attention in Video Transformer. This combination is straightforward and shown to be effective for video recognition.\n\nThe paper is well written and well organized.\n\nUsing the proposed methods, the paper achieves similar performance with much lower computation costs on Kinetics-400, Something-Something v2, and Epic Kitchens datasets.\n\nConcerns:\n1. The experimental results are not sufficient enough. This paper only presents one version of the X-ViT model with different input frames but did not verify the effectiveness of the proposed approach on different sizes of models. \n2. The novelty is somewhat limited. This paper is more like a combination of TSM and Vision Transformer.\n3. In Table 1, a larger temporal window size would lead to worse performance on video recognition. This may limit the application scenarios and hurt the generality of the proposed approach.\n4. Number of parameters and speed of different models should be compared.\n5. Other approaches could benefit from using more temporal clips, such as 4x3. But the presented approach could only achieve slightly better when using more temporal clips. Please explain why.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "uKI6N71NOMo", "reviewer": "Reviewer_kt96", "summary": "In this paper, the authors proposed a Video Transformer model the complexity of which scales linearly (instead of quadratic) with the number of frames in the video sequence. The key ideas are 1) restricting time attention to a local temporal window, and 2) using efficient space-time mixing to attend jointly spatial and temporal locations. They empirically verified that the proposed method outperforms existing state-of-the-art models in popular action recognition datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is well-written overall, with full details of the model in Section 3 and intuitive demonstration in Figure 1.\n\n2. It looks like t_w, the additional hyper-parameter of local window size, is very important and its optimal value may significantly vary dataset by dataset. Although the authors included an empirical evidence on Table 1, I still believe that this may need to be tuned for other datasets differently. Especially, when the dataset contains videos that are relative longer, and thus containing less homogeneous scenes, the optimal value for t_w may be significantly larger than 1. I guess the optimal value was 1 since SSv2 contains mostly homogeneous and short videos. This interpretation is aligned with the main idea of this work that attention only within a short temporal window is enough, as in Figure 1 (d). It is okay to limit the scope of this work to this kinds of short videos only and action recognition (as opposed to general, topical video classification for longer videos like YouTube 8M or TVR), but the scope still should be mentioned clearly.\n\n3. Other than the point in #2, the experiment was conducted clearly, and shows impressive results on multiple datasets. Ablation studies were also conducted nicely.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned in the main review #2, the proposed approach was verified only on a subclass of video datasets (short videos with homogeneous scenes, focusing on action recognition) but it was slightly over-claimed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "TbfiRkupjbk", "reviewer": "Reviewer_fqvk", "summary": "The authors propose a more efficient self-attention operation, specifically for video models, such that the compute requirements increase linearly with the length of the video. The author's contributions consist of two main things: 1) Self-attention along the temporal axis is limited to a smaller window instead of the whole sequence (with multiple transformer layers, the temporal \"receptive field\" grows to cover the whole video) and 2) The authors efficiently perform \"space-time mixing\" by constructing the keys and values for attention by concatenating features from different frames. This operation requires 0 floating point operations, but is still able to combine temporal information from adjacent frames effectively.\n\nThe authors show good experimental results. They achieve similar results to ViViT [1] and TimeSFormer [3] whilst using far fewer GFLOPs, on Kinetics, SSv2 and Epic Kitchens. The paper is also well written in general.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method appears fairly straightforward to implement, is efficient in terms of GFLOPs and achieves comparable results to ViViT and TimeSFormer whilst using less GFLOPs. However, I do think the novelty is quite limited:\n\n- The idea of restricting self-attention to more localised patterns has been explored before in a number of papers, ie [A, B, C, D] among others. The authors should discuss these papers more.\n- The author's proposed method for efficient \"space-time mixing\" is very similar to that of TSM [42]. Moreover, although I am not aware of other transformer papers that do efficient \"space-time mixing\" in exactly the same way, it does seem quite similar in spirit to Model 4 of Vivit [1], and a discussion of this is completely missing in the paper.\n- On a related note, the authors often refer to recent video-transformer papers [1, 3]. However, both of these papers propose multiple different models, and it's not clear which models in the paper the authors are referring to. For example in Figure 1, the attention pattern for TimeSFormer is their \"Divided Space-Time Attention\", and is also Model 3 of Vivit. On Page 5, when the authors are comparing their \"Temporal Attention aggregation\" to Vivit, this is referring only to Model 2 of [1]. These should be clarified in the revision.\n\n\n\nIn the experiments section, the authors state they match state-of-the-art results while being \"significantly faster\". Actually, the authors only report GFLOPs, so it is not correct to say that they are \"faster\". Rather, the proposed model uses less floating point operations, and so the authors should use more precise language here. (The same applies to the discussion of results in the supplementary) A further point here is that the author's \"shift trick\" does not require any FLOPs, but does increase the runtime. Note that TSM [42] also shows that the impact of the \"shift trick\" on runtime is architecture-specific with many caveats.\n\nWhen the authors reported runtime on Page 8, it is reported with \"frames/second\" as the unit. A better unit would be \"video clips / second\" as that is independent of the number of frames used (as this is a model hyperparamter) and thus more comparable across different methods. Am I correct that as the authors process 8 frames per video, 312 frames/second = 39 video clips / second?\n\nTable 4 would also be more clear if the number of tokens and/or number of frames processed by the model were included as separate columns.\n\n\n\nMinor points\n\n- Table 5: Timesformer and Vivit have the same reference.\n- Line 155 to 156: [1] also shows that L_t = 1 works well, and this is identical to the author's temporal aggregation approach. \n\n\n\nReferences\n\n[A] N Parmar et al. Image Transformer. ICML 2018\n\n[B] R Child et al. Generating long sequences with sparse transformers. ICML 2019\n\n[C] I Betalgy et al. Longformer: The Long-Document Transformer (the authors cited this, but did not discuss it in context)\n\n[D] M Zaheer et al. Big Bird: Transformers for Longer Sequences. NeurIPS 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed in the main paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SvIY55riXCn", "reviewer": "Reviewer_qij4", "summary": "In this paper, the authors focus on reducing the computation cost caused by the space-time attention modeling in the Transformer framework for video recognition. Specifically, the proposal includes 1) a space-time mixing attention module that mixes channels from neighboring frames within a local window as key-value tokens, and 2) a temporal attention layer at the end of the model to aggregate global temporal information. The authors provide promising results on Kinetics-400 and SSv2 in the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** The major proposal is the space-time mixing attention module. The authors introduce the “channel shift trick [42]” to the space-time attention module to aggregate both spatial and temporal features while reducing computation complexity. They have shown promising results (Table 5-6) to support their claims. I think the idea is novel and clearly differs from previous works (TimeSformer [3] and ViViT[3]). \n\n**Quality:** The submission is technically sound. The authors have provided extensive experiments and ablation studies to support their major claims. The weaknesses are list as below. \n1. The authors should provide the typical experimental values of T, S, and d (Line 98-101), otherwise, it may be hard to figure out the dominant computation cost of a full model. For example, when $2d > TS$, the complexity of MLP is larger than MSA. \n2. I’m not sure about the complexity mentioned in Line 129. In my understanding, in the local temporal window of Eq. 7, there are $(2t+1)S$ tokens, and I think the complexity should be $O(T(2t+1)^2S^2) \\sim O(Tt^2S^2)$, rather than $O((2t+1)TS^2)$.\n3. In my understanding, the space-temporal mixing attention mainly aggregates temporal information by aggregating a subset of channels from neighboring frames to constitute the features of a new key-value token (Line 137-140). I think another intuitive option is reducing channels of each frame (e.g., reduce to 1/t channels) and then concatenate them together as the final key-value tokens. Such a design can aggregate temporal information while also avoiding introducing additional overhead. We suggest the authors clarify their differences and prove the superiority of their proposal over the abovementioned option. \n4. I think the results of Table 2 (b) look weird as the results of three different models are totally the same. We suggest the authors carefully check the results and report the error bar of the results. \n5. The authors have highlighted the computation complexity and cost multiple times in the paper. Therefore, I think it is important to provide latency (i.e., runtime), the number of input frames, the number of tokens for all the models in Table 5 and Table 6 to support their claims. \n\n\n**Clarity:** I think this paper is well organized, however, some details of the model are not clear enough. \n1. it’s hard for me to totally understand the summary token (Line 159-166). We suggest the authors provide illustrations of the temporal attention aggregation and summary token to demonstrate their functions and differences. \n2. In Table 2 (d), * (44.2) indicates temporal average aggregation, however, it is reported as 55.6% in Line 241. The authors should check it and clarify it. \n\n**Significance:** This paper proposes a novel way to reduce computation cost for spatial-temporal attention modeling. The authors have shown promising FLOPs and accuracy in standard benchmarks. Latency and more details of Table 5 and Table 6 are expected in the final version. Overall, I think it is useful and helpful to others to follow similar ideas to build transformer networks for video tasks. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is good that the authors have discussed the ethical consideration and broader impact of their model in terms of environmental costs and the bias of data-driven systems. We suggest the authors provide specific cases and more details of the mentioned fairness of their model (in Line 314-315). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel transformer architecture for video recognition whose performance scales linearly with the number of frames (as opposed to the usual quadratic scaling). To achieve this the authors restrict the time attention to a local temporal window, and introduce an efficient space-time mixing procedure. The proposed approach offers competitive results in terms of accuracy/flops tradeoffs on several popular video recognition benchmarks.

The paper was reviewed by 4 expert reviewers and received borderline ratings. The rebuttal managed to address points raised by most reviewers, who found it a valuable contribution to the community. One reviewer maintains the view that the work lacks novelty in terms of technical contributions, and that more ablation studies are necessary. After considering the manuscript, the reviews, and the discussion, I felt that the work should be accepted for publication and that a minor revision is sufficient to address the raised criticisms. 
Decision: Accept (Poster)