Summary:
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.

Gedas Bertasius
Heng Wang
Lorenzo Torresani

Accepted to ICML 2021

Is Space-Time Attention All You Need for Video Understanding?
Gedas Bertasius1Heng Wang1Lorenzo Torresani1 2
Abstract
We present a convolution-free approach to video
classiﬁcation built exclusively on self-attention
over space and time. Our method, named “TimeS-
former,” adapts the standard Transformer archi-
tecture to video by enabling spatiotemporal fea-
ture learning directly from a sequence of frame-
level patches. Our experimental study com-
pares different self-attention schemes and sug-
gests that “divided attention,” where temporal
attention and spatial attention are separately ap-
plied within each block, leads to the best video
classiﬁcation accuracy among the design choices
considered. Despite the radically new design,
TimeSformer achieves state-of-the-art results on
several action recognition benchmarks, includ-
ing the best reported accuracy on Kinetics-400
and Kinetics-600. Finally, compared to 3D
convolutional networks, our model is faster to
train, it can achieve dramatically higher test
efﬁciency (at a small drop in accuracy), and
it can also be applied to much longer video
clips (over one minute long). Code and mod-
els are available at: https://github.com/
facebookresearch/TimeSformer .
1. Introduction
Over the last few years, the ﬁeld of natural language pro-
cessing (NLP) has been revolutionized by the emergence of
methods based on self-attention (Vaswani et al., 2017a). Be-
cause of their excellent capabilities at capturing long-range
dependencies among words as well as their training scala-
bility, self-attention architectures, such as the Transformer
model, represent the current state-of-the-art across a wide
range of language tasks, including machine translation (Ott
et al., 2018; Chen et al., 2018a), question answering (De-
vlin et al., 2019; Dai et al., 2019), and autoregressive word
generation (Radford et al., 2019; Brown et al., 2020).
Video understanding shares several high-level similarities
1Facebook AI2Dartmouth College. Correspondence to: Gedas
Bertasius <gberta@seas.upenn.edu>.with NLP. First of all, videos and sentences are both sequen-
tial. Furthermore, precisely as the meaning of a word can
often be understood only by relating it to the other words in
the sentence, it may be argued that atomic actions in short-
term segments need to be contextualized with the rest of the
video in order to be fully disambiguated. Thus, one would
expect the long-range self-attention models from NLP to be
highly effective for video modeling as well. However, in the
video domain, 2D or 3D convolutions still represent the core
operators for spatiotemporal feature learning across differ-
ent video tasks (Feichtenhofer et al., 2019a; Teed & Deng,
2020; Bertasius & Torresani, 2020). While self-attention
has shown beneﬁts when applied on top of convolutional
layers (Wang et al., 2018a), to the best of our knowledge, no
attempt to use self-attention as the exclusive building block
for video recognition models has been reported.
In this work we pose the question of whether it may be
possible to build a performant convolution-free video archi-
tecture by replacing altogether the convolution operator with
self-attention. We argue that such a design has the poten-
tial to overcome a few inherent limitations of convolutional
models for video analysis. First, while their strong inductive
biases (e.g., local connectivity and translation equivariance)
are undoubtedly beneﬁcial on small training sets, they may
excessively limit the expressivity of the model in settings
where there is ample availability of data and “all” can be
learned from examples. Compared to CNNs, Transformers
impose less restrictive inductive biases. This broadens the
family of functions they can represent (Cordonnier et al.,
2020; Zhao et al., 2020), and renders them better suited
to modern big-data regimes where there is less need for
strong inductive priors. Second, while convolutional kernels
are speciﬁcally designed to capture short-range spatiotem-
poral information, they cannot model dependencies that
extend beyond the receptive ﬁeld. While deep stacks of
convolutions (Simonyan & Zisserman, 2015; Szegedy et al.,
2015; Carreira & Zisserman, 2017) naturally extend the
receptive ﬁeld, these strategies are inherently limited in cap-
turing long-range dependencies by means of aggregation
of shorter-range information. Conversely, the self-attention
mechanism can be applied to capture both local as well as
global long-range dependencies by directly comparing fea-
ture activations at all space-time locations, much beyond the
receptive ﬁeld of traditional convolutional ﬁlters. Finally,arXiv:2102.05095v4  [cs.CV]  9 Jun 2021
Is Space-Time Attention All You Need for Video Understanding?
despite the advances in GPU hardware acceleration, training
deep CNNs remains very costly, especially when applied to
high-resolution and long videos. Recent work in the still-
image domain (Dosovitskiy et al., 2020; Carion et al., 2020;
Zhao et al., 2020) has demonstrated that Transformers enjoy
faster training and inference compared to CNNs, making it
possible to construct models with larger learning capacity
for the same computational budget.
Motivated by these observations, we propose a video ar-
chitecture built exclusively on self-attention. We adapt the
image model “Vision Transformer” (ViT) (Dosovitskiy et al.,
2020) to video by extending the self-attention mechanism
from the image space to the space-time 3D volume. Our
proposed model, named “TimeSformer” (from Time-Space
Transformer), views the video as a sequence of patches ex-
tracted from the individual frames. As in ViT, each patch is
linearly mapped into an embedding and augmented with po-
sitional information. This makes it possible to interpret the
resulting sequence of vectors as token embeddings which
can be fed to a Transformer encoder, analogously to the
token features computed from words in NLP.
One downside of self-attention in standard Transformer is
that it requires computing a similarity measure for all pairs
of tokens. In our setting, this is computationally costly due
to the large number of patches in the video. To address these
challenges, we propose several scalable self-attention de-
signs over the space-time volume and empirically evaluate
them over large-scale action classiﬁcation datasets. Among
the proposed schemes, we found that the best design is repre-
sented by a “divided attention” architecture which separately
applies temporal attention and spatial attention within each
block of the network. Compared to the established paradigm
of convolution-based video architecture, TimeSformer fol-
lows a radically different design. Yet, it achieves accuracy
comparable, and in some cases superior, to the state-of-the-
art in this ﬁeld. We also show that our model can be used
for long-range modeling of videos spanning many minutes.
2. Related Work
Our approach is inﬂuenced by recent works that use self-
attention for image classiﬁcation, either in combination with
the convolution operator or even as a full replacement for it.
Within the former class, Non-Local Networks (Wang et al.,
2018b) employ a non-local mean that effectively generalizes
the self-attention function of Transformers (Vaswani et al.,
2017b). Bello et al. (Bello et al., 2019) propose a 2D self-
attention mechanism that is competitive as a replacement of
2D convolution but gives even stronger results when used to
augment convolutional features with self-attention features.
Beyond image categorization, Relation Networks (Hu et al.,
2018) and DETR (Carion et al., 2020) use self-attention on
top of convolutional feature maps for object detection.Our method is more closely related to image networks lever-
aging self-attention as a substitute for convolution (Parmar
et al., 2018; Ramachandran et al., 2019; Cordonnier et al.,
2020; Zhao et al., 2020). Since these works use individual
pixels as queries, in order to maintain a manageable compu-
tational cost and a small memory consumption, they must
restrict the scope of self-attention to local neighborhoods or
use global self-attention on heavily downsized versions of
the image. Alternative strategies for scalability to full im-
ages include sparse key-value sampling (Child et al., 2019)
or constraining the self-attention to be calculated along the
spatial axes (Ho et al., 2019; Huang et al., 2019; Wang
et al., 2020b). A few of the self-attention operators con-
sidered in our experiments adopt similar sparse and axial
computation, although generalized to the spatiotemporal
volume. However, the efﬁciency of our approach stems
mainly from decomposing the video into a sequence of
frame-level patches and then feeding linear embeddings of
these patches as input token embeddings to a Transformer.
This strategy was recently introduced in Vision Transform-
ers (ViT) (Dosovitskiy et al., 2020) which were shown to
deliver impressive performance on image categorization. In
this work, we build on the ViT design, and extend it to video
by proposing and empirically comparing several scalable
schemes for space-time self-attention over videos.
While Transformers have been recently used for video gen-
eration (Weissenborn et al., 2020), we are not aware of prior
video recognition architectures using self-attention as the
exclusive building block. However, we note that Trans-
formers have been adopted on top of convolutional feature
maps for action localization and recognition (Girdhar et al.,
2019), video classiﬁcation (Wang et al., 2018b; Chen et al.,
2018b), and group activity recognition (Gavrilyuk et al.,
2020). We also note that there is a wide literature based on
the use of text Transformers combined with video CNNs
to address various video-language tasks, such as caption-
ing (Zhou et al., 2018), question-answering (Yang et al.,
2020) and dialog (Le et al., 2019). Finally, multimodal
video-text transformers (Sun et al., 2019; Li et al., 2020a)
have also been trained or pretrained in unsupervised fashion
by adopting masked-token pretext tasks adapted from the
language domain (Devlin et al., 2018; Radford et al., 2018).
3. The TimeSformer Model
Input clip. The TimeSformer takes as input a clip X2
RHW3Fconsisting of FRGB frames of size HW
sampled from the original video.
Decomposition into patches. Following the ViT (Doso-
vitskiy et al., 2020), we decompose each frame into N
non-overlapping patches, each of size PP, such that
theNpatches span the entire frame, i.e., N=HW=P2.
We ﬂatten these patches into vectors x(p;t)2R3P2with
Is Space-Time Attention All You Need for Video Understanding?
<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>Time Att.Space Att.<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>MLP<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>Space Att.MLP<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>Joint Space-Time Att.<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>
Space Attention (S)  Joint Space-Time      Attention (ST)Divided Space-Time     Attention (T+S)<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>Time Att.Width Att.<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>
MLP<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>Height Att.<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>
Axial Attention      (T+W+H)Sparse Local Global     Attention (L+G)MLPLocal Att.Global Att.MLPz(`)
<latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit>z(`1)
<latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit>
z(`)
<latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit>z(`)
<latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit>z(`)
<latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit>z(`)
<latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit><latexit sha1_base64="vr9M27hwkmJn9HyewcyBFvyrmCg=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahXkoigh6LXjxWsB/QxLLZTtqlm03Y3RRqyD/x4kERr/4Tb/4bt20O2vpg4PHeDDPzgoQzpR3n2yqtrW9sbpW3Kzu7e/sH9uFRW8WppNCiMY9lNyAKOBPQ0kxz6CYSSBRw6ATj25nfmYBULBYPepqAH5GhYCGjRBupb9uZF4T4KX/Mah5wfp737apTd+bAq8QtSBUVaPbtL28Q0zQCoSknSvVcJ9F+RqRmlENe8VIFCaFjMoSeoYJEoPxsfnmOz4wywGEsTQmN5+rviYxESk2jwHRGRI/UsjcT//N6qQ6v/YyJJNUg6GJRmHKsYzyLAQ+YBKr51BBCJTO3YjoiklBtwqqYENzll1dJ+6LuOnX3/rLauCniKKMTdIpqyEVXqIHuUBO1EEUT9Ixe0ZuVWS/Wu/WxaC1Zxcwx+gPr8wcJUZNB</latexit>z(`1)
<latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit>z(`1)
<latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit>z(`1)
<latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit>z(`1)
<latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit><latexit sha1_base64="PqFOkq34IvzyoSmk0/rjIYa/Lb0=">AAAB+3icbVBNS8NAEN3Ur1q/Yj16WSxCPVgSEfRY9OKxgv2ANpbNdtIu3WzC7kasIX/FiwdFvPpHvPlv3LY5aOuDgcd7M8zM82POlHacb6uwsrq2vlHcLG1t7+zu2fvllooSSaFJIx7Jjk8UcCagqZnm0IklkNDn0PbH11O//QBSsUjc6UkMXkiGggWMEm2kvl1Oe36An7L7tNoDzk/dk6xvV5yaMwNeJm5OKihHo29/9QYRTUIQmnKiVNd1Yu2lRGpGOWSlXqIgJnRMhtA1VJAQlJfObs/wsVEGOIikKaHxTP09kZJQqUnom86Q6JFa9Kbif1430cGllzIRJxoEnS8KEo51hKdB4AGTQDWfGEKoZOZWTEdEEqpNXCUTgrv48jJpndVcp+benlfqV3kcRXSIjlAVuegC1dENaqAmougRPaNX9GZl1ov1bn3MWwtWPnOA/sD6/AHtlpOz</latexit>
Figure 1. The video self-attention blocks that we investigate in this work. Each attention layer implements self-attention (Vaswani et al.,
2017b) on a speciﬁed spatiotemporal neighborhood of frame-level patches (see Figure 2 for a visualization of the neighborhoods). We use
residual connections to aggregate information from different attention layers within each block. A 1-hidden-layer MLP is applied at the
end of each block. The ﬁnal model is constructed by repeatedly stacking these blocks on top of each other.
p= 1;:::;N denoting spatial locations and t= 1;:::;F
depicting an index over frames.
Linear embedding. We linearly map each patch x(p;t)into
an embedding vector z(0)
(p;t)2RDby means of a learnable
matrixE2RD3P2:
z(0)
(p;t)=Ex(p;t)+epos
(p;t)(1)
where epos
(p;t)2RDrepresents a learnable positional embed-
ding added to encode the spatiotemporal position of each
patch. The resulting sequence of embedding vectors z(0)
(p;t)
forp= 1;:::;N , andt= 1;:::;F represents the input to
the Transformer, and plays a role similar to the sequences of
embedded words that are fed to text Transformers in NLP.
As in the original BERT Transformer (Devlin et al., 2018),
we add in the ﬁrst position of the sequence a special learn-
able vector z(0)
(0;0)2RDrepresenting the embedding of the
classiﬁcation token.
Query-Key-Value computation. Our Transformer consists
ofLencoding blocks. At each block `, a query/key/value
vector is computed for each patch from the representation
z(`1)
(p;t)encoded by the preceding block:
q(`;a)
(p;t)=W(`;a)
QLN
z(`1)
(p;t)
2RDh(2)
k(`;a)
(p;t)=W(`;a)
KLN
z(`1)
(p;t)
2RDh(3)
v(`;a)
(p;t)=W(`;a)
VLN
z(`1)
(p;t)
2RDh(4)where LN() denotes LayerNorm (Ba et al., 2016), a=
1;:::;Ais an index over multiple attention heads and A
denotes the total number of attention heads. The latent
dimensionality for each attention head is set to Dh=D=A.
Self-attention computation. Self-attention weights are
computed via dot-product. The self-attention weights
(`;a)
(p;t)2RNF+1for query patch (p;t)are given by:
(`;a)
(p;t)= SM0
@q(`;a)
(p;t)pDh>
"
k(`;a)
(0;0)n
k(`;a)
(p0;t0)o
p0=1;:::;N
t0=1;:::;F#1
A
(5)
where SMdenotes the softmax activation function. Note
that when attention is computed over one dimension only
(e.g., spatial-only or temporal-only), the computation is
signiﬁcantly reduced. For example, in the case of spatial
attention, only N+ 1 query-key comparisons are made,
using exclusively keys from the same frame as the query:
(`;a)space
(p;t)= SM0
@q(`;a)
(p;t)pDh>

k(`;a)
(0;0)n
k(`;a)
(p0;t)o
p0=1;:::;N1
A:
(6)
Encoding. The encoding z(`)
(p;t)at block`is obtained by
ﬁrst computing the weighted sum of value vectors using
self-attention coefﬁcients from each attention head:
Is Space-Time Attention All You Need for Video Understanding?
frame tframe t -
<latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit>frame t +
<latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit>Space Attention (S)   Joint Space-Time       Attention (ST)Divided Space-Time      Attention (T+S)  Sparse Local Global        Attention (L+G)  Axial Attention        (T+W+H)
Figure 2. Visualization of the ﬁve space-time self-attention schemes studied in this work. Each video clip is viewed as a sequence of
frame-level patches with a size of 1616pixels. For illustration, we denote in blue the query patch and show in non-blue colors its
self-attention space-time neighborhood under each scheme. Patches without color are not used for the self-attention computation of the
blue patch. Multiple colors within a scheme denote attentions separately applied along different dimensions (e.g., space and time for
(T+S)) or over different neighborhoods (e.g., for (L+G)). Note that self-attention is computed for every single patch in the video clip, i.e.,
every patch serves as a query. We also note that although the attention pattern is shown for only two adjacent frames, it extends in the
same fashion to all frames of the clip.
s(`;a)
(p;t)=(`;a)
(p;t);(0;0)v(`;a)
(0;0)+NX
p0=1FX
t0=1(`;a)
(p;t);(p0;t0)v(`;a)
(p0;t0):
(7)
Then, the concatenation of these vectors from all heads
is projected and passed through an MLP, using residual
connections after each operation:
z0(`)
(p;t)=WO2
664s(`;1)
(p;t)
...
s(`;A)
(p;t)3
775+z(`1)
(p;t)(8)
z(`)
(p;t)= MLP
LN
z0(`)
(p;t)
+z0(`)
(p;t): (9)
Classiﬁcation embedding. The ﬁnal clip embedding is
obtained from the ﬁnal block for the classiﬁcation token:
y= LN
z(L)
(0;0)
2RD: (10)
On top of this representation we append a 1-hidden-layer
MLP, which is used to predict the ﬁnal video classes.
Space-Time Self-Attention Models. We can reduce the
computational cost by replacing the spatiotemporal atten-
tion of Eq. 5 with spatial attention within each frame only
(Eq. 6). However, such a model neglects to capture temporaldependencies across frames. As shown in our experiments,
this approach leads to degraded classiﬁcation accuracy com-
pared to full spatiotemporal attention, especially on bench-
marks where strong temporal modeling is necessary.
We propose a more efﬁcient architecture for spatiotemporal
attention, named “Divided Space-Time Attention” (denoted
with T+S), where temporal attention and spatial attention
are separately applied one after the other. This architecture
is compared to that of Space and Joint Space-Time attention
in Fig. 1. A visualization of the different attention models
on a video example is given in Fig. 2. For Divided Attention,
within each block `, we ﬁrst compute temporal attention by
comparing each patch (p;t)with all the patches at the same
spatial location in the other frames:
(`;a)time
(p;t)= SM0
@q(`;a)
(p;t)pDh>

k(`;a)
(0;0)n
k(`;a)
(p;t0)o
t0=1;:::;F1
A:
(11)
The encoding z0(`)time
(p;t)resulting from the application of
Eq. 8 using temporal attention is then fed back for spatial
attention computation instead of being passed to the MLP. In
other words, new key/query/value vectors are obtained from
z0(`)time
(p;t)and spatial attention is then computed using Eq. 6.
Finally, the resulting vector z0(`)space
(p;t)is passed to the MLP
of Eq. 9 to compute the ﬁnal encoding z(`)
(p;t)of the patch at
block`. For the model of divided attention, we learn dis-
tinct query/key/value matrices fW(`;a)
Qtime;W(`;a)
Ktime;W(`;a)
Vtimeg
Is Space-Time Attention All You Need for Video Understanding?
Attention Params K400 SSv2
Space 85.9M 76.9 36.6
Joint Space-Time 85.9M 77.4 58.5
Divided Space-Time 121.4M 78.0 59.5
Sparse Local Global 121.4M 75.9 56.3
Axial 156.8M 73.5 56.2
Table 1. Video-level accuracy for different space-time attention
schemes in TimeSformer. We evaluate the models on the valida-
tion sets of Kinetics-400 (K400), and Something-Something-V2
(SSv2). We observe that divided space-time attention achieves the
best results on both datasets.
andfW(`;a)
Qspace;W(`;a)
Kspace;W(`;a)
Vspacegover the time and space
dimensions. Note that compared to the (NF+ 1) compar-
isons per patch needed by the joint spatiotemporal attention
model of Eq. 5, Divided Attention performs only (N+F+2)
comparisons per patch. Our experiments demonstrate that
this space-time factorization is not only more efﬁcient but it
also leads to improved classiﬁcation accuracy.
We have also experimented with a “Sparse Local Global”
(L+G) and an “Axial” (T+W+H) attention models. Their
architectures are illustrated in Fig. 1, while Fig. 2 shows
the patches considered for attention by these models. For
each patch (p;t), (L+G) ﬁrst computes a local attention by
considering the neighboring FH=2W=2patches and
then calculates a sparse global attention over the entire clip
using a stride of 2 patches along the temporal dimension and
also the two spatial dimensions. Thus, it can be viewed as a
faster approximation of full spatiotemporal attention using a
local-global decomposition and a sparsity pattern, similar to
that used in (Child et al., 2019). Finally, “Axial” attention
decomposes the attention computation in three distinct steps:
over time, width and height. A decomposed attention over
the two spatial axes of the image was proposed in (Ho
et al., 2019; Huang et al., 2019; Wang et al., 2020b) and
our (T+W+H) adds a third dimension (time) for the case
of video. All these models are implemented by learning
distinct query/key/value matrices for each attention step.
4. Experiments
We evaluate TimeSformer on four popular action recogni-
tion datasets: Kinetics-400 (Carreira & Zisserman, 2017),
Kinetics-600 (Carreira et al., 2018), Something-Something-
V2 (Goyal et al., 2017b), and Diving-48 (Li et al., 2018). We
adopt the “Base” ViT architecture (Dosovitskiy et al., 2020)
pretrained on either ImageNet-1K or ImageNet-21K (Deng
et al., 2009), as speciﬁed for each experiment. Unless dif-
ferently indicated, we use clips of size 8224224, with
frames sampled at a rate of 1=32. The patch size is 1616
pixels. During inference, unless otherwise noted, we sam-
ple a single temporal clip in the middle of the video. We
use3spatial crops (top-left, center, bottom-right) from the
temporal clip and obtain the ﬁnal prediction by averaging
the scores for these 3crops.
8 32 64 96# of Input frames0510TFLOPsJoint Space-TimeDivided Space-Time224 336 448 560Spatial Crop (Px)0123TFLOPsJoint Space-TimeDivided Space-Time
Out of memory
Out of memoryFigure 3. We compare the video classiﬁcation cost (in TFLOPs) of
Joint Space-Time versus Divided Space-Time attention. We plot
the number of TFLOPs as a function of spatial crop size in pixels
(left), and the number of input frames (right). As we increase the
spatial resolution (left), or the video length (right), our proposed di-
vided space-time attention leads to dramatic computational savings
compared to the scheme of joint space-time attention.
4.1. Analysis of Self-Attention Schemes
For this ﬁrst set of experiments we start from a ViT pre-
trained on ImageNet-21K. In Table 1, we present the results
obtained with TimeSformer for the ﬁve proposed space-time
attention schemes on Kinetics-400 (K400) and Something-
Something-V2 (SSv2). First, we note that TimeSformer
with space-only attention (S) performs well on K400. This
is an interesting ﬁnding. Indeed, prior work (Sevilla-Lara
et al., 2021) has shown that on K400, spatial cues are more
important than temporal information in order to achieve
strong accuracy. Here, we show that it is possible to obtain
solid accuracy on K400 without any temporal modeling.
Note, however, that space-only attention performs poorly on
SSv2. This stresses the importance of temporal modeling
on this latter dataset.
Furthermore, we observe that divided space-time attention
achieves the best accuracy on both K400 and SSv2. This
makes sense because compared to joint space-time attention,
divided space-time attention has a larger learning capacity
(see Table 1) as it contains distinct learning parameters for
temporal attention and spatial attention.
In Figure 3, we also compare the computational cost of joint
space-time versus divided space-time attention when using
higher spatial resolution (left) and longer (right) videos. We
note that the scheme of divided space-time scales gracefully
under both of these settings. In contrast, the scheme of
joint space-time attention leads to a dramatically higher cost
when resolution or video length is increased. In practice,
joint space-time attention causes a GPU memory overﬂow
once the spatial frame resolution reaches 448pixels, or once
the number of frames is increased to 32and thus it is effec-
tively not applicable to large frames or long videos. Thus,
despite a larger number of parameters, divided space-time
attention is more efﬁcient than joint space-time attention
when operating on higher spatial resolution, or longer videos.
Thus, for all subsequent experiments we use a TimeSformer
constructed with divided space-time self-attention blocks.
Is Space-Time Attention All You Need for Video Understanding?
Model Pretrain K400 Training
Time (hours)K400
Acc.Inference
TFLOPsParams
I3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M
I3D 8x8 R50 ImageNet-1K 1440 73.4 1.11 28.0M
SlowFast R50 ImageNet-1K 448 70.0 1.97 34.6M
SlowFast R50 ImageNet-1K 3840 75.6 1.97 34.6M
SlowFast R50 N/A 6336 76.4 1.97 34.6M
TimeSformer ImageNet-1K 416 75.8 0.59 121.4M
TimeSformer ImageNet-21K 416 78.0 0.59 121.4M
Table 2. Comparing TimeSformer to SlowFast and I3D. We ob-
serve that TimeSformer has lower inference cost despite having
a larger number of parameters. Furthermore, the cost of training
TimeSformer on video data is much lower compared to SlowFast
and I3D, even when all models are pretrained on ImageNet-1K.
4.2. Comparison to 3D CNNs
In this subsection we perform an empirical study aimed
at understanding the distinguishing properties of TimeS-
former compared to 3D convolutional architectures, which
have been the prominent approach to video understanding
in recent years. We focus our comparison on two 3D CNN
models: 1) SlowFast (Feichtenhofer et al., 2019b), which is
the state-of-the-art in video classiﬁcation, and 2) I3D (Car-
reira & Zisserman, 2017), which has been shown to beneﬁt
from image-based pretraining, similarly to our own model.
We present quantitative comparisons to these two networks
in Table 2 and highlight key observations below.
Model Capacity. From Table 2, we ﬁrst observe that al-
though TimeSformer has a large learning capacity (the num-
ber of parameters is 121:4M), it has low inference cost
(0:59in TFLOPs). In contrast, SlowFast 8x8 R50 has a
larger inference cost ( 1:97TFLOPs) despite containing only
34:6Mparameters. Similarly, I3D 8x8 R50 also has a larger
inference cost ( 1:11TFLOPs) despite containing fewer pa-
rameters ( 28:0M). This suggests that TimeSformer is better
suited for settings that involve large-scale learning. In con-
trast, the large computational cost of modern 3D CNNs
makes it difﬁcult to further increase their model capacity
while also maintaining efﬁciency.
Video Training Time. One signiﬁcant advantage of Ima-
geNet pretraining is that it enables very efﬁcient training of
TimeSformer on video data. Conversely, state-of-the-art 3D
CNNs are much more expensive to train even if pretrained
on image datasets. In Table 2, we compare the video train-
ing time on Kinetics-400 (in Tesla V100 GPU hours) of
TimeSformer to that of SlowFast and I3D. Starting from a
ResNet50 pretrained on ImageNet-1K, SlowFast 88R50
requires 3 840 Tesla V100 GPU hours in order to reach an
accuracy of 75:6%on Kinetics-400. Training I3D, under
similar settings, requires 1 440 Tesla V100 GPU hours for a
73:4%accuracy. In contrast, TimeSformer, also pretrained
on ImageNet-1K, only requires 416Tesla V100 GPU hours
to achieve a higher 75:8%accuracy (see Table 2). Fur-
thermore, if we constrain SlowFast to be trained under a
somewhat similar computational budget as TimeSformerMethod Pretraining K400 SSv2
TimeSformer ImageNet-1K 75.8 59.5
TimeSformer ImageNet-21K 78.0 59.5
TimeSformer-HR ImageNet-1K 77.8 62.2
TimeSformer-HR ImageNet-21K 79.7 62.5
TimeSformer-L ImageNet-1K 78.1 62.4
TimeSformer-L ImageNet-21K 80.7 62.3
Table 3. Comparing the effectiveness of ImageNet-1K and
ImageNet-21K pretraining on Kinetics-400 (K400) and Something-
Something-V2 (SSv2). On K400, ImageNet-21K pretraining leads
consistently to a better performance compared to ImageNet-1K pre-
training. On SSv2, ImageNet-1K and ImageNet-21K pretrainings
lead to similar accuracy.
(i.e.,448GPU hours), its accuracy drops to 70:0%. Simi-
larly, training I3D using a similar computational budget (i.e.,
444GPU hours) leads to a lower accuracy of 71:0%. This
highlights the fact that some of the latest 3D CNNs (Feicht-
enhofer et al., 2019b; Feichtenhofer, 2020) require a very
long optimization schedule to achieve good performance
(even when using ImageNet pretraining). In contrast, TimeS-
former provides a more efﬁcient alternative to labs that do
not have access to hundreds of GPUs.
The Importance of Pretraining. Due to a large number
of parameters, training our model from scratch is difﬁcult.
Thus, before training TimeSformer on video data, we ini-
tialize it with weights learned from ImageNet. In contrast,
SlowFast can be learned on video data from scratch although
at the expense of a very high training cost (see Table 2). We
also attempted to train TimeSformer on Kinetics-400 di-
rectly, without any ImageNet pretraining. By using a longer
training schedule and more data augmentations, we found
it possible to train the model from scratch, albeit to a much
lower video level accuracy of 64:8%. Thus, based on these
results, for all subsequent studies we continued to use Ima-
geNet for pretraining (Deng et al., 2009)
In Table 3 we study the beneﬁts of ImageNet-1K vs
ImageNet-21K pretraining on K400 and SSv2. For these
experiments, we use three variants of our model: (1) TimeS-
former , which is the default version of our model operating
on8224224video clips, (2) TimeSformer-HR , a high
spatial resolution variant that operates on 16448448
video clips, and lastly (3) TimeSformer-L , a long-range
conﬁguration of our model that operates on 96224224
video clips with frames sampled at a rate of 1=4.
Based on the results in Table 3, we observe that ImageNet-
21K pretraining is beneﬁcial for K400, where it leads to
a consistently higher accuracy compared to ImageNet-1K
pretraining. On the other hand, on SSv2, we observe that
ImageNet-1K and ImageNet-21K pretrainings lead to simi-
lar accuracy. This makes sense as SSv2 requires complex
spatiotemporal reasoning, whereas K400 is biased more to-
wards spatial scene information, and thus, it beneﬁts more
from the features learned on the larger pretraining dataset.
Is Space-Time Attention All You Need for Video Understanding?
60K 120K 180K 240K# of Training Videos60657075AccuracyKinetics
TimeSformerSlowFastI3D42K 85K 127K 170K# of Training Videos354045505560AccuracySomething-Something-V2
TimeSformerSlowFastI3D
Figure 4. Accuracy on Kinetics-400 (K400), and Something-
Something-V2 (SSv2) as a function of the number of training
videos. On K400, TimeSformer performs best in all cases. On
SSv2, which requires more complex temporal reasoning, TimeS-
former outperforms the other models only when using enough
training videos. All models are pretrained on ImageNet-1K.
The Impact of Video-Data Scale. To understand the
effects of video-data scale on performance, we trained
TimeSformer on different subsets of K400 and SSv2:
f25%;50%;75%;100%gof the full datasets. We show
these results in Figure 4, where we also compare our method
with SlowFast R50 (Feichtenhofer et al., 2019b), and I3D
R50 (Carreira & Zisserman, 2017) trained on the same sub-
sets and using the same pretraining. Since we do not have
access to a ResNet pretrained on ImageNet-21K, we use
ImageNet-1K pretraining for all 3architectures.
The results of Figure 4 show that, on K400, TimeSformer
outperforms the other models for all training subsets. How-
ever, we observe a different trend on SSv2, where TimeS-
former is the strongest model only when trained on 75% or
100% of the full data. This may be explained by the fact that
compared to K400, SSv2 requires learning more complex
temporal patterns, and thus more examples are needed by
TimeSformer to learn effectively those patterns.
4.3. Varying the Number of Tokens
The scalability of our model allows it to operate at higher
spatial resolution and on longer videos compared to most 3D
CNNs. We note that both of these aspects affect the length of
the sequence of tokens fed to the Transformer. Speciﬁcally,
increasing the spatial resolution results in a higher number
of patches (N) per frame. The number of input tokens is
also increased when using more frames. To investigate the
beneﬁts, we conduct an empirical study where we separately
increase the number of tokens along each of these two axes.
We report the ﬁndings in Figure 5. We see that increasing
the spatial resolution (up to a certain point) leads to a boost
in performance. Similarly, we observe that increasing the
length of the input clip leads to consistent accuracy gains.
Due to GPU memory constraints, we are not able to test
our model on clips longer than 96frames. Still, we would
like to point out that using clips of 96frames is a signiﬁcant
departure from current convolutional models, which are
typically limited to processing inputs of 832frames.
8 32 64 96Number of Input Frames65707580Clip Accuracy224 336 448 560Spatial Crop Side (Px)65707580Clip AccuracyFigure 5. Clip-level accuracy on Kinetics-400 as a function of spa-
tial crop size in pixels (left), and the number of input frames (right).
Positional Embedding K400 SSv2
None 75.4 45.8
Space-only 77.8 52.5
Space-Time 78.0 59.5
Table 4. Ablation on positional embeddings. The version of TimeS-
former using space-time positional embeddings yields the highest
accuracy on both Kinetics-400 and SSv2.
4.4. The Importance of Positional Embeddings
To investigate the importance of our learned spatiotempo-
ral positional embeddings, we also conduct experiments
with a few variants of TimeSformer that use: (1) no po-
sitional embedding, (2) space-only positional embedding,
and (3) space-time positional embedding. We report these
results in Table 4. Based on these results, we observe
that the variant of our model that uses space-time posi-
tional embeddings produces the best accuracy on both
Kinetics-400, and Something-Something-V2. Interestingly,
we also observe that using space-only positional embed-
dings leads to solid results on Kinetics-400, but much worse
results on Something-Something-V2. This makes sense as
Kinetics-400 is more spatially biased, whereas Something-
Something-V2 requires complex temporal reasoning.
4.5. Comparison to the State-of-the-Art
Kinetics-400 & Kinetics-600. In Table 5 we present our
results on the validation set of K400. For these experiments,
we use TimeSformer pretrained on ImageNet-21K. In ad-
dition to the accuracy metrics, we also include inference
cost, given in TFLOPs. We note that whereas most previous
Method Top-1 Top-5 TFLOPs
R(2+1)D (Tran et al., 2018) 72.0 90.0 17.5
bLVNet (Fan et al., 2019) 73.5 91.2 0.84
TSM (Lin et al., 2019) 74.7 N/A N/A
S3D-G (Xie et al., 2018) 74.7 93.4 N/A
Oct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84
D3D (Stroud et al., 2020) 75.9 N/A N/A
I3D+NL (Wang et al., 2018b) 77.7 93.3 10.8
ip-CSN-152 (Tran et al., 2019) 77.8 92.8 3.2
CorrNet (Wang et al., 2020a) 79.2 N/A 6.7
LGD-3D-101 (Qiu et al., 2019) 79.4 94.4 N/A
SlowFast (Feichtenhofer et al., 2019b) 79.8 93.9 7.0
X3D-XXL (Feichtenhofer, 2020) 80.4 94.6 5.8
TimeSformer 78.0 93.7 0.59
TimeSformer-HR 79.7 94.4 5.11
TimeSformer-L 80.7 94.7 7.14
Table 5. Video-level accuracy on Kinetics-400.
Is Space-Time Attention All You Need for Video Understanding?
Method Top-1 Top-5
I3D-R50+Cell (Wang et al., 2020c) 79.8 94.4
LGD-3D-101 (Qiu et al., 2019) 81.5 95.6
SlowFast (Feichtenhofer et al., 2019b) 81.8 95.1
X3D-XL (Feichtenhofer, 2020) 81.9 95.5
TimeSformer 79.1 94.4
TimeSformer-HR 81.8 95.8
TimeSformer-L 82.2 95.6
Table 6. Video-level accuracy on Kinetics-600.
1 3 5 10
# of Testing Clips65707580Accuracy
TimeSformer-L
TimeSformer
SlowFast-R101+NL
X3D-XL
Figure 6. Video-level accuracy on Kinetics-400 vs the number of
temporal clips used during inference. TimeSformer-L achieves
excellent accuracy using a small number of clips, which leads to
strong performance at low inference cost.
methods use 10temporal clips with 3spatial crops (for a to-
tal of 30space-time views) during inference, TimeSformer
achieves solid accuracy with only 3views (3 spatial crops),
which reduces the inference cost. Our long-range variant,
TimeSformer-L achieves a top-1 accuracy of 80:7%. Fur-
thermore, our default TimeSformer has the lowest inference
cost among recent state-of-the-art models. Yet, it still pro-
vides a solid accuracy of 78:0%, outperforming many more
costly models.
We also measured the actual inference runtime on 20Kvali-
dation videos of Kinetics-400 (using 8Tesla V100 GPUs).
Whereas SlowFast takes 14:88hours to complete the infer-
ence, TimeSformer, TimeSformer-HR, and TimeSformer-L
take36minutes, 1:06hours and 2:6hours, respectively.
Thus, even though SlowFast and TimeSformer-L have com-
parable cost in terms of TFLOPs, in practice the runtimes
of all our versions of TimeSformer are much lower.
In Table 6, we also present our results on Kinetics-600.
Just like on Kinetics-400, we observe that TimeSformer
performs well on this benchmark, outperforming all prior
methods.
Finally, in Figure 6, we study the effect of using multi-
ple temporal clips during inference (each with a single
spatial crop). We plot accuracy using K2f1;3;5;10g
temporal clips for testing. We compare our model against
X3D (Feichtenhofer, 2020), and SlowFast (Feichtenhofer
et al., 2019b). X3D and SlowFast require multiple ( 5)
clips to approach their top accuracy. Conversely, our long-
range variant, TimeSformer-L, does not require multiple
clips to achieve its best performance, since it is able to spanMethod SSv2 Diving-48
SlowFast (Feichtenhofer et al., 2019b) 61.7 77.6
TSM (Lin et al., 2019) 63.4 N/A
STM (Jiang et al., 2019) 64.2 N/A
MSNet (Kwon et al., 2020) 64.7 N/A
TEA (Li et al., 2020b) 65.1 N/A
bLVNet (Fan et al., 2019) 65.2 N/A
TimeSformer 59.5 74.9
TimeSformer-HR 62.2 78.0
TimeSformer-L 62.4 81.0
Table 7. Video-level accuracy on Something-Something-V2 and
Diving-48.Due to an issue with Diving-48 labels used in pre-
viously published results, we only compare our method with a
reproduced SlowFast 168R101 model. All models are pre-
tained on ImageNet-1K.
about 12seconds of a Kinetics video with a single clip.
Something-Something-V2 & Diving-48. In Table 7, we
also validate our model on SSv2 and Diving-48. Since
ImageNet-21K pretraining does not improve accuracy on
SSv2 (see Table 3), in this case, we use TimeSformer pre-
trained on ImageNet-1K. This also allows us to apply the
same pretraining to all other models in this comparison,
using a ResNet pretrained on ImageNet-1K. Our results sug-
gest that TimeSformer achieves lower accuracy than the best
models on this dataset. However, considering that our model
uses a completely different design, we take these results as
suggesting that TimesSformer is a promising approach even
for challenging temporally-heavy datasets, such as SSv2.
In Table 7, we also present our method on another
“temporally-heavy” dataset, Diving-48. Due to a recently
discovered issue with a previous version of Diving-48 labels,
here, we only compare our method with a reproduced Slow-
Fast168R101 model. Our results show that TimeSformer
outperforms SlowFast by a substantial margin.
4.6. Long-Term Video Modeling
Lastly, we evaluate TimeSformer on the task of long-term
video modeling using HowTo100M (Miech et al., 2019).
HowTo100M is an instructional video dataset that contains
around 1M instructional Web videos showing humans per-
forming over 23K different tasks, such as cooking, repairing,
making arts, etc. The average duration of these videos is
around 7minutes, which is orders of magnitude longer than
the duration of videos in standard action recognition bench-
marks. Each HowTo100M video has a label indicating the
task demonstrated in the video (one out of the 23K classes),
which can be used for supervised training. Thus, it is a good
benchmark to assess the ability of a model to recognize
activities exhibited over very long temporal extents.
For this evaluation, we consider only categories that have
at least 100 video examples. This gives a subset of
HowTo100M corresponding to 120Kvideos spanning 1059
task categories. We randomly partition this collection into
85Ktraining videos and 35Ktesting videos.
Is Space-Time Attention All You Need for Video Understanding?
Method # Input
FramesSingle Clip
Coverage# Test
ClipsTop-1
Acc
SlowFast 8 8.5s 48 48.2
SlowFast 32 34.1s 12 50.8
SlowFast 64 68.3s 6 51.5
SlowFast 96 102.4s 4 51.2
TimeSformer 8 8.5s 48 56.8
TimeSformer 32 34.1s 12 61.2
TimeSformer 64 68.3s 6 62.2
TimeSformer 96 102.4s 4 62.6
Table 8. Long-term task classiﬁcation on HowTo100M. Given a
video spanning several minutes, the goal is to predict the long-term
task demonstrated in the video (e.g., cooking breakfast, cleaning
house, etc). We evaluate a few variants of SlowFast and TimeS-
former on this task. “Single Clip Coverage” denotes the number
of seconds spanned by a single clip. “# Test Clips” is the average
number of clips needed to cover the entire video during inference.
All models in this comparison are pretrained on Kinetics-400.
We present our results in Table 8. As our baselines, we use
four variants of SlowFast R101, all operating on video clips
sampled at a frame rate of 1=32but having varying number
of frames: 8;32;64and96. We use the same four conﬁg-
urations for TimeSformer, starting from a ViT pretrained
on ImageNet-21K. All models in this comparison are pre-
trained on Kinetics-400 before ﬁnetuning on HowTo100M.
During inference, for each method, we sample as many
non-overlapping temporal clips as needed to cover the full
temporal extent of a video, e.g., if a single clip spans 8:5
seconds, we would sample 48test clips to cover a video of
410seconds. Video-level classiﬁcation is done by averaging
the clip predictions.
From the results in Table 8 we ﬁrst note that, for the same
single clip coverage, TimeSformer outperforms the corre-
sponding SlowFast by a large margin of 811%. We also
observe that longer-range TimeSformers do better, i.e., our
longest-range variant achieves the best video-level classiﬁca-
tion accuracy. These results suggest that our model is highly
suitable for tasks that require long-term video modeling.
We also experimented with ﬁnetuning TimeSformer directly
from a ViT pretrained on ImageNet-1K and ImageNet-
21K (skipping the Kinetics-400 training). We report that
when pretrained only on ImageNet-1K, our model achieves
top-1 accuracies of 52:8;58:4;59:2;59:4for8;32;64;96
frame inputs, respectively. When considering ImagNet-
21K pretraining, TimeSformer produces top-1 accuracies of
56:0;59:2;60:2;62:1for8;32;64;96frame inputs, respec-
tively. These results demonstrate that our model can effec-
tively exploit long-range temporal dependencies regardless
of the pretraining dataset that we use.
4.7. Additional Ablations
Smaller & Larger Transformers. In addition to the “Base”
ViT model (Dosovitskiy et al., 2020), we also experimented
with the “Large” ViT. We report that this yielded results 1%
Figure 7. Visualization of space-time attention from the output
token to the input space on Something-Something-V2. Our model
learns to focus on the relevant parts in the video in order to perform
spatiotemporal reasoning.
ViT-50 -40 -30 -20 -10 0 10 20 30 40 50-50-40-30-20-1001020304050
-60 -40 -20 0 20 40-40-30-20-1001020304050       TimeSformer w/ Divided            Space-Time Attention-60 -40 -20 0 20 40 60-60-40-200204060    TimeSformer  w/ Space Attention
Figure 8. Feature visualization with t-SNE (van der Maaten & Hin-
ton, 2008) on Something-Something-V2. Each video is visualized
as a point. Videos belonging to the same action category have the
same color. The TimeSformer with divided space-time attention
learns semantically more separable features than the TimeSformer
with space-only attention or ViT (Dosovitskiy et al., 2020).
worse on both Kinetics-400, and Something-Something-V2.
Given that our “Base” model already has 121Mparameters,
we suspect that the current datasets are not big enough to
justify a further increase in model capacity. We also tried
the “Small” ViT variant, which produced accuracies about
5%worse than our default “Base” ViT model.
Larger Patch Size. We also experimented with a different
patch size, i.e., P= 32 . We report that this variant of our
model produced results about 3%worse than our default
variant using P= 16 . We conjecture that the performance
decrease with P= 32 is due to the reduced spatial granular-
ity. We did not train any models with Pvalues lower than
16 as those models have a much higher computational cost.
The Order of Space and Time Self-Attention. Our pro-
posed “Divided Space-Time Attention” scheme applies tem-
poral attention and spatial attention one after the other. Here,
we investigate whether reversing the order of time-space
attention (i.e., applying spatial attention ﬁrst, then tempo-
ral) has an impact on our results. We report that apply-
ing spatial attention ﬁrst, followed by temporal attention
leads to a 0:5%drop in accuracy on both Kinetics-400, and
Something-Something-V2. We also tried a parallel space-
time self-attention. We report that it produces 0:4%lower
accuracy compared to our adopted “Divided Space-Time
Attention” scheme.
Is Space-Time Attention All You Need for Video Understanding?
4.8. Qualitative Results
Visualizing Learned Space-Time Attention. In Figure 7,
we present space-time attention visualizations obtained by
applying TimeSformer on Something-Something-V2 videos.
To visualize the learned attention, we use the Attention
Rollout scheme presented in (Abnar & Zuidema, 2020).
Our results suggest that TimeSformer learns to attend to the
relevant regions in the video in order to perform complex
spatiotemporal reasoning. For example, we can observe that
the model focuses on the conﬁguration of the hand when
visible and the object-only when not visible.
Visualizing Learned Feature Embeddings. In Figure 8,
we also visualize the features learned by TimeSformer on
Something-Something-V2. The visualization is done using
t-SNE (van der Maaten & Hinton, 2008) where each point
represents a single video, and different colors depict differ-
ent action categories. Based on this illustration, we observe
that TimeSformer with divided space-time attention learns
semantically more separable features than the TimeSformer
with space-only attention or ViT (Dosovitskiy et al., 2020).
5. Conclusion
In this work, we introduced TimeSformer, a fundamentally
different approach to video modeling compared to the es-
tablished paradigm of convolution-based video networks.
We showed that it is possible to design an effective, and
scalable video architecture built exclusively on space-time
self-attention. Our method (1) is conceptually simple, (2)
achieves state-of-the-art results on major action recognition
benchmarks, (3) has low training and inference cost, and (4)
can be applied to clips of over one minute, thus enabling
long-term video modeling. In the future, we plan to ex-
tend our method to other video analysis tasks such as action
localization, video captioning and question-answering.
Appendix
A. Implementation Details
Our TimeSformer implementation is built using PySlow-
Fast (Fan et al., 2020) and pytorch-image-models (Wight-
man, 2019) packages. Below, we describe speciﬁc imple-
mentation details regarding the training and inference pro-
cedures of our model.
Training. We train our model for 15epochs with an initial
learning rate of 0:005, which is divided by 10at epochs
11;and14. During training, we ﬁrst resize the shorter
side of the video to a random value in [256;320]. We then
randomly sample a 224224crop from the resized video.
For our high-resolution model, TimeSformer-HR, we resize
the shorter side of the video to a random value in [448;512],
and then randomly sample a 448448crop. We randomlysample clips from the full-length videos with a frame rate
of1=32. The batch size is set to 16. We train all our models
using synchronized SGD across 32GPUs. The momentum
is set to 0:9, while the weight decay is set to 0:0001 .
Unless otherwise noted, in our experiments we use the
“Base” ViT model (Dosovitskiy et al., 2020). Temporal and
spatial attention layers in each block are initialized with the
same weights, which are obtained from the corresponding
attention layer in ViT.
Inference. As discussed in the main draft, during inference
we sample a single temporal clip in the middle of the video.
We scale the shorter spatial side of a video to 224pixels (or
448for TimeSformer-HR) and take 3crops of size 224224
(448448for TimeSformer-HR) to cover a larger spatial
extent within the clip. The ﬁnal prediction is obtained by
averaging the softmax scores of these 3predictions.
Other models in our comparison. To train I3D (Carreira
& Zisserman, 2017), and SlowFast (Feichtenhofer et al.,
2019b), we use the training protocols that were used in the
original papers. For I3D, we initialize it with a 2D ImageNet
CNN, and then train it for 118epochs with a base learning
rate of 0:01, which is divided by 10at epochs 44and88.
We use synchronized SGD across 32GPUs following the
linear scaling recipe of Goyal et al. (2017a). We set the
momentum to 0:9, and weight decay to 0:0001 . The batch
size is set to 64. For the SlowFast model, when initialized
from ImageNet weights, we use this same exact training
protocol. When training SlowFast from scratch, we use the
training protocol described by the authors (Feichtenhofer
et al., 2019b). More speciﬁcally, in that case, the training
is done for 196epochs with a cosine learning rate schedule,
and the initial learning rate is set to 0:1. We use a linear
warm-up for the ﬁrst 34epochs starting with a learning rate
of0:01. A dropout of 0:5is used before the ﬁnal classiﬁca-
tion layer. The momentum is set to 0:9, the weight decay is
0:0001 , and the batch size is set to 64. Just as before, we
adopt the linear scaling recipe (Goyal et al., 2017a).
Datasets. Kinetics-400 (Carreira & Zisserman, 2017) con-
sists of 240Ktraining videos and 20Kvalidation videos
that span 400human action categories. Kinetics-600 (Car-
reira et al., 2018) has 392Ktraining videos and 30Kvali-
dation videos spanning 600action categories. Something-
Something-V2 (Goyal et al., 2017b) contains 170Ktraining
videos and 25Kvalidation videos that span 174action cate-
gories. Lastly, Diving-48 (Li et al., 2018) has 16Ktraining
videos and 3Ktesting videos spanning 48ﬁne-grained div-
ing categories. For all of these datasets, we use standard
classiﬁcation accuracy as our main performance metric.
Is Space-Time Attention All You Need for Video Understanding?
References
Abnar, S. and Zuidema, W. Quantifying attention ﬂow in
transformers, 2020.
Ba, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization.
CoRR , 2016.
Bello, I., Zoph, B., Le, Q., Vaswani, A., and Shlens, J.
Attention augmented convolutional networks. In 2019
IEEE/CVF International Conference on Computer Vision,
ICCV , 2019.
Bertasius, G. and Torresani, L. Classifying, segmenting, and
tracking object instances in video with mask propagation.
InThe IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) , June 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,
Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
S., Radford, A., Sutskever, I., and Amodei, D. Language
models are few-shot learners. 2020.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov,
A., and Zagoruyko, S. End-to-end object detection with
transformers. In European Conference Computer Vision
(ECCV) , 2020.
Carreira, J. and Zisserman, A. Quo vadis, action recogni-
tion? A new model and the kinetics dataset. In 2017 IEEE
Conference on Computer Vision and Pattern Recognition,
CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , 2017.
Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., and
Zisserman, A. A short note about kinetics-600. CoRR ,
2018.
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey,
W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Par-
mar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Chen,
Z., Wu, Y ., and Hughes, M. The best of both worlds:
Combining recent advances in neural machine translation.
InProceedings of the 56th Annual Meeting of the Asso-
ciation for Computational Linguistics . Association for
Computational Linguistics, 2018a.
Chen, Y ., Kalantidis, Y ., Li, J., Yan, S., and Feng, J. Aˆ2-
nets: Double attention networks. In Advances in Neural
Information Processing Systems 31 , 2018b.
Chen, Y ., Fan, H., Xu, B., Yan, Z., Kalantidis, Y ., Rohrbach,
M., Yan, S., and Feng, J. Drop an octave: Reducing
spatial redundancy in convolutional neural networks with
octave convolution. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV) ,
October 2019.
Child, R., Gray, S., Radford, A., and Sutskever, I. Gener-
ating long sequences with sparse transformers. CoRR ,
2019.
Cordonnier, J., Loukas, A., and Jaggi, M. On the relation-
ship between self-attention and convolutional layers. In
8th International Conference on Learning Representa-
tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
2020 , 2020.
Dai, Z., Yang, Z., Yang, Y ., Carbonell, J., Le, Q., and
Salakhutdinov, R. Transformer-XL: Attentive language
models beyond a ﬁxed-length context. In Proceedings of
the 57th Annual Meeting of the Association for Computa-
tional Linguistics , 2019.
Deng, J., Dong, W., Socher, R., Li, L., Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database. In
2009 IEEE Conference on Computer Vision and Pattern
Recognition , pp. 248–255, 2009. doi: 10.1109/CVPR.
2009.5206848.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert:
Pre-training of deep bidirectional transformers for lan-
guage understanding. arXiv preprint arXiv:1810.04805 ,
2018.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:
Pre-training of deep bidirectional transformers for lan-
guage understanding. In Proceedings of the 2019 Confer-
ence of the North American Chapter of the Association
for Computational Linguistics: Human Language Tech-
nologies, Volume 1 (Long and Short Papers) , 2019.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.
An image is worth 16x16 words: Transformers for image
recognition at scale. CoRR , 2020.
Fan, H., Li, Y ., Xiong, B., Lo, W.-Y ., and Feichten-
hofer, C. Pyslowfast. https://github.com/
facebookresearch/slowfast , 2020.
Fan, Q., Chen, C.-F. R., Kuehne, H., Pistoia, M., and Cox,
D. More is less: Learning efﬁcient video representations
by big-little network and depthwise temporal aggregation.
InAdvances in Neural Information Processing Systems ,
volume 32, 2019.
Feichtenhofer, C. X3d: Expanding architectures for efﬁcient
video recognition. CVPR , pp. 200–210, 2020.
Feichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast
networks for video recognition. In Proceedings of the
Is Space-Time Attention All You Need for Video Understanding?
IEEE/CVF International Conference on Computer Vision
(ICCV) , 2019a.
Feichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast
networks for video recognition. In 2019 IEEE/CVF Inter-
national Conference on Computer Vision, ICCV , 2019b.
Gavrilyuk, K., Sanford, R., Javan, M., and Snoek, C. G. M.
Actor-transformers for group activity recognition. In 2020
IEEE/CVF Conference on Computer Vision and Pattern
Recognition, CVPR , 2020.
Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A.
Video action transformer network. In IEEE Conference on
Computer Vision and Pattern Recognition, CVPR , 2019.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P.,
Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y ., and
He, K. Accurate, large minibatch sgd: Training imagenet
in 1 hour. arXiv preprint arXiv:1706.02677 , 2017a.
Goyal, R., Kahou, S. E., Michalski, V ., Materzynska, J.,
Westphal, S., Kim, H., Haenel, V ., Fründ, I., Yianilos,
P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I.,
and Memisevic, R. The "something something" video
database for learning and evaluating visual common sense.
CoRR , 2017b.
Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T.
Axial attention in multidimensional transformers. CoRR ,
2019.
Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y . Relation net-
works for object detection. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR , 2018.
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y ., and
Liu, W. Ccnet: Criss-cross attention for semantic seg-
mentation. 2019.
Jiang, B., Wang, M., Gan, W., Wu, W., and Yan, J. Stm:
Spatiotemporal and motion encoding for action recog-
nition. In Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV) , October 2019.
Kwon, H., Kim, M., Kwak, S., and Cho, M. Motionsqueeze:
Neural motion feature learning for video understanding.
InECCV , 2020.
Le, H., Sahoo, D., Chen, N., and Hoi, S. Multimodal trans-
former networks for end-to-end video-grounded dialogue
systems. In Proceedings of the 57th Annual Meeting of
the Association for Computational Linguistics , 2019.
Li, L., Chen, Y .-C., Cheng, Y ., Gan, Z., Yu, L., and
Liu, J. Hero: Hierarchical encoder for video+ lan-
guage omni-representation pre-training. arXiv preprint
arXiv:2005.00200 , 2020a.Li, Y ., Li, Y ., and Vasconcelos, N. Resound: Towards action
recognition without representation bias. In The Euro-
pean Conference on Computer Vision (ECCV) , September
2018.
Li, Y ., Ji, B., Shi, X., Zhang, J., Kang, B., and Wang, L. Tea:
Temporal excitation and aggregation for action recogni-
tion. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR) , June
2020b.
Lin, J., Gan, C., and Han, S. Tsm: Temporal shift module for
efﬁcient video understanding. In Proceedings of the IEEE
International Conference on Computer Vision , 2019.
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev,
I., and Sivic, J. HowTo100M: Learning a Text-Video Em-
bedding by Watching Hundred Million Narrated Video
Clips. In ICCV , 2019.
Ott, M., Edunov, S., Grangier, D., and Auli, M. Scaling
neural machine translation. In Proceedings of the Third
Conference on Machine Translation: Research Papers ,
2018.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer,
N., Ku, A., and Tran, D. Image transformer. In Dy, J. G.
and Krause, A. (eds.), Proceedings of the 35th Interna-
tional Conference on Machine Learning, ICML , 2018.
Qiu, Z., Yao, T., Ngo, C.-W., Tian, X., and Mei, T. Learn-
ing spatio-temporal representation with local and global
diffusion. In CVPR , 2019.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever,
I. Improving language understanding by generative pre-
training. 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and
Sutskever, I. Language models are unsupervised multitask
learners. 2019.
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Lev-
skaya, A., and Shlens, J. Stand-alone self-attention in
vision models. In Advances in Neural Information Pro-
cessing Systems , pp. 68–80, 2019.
Sevilla-Lara, L., Zha, S., Yan, Z., Goswami, V ., Feiszli,
M., and Torresani, L. Only time can tell: Discovering
temporal data for temporal modeling. In Proceedings
of the IEEE/CVF Winter Conference on Applications of
Computer Vision (WACV) , pp. 535–544, January 2021.
Simonyan, K. and Zisserman, A. Very deep convolutional
networks for large-scale image recognition. In ICLR ,
2015.
Is Space-Time Attention All You Need for Video Understanding?
Stroud, J., Ross, D., Sun, C., Deng, J., and Sukthankar, R.
D3d: Distilled 3d networks for video action recognition.
InProceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV) , March 2020.
Sun, C., Myers, A., V ondrick, C., Murphy, K., and Schmid,
C. Videobert: A joint model for video and language
representation learning, 2019.
Szegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S.,
Anguelov, D., Erhan, D., Vanhoucke, V ., and Rabinovich,
A. Going deeper with convolutions. In Computer Vision
and Pattern Recognition (CVPR) , 2015.
Teed, Z. and Deng, J. RAFT: recurrent all-pairs ﬁeld trans-
forms for optical ﬂow. In Computer Vision - ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part II , 2020.
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y ., and
Paluri, M. A closer look at spatiotemporal convolutions
for action recognition. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, Salt Lake City,
USA, 2018 , 2018.
Tran, D., Wang, H., Feiszli, M., and Torresani, L. Video
classiﬁcation with channel-separated convolutional net-
works. ICCV , pp. 5551–5560, 2019.
van der Maaten, L. and Hinton, G. Visualizing data us-
ing t-SNE. Journal of Machine Learning Research , 9:
2579–2605, 2008. URL http://www.jmlr.org/
papers/v9/vandermaaten08a.html .
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Atten-
tion is all you need. In Advances in Neural Information
Processing Systems , 2017a.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Atten-
tion is all you need. In Advances in Neural Information
Processing Systems 30 . 2017b.
Wang, H., Tran, D., Torresani, L., and Feiszli, M. Video
modeling with correlation networks. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pat-
tern Recognition (CVPR) , June 2020a.
Wang, H., Zhu, Y ., Green, B., Adam, H., Yuille, A. L., and
Chen, L. Axial-deeplab: Stand-alone axial-attention for
panoptic segmentation. In Computer Vision - ECCV 2020
- 16th European Conference , 2020b.
Wang, X., Girshick, R., Gupta, A., and He, K. Non-local
neural networks. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) ,
June 2018a.Wang, X., Girshick, R. B., Gupta, A., and He, K. Non-local
neural networks. In 2018 IEEE Conference on Computer
Vision and Pattern Recognition, CVPR 2018, Salt Lake
City, UT, USA, June 18-22, 2018 , 2018b.
Wang, X., Xiong, X., Neumann, M., Piergiovanni, A. J.,
Ryoo, M. S., Angelova, A., Kitani, K. M., and Hua, W.
Attentionnas: Spatiotemporal attention cell search for
video classiﬁcation. In Computer Vision - ECCV 2020 -
16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part VIII , 2020c.
Weissenborn, D., Täckström, O., and Uszkoreit, J. Scal-
ing autoregressive video models. In 8th International
Conference on Learning Representations, ICLR , 2020.
Wightman, R. Pytorch image models. https://github.
com/rwightman/pytorch-image-models ,
2019.
Xie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K.
Rethinking spatiotemporal feature learning: Speed-
accuracy trade-offs in video classiﬁcation. In Com-
puter Vision - ECCV 2018 - 15th European Confer-
ence, Munich, Germany, September 8-14, 2018, Pro-
ceedings, Part XV , pp. 318–335, 2018. doi: 10.1007/
978-3-030-01267-0\_19. URL https://doi.org/
10.1007/978-3-030-01267-0_19 .
Yang, Z., Garcia, N., Chu, C., Otani, M., Nakashima, Y ., and
Takemura, H. Bert representations for video question an-
swering. In The IEEE Winter Conference on Applications
of Computer Vision , 2020.
Zhao, H., Jia, J., and Koltun, V . Exploring self-attention
for image recognition. In 2020 IEEE/CVF Conference on
Computer Vision and Pattern Recognition, CVPR , 2020.
Zhou, L., Zhou, Y ., Corso, J. J., Socher, R., and Xiong, C.
End-to-end dense video captioning with masked trans-
former. In Proceedings of the IEEE Conference on Com-
puter Vision and Pattern Recognition , 2018.

Quantifying Attention Flow in Transformers

Layer Normalization

Lei Jimmy Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Attention Augmented Convolutional Networks

Bello, I., Zoph, B., Le, Q., Vaswani, A., Shlens, J.

Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

Language Models are Few-Shot Learners