Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
Authors: Mandela Patrick
Dylan Campbell
Yuki Asano
Ishan Misra
Florian Metze
Christoph Feichtenhofer
Andrea Vedaldi
Joao F. Henriques
Venue: NeurIPS 2021 Oral (2021)
Keywords: xformers
video action recognition
transformers
attention
efficient attention

Abstract: In video transformers, the time dimension is often treated in the same way as the two spatial dimensions. However, in a scene where objects or the camera may move, a physical point imaged at one location in frame $t$ may be entirely unrelated to what is found at that location in frame $t+k$. These temporal correspondences should be modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers - trajectory attention - that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas are useful in a range of settings, we apply them to the specific task of video action recognition with a transformer model and obtain state-of-the-art results on the Kinetics, Something-Something V2, and Epic-Kitchens datasets.

Keeping Your Eye on the Ball:
Trajectory Attention in Video Transformers
Mandela Patrick*
Dylan Camphell University of Oxford dy fylan@robots.commm ox. ac
Yuki Asano" University of Oxford yuki quki@robots.c ox. ac uk
Facebook AI mandelapatrickkff.o com
Ishan Misra
Florian Metze
Christoph Feichtenhofer
Facebook AI imisra@fb... com
Facebook AI fmetze0fb. com
Facebook AI feichtenhooerefb.. com
Andrea Vedaldi Facebook Al vedaldi@fb.com
João F. Henriques University of Oxford
joao@robots ox ac uk
Abstract
In video transformers, the time dimension often treated in the same way the two spatial dimensions. However, in a scene where objects or the camera may move, physical point imaged one location in frame may be entirely unrelated what is found at that location in frame t + k. These temporal correspondences should modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers trajecettrtttttetoorrr attention that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas usefuu in range of settings, apply them to the specific task video action recognition with a transformer model and obtain state-of-the-an results on the Kinetics, BoneehinggSSomtthinn V2, and Epic-Kitchens Kitchens datasets. comm/facebokrrseercchh
Code and models are available at: Mot onf ormer!
Introduction
Transformers 75 have become a popular architecture across NLP [32], vision (0) and speech (5]. The self-attention mechanism in the transformer works well for different types of data and across domains. However, its generic nature and its lack of inductive biases also mean that transformers typically require extremely large amounts of data for training [56] 9 aggressive domain-specific augmentations [7]. This is particularly true for video data, for which transformers are also appli- cable [50]. but where statistical inefficiencies are exacerbated. While videos carry rich temporal information, they can also contain redundant spatial information from neighboring frames. Vanilla self-attention applied to videos compares pairs of image patches extracted all possible spatial locations and frames. This can lead it to focus on the redundant spatial information rather than the temporal information, as we show by comparing normalization strategies our experiments.
We therefore contribute a variant of self-attention, called trajectory attention, which is better able characterize the temporal information contained videos. For the analysis of still images,
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

time
Figure 1: Trajectory attention. In this sequence of frames from the Kinetics-400 dataset, depicting the action "kicking soccer ball', the ball does not remain stationary with respect to the camera, but instead moves different locations in each frame. Trajectory attention aims to share information along the motion path of the ball, more natural inductive bias for video data than pooling axially along the temporal dimension over the entire space-time feature volume. This allows the network to aggregate information from multiple views of the ball, to reason about its motion characteristics, and to be less sensitive to camera motion.
spatial locality is perhaps the most important inductive bias, motivating the design of convolutional networks [42] and the use of spatial encodings in vision transformers [20]. This is a direct consequence of the local structure of the physical world: points that belong to the same 3D object tend project to pixels that are close to each other in the image. By studying the correlation of nearby pixels, we thus learn about the objects.
Videos are similar, except that 3D points move over time, thus projecting on different parts of the image along certain 2D trajectories. Existing video transformer methods [8, 3
50) disregard these
trajectories, pooling information over the entire 3D space-time feature volume B 50, or pooling axially across the temporal dimension [8]. We contend that pooling along motion trajectories would provide a more natural inductive bias for video data, allowing the network to aggregate information from multiple views of the same object region, to reason about how the object or region is moving (for example, the linear and angular velocities), and to invariant to camera motion.
We leverage attention itself as mechanism to find these trajectories. This is inspired by methods such as RAFT (70,, which showed that excellent estimates of optical flow can be obtained from the correlation volume obtained by comparing local features across space and time. We observe that the joint attention mechanism for video transformers computes such a correlation volume as an intermediate result. However, subsequent processing collapses the volume without consideration for its particular structure. In this work, we seek instead to use the correlation volume to guide the network to pool information along motion paths.
We also note that visual transformers operate on image patches which, differently from individual pixels, cannot be assumed to correspond to individual 3D points and thus to move along simple ID trajectories. For example, in Figure [ depicting the action 'kicking soccer ball', the ball spans up to four patches, depending foreground (the ball) and background objects, thus at least two distinct motions. Fortunately, we are not forced to select single putative motion: the attention mechanism allows to assemble a motion feature from all relevant 'ball regions'.
the specific video frame. Furthermore, these patches contain a mix of
Inspired by Nyströmformer 85 we also propose a principled approximation to self-attention, Orthoformer. Our approximation sets state-of-the-ee performance on the recent Long Range Arena (LRA) benchmark [69] for evaluating efficient attention approximations and generalizes beyond the video domain to long text and high resolution images, with lower FLOPS and memory requirements compared to alternatives, Nyströmformer and Performer [15]. Combining our approximation with trajectory attention allows to significantly improve computational and memory efficiency. With contributions, set state-of-the- the- results on four video action recognition benchmarks.

---

2
Related Work
Video representations and 3D-CNNs. Hand-crafted features were originally used to convert video data into a representation amenable to analysis by a shallow linear model. Such representations include SIFT-3D [60]. HOG3D [39,, and IDT [76]. Since the breakthrough of AlexNet [40] on the ImageNet classification benchmark (58]. which demonstrated the empirical benefits of deep neural networks to learn representations end-to-end, there have been many attempts to do the same for video. Architectures with 3D convolutions-3D1 -3D-CN--------N------ weere originally proposed to learn deep video representations 772. Since then, improvements to this paradigm include the use of ImageNet-inflated 3D convolutions
[54 74, 84], channel- -separatd convolutions [73), non-local blocks [79), and attention layers [13] Optical flow-based pooling can be used instead temporal convolutions to improve the representation' 's robustness to camera and object motions [.. Our approach shares this motivation.
weights [Л the space-time decomposition
The transformer architecture [75], originally proposed for natural language processing, has recently gained traction in the computer vision domain. The vision transformer (ViT) 20 decomposes an image into a sequence of 16 x 16 words and uses multi-layer transformer to perform image classification. To improve ViT's data efficiency, DeiT [71 used distillation from a strong teacher model and aggressive data augmentation. Transformers have also been used in variety of vision image tasks, such as image representation learning [12
Vision transformers.
82 18]
[9] image
generation [51], object detection [48 [0], video question-asswering (35), few-shot learning [9 and
image-text 449 63 67 44. 68 video-text [65 64 89 25 53
and video-audio 43 52 29)
representation learning.
Attention for video recognition. The self-attention operation proposed in the transformer 75 have been adapted to video recognition tasks. Wang al. [79] propose the non-local mean operation for video action recognition, which is equivalent to the standard transformer self-attention applied uniformly across space and time, while our proposed trajectory attention does not treat the space and time dimensions equivalently. Zhao et al. [87] propose CNN architecture that explicitly predicts trajectories and aggregates information along them using a convolution operation. In contrast, our transformer architecture does not explicitly predict trajectories, but instead provides an inductive bias that encourages the network to consider motion trajectories where useful. Concurrent works B 3 50 21 have also adapted the self-attention operation to the spatio-temporal nature of videos, however, these approaches do not have a mechanism for reasoning about motion paths, treating time as just another dimension, unlike our approach.
Efficient attention. Due the quadratic complexity of self-attention, there has been significant amount of research on how to reduce its computational complexity with respect to time and memory use. Sparse attention mechanisms [4] were used reduce self-attention complexity to (nn/n). and sensiity-ensittiviyy hashing was used by Reformer (38) to further reduce this to O log n). More
recently, linear attention mechanisms have been introduced, namely Longformer 77, Linformer 778)
Performer [15] and Nyströmformer 885 The Long Range Arena benchmark 669 was recently introduced to compare these different attention mechanisms.
Temporal correspondences and optical flow. There are many approaches that aim to establish explicit correspondences between video frames as a way reason about camera and object motion. For short-range correspondences across time, optical flow algorithms (30] [0] are highly effective. In particular, RAFT 770 showed the effectiveness of an all all-pairs inter-frame correlation volume as an encoding, which is essentially an attention map. All-pairs intra-frame correlations were subsequently shown to help resolve correspondence ambiguities [4 For longer-range correspondences, object tracking by repeated detection (57) and data association can be used. In contrast to these approaches, our work does not explicitly establish temporal correspondences, but facilitates implicit correspon- dence learning via trajectory attention. Jabri et al. [i estimate correspondences in similar way, framing the problem as a contrastive random walk on graph and apply explicit guidance via a cycle consistency loss. Incorporating such guidance into a video transformer is interesting direction.
3 Trajectory Attention for Video Data
Our goal is to modify the attention mechanism in transformers to better capture the information contained in videos. Consider input video I T X consisting of I" frames of resolution
3

---

Spatial Attendon
Temperal Attention
t=
T
Y: TSxx
TSx
Figure 2: Trajectory attention flowchart. We divide the attention operation into stages: the first forming a set ST trajectory tokens for every space-time location st-a a spatial attention operation between pairs of frames and the second pooling along these trajectories with ID temporal attention operation. In this way, we accumulate information along the motion paths of objects in the video. The softmax operations
computed over the last dimension.
H W. As in existing video transformer models g 3], we pre-process the video into a sequence of ST tokens X,, E RD, for spatial resolution of S and a temporal resolution of T.. We use cuboid embedding 3 21. where disjoint spatio-temporal cubes from the input volume are linearly projected to RR (equivalent to a 3D convolution with downsampling). We also test an embedding of disjoint image patches (00] A learnable positional encoding
TR. is added the video embeddings for spatial and temporal dimensions separately, resulting in the code Finally, a learnable classification token Zets added to the sequence of tokens, like in the BERT Transformer [2), to reason about the video whole. For clarity, we elide the classification token from our treatment in the sequel.
Xst + e + e
We now have set of tokens that form the input to a sequence of transformer layers that, as in ViT 20), consist of Layer Norm (LN) operations 44, multi-head attention (MHA) [75] residual connections 27 and feed-forward network (MLP):
y MHA(LN(Z)) z: z' = MLP(LN(()) + yy.
In the next section, we shall focus on a single head of the attention operation, and demonstrate how self-attention can realize suitable inductive bias for video data. For clarity of exposition, abuse the notation slightly, neglecting the layer norm operation and using the same dimensions for single-head attention for multi-head attention.
3.1 Video self-attention
The self-attention operation begins by forming set of svery-key-valus vectors qst, kat: Vat R.D. one for each sime location st in the video. These are computed as linear projections of the input
zet that is, q.r W zzt. k,, Zgz and as W Zst. for projection matrices W; IR DXD
A direct application of attention across space-time (called joint space-time atlention S 3) computes:
exp( k.r)
st
Vere
Expexp(a kgr)
In this way, each query 9st is compared to all keys k.' using dot products, the results are normalized using the softmax operator, and the weights thus obtained are used to average the values corresponding to the keys. Compared to standard transformer, we have omitted for brevity the softmax temperature parameter 1/2 and instead assume that the queries and keys have been divided by 1/4
One issue with this formulation is that it has quadratic complexity in both space and time, i.c., (SST2)) An alternative is to restrict attention to either space or time (called divided space-time attention):
exp( kvtt) Y = Σ Vot E. exp( k())
(space); Yst = Σ Var exp(q ksv) Σ, exp(qnn k,i
(time).
(3)

---

This reduces the complexity to T(S and (SST)), respectively, but only allows the model to analyse time and space independently. This is usually addressed by interleaving [8] or stacking the two attention modules in sequence.
Different to both of these approaches, we perform attention along trajectories, the probabilistic path of a token between frames For each space-time location st (the trajectory 'reference point') and corresponding query 9st, we construct a set of trajectory tokens Ysu, representing the pooled information weighted by the trajectory probability. The trajectory extends for the duration of the at different times t' are given by:
video sequence and its tokens You IR
exp(qst, kse)
Ysu Σ Var
(4)
Σ; exp(qqe kse)
Note that the attention in this formula is applied spatially (index s) and independently for each frame. Intuitively, this pooling operation implicitly seeks the location of the trajectory at time e' by comparing the trajectory query qst to the keys ky' at time t!.
Once the trajectories are computed, we further pool them across time to reason about intra-frame information/ccomecioon., To do so, the trajectory tokens are projected to a new set of queries, keys and values as usual:
Tet W, Nott, katt' y
Vatt
(5)
Like qst before, the updated reference query Åst corresponds to the trajectory reference point st and contains information cettially-poolee from across the reference frame t. This new query is used to pool across the new time (trajectory) dimension by applying ID attention:
exp(ân: kate)
Sst Σ Stt! Σ, explpst. kar)
(6)
Like joint space-time attention, our approach has quadratic complexity in both space and time, ((STT2), so has no computational advantage and is in fact slower than divided space-time attention. However, we demonstrate better accuracy than both joint and divided space-time attention mecha- nisms. We also provide fast approximations in Section 3.2 A flowchart of the full trajectory attention operation shown in tensor form in Figure
3.2 Approximating attention
To complement our trajectory attention, we also propose an approximation scheme to speed up calculations. This scheme is generic and applies to any attention-like pooling mechanism. We thus switch to a generic transformer-like notation to describe it. Namely, consider query-key query-key-valuue matrices Q, V e IR DxN such that the query-key-value vectors are stored as columns q k,, V, RD in these matrices.
In order obtain efficient decomposition of the attention operator, we will rewrite it using probabilistic formulation. Let A. E (0, be a categorical random variable indicating whether the jth input (with key vector k, E R) is assigned to the ith output (with query vector q, E R²), with parametric model of the probability of this event based on
Σ, Aij 1. The attention operator the multinomial logistic function, i.e., the softmax operator S(-) 3
P(A.) = K)q
where the subscript: denotes a full slice of the input tensor in that dimension. We now introduce the latent variables U2j E {0, 1}. which similarly indicate whether the jth input is assigned to the fth prototype, auxiliary vector which denote by P R We can use the laws of total and conditional probability to obtain:
P(Ai)) = E, (U)))(
(8)
Note that the latent variables that we chose are independent of the inputs (keys). They use the same parametric model, but with parameters P D (the concatenated prototype vectors P.):
Heree we refer to the trajectory as the motion between pairs frames, rather than multi-frame path. Le. [S(z)]: exp(zi/VD) D, exp(2;/VD), For matrix inputs, the sum is over the columns.
5

---

P(U) S(PTK). Eq. 8 is exact, even under the parametric model for P(U), though the corre- sponding true distribution P(A] U) is intractable. We now approximate the conditional probability P(A ()) with a similar parametric model:
S(Q
(9)
where Q E IR DxN concatenates all query vectors horizontally. Substituting equations 7 we write the full approximate attention A, multiplied by an arbitrary matrix V (which in the case of transformer contains the values of the key-value pairs stacked as rows):
P(A)V = -(Q P)) (S(PTK)V).
(10)
Computational efficiency. One important feature of the approximation in eq. 10 is that it can be computed in tWO steps. First the values V are multiplied by prototypes-keys attention matrix K) S(PTK)
E R RRXN which can be much smaller than the full attention matrix S(Q' K) E N X (eq. i.c., R & N. Finally, this product is multiplied by a queries-prototyype attention matrix S(Q P)
WERT NN which is also small. This allows us to sidestep the quadratic dependency of full attention over the input and output size (NN2). replacing with linear complexity (O(N) as long as Ri is kept constant.
Prototype selection. The aim for prototype- based attention approximation schemes is to as few prototypes as possible while reconstructing the attention operation as accurately as possible. As such, it behooves to select prototypes efficiently. We have two priorities for the prototypes: to dynamically adjust to the query and key vectors so that their region of space is well-reconstructed, and to minimize redundancy. The latter is important because the relative probability of query-key pair may be over-estimated if many prototypes are clustered near that query and key. To address these criteria, we incrementally build set of prototypes from the set of queries and keys such that a new prototype is maximally orthogonal to the prototypes already selected, starting with a query or key at random. This greedy strategy dynamic, since selects prototypes from the current set of queries and keys, and has high entropy, since it preferences well-separated prototypes. Moreover, it balances speed and performance by using greedy strategy, rather than finding a globally-optimal solution to the maximum entropy sampling problem [6]], making it suitable for use in transformer.
Naïvely applying prototype-based attention approximation techniques to video transformers would involve creating a unique set of prototypes for each frame in the video. However, additional memory savings can be realized by sharing prototypes across time. Since there is significant information redundancy between frames, video data is opportune for compression via temporally- shared prototypes.
Orthoformer algorithm. The proposed approximation algorithm is outlined in Algorithm I The attention matrix is approximated using intermediate prototypes, selected the most orthogonal subset of the queries and keys, given a desired number of prototypes R. To avoid a linear dependence on the sequence length N, we first randomly subsample eR queries and keys, for constant before selecting the most orthogonal subset, resulting in a complexity quadratic in the number of prototypes ((R2). The algorithm then computes two attention matrices, much smaller than the original problem, and multiplies them with the values. The most related approach in the literature is Nyströmformer (85) This approach involves a pseudoinverse attenuate the effect of near-paralle prototypes, has more operations, and a greater memory footprint.
attention, outlined in Algorithm 2
Algorithm 1 Orthoformer (proposed) attention
Algorithm 2 Nyströmformer [85 attention
I: P - MostOrthogonaaSuuse(( Q, K,
I: P,, P. <- SegmentMeans Q R)
2: Z. S(Q"P /VD)
2: Ωh = S(Q P//PP//DD
3: Ω. S(P'K/VD
3: Ωz Iteratieenneeee((PPPPP////// Nus)
4: Y = (
4: Ωr S(P].K/D)
5: Y Ω (Ω2 (N&V))
3.3 The Motionformer model
Our full video transformer model builds on previous work, as shown in Table In particular, we use the ViT image transformer model (20 as the base architecture, the separate space and time positional encodings of TimeSformer (8), and the cubic image tokenization strategy as in ViVIT [. These
design choices are ablated in Section The crucial difference for our model is the trajectory attention mechanism, with which we demonstrate greater empirical performance than the other models.

---

Table 1: Comparison of recent video transformer models. We show the different design choices
of recent video transformer models and how they compare to our proposed Motionformer model.
Model
Base Model
Attention
Pos. Encoding
Tokenization
TimeSforerr 8
ViT-B
Divided Space- Time Joint/Divided Space-Time
Separate
Square
ViVIT (
ViT-L
Joint
Cubic
Motionformer
ViT-B
Trajectory
Separate
Cubic
4
Experiments
Datasets. Kinetics [36] is large-scale video classification dataset consisting of short clips collected from YouTube, licensed by Google under Creative Commons. As is a dataset of human actions, it potentially contains personally identifiable information such as faces, names and license plates. Something-Sometting V2 [26] is video dataset containing more than 200,000 videos across 174 classes, with greater emphasis on short temporal clips. In contrast to Kinetics, the background and objects remain consistent across different classes, and therefore models have to reason about fine- fine-grained motion signals. We verified the importance of temporal reasoning on this dataset by showing that single frame model gets significantly worse results, a decrease of 39% top-1 accuracy. In contrast, a drop of only 7% is seen on the Kinetics-400 dataset, showing that temporal information is much less relevant there. We obtained a research license for this data from https://20bn. com, the data was collected with consent. Epic Kitchens- -o0eens--000 [16] is an egocentric video dataset capturing daily kitchen activities. The highest scoring verb and action pair predicted by the network constitutes action, for which we report top-1 accuracy. The data is licensed under Creative Commons and was collected with consent by the Epic Kitchens teams.
Implementation details. We follow a standard training and augmentation pipeline B.. as detailed in the appendix. For ablations, our default Motionformer model is the Vision Transformer Base architecture 20 (ViT/B), pretrained on ImageNet-218 17 patch-size 2 16x6xx 16 with central frame initialization 3 separate space-time positional embedding and our trajectory attention. The base architecture has 12 layers, 12 attention heads, and an embedding dimension of 768. Our default Motionformer model operates on 16x224x 2244 videos with temporal stride i.e. temporal extent of 2s. For comparisons with state-of-the-art, we report results on two additional variants: Motionformer- HR, which has a high spatial resolution (16 x333333636 videos with temporal stride 4 i.e. temporal extent of 2s), and Motionformer-L, which has long temporal range (32x224xx x videos with temporal stride 3 i.c. temporal extent of 3s). Experiments with the large ViT architecture are deferred to the appendix.
4.1 Ablation studies
Input: tokenization. We consider the effect of different input tokenization approaches for both joint and trajectory attention on Kinetics-400 (K-400) and Something-SSmethhing V2 (SSv2) in Table 2b For patch tokenization (1 1xx16x10 16), we use inputs of size 8x224x222 while for cubic [3] 21 tokenization (2x16x 16x 16), we use inputs of size 16x224x224 to ensure that the model has the same number of input tokens over the same temporal range of 2 seconds. For both attention types, that cubic tokenization gives a 1% accuracy improvement over square tokenization on SSv2, a dataset for which temporal information is critical. Furthermore, our proposed trajectory attention using cubic tokenization outperforms joint space-time attention on both datasets.
Input: positional encoding. Here, we ablate using joint or separate 21 (default) space-time positional encoding in Table Similar to the results for input tokenization, the choice of positional encoding is particularly important for the fine-grained motion dataset, SSv2. Since joint space-time attention treats tokens in the space-time volume equally, it benefits particularly from separating the positional encodings, allowing it to differentiate between space and time dimensions, with 4% improvement on SSv2 over joint space- space-time encoding. Our proposed trajectory attention elicits more modest improvement of 1% from using separated positional encodings SSv2, and outperforms joint space-time attention in both settings on both datasets.
Attention block: comparisons. We compare our proposed trajectory attention to joint space-time attention 3 and divided space-time attention [8 in Table 4 Our trajectory attention (bottom row) outperforms both alternatives on the K-400 and SSv2 datasets. While we see only modest
7

---

Table 2: Input encoding ablations: Comparison of input tokenization and positional encoding design choices. We report GFLOPS and top-1 accuracy (%) on K-400 and SSv2.
(a) Cubic tokenization works best for trajectory attn.
(h) Trajectory atm. works well with both encodings.
Attention
Tokenization
GFlops
K-400
SSv2
Attention
Pos. Encoding
GFlops
K-400
SSv2
Joint ST
Square (1x 1
179.7
79.4
63.0
Joint ST
Joint ST
180.6
79.1
60.8
Cubic (2x16²)
180.6
79.2
64.0
Separate ST 21
180.6
79.2
64.0
Trajectory Square (Ix 162)
368.5
79.4
65.8
Trajectory Joint ST
369.5
79.6
65.8
Cubic 12x)
369.5
79.7
66.5
Separate ST
21
369.5
79.7
66.5
Table 3: Orthoformer ablations: We ablate various aspects of our Orthoformer approximation. E denotes exact attention and A denotes approximate attention. We report max CUDA memory consumption (GB) and top-1 accuracy (%) on K-400 and SSv2.
(a) Orthoformer competitive with Nyström.
(b) Selecting orthogonal prototypes is the best strategy.
Attention
Approx.
Mem.
K-400
SSv2
Attention
Selection
Mem.
K-400
SSv2
Trajectory (E) N/A
7.4
79.7
66.5
Trajectory (E) N/A
7.4
79.7
66.5
Trajectory (A) Performer
5.1
72.9
52.7
Trajectory (A) Seg-Means
3.6
75.8
60.3
Nyströmformer
3.8
77.5
64.0
Random
3.6
76.5
62.5
Orthoformer
3.6
77.5
63.8
Orthogonal
3.6
77.5
63.8
(c) Approximation improves with more protolypes.
(d) Temporal sharing is the best strategy.
Attention
# Prototypes
Mem.
K-400
SSv2
Attention
Sharing
Mem.
K-400
SSv2
Trajectory (E)
N/A
7.4
79.7
66.5
Trajectory (E)
N/A
7.4
79.7
66.5
Trajectory (A)
16
3.1
73.9
59.2
Trajectory (A)
x
16.5
77.3
61.5
64
3.3
74.9
63.0
/
3.6
77.5
63.8
128
3.6
77.5
63.8
improvements on the appearance cue-reliant K-400 dataset, our trajectory attention significantly outperforms (+2%) the other approaches on the motion cuc-reliant SSv2 dataset. This dataset requires fine-grained motion understanding, something explicitly singled out by previous video transformer works [ g as a challenge for their models. In contrast, our trajectory attention excels on this dataset, indicating that its motion-based design is able capture some of this information.
Attention block: trajectory attention design. We ablate two design choices for our trajectory attention: the per-frame softmax normalization and the ID temporal attention. Unlike joint space-time attention, which normalizes the attention map over all tokens in space and time, trajectory attention normalizes independently per frame, allowing to implicitly track the trajectories of query patches in time. In row of Table 4 we ablate the benefits of this design choice. We observe a reduction of 2.5% on K-400 and 5.6% on SSv2 by normalizing over space and time (NormsT) compared with normalizing over space alone (Norms). In row 4, we show the benefit of using ID temporal attention (Atty) aggregate temporal features, compared to average pooling (Avgr). We observe reductions of 3. 3.7% on K-400 and 6.5% on SSv2 when using average pooling instead of temporal attention applied to the motion trajectories, although it saves computing the additional query/key/value projections.
4.2 Orthoformer approximated attention
Approximation comparisons. In Table 3a we compare our Orthoformer algorithm to alternative strategies: Nyströmformer [85] and Performer [15]. Our algorithm performs comparably with Nyströmformer with a reduced memory footprint. In Table 5 we also compare these attention mechanisms on the Long Range Arena benchmark [69] to show applicability to other tasks and data types. Orthoformer is able to effectively approximate self-attention, outperforming the state-of-the-an despite using far fewer prototypes (64) and so gaining significant computational and memory benefits.
Prototype selection. A key part of our Orthoformer algorithm is the prototype selection procedure. In Table 3b) we ablate three prototype selection strategies: segment-mean random, and greedy most-orthogonal selection. Segment-means, the strategy used in Nyströmformer, performs poorly because it can generate multiple parallel prototypes, which will over-estimate the relative probability
8

---

Table 4: Attention ablations: We compare trajectory attention with alternatives and ablate its design choices. We report GFLOPS and top-1 accuracy (%) on K-400 and SSv2. Atty: temporal attention, Avgr: temporal averaging, Norm space-time normalization, Norms: spatial normalization.
Attention
Altr
Avgr
Norms
Normst
GFLOPS
K-400
SSv2
Joint Space-Time
180.6
79.2
64.0
Divided Space-Time
185.8
78.5
64.2
X
180.6
76.0
60.0
x
V
369.5
77.2
60.9
Trajectory
X
X
369.5
79.7
66.5
Table 5: Comparison to the state-of-the-art on Long Range Arena benchmark. GFLOPS and CUDA maximum Memory (MB) are reported for the ListOps task. Note that our algorithm achieves the best overall results with far fewer prototypes (64) than the other methods.
Model
ListOps
Text
Retrieval
Image
Pathfinder
Avgī
GFL.OS|
Mem.
Exact
36.69
63.09
78.22
31.47
66.35
55.16
1.21
4579
Performer-22 256
15
36.69
63.22
78.98
29.39
66.55
54.97
0.49
885
Nyströmformer- 128
85
36.90
64.17
78.67
36.16
52.32
53.64
0.62
745
Orthoformer-64
33.87
64.42
78.36
33.26
66.41
55.26
0.24
344
of query-key pairs those redundant prototypes. In contrast, proposed strategy of selecting the most orthogonal prototypes from the query and key set works the best across both datasets, because explicitly minimises prototype redundancy with respect to direction.
Table 3c we show that Orthoformer improves monotonically as number of prototypes is increased. In particular, we see an average performance improvement of 4% on both datasets as we increase the number of prototypes from 16 to 128.
Number of prototypes.
Temporally-shared prototypes. In Table 3d we demonstrate the memory savings and perfor- mance benefits of sharing prototypes across time. On SSv2, we observe a 2% improvement in performance and a 5x decrease in memory usage. These gains may be attributed to the regularization effect of having prototypes leverage redundant information across frames.
Scaling transformer models with approximated trajectory attention. The Orthoformer atten- tion approximation algorithm allows given GPU memory budget. Here, we verify this, by training a large vision transformer model (ViT-1/16) 20 with higher resolution input (336 336 pixels) the Kinetics-400 dataset, using the Orthoformer approximation with 196 temporally-sharee the base model. We use a fixed patch size (in pixels) for all models, and the number of input tokens to the transformer scales with the square of the image resolution. As shown in Table 7 this model achieves a competitive accuracy without fine-tuning the training schedule, hyperparameters or data augmentation strategy. We expect that fine-tuning these on a validation set would greatly improve the model's performance, based on results from contemporary work [3) Obviously such parameter sweep is more time-consue consuminn for these large models, however, these preliminary results are indicative that higher accuracies are attainable these parameters were to optimized.
train larger models and higher resolution inputs for
prototypes and the same schedule as
4.3 Comparison to the state-of-the-art
In 6 we compare our method against the current state-of-the-art on four common benchmarking datasets: Kinetics-400, Kinetics-600, Something-SSoethhigg v2 and Epic-Kitchens. We find that our method performs favorably against current methods, even when compared against much larger models such as ViViT-L. In particular, achieves strong top-1 accuracy improvements of 1.0% and 2.3% for SSv2 and Epic-Kite Kitchen Nouns, respectively. These datasets require greater motion reasoning than Kinetics and so are more challenging benchmark for video action recognition.
Conclusion
We have presented a new general-purge purpose attention block for video data that aggregates information along implicitly determined motion trajectories, lending a realistic inductive bias to the model. We further address its quadratic dependence on the input size with new attention approximation

---

Table 6: Comparison to the state-of-the-art on video action recognition. We report GFLOPS and top-1 (%) and top-5 (%) video action recognition accuracy on K-400/600, and SSv2. On Epic-Kitchent we report top-1 (%) action (A), verb (V), and noun (N) accuracy.
(a) Soonething-Sometiie V2
(b) Kinetics-400
Model
Pretrain
Top-1
Top-5
GFLOPs X viewws
Method
Pretrain
Top
Top-5
GFLOPs viewws
Fast
K-400
61.7
x
IN-IK
72.1
NIA
TSM
47
K-400
63.4
88.5
62.4x3x2
R(2+DD
72.0
90.0
STM
83
IN-IK
64.2
89.8
10
S3D-G 84
IN-IK
74.7
93.4
142.8xN/A
MSNet 41
IN-IK
64.7
89.4
67x1x1
X3D XL
25
79.1
93.9
48.4x3x
TEA
[6
TN-IK
65.1
x0 10
SlowFast
79.8
93.9
234 3
bLVNet
128.6x3x10
VTN
IN-21K
VidT=L
45
IN-2IK+K-400
60.2
x51 10
VidTeL 351
IN-21K
79.1
93.9
x92x 3x 10
Tormer--
8
IN-21K
62.5
1703x3x1
Tormer-
IN-21K
80.7
94.7
x
VIVIT-L
8
IN-2IK+K-400
65.4
89.8
3992x4x3
MVIT-
81.2
95.1
433
MVTTBB
K-400
67.1
90.8
170x3x1
VIVIT- 1
IN-21K
$1.3
94.7
x992x3xx
Mfurmer
IN-2IK+K-400 400
66.5
369.5x3x1
Miormer
IN-21K
3
Mformer-I IN-2IK+K-400
68.1
91.2
1
Mformer-I.
IN.21K
80.2
94.8
1185. x 3 10
Mformer-HR
IN-21K+K-400
67.1
90.6
958.8x3x1
Mffomer-HR
IN-21K
81.1
95.2
958.8
(c) Epic-Kitchens
(d) Kinetics-600
Method
Pretrain
V
Model
Top-1
Top-5
GFLOPs
TSN
77
IN-IK
33.2
60.2
46.0
AttaNAS
80
79.8
94.4
IRN 88
IN-IK
35.3
65.9
45.4
LGD-3D
55
IN-IK
81.5
95.6
TBN
IN-IK
36.7
66.0
47.2
SlowFast
24
81.8
95.1
234x3x 10
TSM
IN-IK
38.3
67.9
49.0
X3D-XI
21
81.9
95.5
48.4xx3 10
Slow
K-400
38.5
65.6
50.0
ToormerHHR
IN-21K
82.4
x
VIVIT-L 3
IN-21K+K-400
44.0
66.4
56.8
VIVT-L X
IN-21K
83.0
95.7
x992xxxx4
Mformer
IN-21K+K-100
43.1
66.7
56.5
MVIT-B-24 21
83,8
96.3
236x 1x5
Mformer-L
IN K-00 400
44.1
67.1
57.6
Mformer
IN-21K
81.6
95.6
369.5 x
Mformer-HR
KN-2
44.5
67.0
58.5
Mformer-I.
IN-21K
82.2
96.0
1185. x 10
1
Mformer-HR
IN-21K
82.7
96.1
958.8x3x100
Table 7: Can train larger models using approximated trajectory attention? We report top-1 and top-5 accuracy (%) on the Kinetics-400 dataset two variants our Motionformer model: Motionformer-B and Motionformer-H. The former uses the base model with exact (E) trajectory attention, while the latter uses a much larger model (ViT-L) and a higher resolution input (336 x 336 pixels) with approximate (A) trajectory attention, i.e., using Orthoformer. We reduce this linear relationship with the Orthoformer approximation, which allows us to fit the model on the GPU.
Model
Base model
Params
Attention
Max memory (GB)
Top-1
Top-5
Mformer-B
ViT-B/224
109. IM
Trajectory (E)
7.3
79.7
94.2
Mformer-H
ViT-L/336
381.9M
Trajectory (A)
22.2
80.0
94.5
algorithm that significantly reduces the memory requirements, the largest bottleneck for transformer models. With these contributions, we obtain state-of-the- results on several benchmark datasets. Nonetheless, approach inherits many of the limitations of transformer models, including poor data efficiency and slow training. Specific to this work, trajectory attention has higher computational complexity than alternative attention operations used for video data. This is attenuated by the proposed approximation algorithm, with significantly reduced memory and computation requirements. However, its runtime is bottlenecked by prototype selection, which is not easily parallelized.
Future work. There are many applications of trajectory attention beyond video action classification, such as those tasks where temporal context is highly important. We see significant potential for using trajectory attention for tracking (28), temporal action localization [83,8] and online action detection 86 62]. among other settings, and leave these as avenues for future work.
Potential negative societal impacts. One negative impact of this research is the significant envi- ronmental impact associated with training transformers, which are large and models. Compared to 3D-CNNs where the compute scales linearly with the sequence length, video transformers scale quadratically. To mitigate this, we proposed an approximation algorithm with linear complexity that greatly reduces the computational requirements. There is also potential for video action recognition models to be misused, such as for unauthorized surveillance.
10

---

Acknowledgments and Disclosure of Funding
We are grateful for support from the Rhodes Trust (M.P.,, the European Research Council Starting Grant (IDIU 638009, D.C.), Qualcomm Innovation Fellowship (Y.A.), the Royal Academy of J.H.), and EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems (EP/L005897// M.P. and Y.A.). Funding for M.P. was received under his Oxford affiliation. We thank Bernie Huang, Dong Guo, Rose Kanjirathnkall, Gedas Bertasius, Mike Zheng Shou, Mathilde Caron, Hugo Touvron, Benjamin Lefaudeux, Haoqi Fan, and Geoffrey Zweig from Facebook Al for their help, support, and discussion around this project. We also thank Max Bain and Tengda Han from VGG for fruitful discussions.
Engineering (RF201819/18/1))

---

References
[1] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- surerripevi learning
of qusuol objects from video. In ECCV. 2020.
[2] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.
Vatt: Transformers for multimodal self-supervised I from raw video, audio and text. 2021.
[3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit:
vidco vision transformer, 2021.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. In NewrlPS, 2020.
[6] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval, 2021.
[7] lz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. 2020.
[8] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemnr, Sam McCandlish, Alec Radford, lya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[10] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[11] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset.
In CVPR, 2017. 2017.
[12] Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[13] Yunpeng Chen, Yannis Kalantidis, Jianshu Li. Shuicheng Yan, and Jiashi Feng. a²-nets: Double attention networks. In NeurlPS, 2018.
[14] Rewon Child, Scott Gray, Alee Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. URL https://ppenct.com/bli wwt..///wwoo/spppssetttss/rrrraa
2019.
[15] Krzysztof Marcin Choromanski, Valeri Likhosherstov, David Dohan, Xingyou Song. Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger. Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR, 2021. [16] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision. In ECCV, 2020.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009,
[18]
Karan Desai and Justin Johnson. VirTex: Learning Visual Representations from Textual Annotations. In CVPR, 2021.
[19]
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer.
In NeurlPS, 2020.
[20]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[21]
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer: Multiscale vision transformers, 2021.
[22] Quanfu Fan, Chun-Fu (Ricarhd) Chen, Hilde Kuchne, Marco Pistoia, and David Cox. More Is Less:
Learning Efficient Video Representations by Temporal Aggregation Modules. In NeurlPS, 2019.
[23] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020.
[24]
Christoph Feichtenhofer. Haoqi Fan. Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
[25] Valentin Gabeur. Chen Sun. Karteek Alahari, and Cordelia Schmid, Multi-modal transformer for video
retrieval. In ECCV, 2020.
[26] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska. Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, and al. The "something something" " video database for learning and evaluating visual common sense. In ICCV, 2017.
[27] Kaiming He. Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[28] João F Henriques, Rui Casciro, Pedro Martins, and Jorge Batista. High-speed tracking with kernelized correlation filters. IEEE fransactions on pattern analysis and machine intelligence, 37(3):883-59, 2014. [29] Po- Yao Huang. Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze. and Alexander Hauptmann. Multilingual multimodal pre-training for shot gross-lingus transfer of vision- language models. In NAACL, 2021.
12

---

[30]
Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper. Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2016.
[31]
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence contrastive random walk.
In NeurlPS, 2020.
[32]
Kenton Lee Jacob Devlin, Ming-Worr Wei Chang and Kristina Toutanova. BERT: Pre-training of deep bidirec-
tional transformers for language understanding, In NAACL, 2018.
[33] Boyuan Jiang, Mengmeng Wang. Weihao Gan, Wei Wu, and Junjic Yan. Stm: Spatiotemporal and motion encoding for action recognition. In ICCV, 2019.
[34] Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning to estimate hidden motions with global motion aggregation. In ICCV, 2021.
[35] Yash Kant, Dhruv Batra, Peter Anderson, Alex Schwing, Devi Parikh, Jiasen Lu, and Harsh Agrawal.
Spatially aware multimodal transformers for textvqa. In ECCV, 2020.
[36] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Viayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, ct al. The kinetics human action video dataset, 2017.
[37] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual
temporal binding for egocentric action recognition. In ICCV, 2019.
[38]
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020. Alexander Klaser, Marcin Marszatek, and Cordelia Schmid. A spatio-temporal descriptor based on 3d- ul-gradients. In BMVC, 2008.
[39]
[40] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[41] Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. In ECCV, 2020.
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-223 -2224, 1998.
[43] Sangho Lee, Youngjae Yu, Gunhee Kim. Thomas Breuel,, Jan Kautz, and Yale Song. Parameter efficient multimodal transformers for video representation learning. In JCLR, 2021.
[44] Liunaan Harold Li, Mark Yatskar, Da Yin. Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and
performant baseline for vision and language, 2019.
[45] Xinyu Li. Yanyi Zhang, Chunhui Liu. Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and
Joseph Tighe. Vidtr: Video transformer without convolutions, 2021.
[46] Yan Li, Bin Ji. Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, 2020.
[47] Ji Lin. Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. 2019. [48] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkorcit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, 2020.
[49] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision- and language tasks. In NeurlPS, 2019.
[50]
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network, 2021.
[51]
Niki Parmar, Ashish Vaswani. Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In JCML, 2018
[52] Mandela Patrick, Yuki M. Asano, Bernie Huang, Ishan Misra, Florian Metze, Joao Henriques, and Andrea
Vedaldi. Space-time crop & attend: Improving cross-modal video representation learning. 2021.
[53] Mandela Patrick, Po- Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F. Henriques,
and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2021.
[54] Zhaofan Qiu, Ting Yao, and Tao Mci. Learning spatio-temporal representation with pscudo-3d residual networks. ICCV. 2017.
[55] Zhaofan Qiu, Ting Yao, Chong Wah Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representa- tion with local and global diffusion. CVPR, 2019.
[56] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):, 2019.
[57] Deva Ramanan, David A. Forsyth. and Andrew Zisserman. Strike a pose: Tracking people by finding stylized poses. In Proc. CVPR, 2005.
[58] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
[59] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, 2020.
[60] P. Scovanner, S. Ali, and M. Shah. A 3-dimensional sift descriptor and its application to action recognition.
In ACM MM, 2007.
[61] Michael Shewry and Henry P Wynn. Maximum entropy sampling. Journal of Applied Statistics, 14(2):665-170, 1987.
[62] Gurkirt Singh. Suman Saha, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin. Online real-time multiple spatiotemporal action localisation and prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 3637-3646, 2017.
[63] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2020.
13

---

[64]
Chen Sun, Fabien Baradel. Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. 2019.
[65]
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019.
[66]
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cans for optical flow using pyramid, warping, and cost volume. CVPR, 2018.
[67]
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers.
In EMNLP, 2019.
[68]
Hao Tan and Mohit Bansal. Vokenization: Improving language understanding with contextualized,
Disual-groundee supervision. In EMNLP. 2020.
[69] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In ICLR,
2021.
[70] Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow, In Proc. ECCV, 2020.
[71] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In ICMI., 2021.
[72] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[73] Du Tran, Heng Wang, Matt Feiszli, and Lorenzo Torresani. Video classification with channel-separated
convolutional networks. In JCCV, 2019.
[74] Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition.
CVPR, 2018.
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In 2017.
[76
Heng Wang and Cordelia Schmid, Action recognition with improved trajectories. In ICCV, 2013, Limin Wang. Yuanjun Xiong, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
[77]
[78] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. In NeurlPS, 2020.
[79] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR,
2018.
[80] Xiaofang Wang. Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani, and Wei Hua. Attentionnas: Spatiotemporal attention cell search for video classification. In ECCV, 2020.
[81] Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Learning track for spatio-temporal action localization. Proceedings of the IEEE international conference on computer vision, pages 3164-3172,
2015.
[82] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
[83] Chao- Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming Hc, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In Proceedings of the IEEF/CVF Conference
on Computer Vision and Pattern Recognition, pages 284-293, 2019.
[84] Saining Xic, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed accuracy trade-offss in vidco classification. In ECCV, 2018
[85]
Yunyang Xiong. Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung. Yin Li, and Vikas
Singh. Nyströmformer: A nyström-basee algorithm for approximating self-attention. In AAAI, 2021. [66] Mingze Xu, Yuanjun Xiong. Hao Chen, Xinyu Li. Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. arXiv preprint arXiv:2 107.03377, 2021.
[87] Yue Zhao, Yuanjun Xiong, and Dahua Lin. Trajectory convolution for action recognition. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2208-2219, 2018. Bolci Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos.
[88]
In ECCV, 2018.
[89] Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In CVPR, 2020.
14


Reviews:
[{"review_id": "mxqbqWWUQk", "reviewer": "Reviewer_8Cfh", "summary": "The paper focused on model related motion information along the temporal domain. In particular, they proposed new drop-in block for video transformers that aggregates information along implicitly determined motion paths. The network is based on attentive mechanisms with Transformer. The new model doesn't rely on quadratic dependence of computation and memory on the input size, and is efficient to be applied to high resolution or long videos. The model was evaluated on multiple standard benchmarks, and it achieved convincing results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The idea of modeling temporal information along the object moving trajectory is convincing.\n\n- The paper writing is fairly good, and the proposed method is novel.\n\n- The proposed method was evaluated on most of standard action recognition benchmarks, and the performance is reasonable and convincing.\n\nWeakness:\n\n- The qualitative results provided in the appendix is too simple. There is only one moving object in the videos, and the attention mainly focuses on the object. It's better to show some complicated cases that at least multiple objects are moving and their corresponding attention maps.\n\n- The proposed method is only evaluated on action recognition, which is not a problem. But as the authors mentioned that their method is efficient and effective for long videos, in the future, it is interesting to show its performance on some untrimmed video tasks, such as temporal action localization [1,2] and online action detection [3,4]. As they are more challenging due to the complicated temporal context, and modeling with trajectory attention could benefit more on these tasks.\n\n[1] Long-Term Feature Banks for Detailed Video Understanding. Wu et al. CVPR.\n\n[2] Learning to track for spatio-temporal action localization. Weinzaepfel et al. ICCV.\n\n[3] Long Short-Term Transformer for Online Action Detection. Xu et al. arXiv.\n\n[4] Online Real-time Multiple Spatiotemporal Action Localisation and Prediction. Singh et al.\n\n- Trajectory convolution [5] shares similar ideas with the proposed method. Reviewing it as related work would be necessary.\n\n[5] Trajectory Convolution for Action Recognition. Zhao et al. NeurIPS.\n\n- It seems that the attention focuses on the \"key\" moving object in the video, could the authors explain more about the difference between the proposed method and non-local?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitations are explained.", "review_text": "", "extra_scores": {"ethical_concerns": "I didn't see any concern about this.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "gpjiiNuUCuw", "reviewer": "Reviewer_CQwE", "summary": "This work proposed a novel drop-in block for video transformers. Different from transformers that work on image or language modality, this block aggregates information along motion paths. The key idea is to design an efficient algorithm to approximation the path (trajectory attention), which requires quadratic dependence of computation and memory on the input size. As a solution, the proposed Motionformer model uses the (1) ViT image transformer model as the base architecture, (2) the separate space and time positional encodings of TimeSformer, (3) and the cubic image tokenization strategy as in ViViT. Experimental results on four video action recognition benchmarks show Motionformer can achieve state-of-the-art performance. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) This work brings a novel idea to video understanding. It implicitly determined motion paths by trajectory attention. It allows the network to (1) aggregate information from multiple views of the object/region, (2) reason the movement of the object/region, and (3) be invariant to camera motion.\n\n(+) The approximation scheme can speed up the calculation of trajectory attention. It reduces the complexity in time and space, which is essential in transformer-based models.\n\n(+) Extensive experiments are conducted on popular video action recognition benchmarks, justifying the effectiveness of this work. \n\n(+) The authors also provide code in the supplementary material. It is beneficial for the video understanding community. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As explained in Section 5, Motionformer has its limitation of poor data efficiency and slow training, which is common for transformer-based methods. Since this work needs to compute trajectory attention, it has higher computational complexity.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "C2p5n0WeO_T", "reviewer": "Reviewer_m5y4", "summary": "The paper proposes a new component for transformer architectures to model trajectory attention by modeling temporal correspondences in dyanmic scenes. They also present approximation techniques to address computation and memory challenges for high-resolution videos. The proposed contributions are demonstrated on video action recognition tasks with SOTA results on multiple datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main contributions: \n- a new block which can be easily integrated into transformer architectures to mode the temporal correpsondence in dynamic scenes (trajectory attention) \n- approximation techqniques (Orthoformer algorithm) to reduce memory usage and minimize quadratic dependency to facilate the handling of long duration, high resolution videos \n- Integration of these contributions into Motionformer model and demonstration on video action recognition task to achieve SOTA results on Kinetics, Something-Something V2, and Epic-Kitchens datasets.\n\nThe trajectory attention is implemented by using the two stages (a spatial attention operation and a 1D temporal attention) as shown in the Figure2.  For the trajectory attention test, the paper proposed Motionformer model which is using the VIT image transformer model as the base architecture and the trajectory attention for the attention mechanism. Through ablation studies, the paper proved that the trajectory attention showed better accuracy than other attention methods (Table 4). In particular, there was a significant improvement in the SSv2 data test. This shows that the proposed model with the trajectory attention can understand fine-grained motion signals more effectively than other attention mechanisms. \nThe limitation of the trajectory attention occurred when it computed the trajectory attention because it’s causing quadratic complexity in both space and time. To overcome this, the approximation algorithm is proposed. It was shown that the proposed approximation algorithm used less memory than other models and showed better results overall in Tables 3 and 5. \nAt the end, Table 6 showed the proposed model (Motionformer) achieved comparable or better performance with fewer parameters than current models (ViViT-L, Tformer-L and so on) for each dataset (Kinetics 400/600 and Something-Something V2)\n\nOverall, I believe the paper makes a strong contribution and should be of interest to the wider community. \n\nQuestions/concerns: \n\n- I wonder how much the training time is different due to the prototype selection runtime bottleneck. How big is the time difference to get results in table3 (c) for 16, 64, and 128 respectively?\n- Although it's already explained in the paper, I want to know how slow the training speed of the proposed method is compared to other models. I wonder how long it took to train each baseline model and the proposed model to get the last table result.\n- I am curious how the proposed trajectory attention mechanism can be applied to other tasks (e.g., human trajectory prediction). Some commentary on the generality of the proposed modeling contributions on other dynamic tasks would be useful. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would encourage the authors to provide more commentary on how the trajectory attention model and approximation techiques can be applied to other domains and tasks which benefit from modeling temporal correspondence. \n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "28XUWif66fJ", "reviewer": "Reviewer_E1g8", "summary": "The paper introduces a novel transformer architecture in which trajectories in video are considered. The model is able to leverage its own self-attention module to build trajectories around relevant objects in the video. Furthermore, authors introduce an approximation through a set of intermediate variables which enables them to reduce the computational cost of the transformer. Authors validate their model against state-of-the-art models in three main video benchmarks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Summary: The architecture proposed is interesting and well validated through ablation experiments. Furthermore, authors show that the proposed model improves performance over the baselines. Finally, the approximation technique introduced is useful to reduce the computational needs. For all those reasons, I believe the paper should be accepted.\n\nStrengths:\n\n- The paper is well written and the novelty aspects are well highlighted and contextualised. \n\n- Introducing elements in the transformer which relates to the nature of the input is an interesting and useful exercise. The community has commonly relied on the transformer architecture as a black box and authors are introducing some previous knowledge in order to improve the performance by tracking trajectories through time. I think this is a good direction and authors show that can have positive impact to the final outcome of the model.\n\n- The paper presents a good ablation study to better understand the design decisions through the paper. Furthermore, they report GFLOPS and memory used, which is very useful for the reader to understand the different models and its advantages and disadvantages. \n\n- The approximation through prototype vectors is a simple but effective mesure to reduce computational complexity. I think this idea can be useful for the community to not only improve this model but also other with similar operations on it. \n\n- The results compared to the state-of-the-art models are strong, specially in the Something-Something and EpicKitchens benchmarks.\n\nWeaknesses / Things to improve: \n\n- The paper story relies on the idea that the model is able to track trajectories of objects through the video. However, only a figure in the supplemental show examples of this tracked objects. I wonder if authors could quantitatively evaluate the tracking ability with a tracking dataset.\n\n- Some of the ablations such as Input Tokenisation or Position Encoding are not strongly related with the main innovation of the paper. I believe that it's very interesting to report those numbers for the readers, but moving those experiments to supplemental would give authors a bit more space to discuss some of the results in the supplemental such as scaling-up the transformer models like in Table 2 of the supplemental.\n\n- It would be interesting for the reader to see a more detailed analysis on the classes where the performance difference is larger with and without the trajectory block. I think it would improve the understanding of the model by the reader. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the societal impact and limitations of the work in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on attention in video transformers.  In particular, a novel trajectory-focused self-attention approach is developed, which essentially tracks space-time patches.  The reviewers appreciated the novel method, clear presentation, and effectiveness in empirical evaluations.  The new approach forms a drop-in block that could be used in video architectures.  The reviewers were unanimous in recommending acceptance for the paper.

Decision: Accept (Oral)