Long Short-Term Transformer for Online Action Detection
Authors: Mingze Xu
Yuanjun Xiong
Hao Chen
Xinyu Li
Wei Xia
Zhuowen Tu
Stefano Soatto
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Online Action Detection
Action and Behavior Recognition
Video Analysis
Transformers
Recurrent Neural Networks

Abstract: We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e.g., 2048 frames spanning of up to 8 minutes), together with an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared to prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS'14, TVSeries, and HACS Segment. Code has been made available at: https://xumingze0308.github.io/projects/lstr.

Long Short-Term Transformer for Online Action Detection
Mingze Xu
Yuanjun Xiong
Hao Chen Stefano Soatto
Xinyu Li
Wei Xia
Zhuowen Tu Amazon/AWS AI hxen, xxnl, wxia, ztu, soattos))amazon.ccooom
(xumingze, yuanjx, yuan
Abstract
We present Long Short-term TRansformer (LSTR), temporal modeling algo- rithm for online action detection, which employs a long- and short-term memory mechanism model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e.g., 2048 frames spanning of up to minutes), together with an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS' TVSeries, and HACS Segment. Code has been made available https::/xuminggz000830 www.eb....b............... 10/ wo/prooectss/stzz
Introduction
Given an incoming stream of video frames, online action detection [4] is concerned with the task of classifying what is happening each frame without seeing the future. Unlike offline methods that assume the entire video is available, online methods process the data causally, up to the current time. In this paper, we present an online temporal modeling algorithm capable of capturing temporal relations on prolonged sequences up to 8 minutes long, while retaining fine granularity of the event in the representation. This is achieved by modeling activities different temporal scales, so to capture variety of events ranging from bursts to slow trends.
Specifically, propose method, named Long Short-term TRansformer (LSTR), to jointly model long- and short-term temporal dependencies. LSTR has two main advantages over prior work. 1) It stores the history directly thus avoiding the pitfalls of recurrent models [18] 50 28 T0] Back-
propagation through time, BPTT, not needed as the model can directly attend to any useful frames from memory. 2) It separates long- and short-term memories, which allows modeling short-term context while extracting useful correlations from the long-term history. This allows us to compress the long-term history without losing important fine-scale information.
As shown in Fig. we explicitly divide the entire history into the long- and short-term memories and build our model with encoder-decoder architecture. Specifically, the LSTR encoder compresses and abstracts the long-term memory into a latent representation of fixed length, and the LSTR decoder uses short window of transient frames perform self-attention and cross-attention operations on the extracted token embeddings from the LSTR encoder. In the LSTR encoder, an extended temporal support becomes beneficial in dealing with untrimmed, streaming videos by devising two- stage memory compression, which is shown to be computationally efficient in both training and inference. Our overall long short-term Transformer architecture gives rise an effective and efficient representation for modeling prolonged sequence data.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Long Short-term TRansformer (LSTR)
Long-Tern Memary
LSTR Encoder
Classifcation time
LSTR Decoder
Store
Short- Temt Memory
Figure 1: Overview of Long Short-term TRansformer (LSTR). Given a live streaming video, LSTR sequentially identifies the actions happening in each incoming frame by using an encoder- decoder architecture, without future context. The dashed brown arrows indicate the data flow of the long- and short-term memories following the frst-in-first-our (FIFO) logic. (Best viewed in color.)
We validate LSTR on standard benchmark datasets (THUMOS' 14 30 TVSeries [4], and HACS Segment (76)). These have distinct characteristies such video length spanning from few seconds to tens of minutes. Experimental results establish LSTR as the state-of-the-an for online action detection, Ablation studies further showcase LSTR's abilities in modeling long video sequences.
2 Related Work
Online Action Detection. Temporal action localization aims detect the onset and termination of action instances after observing the entire video [22
38 Embodied perception, however, requires causal processing 73 where we can only process the data up the present. Online action detection focuses on this setting 14). RED 222 uses reinforcement loss to encourage recognizing actions early possible. TRN 722 models greater temporal context by simultaneously performing online action detection and anticipation. IDN [19] learns discriminative features and accumulates only relevant information for the present. LAP-Net [46] proposes an adaptive sampling strategy to obtain optimal features. PKD [77 transfers knowledge from offline to online models using curriculum learning. As with early action detection [27 40), Shou er al. [54] focus on online detection of action start (ODAS). StartNet (4 decomposes ODAS into two stages and learns with policy gradient. WOAD 25 uses weakly-supervised learning with video-level labels.
71
23
53
78
5 37
Temporal/Sequence Modeling. Causal time series analysis has traditionally assumed the existence of latent "state" variable that captures all information in past data, and is updated using only the current datum [49 28 10]. While the Separation Principle ensures that such a state exists for linear- Gaussian time series, in general it is not possible summarize all past history of complex data in finite-dimensional sufficient statistic. Therefore, we directly model the history, in accordance with other work on video understanding (69 45 68 Earlier work on action recognition usually relies on heuristic sub-sampling (typically 3 to 7 video frames) for more feasible training [74] 62 21 41 (57]. 3D ConvNets
[59 60] are used to perform spatio-temporal feature modeling on more frames, but they fail to capture temporal correlations beyond their receptive field. Recently, Wu et al. [69] propose long-term feature banks to capture objects and scene features, but discarding their temporal order which is clearly informative. Most of work above does not explicitly separate the long- and short-term context modeling, but instead integrates all observed features with simple mechanisms such as pooling or concatenation. We are motivated by work in Cognitive Science [44, 12 9 35 that has shed light on the design principles for modeling long-term dependencies with attention
mechanism 666 13 48 75).
Transformers for Action Understanding. Transformers have achieved breakthrough success in and are adopted in computer vision for image recognition [17 58) and object detec- tion Recent papers exploit Transformers for temporal modeling tasks in videos, such as action recognition 43
NLP 47 [5]
36
3 and temporal action localization [2] 56), and achieve promising results.
However, computational and memory demands result in most work being limited to short video clips, with few exceptions [3 6 that focuses on designing Transformers to model long-range context. The mechanism for aggregating long- and short-term information is relatively unexplored 32
3 Long Short-Term Transformer
Given a live streaming video, our goal is to identify the actions performed in each video frame using only past and current observations. Future information not accessible during inference. Formally, a streaming video at time t is represented by batch of past frames Is (lerr.-.. .4c.

---

LSTR Encoder
LSTR Decoder
Transformer Decoder Unit
Long- Merrory
Compreeed Memory
Meerory
1x Transfommer Decoder Unit
I enc X Transfomeer Units
doc
mu
Transformer Decoder Units
fication
eddy Murrn
Feee Farwert
m
Malt Freed
Atsention
Fiitwdinge
Torn Monory
AAoeE Mask
Pastooes
Teokinal
Mell emeedoor
Live Streaming Video
Carren
Figure 2: Visualization of Long Short-Term Transformer (LSTR), which is formulated in an encoder-decoder manner. Specifically, the LSTR encoder compresses the long-term memory of size my to nı encoded latent features, and the LSTR decoder references related context information from the encoded memory with the short-term memory of size my for action recognition of the present. The LSTR encoder and decoder are built with Transformer decoder units [61], which take the input tokens (dark green arrows) and output tokens (dark blue arrows) as inputs. During inference, LSTR processes every incoming frame in online manner, absent future context. (Best viewed in color.)
which reads "7 up to time 1." The online action detection system receives I input, and classifies the action category ût belonging to one of K + 1) classes, út E {0. 1, K). ideally using the posterior probability P(o- k[I'), where k 0 denotes the probability that no event is occurring at frame t. We design our method by assuming that there is pretrained feature extractor [62] that processes each video frame into a feature vector f, E RC of C' dimensions" These vectors form (T x dimeennsio temporal sequence that serves as the input of our method.
3.1 Overview
Our method is based on the intuition that frames observed recently provide precise information about the ongoing action instance, while frames over an extended period offer contextual references for actions that are potentially happening right now. We propose Long Short-term T'Ransformer (LSTR) in an explicit incoder-decoder manner, as shown in Fig. 2 In particular, the feature vectors of m frames in the distant past are stored in a long-term memory, and short-term memory stores the features of my recent frames. The LSTR encoder compresses and abstracts features in long-term memory to an encoded latent representation of n4 vectors. The LSTR decoder queries the encoded long-term memory with the short-term memory for decoding, leading to the action prediction ÿt- This design follows the line of thought in combining long- and short-term information for action understanding [16, 62 69), but addresses several key challenges to efficiently achieve this goal, exploiting the flexibility of Transformers [61].
3.2 Long- and Short-Term Memories
We store the streaming input of feature vectors into two consecutive memories. The first is the short-term memory which stores only a small number of frames that are recently observed. We implement it with ienss--i-fiis--ouu (FIFO) queue of ms slots. At time T, it stores the feature vectors as Mş = {fr,... fm1m+ When a frame becomes "older" than mg time steps, it graduates from Ms and enters into the long-term memory, which is implemented with another FIFO queue of m. slots. The long term memory stores M, = (fr-my- mmg.-.... fr-my-mm my +1}. The long-term memory serves as the input memory the LSTR encoder and the short-term memory serves the queries for the LSTR decoder. In practice, the long-term memory stores a much longer time span than the short-term memory (mg < m.). A typical choice m, 2048, which represents 512 seconds worth of video contents with 4 frames per second (FPS) sampling rate, and mg = 32 representing
In practice, some feature extractors (59 take consecutive frames to produce one feature vector. Nonetheless, it is still temporally "centered" single frame. Thus the single frame notation here for simplicity.
3

---

8 seconds. We add a sinusoidal positional encoding [6]] to each frame feature in the memories relative to current time T (i.e., the frame at T 7 receives positional embedding of s,).
3.3 LSTR Encoder
The LSTR encoder aims encoding the long-term memory of my feature vectors into latent representation that LSTR can use for decoding useful temporal context. This task requires a large capacity in capturing the relations and temporal context across span of hundreds or even thousands of frames. Prior work on modeling long-term context for action understanding relies on heuristic temporal sub-sampling 66 62 or recurrent networks [16] to make training feasible, at the cost of
losing specific information of each time step. Attention-based architectures, such as Transformer [61] have recently been shown promising for similar tasks that require long-range temporal modeling [51] A straightforward choice for LSTR encoder would be to use a Transformer encoder based on self-attention. However, its time complexity. (m C), grows quadratically with the memory sequence length m,, This limits our ability to model long-term memory with sufficient length to cover long videos. Though recent work [64] has explored self-attention with linear complexity, repeatedly referencing information from the long-term memory with multi-layer Transformers is still computationally heavy. LSTR, we propose to use the two-stage memory compression mechanism based on Transformer decoder units [61 to achieve more effective memory encoding.
The Transformer decoder unit [6]] takes two sets of inputs. The first set includes a fixed number of n learnable output tokens À E _aaxC where C' is the embedding dimension. The second set includes another m input tokens 0 E IR m where m can a rather large number. It first applies one layer of multi-head self- attfetttention on A. The outputs X are then used as queries in an "QKV cross-attention" operation and the input embeddings A serve as key and value. The two steps can be written as
X = SelfAtt((A = Softmax( Ax ) and CrossAttm (XX)) 0) = Softmax( (X') 0"
)0
VC
VC
where a : 1RaxC + gnxC' denotes the intermediate layers between the two attention operations. One appealing property of this design is that it transforms the x C' dimensional input tokens into output tokens n x C dimensions in Onn²C + nmCC time complexity. When n < m, the time complexity becomes linear to m, making it an ideal candidate for compressing long-term memory. This property is also utilized in 32 to efficiently process large volume inputs, such as image pixels.
Two-Stage Memory Compression. Stacking multiple Transformer decoder units on the long-term memory, as 61, can form memory encoder with linear complexity with respect to the memory size m.. However, running the encoder at each time step can still be time consuming. We can further reduce the time complexity with two-stage memory compression design. The first stage has one Transformer decoder unit with no output tokens. Its input tokens are the entire long-term memory of size my. The outputs of the first stage are used as the input tokens to the second stage, which has lene stacked Transformer decoder units and n output tokens. Then, the long-term memory of size m x C is compressed into a latent representation of size n1 x C, which can then be efficiently queried in the LSTR decoder later. This two-stage memory compression design is illustrated in Fig, 2
Compared to an (1 + lene))layer Transformer encoder with Omm) (1 (1+ (enc) time complexity or stacked Transformer decoder units with n output-tokens having O( + nm + (ene) time complexity, the proposed LSTR encoder has complexity of O(n5C' +nnnngCC (nf + (ing) Because both no and ni are much smaller than m., and lane is usually larger than 1, using two-stage memory compression could be more efficient. In Sec. Sec.3.66 we will show that, during online inference, it further enables us to reduce the runtime of the Transformer decoder unit of the first stage. In we empirically found this design also leads to better performance for online action detection.
Sec. 4.5
3.4
LSTR Decoder
The short-term memory contains informative features for classifying actions on the latest time step. The LSTR decoder uses the short-term memory as queries to retrieve useful information from the encoded long-tern memory produced by the LSTR encoder. The LSTR decoder formed by stacking ldec layers of Transform decoder units. It takes the outputs of the LSTR encoder as input tokens and the my feature vectors the short-term memory as output tokens. It outputs m, probability vectors {PT," ; pr-ms+1> } E [0. 1] 1] each p. representing the predicted probability distribution of K action categories and one "background" class at time During inference, we only take the probability vector pT from the output token corresponding to the current time T' for the classification result.

---

However, having the additional outputs on the older frames allows the model to leverage more supervision signals during training. The details will be described below.
3.5 Training LSTR
LSTR can be trained without temporal unrolling and Backpropagation Through Time (BPTT) as in LSTM [28), which is a common property of Transformers [61]. We construct each training sample by randomly sampling an ending time 7 and filling the long- and short-term memories by tracing back in time for mş + m, frames. We use the empirical cross entropy loss between the predicted probability distribution Pr at time T and the ground truth action label yr E {0,1,--, .K]
K
L(yr, (yrrpr;:1
Σ
8(k- yr) log log
k=0
where pl. is the k-th element of the probability vector Py. predicted on the latest frame T. Additionally, we add directional attention mask [6]] to the short-term memory so that any frame in the short-term memory can only depend on its previous frames. In this way, we can make predictions on all frames in the short-term memory as if they are the latest ones. Thus we can provide supervision on every frame in the short-term memory, and the complete loss function L is then
Lr
L(ye,Pe: P::
(2)
T= -m-++]
where Pt. denotes the prediction from the output token corresponding to time t.
3.6 Online Inference with LSTR
During online inference, the video frame features are streamed to the model as time passes. Running LSTR's long-term memory encoder from scratch for each frame results in a time complexity of C + nomr,, and ((n) Reno) for the first and second memory compression stages, respectively. However, at each time step, there is only one new video frame to be updated. We show it is possible to achieve even more efficient online inference by storing the intermediate results for the Transformer decoder unit of the first stage. First, the queries of the first Transformer decoder unit are fixed. So their self-attention outputs can be pre-computed and used throughout the inference. Second, the cross-attention operation in the first stage can be written as
mg+mp-1
exp( +.) V./VC) ma exp((fr-, + 87) q//VC) (fr-r +.). (3) m-mig
CrossAltu(q,, (fr-r + ss
where the index T t is the relative position of a frame in the long-term memory to the latest time T. This calculation depends on the un- nn-nomaaliee attention weight matrix A E mm xno with elements azi (f + s,) q. A can be decomposed into the sum of two matrices A and A We have their elements as or = fy- - qi and at = 8, qi The queries after the first self-attention operation, Q [).. ( Ino and the position embedding 8, fixed during inference. Thus the matrix A can be pre-computed and used for every incoming frame. We additionally maintain a FIFO queue of vectors a, = Q f, of size my. A any time step T can be obtained by stacking all vectors currently in this queue. Updating this queue at each time step requires O(noC") time complexity for the matrix-vector product. Now can obtain the matrix A with only no m. additions by adding A and A together, instead of no X my C' multiplications and additions using Eq. (3). This means the amortized time complexity for computing the attention weights can be reduced to Ono (mL + C)). Although the time complexity of the gross-attentio operation is still O(ngm,,C) C) due the inevitable operation of weighted sum, since usually larger than 1024 (26.. this is still a considerable reduction of runtime. LSTR's 's walltime efficiency is discussed in Sec. 4.6
4 Experiments
4.1 Datasets
We evaluate our model on three publicly-avallabl datasets: THUMOS'14 (30, TVSeries [1] and HACS Segment (76). THUMOS' 14 includes over 20 hours of sports video annotated with 20 actions.
We follow prior work 772 19 and train on the validation set (200 untrimmed videos) and evaluate on the test set (213 untrimmed videos). TVSeries contains 27 episodes of 6 popular TV series, totaling 16 hours of video. The dataset is annotated with 30 realistic, everyday actions (e.g., open door).
HACS Segment is a large-scals scale dataset of web videos. It contains 35,300 untrimmed videos over 200 human action classes for training and 5,530 untrimmed videos for validation.
5

---

4.2 Settings
Feature Encoding. We follow the experimental settings of state-of-the-art methods 72 We extract video frames at 24 FPS and set the video chunk size to 6. Decisions are made at the chunk level, and thus accuracy is evaluated at every 0.25 second. For feature encoding, we adopt the TSN [62] models implemented in open-source toolbox [I]]. Specifically, the features are extracted by one visual model with the ResNet-50 (26) architecture from the central frame of each chunk and one motion model with the BN-Inception Ki architecture from the stacked optical flow fields between consecutive frames [62]. The visual and motion features are concatenated along the channel dimension the final feature f. We experiment with feature extractors pretrained on two
datasets, ActivityNet and Kinetics.
Implementation Details. We implemented our proposed model in PyTorch I1 and performed all experiments on a system with Nvidia V100 graphics cards. For Transformer units, we set their number of heads as 16 and hidden units 1024 dimensions. To learn model weights, we used the Adam 334 optimizer with weight decay x 10-5 The learning rate was linearly increased from zero to x 10 in the first 2/5 of training iterations and then reduced to zero following cosine function. Our models were optimized with batch size of 16, and the training was terminated after 25 epochs.
Evaluation Protocols. We follow prior work and use per-frame mean average precision (mAP) evaluate the performance of online action detection. We also use per-frame calibrated average precision (cAP) [14] that was proposed for TVSeries to correct the imbalance between positive and negative samples, cAP = E, ePrec(k) I [k) P, where cPrec TP/ FP/w), T(k) is if frame l is true positive, P is the number of true positives, and w is the negative and positive ratio.
4.3 Comparison with the State-of-the-art Methods
Table 1: Online action detection and anticipation results on THUMOS'I 14 and TVSeries in terms of mAP and cAP, respectively. For online action detection, LSTR outperforms the state-of-the- methods on THUMOS' 14 by 3.7% and 2.4% in mAP and on TVSeries by 2.8% and 2.7% in cAP, using ActivityNet and Kinetics pretrained features, respectively. LSTR also achieves promising results for action anticipation. *Results reproduced by using their papers' default settings.
(a) Results of online action detec-
(b) Results of online action detec-
(c) Results of action anticipation using Activity Net features
tion using ActivityNet features
tion using Kinetics features
THUMOS' TVSeries
THUMOS' TVSeries
THUMOS' TVSeries
mAP (%)
meAP (%)
mAP (%)
mcAP (%)
mAP (%)
mcAP (%)
CDC
53
44.4
FATS A3
59.0
84.6
22
34.4
72.5
RED
22
45.3
IDN
60.3
86.1
ED 22
36.6
74.5
TRN
72
47.2
83.7
TRN 72
62.1
86.2
RED 22
37.5
75.1
FATS
33
$1.6
81.7
PKD 7
64.5
86.4
TRN
38.9
75.7
IDN
19
50.0
84.7
WOAD
67.1
TTM
40.9
77.9
LAP
46
53.3
85.3
LFB 69
64,8
85.8
LAP
46
42.6
78.7
TFN
20
55.7
85.0
LSTR (ours)
69.5
89.1
LSTR (ours)
50.1
80.8
LFB
61.6
84.8
LSTR (ours)
65.3
88.1
Table 2: Online action detection results when only portions of videos considered in cAP (%) on T'Series (e.g., 80%-90% means only frames of this range action instances were evaluated). LSTR outperforms existing methods every time stage, especially boundary locations.
Portion of Video
Features
0-10%
10-20%
20-30%
30-40%
40-50%
50-60%
60-70%
70-80%
80-90%
90-1009
TRN
72
78.8
79.6
80.4
81.0
81.6
81.9
82.3
82.7
82,9
83.3
IDN
19
80.6
81.1
81.9
82,3
82.6
82.8
82.6
82.9
83.0
83.9
AetivityNet
TFN
20
83.1
84.4
85.4
85,8
87.1
88.4
87.6
87.0
86.7
85.6
LSTR (ours)
83.6
85.0
86,3
87.0
87.8
88,5
88.6
88.9
89.0
88.9
IDN
81.7
83.1
82.9
83.2
83.2
83.0
83.3
86.6
PKD 7
Kinetics
82.1
83.5
86.1
87.2
88.3
88.4
89.0
88.7
88.9
87.7
LSTR (ours)
84.4
85.6
87.2
87.8
88.8
89.4
89.6
89.9
90.0
90.1
We compare LSTR against other state-of-the-arr methods 72 and HACS Segment. Specifically, on THUMOS' 14 and TVSeries, we implement LSTR with the long- and short-term memories of 512 and 8 seconds, respectively, On HACS Segment, we reduce
19
25
THUMOS' 14, TVSeries,

---

the long-term memory to 256 seconds, considering that its videos are strictly shorter than 4 minutes. For LSTR, we implement the two-stage memory compression using Transformer decoder units. We set the token numbers to no = 16 and n = 32 and the Transformer layers to lone 2 and &dee 2.
4.3.1 Online Action Detection
THUMOS' 14. We compare LSTR with recent work on THUMOS' 14, including methods that use
3D ConvNets [33 and RNNs 170 19 25], reinforcement learning [22), and curriculum learning 777] Table Ta and Tb shows that LSTR significantly outperforms the the state-of-the-ar methods (69 25
by 3.7% and 2.4% in terms of mAP using ActivityNet and Kinetics pretrained features, respectively.
TVSeries. Table la and Tb show the online action detection results that LSTR outperforms the state-of-the-art methods 20 77 by 2.8% and 2.7% in terms of cAP using ActivityNet and Kinetics pretrained features, respectively. Following prior work [14), we also investigate LSTR's performance at different action stages by evaluating each decile (ten-percent interval) of the video frames separately. Table shows that LSTR outperforms existing methods at every stage of action instances.
HACS Segment. LSTR achieves 82.6% on HACS Segment in term of mAP using Kinetics pretrained features. Note that HACS Segment new large-scale dataset with only a few previous results. LSTR outperforms existing methods RNN 28 (77.6%) by 5.0% and TRN 72 (78.9%) by 3.7%.
4.3.2 Action Anticipation
We extend the idea LSTR to action anticipation for up to seconds (i.e., 8 steps in 4 FPS) into the future. Specifically, concatenate another 8 learnable output tokens (with positional embedding) after the short-term memory in the LSTR decoder to produce the prediction results accordingly. Table shows that LSTR significantly outperforms the state-of of-the-art methods [72] 46 by 7.5% mAP on THUMOS and 2.1 % cAP on T'Series, using ActivityNet pretrained features.
4.4 Design Choices of Long- and Short-Teree Memories
Table 3: Results of LSTR using downsampled long-term memory on THUMOS' 14 in mAP (%). In particular, we use long-term memory of 512 seconds and short-term memory as 8 seconds.
Temporal Stride
32
64
128
LSTR
69.5
69.5
69.5
69.2
68.7
67.3
66.6
65.9
We experiment for design choices long- and short-term memories. Unless noted otherwise, we use THUMOS' 14, which contains various video lengths, and Kinetics pretrained features.
Lengths of long- and short-term memories. We first analyze the effect of different lengths of long-term m, and short-term my memory. In particular, we test mg E {4, 8, 16) seconds with my starting from 0 second (no long-term memory). Note that we choose the max length (1024 seconds for THUMOS' 14 and 256 seconds for HACS Segment) to cover lengths of 98% videos, and do not have proper datasets to test longer my Fig. 3 shows that LSTR is beneficial from larger my in most cases. In addition, when my is short ( 16 in our cases), using larger my obtains better results and when my is sufficient (2 32 in our cases), increasing my does not always guarantee better performance.
Effect of Memory Length THUMOS'14 14
Effect of Memory Length HACS Segment
70.0
83.0
69.0
82.0
81.0
mAP (%)
80.0
67.0
66.0
secs Short-Term Memory secs Short-
79.0
Term Memory secs Short-Term Memory w/ RNN
78.0
- 4 secs Short- Torm Memory
65.0
77.0
8 secs Short: Term Memory
64.0
16 secs Short- Term Memory
76.0
16 secs Short- Term Memory
63.0
75.0
0
8
16
32
64
128
256 5i2 1024
16
32
64
128
256
Long- Term Memory Length (secs)
Long- Term Memory Length (secs)
Figure 3: Effect of using different lengths of long- and short-term memories.
Can we downsample long-term memory? We implement LSTR with vns as 8 seconds and as 512 seconds, and test the effect of downsampling long-term memory. Table 3 shows the results that downsampling with strides smaller than 4 does not cause performance drop, but more aggressive
7

---

strides dramatically decrease the detection accuracy. Note that, when extracting frame features in 4 FPS, both LSTR encoder (no 16) and downsampling with stride 128 compress the long-term memory to 16 features, but LSTR achieves much better performance (69.5% vs. 65.9% in mAP). This demonstrates the effectiveness of our "adaptive compression" compared to heuristics downsampling.
Can we compensate reduced memory length with RNN? We note that LSTR's performance notably decreases when it can only access very limited memory (e.g., my + ms S 16 seconds). Here we test if RNN can compensate LSTR's 's reduced memory or even fully replace the LSTR encoder. We implement LSTR using ms seconds with an extra Gated Recurrent Unit (GRU) [0] (its architecture is visualized in the Supplementary Material) to capture all history outside the long- and short-term memories. The dashed line in Fig. F shows the results. Plugging-in in RNNs indeed improves the performance when m is small, but when m, is large (2 64 seconds), it does not improve the accuracy anymore. Note that RNNs are not used in any other experiments this paper.
4.5 Design Choices of LSTR
Table 4: Results of different designs of the LSTR encoder and decoder. The length of short-term memory is set to 8 seconds. "TR" denotes Transformer. The last row is our proposed LSTR design.
Length of Long- Term Memory my (secs)
LSTR Encoder
LSTR Decoder
16
32
64
128
256
512
1024
N/A N/A TR Encoder Projection Layer
TR Encoder
65.7
66.8
67.1
67.2
67.3
66.8
66.5
66.2
TR Decoder
66.5
67.3
67.7
68.1
68.3
67.9
67.0
66.5
TR Decoder
65.9
66.4
66.7
67.4
67.5
67.2
67.0
66.6
TR Decoder
66.2
67.1
67.4
67.7
67.5
67.2
66.9
66.8
TR Decoder
TR Decoder
66.1
67.1
67.4
68.0
68.5
68.6
68.7
68.7
TR Decoder + TR Encoder
TR Decoder
66.2
67.3
67.6
68.4
68.6
68.8
68.9
69.0
TR Decoder + TR Decoder
N/A
64.0
64.7
65.9
66.1
66.5
66.2
65.4
65.2
TR Decoder + TR Decoder
TR Decoder
66.6
67.8
68.2
68.8
69.2
69.4
69.5
69.5
We continue to explore the design trade-offss of LSTR. Unless noted otherwise, we use short-term memory of seconds, long-term memory of 512 seconds, and Kinetics pretrained features.
Number of layers and tokens. First, we assess using different numbers of token embeddings (i.e., ny and n2) in LSTR encoder. Fig 4 (left) shows that LSTR is quite robust different choices (the best and worst performance gap is only about 1.5%). but using no = 16 and accuracy. Second, we experiment for the effect of using different numbers of Transformer decoder units (i.e., lene and dee)). As shown in Fig (right), LSTR does not need a large model to get the best performance, and in practice, using more layers can cause overfitting.
32 gets highest
Different Number of Token Embeddings
Different Number of Transformer Decoder Units
70.0
70.0
69.5
69.5
mAP (%)
69.0
69.0
mAp
68.5
68.5
71
16
Cese =1 fas 2
68.0
7kg
32
68.0
70 64
feee
67.5
67.5
1
16
32
64
7gg
Edst
Figure 4: Left: Results of different number of token embeddings for our two-stage memory compres- sion. Right: Results of different number of Transformer decoder units for lene and &
Can we unify the temporal modeling using only self-attention models? We test long-term M, and short-term Ms memory can be learned whole using self-attention models. Specifically, we concatenate M, and Mg and feed them into a standard Transformer Encoder [6]] with a similar model size to LSTR. Table 4 (row 8 vs. especially when m, is large (e.g.. 69.5% vs. 66.5% when my = 512 and 66.6% vs. 65.7% when my, 8). This shows the advantage of LSTR for temporal modeling on long- and short-term context.
1) shows that LSTR achieves better performance
Can we remove the LSTR encoder? We explore this by directly feeding M,, into the LSTR decoder, and using Mg as tokens to reference useful information. Table shows that LSTR outperforms this
8

---

baseline (row 8 vs. row 2), especially when my is large. We also compare it with self-attention models (row 1) and observe that although neither of them can effectively model prolonged memory, this baseline outperforms the Transformer Encoder. This also demonstrates the effectiveness of our idea of using short-term memory to query related context from long-range context.
Can the LSTR encoder learn effectively using sell-attention? To evaluate the "bottleneck" design with cross-attention LSTR encoder, we try modeling Mr, using standard Transformer Encoder units 6n. Note that this still captures M,, and Ms with the similar workflow of LSTR, but does not compress and encode M2 using learnable tokens. Table (row 8 vs. row 3) shows that LSTR outperforms this baseline with all my settings. In addition, the performance of this baseline decreases when my gets larger, which suggests the superior ability of LSTR for modeling long-range patterns.
How to design the memory compression for the LSTR encoder? First, we use projection layer, consisting of a learnable matrix of size ni my followed by MLP layers, compress the long-term memory along the temporal dimension. Wable|4shows that using this simple projection layer (row 4) slightly outperforms the model without long-term memory (row 1). but worse than iitention-based compression methods. Second, we evaluate the one-stage design with leno + 1 Transformer decoder units. Table (row 8 vs. row 5) shows that two-stage compression is stably better than one-stage, and their performance gap gets larger when using larger m (0.5% when m, = 8 and 0.8% when my 512). Third, we compare cross- cross-attentin attention and self-attention for two-stage compression by replacing the second Transformer decoder with Transformer encoder. LSTR stably outperforms this baseline (row 6) by about 0.5% in mAP, However, its performance is still better than models with one-stage compression of about 1 1.3% (row 6 vs. row 3) and 0.3% (row 6 vs. row 5) on average.
Can we remove the LSTR decoder? We remove the LSTR decoder to evaluate its contribution. Specifically, we feed the entire memory to LSTR encoder and attach multi-layer (MLP) classifier on its output tokens embeddings. Similar the above experiments, we increase the model size to ensure fair comparison. Table shows that LSTR outperforms this baseline (row 7) by about 4% on large m (e.g.. 512 and 1024) and about 2.5% on relative small my (e.g.. 8 and 16).
Table 5: Results of LSTR using different memory integration methods in mAp (%). Our pro- posed integration method using cross-attention stably outperforms the heuristic methods.
Length of Long-Term Memory my (secs)
Memory Integration Methods
8
16
32
64
128
256
512
1024
Average Pooling
66.1
67.0
67.3
67.5
68.2
68.4
68.6
68.6
Concatenation Cross-Attention (ours)
65.9
67.2
67.5
67.7
68.4
68.5
68.7
68.6
66.6
67.8
68.2
68.8
69.2
69.4
69.5
69.5
Cross-attention vs. heuristics for integrating long- and short-term memories. We explore inte- grating the long- and short-term memories by using average pooling and concatenation. Specifically, converted a vector of C elements by channel-wise
the encoded long-term features of size n1 averaging, and the short-term memory of size ms x C' is encoded by Eaee Transformer encoder units. Then, each slot of the short-term features either averaged or concatenated with the long-term feature vector for action classification. Note that these models still benefit from LSTR's effectiveness for long-term modeling. Table 5 shows that using average pooling and concatenation obtain comparable results, but LSTR with cross-attention stably outperforms these baselines.
4.6 Runtime
Table 6: Runtime of LSTR with different design choices. The last row is our proposed LSTR design.
Frames Per Second (FPS)
LSTR Encoder LSTR Decoder
OptFlow Computation
RGB Feature Extraction
OptFlow Feature
LSTR
Extraction
N/A TR Encoder
TR Encoder
43.2
TR Decoder
50.2
8.1
70.5
14.6
TR Decoder
TR Decoder
59.5
TR Decoder + TR Decoder
TR Decoder
91.6
We report LSTR's runtime in frames per second (FPS) on a system with a single V100 GPU, and use the videos from THUMOS' 14 dataset. The results are shown in Table

---

We start by comparing the runtime between LSTR's different design choices without considering the pre-processing (e.g., feature extraction). First, LSTR runs at 91.6 FPS using our two-stage memory compression (row 4), whereas using the one-stage design runs at slower 59.5 FPS (row 3). Our two-stage design is more efficient because does not need reference information from the long-term memory multiple times, and can be further accelerated during online inference (see Sec.B.6). Second, we test the LSTR encoder using self-attention mechanisms (row 2). This design does not compress the long-term memory, thus increasing the computational cost of both the LSTR encoder and decoder, leading to slower speed 50.2 FPS. Third, we test the standard Transformer Encoder [61] (row 1), whose runtime speed, 43.2 FPS, is about 2x slower than LSTR.
We also compare LSTR with state-of -ate-offtheee-ee the-hhear recurrent models. As we are not aware of any prior work that reports their runtime, we test TRN (72) using their official open-source code [2]. The result shows that TRN runs at 123.3 FPS, which is faster than LSTR. This is because recurrent models abstract the entire history compact representation but LSTR needs to process much more information. On the other hand, LSTR achieves much higher performance, outperforming TRN by about 7.5% in mAp THUMOS 14 and about 4.5% in cAP on TVSeries.
For end-to-end online inference, we follow the state-of-the-art methods
25 and build LSTR
on two-stream features [62]. LSTR together with pre-processing techniques run at 4.6 FPS. Table 6 shows that the speed bottleneck is the motion feature extraction - it accounts for about 90% of the total runtime including the optical flow computation with DenseFlow [63]. One can improve the
efficiency largely by using taal-tmme optical flow extractors (e.g., PWC-Net (55)) or using only visual
features extracted by a light- backbone (e.g., MobileNet 29 and FBNet 67
4.7
Error Analysis
Table 7: Action classes with highest and lowest performance on THUMOS' 14.
Action Classes HammerThrow PoleVault LongJump
Diving BaseballPitch FrisheeCatch
Billiands CricketShot
AP (%)
92.8
89.7
86.9
86.7
55.4
49.4
39.8
38.6
25
Figure 5: Failure cases on THUMOS' 14. Action classes from left right are "BaseballPitch", "FrisbeeCatch", "Billiards", and "CricketShot". Red circle indicates where the action is happening.
In Table 7 we list the action classes from THUMOS' 14 where LSTR gets the highest (color green) and the lowest (color red) per-frame APs. In Fig. 5 we illustrate four sample frames with incorrect predictions. More visualizations are included in the Supplementary Material. We observe that LSTR sees decrease in detection accuracy when the action incurs only tiny motion or the subject is very far away from the camera, but excels at recognizing actions with long temporal span and multiple stages, such as Vault" and "Long Jump". This suggests we may explore extending the temporal modeling capability of LSTR to both spatial and temporal domains.
5 Conclusion
We present LSTR which captures both long- and short-term correlations in past observations of time series by compressing long-term memory into encoded latent features and referencing related temporal context from them with short-term memory. This demonstrates the importance of separately modeling long- and short-term information and then integrating them for online inference tasks. Experiments on multiple datasets and ablation studies validate the effectiveness and efficiency of the I.STR design in dealing with prolonged video sequences. However, we note that LSTR is operating only on the temporal dimension. An end-to-end video understanding system requires simultancous spatial and temporal modeling for optimal results. Therefore extending the idea of LSTR to spatio-temporal modeling remains an open yet challenging problem.
6 Acknowledgment and Disclosure of F Funding
We thank the anonymous reviewers for their helpful suggestions. This work was funded by Amazon.
10

---

References
[1] http://pptorch.orr//
[2]
pytorch
[3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. htt:::003./59/// 2021.
[4] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? wxXi:2202.05095. 2021.
[5] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. SST: Single-stream temporal action proposals. In CVPR, 2017.
[6] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv:2006. 11527, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the Kinetics dataset. In CVPR, 2017.
[9] Marvin M Chun. Visual working memory visual attention sustained internally over time. Neuropsychologia, 2011.
[10] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014.
[11] MMAction2 Contributors. Openmmlab's next generation video understanding toolbox and benchmark. htttps://iite com/ wwwooppeaanaaaaeeooa.o..ea...aae www.aa//mmaaaaee.. 2020.
[12] Nelson Cowan. Attention and memory: An integrated framework. Oxford University Press, 1998.
[13] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov, Transformer-XL: Attentive language models beyond fixed-length context. ACL, 2019.
[14] Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snock, and Tinne Tuytelaars. Online action detection. In ECCV, 2016.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACI, 2019.
[16] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
[18] Jeffrey L Elman. Finding structure in time. Cognitive science, 1990.
[19] Hyunjun Eun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Learning to discriminate information for online action detection. In CVPR, 2020.
[20] Hyunjum Eun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Temporal filtering networks for online action detection. Pattern Recognition, 2021.
[21] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recognition. CVPR, 2017.
[22] Jiyang Gao, Zhenheng Yang, and Ram Nevatia. RED: Reinforced encoder-decoder networks for action anticipation. In BMVC, 2017.

---

[23] Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, and Ram Nevatia. TURN TAP: Temporal unit regression network for temporal action proposals. ICCV, 2017.
[24] Mingfei Gao, Mingze Xu, Larry S Davis, Richard Socher, and Caiming Xiong. Startnet: Online detection of action start in untrimmed videos. In ICCV, 2019.
[25] Mingfei Gao, Yingbo Zhou, Ran Xu, Richard Socher, and Caiming Xiong. WOAD: Weakly supervised online action detection in untrimmed videos. In CVPR, 2021.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Minh Hoai and Fernando De la Torre. Max-margin early event detectors. In IJC,, 2014.
[28] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[29] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXivv1704.04880 2017.
[30] Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos "in the wild". CVIU, 2017.
[31] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICMI., 2015.
[32] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv:2103..3226, 2021.
[33] Young Hwi Kim, Seonghyeon Nam, and Seon Joo Kim. Temporally smooth online action detection using cycle-consistent future anticipation. Pattern Recognition, 2021.
[34] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[35] Anastasia Kiyonaga and Tobias Egner. Working memory as internal attention: Toward an integrative account of internal and external selection processes. Psychonomic bulletin & review, 2013.
[36] Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. 1144 2021.
[37] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: Boundary sensitive network for temporal action proposal generation. In ECCV, 2018.
[38] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. BMN: Boundary-matching network for temporal action proposal generation. In ICCV, 2019.
[39] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multi-granularity generator for temporal action proposal. In CVPR, 2019.
[40] Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in Istms for activity detection and early detection. In CVPR, 2016.
[41] Antoine Miech, Ivan Laptev, and Josef Sivic. Learnable pooling with context gating for video classification. urXiv:1706.06600 2017.
[42] Megha Nawhal and Greg Mori. Activity graph transformer for temporal action localization. arXiv:2101.08540, 2021.
[43] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. arXiv:2102.00719, 2021.
12

---

[44] Klaus Oberauer. Working memory and attention conceptual analysis and review. Journal of cognition, 2019.
[45] Seoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time time memory networks. CVPR, 2019.
[46] Sanqing Qu, Guang Chen, Dan Xu, Jinhu Dong, Fan Lu, and Alois Knoll. LAP-Net: Adaptive features sampling via learning action progression for online action detection. arXiv:2011.07915, 2020.
[47] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[48] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. ICLR, 2020.
[49] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by pack-propagating errors. Nature, 1986.
[50] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. ITSS, 1997.
[51] Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor An image is worth 16x16 words, what is video worth? arXiv: 13915, 2021.
[52] Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cans. In CVPR, 2016.
[53] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. CDC: noavolutional-de-coovlurrr networks for precise temporal action localization untrimmed videos. In CVPR, 2017.
[54] Zheng Shou, Junting Pan, Jonathan Chan, Kazuyuki Miyazawa, Hassan Mansour, Anthony Vetro, Xavier Giro-i Nieto, and Shih-Fu Chang. Online action detection in untrimmed, streaming videos-modeling and evaluation. In ECCV, 2018
[55] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: Cans for optical flow using pyramid, warping, and cost volume. In CVPR, 2018
[56) Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Relaxed transformer decoders for direct action proposal generation. arXiv:2102.01894, 2021.
[57] Yongyi Tang, Xing Zhang, Lin Ma, Jingwen Wang, Shaoxiang Chen, and Yu-Gang Jiang. Non-local netvlad encoding for video classification. ECCVW, 2018.
[58] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012. /2877, 2020.
[59] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[60] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz. Kaiser, and Illia Polosukhin. Attention is all you need. arXiv: /706v:/706.033 037622 2017.
[62] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Lue Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
(63] Shiguang Wang*, Zhizhong Li*, Yue Zhao, Yuanjun Xiong, Limin Wang, and Dahua Lin. denseflow. attps:///githn cwm/
2020.
13

---

[64] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. 04440,,006.04768, 2020.
[65] Wen Wang, Xiaojiang Peng, Yanzhou Su, Yu Qiao, and Jian Cheng. TTPP: Temporal transformer with progressive prediction for efficient action anticipation. Neurocomputing, 2021.
[66] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[67] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019.
[68] Chao- Yuan Wu and Philipp Krähenbühl. Towards long-form video understanding, In CVPR, 2021.
[69] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and
Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, 2019.
[70] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In CVPR, 2017.
[71] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region convolutional 3d network for temporal activity detection. In ICCV, 2017.
[72] Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. In ICCV, 2019.
[73] Yu Yao, Mingze Xu, Yuchen Wang, David J Crandall, and Ella M Atkins. Unsupervised traffic accident detection in first-person videos. In IROS, 2019.
[74] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijaanarasimman, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
[75] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. Big bird: Transformers for longer sequences. In NeurlPS, 2020.
[76] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. ICCV, 2019.
[77] Peisen Zhao, Jiajie Wang, Lingxi Xie, Ya Zhang, Yanfeng Wang. and Qi Tian. Privileged knowledge distillation for online action detection. arXiv:2011.09158, 2020.
[78] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, 2017.
14


Reviews:
[{"review_id": "xe27SeNazQY", "reviewer": "Reviewer_4xKN", "summary": "This work proposes a two-stage processing of video frames by Transformer decoders for video classification.  In the first stage a set of learnt queries “decode” a sequence of fixed-number of earlier video frames. Then in the second stage, features from recent frames act as queries and further “decode” the vectors output in the first stage. At the end of the second stage, final classification is carried out. \n\nThe main contribution is this two-stage design of the video classification model.\nThe fixed-length “long-term” context of video frames is shown to be useful for “online” frame classification (i.e., w/o access to the future frames).\n\nThis work presents a set of experiments for different design choices, which is also informative.\n\nThe key experiments are online video classification on — THUMOS’14, TVSeries, and HACS datasets, where the proposed model outperforms the baselines reported in this work. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe use of slot-based video features for long-term memory and its combination with short-term features using attention-based non-local blocks was introduced in long-term feature banks [28]. However, this work proposes a specific architecture using Transformer Decoders which is novel.\nIn the online action “detection” space, this method achieves strong results.\n&nbsp;\n\n### Quality\nThe design choices are validated through a good set of experiments. However, key weaknesses include: (i) use of non-standard datasets (however, these datasets seem to be standard for “online action detection”), and (ii) unclear if baseline methods were matched in capacity/modality etc. (see detailed comments below). \n&nbsp;\n\n### Clarity\nThe writing is concise and easy to understand. However, there are a few potential errors (e.g., (3) in “clarifications/typos” below).\nSignificance\nGiven that the using slot-based video features for long-term context, and absence of results on more popular/standard datasets might limit the impact of this work.\n&nbsp;\n\n### Strengths\nThe proposed method is simple/easy to understand and implement. \nA number of experiments validate the various design choices.\nStrong results are achieved by the proposed model on a number of popular “online action detection” benchmarks.\n&nbsp;\n\n### Weakness\n1. The proposed model has a fixed context length (although this can be large).  This means that information from frames older than a certain length ($m_L + m_s$) do not influence the current decision. In LSTMs, for example, this context-length is continuous and not fixed (at least in theory). Hence, the model design does not accurately represent the name of the proposed model “LSTR”. \n\n2. On L268, an experiment is performed with recurrent models to compensate for this limited context length. However, it is found that this has no benefit over long LSTR memory sizes. This could be a limitation of the tasks in the experiment / difficulty in training the RNNs. \n\n\n3. A second limitation is that experimental comparisons against baselines are provided under non-standard settings. For example, it is not clear whether the baselines (e.g., LFB [58]) have a similar number of model parameters, model depth/backbone, modality ( i.e., two-stream (RGB + flow) or not), and inference time.  If the authors can add this information to the tables 1(a,b) and 2, it would help put the proposed method in context.\n\n\n4. Further to the point above, a main reference model is the LFB [58], as it similarly has slot-based long-term memory.  However, the proposed work does not compare to LFB in the settings explored in the original LFB paper (e.g., Charades / EPIC-Kitchens – i.e. action recognition without localisation).\n\n\n5. The comparison against “standard Transformer Encoder units” is provided on L297-L303. However, no comparisons are provided against standard Transformer **decoder** units, which are what the proposed model is based on. Comparison against a stack of Transformer decoder units (with learned query vectors, very much like the model proposed in [28]), without the late-stage fusion with recent features will be informative. This is a key ablation.\n\n\n6. While the use of Transformers with “two-stage” video memory is novel, it is not very different in principle from the LFB [58] model, which also uses attention-based non-local blocks for fusing short-term and long-term features.   This might limit the impact of the proposed work.\n\n\n7. THUMOS’14, and TVSeries are relatively small datasets (although popularly used in “online action detection” works), as compared to more “standard” datasets, like EPIC-kitches, Charades, AVA etc., which might further limit the impact of the proposed work. Authors are encouraged to adapt their method to the above datasets for greater impact.\n&nbsp;\n\n### Clarifications / Typos:\n1. L79:  “most of these work only performs …” → these works … perform.\n\n\n2. L92: what is ‘t’ in (txC) dimensional temporal sequence? Did you mean \\tau instead?\n\n\n3. L159: Unclear how $O(n_0 * (m_L +C))$ is “sub-linear w.r.t $m_L$”; clearly it grows linearly with $m_L$. Please clarify. \n\n\n4. L166: Saying $m_S$ are “output tokens” can be improved, by using the term “queries” instead.\n\n\n5. L173:  Not having to do temporal unrolling / BPTT has more to do with the fact that the proposed architecture drops features older than $m_L$ time-steps (due to the FIFO queue), than using Transformers. \n\n6. Sec. 3.5:  Training for all frames in the short-term memory implies that the context length is shorter for the earlier $m_S - 1$ frames – this may be beneficial due to regularization, however, this point should be discussed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.  \n\n1. Potential applications in online video classification (e.g., surveillance).\n", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "lIeAezarMs7", "reviewer": "Reviewer_eqy6", "summary": "This paper presents a novel long video model named long short term transformer, with online action detection as the study case. The proposed transformer uses the encoder to model long-term historical information and uses the decoder to model the short-term information i.e. a few frames right before the current time. Experiments have been done on three standard benchmarks (THUMOS, TV, HACS) and the performance gains are clear compared to SOTA. Detailed analysis has been conducted as well.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper is well written and organized.\n2. The problem of online action detection is important in real-world applications and the proposed idea is interesting in the sense of being capable of dealing with very long video by decoupling long and short memories.\n3. This model outperforms all of the sota methods on three benchmark datasets with clear gains, confirming the effectiveness of the proposed approach. Notably, the paper adopted LFB and tried to reproduce it on the THUMOS dataset for comparison.\n4. The ablation experiments are extensive and can demonstrate the advantages of each component of the LSTR architecture.\n\nWeakness:\n1. The time complexity analysis in Sec 3.3 and Sec 3.6 is concrete and direct, however, there are no related experiments to evaluate the reduced time complexity in the paper. Although the runtime is shown in the Appendix, it is highly recommended to include the results in the experiment section since one important novelty of this model is its efficiency (as claimed in line 9 and line 40). Also, it is better to compare the efficiency of the memory compression with the single-stage design.\n2. The idea of compressing historical information and decoupling long and near term information is not new for temporal sequence modeling. “Compressive transformers for long-range sequence modelling. ICLR 2020.” has similar motivation. I do acknowledge there are significant efforts in adapting the similar idea to online action detection, such a specific topic. But it's worthwhile mentioning and discussing these NLP works about dealing with long sequences.\n3. It is nice that the paper compares with LFB [58] on THUMOS’14 because the non-local block can also aggregate features based on the attention operator. But in addition to TH’14, it will better convince me if there are comparisons with LFB on TVseries and HACS datasets as well.\n4. This work only focuses on online action detection - how about offline temporal localization? In other words, I wonder if this kind of long short term modeling might also be useful for offline scenarios such as modeling long-term dependencies of sub-events in a long event while at the same time capturing fine-grained, localized information from short yet key moments. If space allowed, it would be interesting to discuss for future work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have described the limitations of the model. As described in Sec 4.6, this model is not robust for tiny motion and a lower performance when the subject is very far away from the camera. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "cHokRLUM7E", "reviewer": "Reviewer_Zx4R", "summary": "This paper proposed a transformer-based architecture for online action detection. The architecture contains a long-term memory that stores information from the past 8 minutes and a short-term memory that processes the recent 8 seconds. Some techniques are used to reduce the complexity of transformer operations. The proposed method gets good results on THUMOS’14, TVSeries and HACS datasets. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using transformers to model long-short term temporal information is a novel architecture design, and I think the attention mechanism is suitable for retrieving information from long-short term memories. The proposed memory compression method for complexity reduction is a useful tool for long-term video modelling.\n\nOverall, the experiments are well-conducted to show the effect of design choices. The analysis of runtime in the supplementary material is valuable for future research to follow. \n\nThe submission is clearly written. As a suggestion, Fig 1 can be made more concise by avoiding repeating the video stream. \n\nThe proposed LSTR model achieves state-of-the-art results on online action detection tasks on three common datasets, showing the effectiveness of the transformers and the long-short term design. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Generally, the paper is good. Two limitations (suggestions) below:\n\n* In the online inference setting (section 3.6), the current visual feature predicts actions conditioning on both the short and long memory. But in section 3.5 (Training LSTR), it is said a directional attention mask is used such that each element in the short-term memory (current -- 8s ago) can have an action prediction. In the training case, the earlier visual features are having less short-term memory, e.g. the earliest feature at 8s ago is not utilizing any short-term memory at all - which is different with inference. I wonder if this directional attention-based training setting is the most efficient. \n\n* The purpose of the first transformer decoder unit is to apply a dimension reduction for the long-term memory. Have authors tried other simpler alternative designs, like a projection layer?", "review_text": "", "extra_scores": {"ethical_concerns": "There is no major ethical concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Rw4Z6tuD12", "reviewer": "Reviewer_eKmB", "summary": "The paper addresses an interesting and practical problem in videos – online detection of actions by examining only frames available till the present time and without any access to future frames. The problem is not only challenging but has practical significance too as online action detection is important in autonomous driving or surveillance scenarios and handling this with fast and light memory-weight training and inference enables one to address the problem real-time. The solution approach is based on the intuition that recent frames are more instrumental about the ongoing action while frames from extended past offer potential context to the present action. While differential exploitation of recent and extended past can definitely help (compared to using frames only from the recent past), it comes with extra computation and memory overhead. Based on the recently popular transformer philosophy, the authors came up with an encoder-decoder architecture with short and long term memories for recent and extended memories respectively. While related works have explored long and short term memories, the integration of the two outputs from them is not explicit and rather ad-hoc (pooling, concatenation etc.). The proposed work, on the other hand, made intelligent use of transformers with a two-stage memory compression design based on transformer decoders. Such a design is shown to be not only effective but also both memory and compute efficient. The experiments are well-designed. Extensive comparison with SOTA approaches and well-thought ablations show its superiority and robustness of design choices. The writing is crisp and adequate with insightful experimental analysis. The figures are also self-explanatory with descriptive captions. The related works are also covered good.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]:\n1. The paper addresses a very practical problem and proposes an efficient transformer based architecture for the purpose.\n2. The experiments are well-designed and executed showing the superiority of the proposed approach.\n3. The paper is well-written, the figures are well-illustrated and related works are well documented.\n4. The long and short term memory with two-stage memory compression design based on transformer decoders for online action detection is novel and effective.\n\n[Weaknesses]:\n\nThough I am writing these under the heading ‘weaknesses’, they are mostly my queries for a better understanding.\n1. Line 69-71 mentions about existing non-explicit integration of long and short term features using simple pool or concat operations. An ablation experiment with such heuristics for the integration would have been good.\n2. Though online action detection is a very good usecase, is it possible to show the applicability of LSTR's superior modeling of long videos in other video applications?\n3. Figure 1 (since the figure is said as best viewed in color): Does the different arrow colors mean anything?\n\nIn short, the paper seems to have addressed a practical problem with well-supported experiments and analysis and is also well written. Thus, I am in favor of accepting it at the moment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is described adequately in section 4.6 and possible future exploration of the temporal modeling capability of LSTR to both the spatial and the temporal domains is planned.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The LSTR model is original and well explained and explored in the paper. It was well received by all reviewers, and the rebuttal has addressed most of the concerns raised.

One outstanding issue is that the model is not applied to several of the current datasets and tasks of interest (e.g. AVA, EPIC-Kitchens) and this will somewhat limit interest in it. The authors are encouraged to address this in the final version of the paper.
Decision: Accept (Spotlight)