Long Short-Term Transformer for Online Action Detection
Year: 2021

Abstract:
We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a longand short-term memory mechanism to model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e.g., 2048 frames spanning of up to 8 minutes), together with an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared to prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS’14, TVSeries, and HACS Segment. Code has been made available at: https://xumingze0308.github.io/projects/lstr.

heading: 1 Introduction
text: Given an incoming stream of video frames, online action detection [14] is concerned with the task of classifying what is happening at each frame without seeing the future. Unlike offline methods that assume the entire video is available, online methods process the data causally, up to the current time. In this paper, we present an online temporal modeling algorithm capable of capturing temporal relations on prolonged sequences up to 8 minutes long, while retaining fine granularity of the event in the representation. This is achieved by modeling activities at different temporal scales, so as to capture a variety of events ranging from bursts to slow trends.
Specifically, we propose a method, named Long Short-term TRansformer (LSTR), to jointly model long- and short-term temporal dependencies. LSTR has two main advantages over prior work. 1) It stores the history directly thus avoiding the pitfalls of recurrent models [18, 50, 28, 10]. Backpropagation through time, BPTT, is not needed as the model can directly attend to any useful frames from memory. 2) It separates long- and short-term memories, which allows modeling short-term context while extracting useful correlations from the long-term history. This allows us to compress the long-term history without losing important fine-scale information.
As shown in Fig. 1, we explicitly divide the entire history into the long- and short-term memories and build our model with an encoder-decoder architecture. Specifically, the LSTR encoder compresses and abstracts the long-term memory into a latent representation of fixed length, and the LSTR decoder uses a short window of transient frames to perform self-attention and cross-attention operations on the extracted token embeddings from the LSTR encoder. In the LSTR encoder, an extended temporal support becomes beneficial in dealing with untrimmed, streaming videos by devising twostage memory compression, which is shown to be computationally efficient in both training and inference. Our overall long short-term Transformer architecture gives rise to an effective and efficient representation for modeling prolonged sequence data.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
We validate LSTR on standard benchmark datasets (THUMOS’14 [30], TVSeries [14], and HACS Segment [76]). These have distinct characteristics such as video length spanning from a few seconds to tens of minutes. Experimental results establish LSTR as the state-of-the-art for online action detection. Ablation studies further showcase LSTR’s abilities in modeling long video sequences.

heading: 2 Related Work
text: Online Action Detection. Temporal action localization aims to detect the onset and termination of action instances after observing the entire video [52, 71, 23, 53, 78, 5, 37, 39, 38]. Embodied perception, however, requires causal processing [73], where we can only process the data up to the present. Online action detection focuses on this setting [14]. RED [22] uses a reinforcement loss to encourage recognizing actions as early as possible. TRN [72] models greater temporal context by simultaneously performing online action detection and anticipation. IDN [19] learns discriminative features and accumulates only relevant information for the present. LAP-Net [46] proposes an adaptive sampling strategy to obtain optimal features. PKD [77] transfers knowledge from offline to online models using curriculum learning. As with early action detection [27, 40], Shou et al. [54] focus on online detection of action start (ODAS). StartNet [24] decomposes ODAS into two stages and learns with policy gradient. WOAD [25] uses weakly-supervised learning with video-level labels.
Temporal/Sequence Modeling. Causal time series analysis has traditionally assumed the existence of a latent “state” variable that captures all information in past data, and is updated using only the current datum [49, 28, 10]. While the Separation Principle ensures that such a state exists for linearGaussian time series, in general it is not possible to summarize all past history of complex data in a finite-dimensional sufficient statistic. Therefore, we directly model the history, in accordance with other work on video understanding [69, 45, 68]. Earlier work on action recognition usually relies on heuristic sub-sampling (typically 3 to 7 video frames) for more feasible training [74, 62, 21, 41, 57]. 3D ConvNets [59, 8, 60] are used to perform spatio-temporal feature modeling on more frames, but they fail to capture temporal correlations beyond their receptive field. Recently, Wu et al. [69] propose long-term feature banks to capture objects and scene features, but discarding their temporal order which is clearly informative. Most of work above does not explicitly separate the long- and short-term context modeling, but instead integrates all observed features with simple mechanisms such as pooling or concatenation. We are motivated by work in Cognitive Science [44, 12, 9, 35] that has shed light on the design principles for modeling long-term dependencies with attention mechanism [66, 13, 48, 75].
Transformers for Action Understanding. Transformers have achieved breakthrough success in NLP [47, 15] and are adopted in computer vision for image recognition [17, 58] and object detection [7]. Recent papers exploit Transformers for temporal modeling tasks in videos, such as action recognition [43, 51, 36, 4, 3] and temporal action localization [42, 56], and achieve promising results. However, computational and memory demands result in most work being limited to short video clips, with few exceptions [13, 6] that focuses on designing Transformers to model long-range context. The mechanism for aggregating long- and short-term information is relatively unexplored [32].

heading: 3 Long Short-Term Transformer
text: Given a live streaming video, our goal is to identify the actions performed in each video frame using only past and current observations. Future information is not accessible during inference. Formally, a streaming video at time t is represented by a batch of τ past frames It = {It−τ , · · · , It},
which reads “I up to time t.” The online action detection system receives It as input, and classifies the action category ŷt belonging to one of (K + 1) classes, ŷt ∈ {0, 1, · · · ,K}, ideally using the posterior probability P (ŷt = k|It), where k = 0 denotes the probability that no event is occurring at frame t. We design our method by assuming that there is a pretrained feature extractor [62] that processes each video frame It into a feature vector ft ∈ RC of C dimensions1. These vectors form a (τ × C)-dimensional temporal sequence that serves as the input of our method.

heading: 3.1 Overview
text: Our method is based on the intuition that frames observed recently provide precise information about the ongoing action instance, while frames over an extended period offer contextual references for actions that are potentially happening right now. We propose Long Short-term TRansformer (LSTR) in an explicit encoder-decoder manner, as shown in Fig. 2. In particular, the feature vectors of mL frames in the distant past are stored in a long-term memory, and a short-term memory stores the features of mS recent frames. The LSTR encoder compresses and abstracts features in long-term memory to an encoded latent representation of n1 vectors. The LSTR decoder queries the encoded long-term memory with the short-term memory for decoding, leading to the action prediction ŷt. This design follows the line of thought in combining long- and short-term information for action understanding [16, 62, 69], but addresses several key challenges to efficiently achieve this goal, exploiting the flexibility of Transformers [61].

heading: 3.2 Long- and Short-Term Memories
text: We store the streaming input of feature vectors into two consecutive memories. The first is the short-term memory which stores only a small number of frames that are recently observed. We implement it with a first-in-first-out (FIFO) queue of mS slots. At time T , it stores the feature vectors as MS = {fT , · · · , fT−mS+1}. When a frame becomes “older” than mS time steps, it graduates from MS and enters into the long-term memory, which is implemented with another FIFO queue of mL slots. The long term memory stores ML = {fT−mS , · · · , ft−mS−mL+1}. The long-term memory serves as the input memory to the LSTR encoder and the short-term memory serves as the queries for the LSTR decoder. In practice, the long-term memory stores a much longer time span than the short-term memory (mS mL). A typical choice is mL = 2048, which represents 512 seconds worth of video contents with 4 frames per second (FPS) sampling rate, and mS = 32 representing
1In practice, some feature extractors [59] take consecutive frames to produce one feature vector. Nonetheless, it is still temporally “centered” on a single frame. Thus we use the single frame notation here for simplicity.
8 seconds. We add a sinusoidal positional encoding s [61] to each frame feature in the memories relative to current time T (i.e., the frame at T − τ receives a positional embedding of sτ ).

heading: 3.3 LSTR Encoder
text: The LSTR encoder aims at encoding the long-term memory of mL feature vectors into a latent representation that LSTR can use for decoding useful temporal context. This task requires a large capacity in capturing the relations and temporal context across a span of hundreds or even thousands of frames. Prior work on modeling long-term context for action understanding relies on heuristic temporal sub-sampling [69, 62] or recurrent networks [16] to make training feasible, at the cost of losing specific information of each time step. Attention-based architectures, such as Transformer [61], have recently been shown promising for similar tasks that require long-range temporal modeling [51]. A straightforward choice for LSTR encoder would be to use a Transformer encoder based on self-attention. However, its time complexity, O(m2LC), grows quadratically with the memory sequence length mL. This limits our ability to model long-term memory with sufficient length to cover long videos. Though recent work [64] has explored self-attention with linear complexity, repeatedly referencing information from the long-term memory with multi-layer Transformers is still computationally heavy. In LSTR, we propose to use the two-stage memory compression mechanism based on Transformer decoder units [61] to achieve more effective memory encoding.
The Transformer decoder unit [61] takes two sets of inputs. The first set includes a fixed number of n learnable output tokens λ ∈ Rn×C , where C is the embedding dimension. The second set includes another m input tokens θ ∈ Rm×C , where m can be a rather large number. It first applies one layer of multi-head self-attention on λ. The outputs λ′ are then used as queries in an “QKV cross-attention” operation and the input embeddings θ serve as key and value. The two steps can be written as
λ′ = SelfAttn(λ) = Softmax( λ · λT√
C )λ and CrossAttn(σ(λ′),θ) = Softmax( σ(λ′) · θT√ C )θ,
where σ : Rn×C → Rn×C denotes the intermediate layers between the two attention operations. One appealing property of this design is that it transforms the m× C dimensional input tokens into output tokens of n× C dimensions in O(n2C + nmC) time complexity. When n m, the time complexity becomes linear to m, making it an ideal candidate for compressing long-term memory. This property is also utilized in [32] to efficiently process large volume inputs, such as image pixels.
Two-Stage Memory Compression. Stacking multiple Transformer decoder units on the long-term memory, as in [61], can form a memory encoder with linear complexity with respect to the memory size mL. However, running the encoder at each time step can still be time consuming. We can further reduce the time complexity with a two-stage memory compression design. The first stage has one Transformer decoder unit with n0 output tokens. Its input tokens are the entire long-term memory of size mL. The outputs of the first stage are used as the input tokens to the second stage, which has `enc stacked Transformer decoder units and n1 output tokens. Then, the long-term memory of size mL × C is compressed into a latent representation of size n1 × C, which can then be efficiently queried in the LSTR decoder later. This two-stage memory compression design is illustrated in Fig. 2.
Compared to an (1 + `enc)-layer Transformer encoder with O(m2L(1 + `enc)C) time complexity or stacked Transformer decoder units with n output-tokens having O((n2 + nmL)(1 + `enc)C) time complexity, the proposed LSTR encoder has complexity of O(n20C+n0mLC+(n 2 1+n1n0)`encC). Because both n0 and n1 are much smaller than mL, and `enc is usually larger than 1, using two-stage memory compression could be more efficient. In Sec. 3.6, we will show that, during online inference, it further enables us to reduce the runtime of the Transformer decoder unit of the first stage. In Sec. 4.5, we empirically found this design also leads to better performance for online action detection.

heading: 3.4 LSTR Decoder
text: The short-term memory contains informative features for classifying actions on the latest time step. The LSTR decoder uses the short-term memory as queries to retrieve useful information from the encoded long-term memory produced by the LSTR encoder. The LSTR decoder is formed by stacking `dec layers of Transform decoder units. It takes the outputs of the LSTR encoder as input tokens and the mS feature vectors in the short-term memory as output tokens. It outputs mS probability vectors {pT , · · · ,pT−mS+1} ∈ [0, 1]K+1, each pt representing the predicted probability distribution of K action categories and one “background” class at time t. During inference, we only take the probability vector pT from the output token corresponding to the current time T for the classification result.
However, having the additional outputs on the older frames allows the model to leverage more supervision signals during training. The details will be described below.

heading: 3.5 Training LSTR
text: LSTR can be trained without temporal unrolling and Backpropagation Through Time (BPTT) as in LSTM [28], which is a common property of Transformers [61]. We construct each training sample by randomly sampling an ending time T and filling the long- and short-term memories by tracing back in time for mS +mL frames. We use the empirical cross entropy loss between the predicted probability distribution pT at time T and the ground truth action label yT ∈ {0, 1, · · · ,K} as
L(yT ,pT ;T ) = − K∑ k=0 δ(k − yT ) log pkT , (1)
where pkT is the k-th element of the probability vector pT , predicted on the latest frame at T . Additionally, we add a directional attention mask [61] to the short-term memory so that any frame in the short-term memory can only depend on its previous frames. In this way, we can make predictions on all frames in the short-term memory as if they are the latest ones. Thus we can provide supervision on every frame in the short-term memory, and the complete loss function L is then
LT = T∑
t=T−ms+1
L(yt,pt;T ), (2)
where pt denotes the prediction from the output token corresponding to time t.

heading: 3.6 Online Inference with LSTR
text: During online inference, the video frame features are streamed to the model as time passes. Running LSTR’s long-term memory encoder from scratch for each frame results in a time complexity of O(n20C + n0mLC) and O((n 2 1 + n1n0)`encC) for the first and second memory compression stages, respectively. However, at each time step, there is only one new video frame to be updated. We show it is possible to achieve even more efficient online inference by storing the intermediate results for the Transformer decoder unit of the first stage. First, the queries of the first Transformer decoder unit are fixed. So their self-attention outputs can be pre-computed and used throughout the inference. Second, the cross-attention operation in the first stage can be written as
CrossAttn(qi, {fT−τ + sτ}) = mS+mL−1∑ τ=mS exp((fT−τ + sτ ) · qi/ √ C)∑mS+mL−1 τ=mS exp((fT−τ + sτ ) · qi/ √ C) · (fT−τ + sτ ), (3)
where the index τ = T − t is the relative position of a frame t in the long-term memory to the latest time T . This calculation depends on the un-normalized attention weight matrix A ∈ RmL×n0 , with elements aτi = (fT−τ + sτ ) · qi. A can be decomposed into the sum of two matrices Af and As. We have their elements as afτi = fT−τ · qi and asτi = sτ · qi. The queries after the first self-attention operation, Q = [q1, . . . ,qn0 ], and the position embedding sτ are fixed during inference. Thus the matrix As can be pre-computed and used for every incoming frame. We additionally maintain a FIFO queue of vectors at = Q>ft of size mL. Af at any time step T can be obtained by stacking all vectors currently in this queue. Updating this queue at each time step requires O(n0C) time complexity for the matrix-vector product. Now we can obtain the matrix A with only n0 × mL additions by adding As and Af together, instead of n0 ×mL × C multiplications and additions using Eq. (3). This means the amortized time complexity for computing the attention weights can be reduced to O(n0(mL + C)). Although the time complexity of the cross-attention operation is still O(n0mLC) due to the inevitable operation of weighted sum, since C is usually larger than 1024 [26], this is still a considerable reduction of runtime. LSTR’s walltime efficiency is discussed in Sec. 4.6.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our model on three publicly-available datasets: THUMOS’14 [30], TVSeries [14] and HACS Segment [76]. THUMOS’14 includes over 20 hours of sports video annotated with 20 actions. We follow prior work [72, 19] and train on the validation set (200 untrimmed videos) and evaluate on the test set (213 untrimmed videos). TVSeries contains 27 episodes of 6 popular TV series, totaling 16 hours of video. The dataset is annotated with 30 realistic, everyday actions (e.g., open door). HACS Segment is a large-scale dataset of web videos. It contains 35,300 untrimmed videos over 200 human action classes for training and 5,530 untrimmed videos for validation.

heading: 4.2 Settings
text: Feature Encoding. We follow the experimental settings of state-of-the-art methods [72, 19]. We extract video frames at 24 FPS and set the video chunk size to 6. Decisions are made at the chunk level, and thus accuracy is evaluated at every 0.25 second. For feature encoding, we adopt the TSN [62] models implemented in an open-source toolbox [11]. Specifically, the features are extracted by one visual model with the ResNet-50 [26] architecture from the central frame of each chunk and one motion model with the BN-Inception [31] architecture from the stacked optical flow fields between 6 consecutive frames [62]. The visual and motion features are concatenated along the channel dimension as the final feature f. We experiment with feature extractors pretrained on two datasets, ActivityNet and Kinetics.
Implementation Details. We implemented our proposed model in PyTorch [1], and performed all experiments on a system with 8 Nvidia V100 graphics cards. For all Transformer units, we set their number of heads as 16 and hidden units as 1024 dimensions. To learn model weights, we used the Adam [34] optimizer with weight decay 5× 10−5. The learning rate was linearly increased from zero to 5× 10−5 in the first 2/5 of training iterations and then reduced to zero following a cosine function. Our models were optimized with batch size of 16, and the training was terminated after 25 epochs.
Evaluation Protocols. We follow prior work and use per-frame mean average precision (mAP) to evaluate the performance of online action detection. We also use per-frame calibrated average precision (cAP) [14] that was proposed for TVSeries to correct the imbalance between positive and negative samples, cAP = ∑ k cPrec(k) ∗ I(k)/P , where cPrec = TP/(TP + FP/w), I(k) is 1 if frame k is a true positive, P is the number of true positives, and w is the negative and positive ratio.

heading: 4.3 Comparison with the State-of-the-art Methods
text: Table 1: Online action detection and anticipation results on THUMOS’14 and TVSeries in terms of mAP and cAP, respectively. For online action detection, LSTR outperforms the state-of-the-art methods on THUMOS’14 by 3.7% and 2.4% in mAP and on TVSeries by 2.8% and 2.7% in cAP, using ActivityNet and Kinetics pretrained features, respectively. LSTR also achieves promising results for action anticipation. *Results are reproduced by us using their papers’ default settings.
(a) Results of online action detection using ActivityNet features
(b) Results of online action detection using Kinetics features
(c) Results of action anticipation using ActivityNet features
Features Portion of Video
0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90-100%
TRN [72]
ActivityNet 78.8 79.6 80.4 81.0 81.6 81.9 82.3 82.7 82.9 83.3 IDN [19] 80.6 81.1 81.9 82.3 82.6 82.8 82.6 82.9 83.0 83.9 TFN [20] 83.1 84.4 85.4 85.8 87.1 88.4 87.6 87.0 86.7 85.6 LSTR (ours) 83.6 85.0 86.3 87.0 87.8 88.5 88.6 88.9 89.0 88.9
IDN [19] Kinetics 81.7 81.9 83.1 82.9 83.2 83.2 83.2 83.0 83.3 86.6 PKD [77] 82.1 83.5 86.1 87.2 88.3 88.4 89.0 88.7 88.9 87.7 LSTR (ours) 84.4 85.6 87.2 87.8 88.8 89.4 89.6 89.9 90.0 90.1
We compare LSTR against other state-of-the-art methods [72, 19, 25] on THUMOS’14, TVSeries, and HACS Segment. Specifically, on THUMOS’14 and TVSeries, we implement LSTR with the long- and short-term memories of 512 and 8 seconds, respectively. On HACS Segment, we reduce
the long-term memory to 256 seconds, considering that its videos are strictly shorter than 4 minutes. For LSTR, we implement the two-stage memory compression using Transformer decoder units. We set the token numbers to n0 = 16 and n1 = 32 and the Transformer layers to `enc = 2 and `dec = 2.

heading: 4.3.1 Online Action Detection
text: THUMOS’14. We compare LSTR with recent work on THUMOS’14, including methods that use 3D ConvNets [53] and RNNs [70, 19, 25], reinforcement learning [22], and curriculum learning [77]. Table 1a and 1b shows that LSTR significantly outperforms the the state-of-the-art methods [69, 25] by 3.7% and 2.4% in terms of mAP using ActivityNet and Kinetics pretrained features, respectively.
TVSeries. Table 1a and 1b show the online action detection results that LSTR outperforms the state-of-the-art methods [20, 77] by 2.8% and 2.7% in terms of cAP using ActivityNet and Kinetics pretrained features, respectively. Following prior work [14], we also investigate LSTR’s performance at different action stages by evaluating each decile (ten-percent interval) of the video frames separately. Table 2 shows that LSTR outperforms existing methods at every stage of action instances.
HACS Segment. LSTR achieves 82.6% on HACS Segment in term of mAP using Kinetics pretrained features. Note that HACS Segment is a new large-scale dataset with only a few previous results. LSTR outperforms existing methods RNN [28] (77.6%) by 5.0% and TRN [72] (78.9%) by 3.7%.

heading: 4.3.2 Action Anticipation
text: We extend the idea of LSTR to action anticipation for up to 2 seconds (i.e., 8 steps in 4 FPS) into the future. Specifically, we concatenate another 8 learnable output tokens (with positional embedding) after the short-term memory in the LSTR decoder to produce the prediction results accordingly. Table 1c shows that LSTR significantly outperforms the state-of-the-art methods [72, 46] by 7.5% mAP on THUMOS and 2.1% cAP on TVSeries, using ActivityNet pretrained features.

heading: 4.4 Design Choices of Long- and Short-Term Memories
text: We experiment for design choices of long- and short-term memories. Unless noted otherwise, we use THUMOS’14, which contains various video lengths, and Kinetics pretrained features.
Lengths of long- and short-term memories. We first analyze the effect of different lengths of long-term mL and short-term mS memory. In particular, we test mS ∈ {4, 8, 16} seconds with mL starting from 0 second (no long-term memory). Note that we choose the max length (1024 seconds for THUMOS’14 and 256 seconds for HACS Segment) to cover lengths of 98% videos, and do not have proper datasets to test longer mL. Fig. 3 shows that LSTR is beneficial from larger mL in most cases. In addition, when mL is short (≤ 16 in our cases), using larger mS obtains better results and when mL is sufficient (≥ 32 in our cases), increasing mS does not always guarantee better performance.
Can we downsample long-term memory? We implement LSTR with mS as 8 seconds and mL as 512 seconds, and test the effect of downsampling long-term memory. Table 3 shows the results that downsampling with strides smaller than 4 does not cause performance drop, but more aggressive
strides dramatically decrease the detection accuracy. Note that, when extracting frame features in 4 FPS, both LSTR encoder (n0 = 16) and downsampling with stride 128 compress the long-term memory to 16 features, but LSTR achieves much better performance (69.5% vs. 65.9% in mAP). This demonstrates the effectiveness of our “adaptive compression” compared to heuristics downsampling.
Can we compensate reduced memory length with RNN? We note that LSTR’s performance notably decreases when it can only access very limited memory (e.g., mL+mS ≤ 16 seconds). Here we test if RNN can compensate LSTR’s reduced memory or even fully replace the LSTR encoder. We implement LSTR using mS = 8 seconds with an extra Gated Recurrent Unit (GRU) [10] (its architecture is visualized in the Supplementary Material) to capture all history outside the long- and short-term memories. The dashed line in Fig. 3 shows the results. Plugging-in RNNs indeed improves the performance when mL is small, but when mL is large (≥ 64 seconds), it does not improve the accuracy anymore. Note that RNNs are not used in any other experiments in this paper.

heading: 4.5 Design Choices of LSTR
text: We continue to explore the design trade-offs of LSTR. Unless noted otherwise, we use short-term memory of 8 seconds, long-term memory of 512 seconds, and Kinetics pretrained features.
Number of layers and tokens. First, we assess using different numbers of token embeddings (i.e., n0 and n1) in LSTR encoder. Fig 4 (left) shows that LSTR is quite robust to different choices (the best and worst performance gap is only about 1.5%), but using n0 = 16 and n1 = 32 gets highest accuracy. Second, we experiment for the effect of using different numbers of Transformer decoder units (i.e., `enc and `dec). As shown in Fig 4 (right), LSTR does not need a large model to get the best performance, and in practice, using more layers can cause overfitting.
Can we unify the temporal modeling using only self-attention models? We test if long-term ML and short-term MS memory can be learned as a whole using self-attention models. Specifically, we concatenate ML and MS and feed them into a standard Transformer Encoder [61] with a similar model size to LSTR. Table 4 (row 8 vs. row 1) shows that LSTR achieves better performance especially when mL is large (e.g., 69.5% vs. 66.5% when mL = 512 and 66.6% vs. 65.7% when mL = 8). This shows the advantage of LSTR for temporal modeling on long- and short-term context.
Can we remove the LSTR encoder? We explore this by directly feeding ML into the LSTR decoder, and using MS as tokens to reference useful information. Table 4 shows that LSTR outperforms this
baseline (row 8 vs. row 2), especially when mL is large. We also compare it with self-attention models (row 1) and observe that although neither of them can effectively model prolonged memory, this baseline outperforms the Transformer Encoder. This also demonstrates the effectiveness of our idea of using short-term memory to query related context from long-range context.
Can the LSTR encoder learn effectively using self-attention? To evaluate the “bottleneck” design with cross-attention in LSTR encoder, we try modeling ML using standard Transformer Encoder units [61]. Note that this still captures ML and MS with the similar workflow of LSTR, but does not compress and encode ML using learnable tokens. Table 4 (row 8 vs. row 3) shows that LSTR outperforms this baseline with all mL settings. In addition, the performance of this baseline decreases when mL gets larger, which suggests the superior ability of LSTR for modeling long-range patterns.
How to design the memory compression for the LSTR encoder? First, we use a projection layer, consisting of a learnable matrix of size n0 ×mL followed by MLP layers, to compress the long-term memory along the temporal dimension. Table 4 shows that using this simple projection layer (row 4) slightly outperforms the model without long-term memory (row 1), but is worse than attention-based compression methods. Second, we evaluate the one-stage design with `enc + 1 Transformer decoder units. Table 4 (row 8 vs. row 5) shows that two-stage compression is stably better than one-stage, and their performance gap gets larger when using larger mL (0.5% when mL = 8 and 0.8% when mL = 512). Third, we compare cross-attention and self-attention for two-stage compression by replacing the second Transformer decoder with Transformer encoder. LSTR stably outperforms this baseline (row 6) by about 0.5% in mAP. However, its performance is still better than models with one-stage compression of about 1.3% (row 6 vs. row 3) and 0.3% (row 6 vs. row 5) on average.
Can we remove the LSTR decoder? We remove the LSTR decoder to evaluate its contribution. Specifically, we feed the entire memory to LSTR encoder and attach a multi-layer (MLP) classifier on its output tokens embeddings. Similar to the above experiments, we increase the model size to ensure a fair comparison. Table 4 shows that LSTR outperforms this baseline (row 7) by about 4% on large mL (e.g., 512 and 1024) and about 2.5% on relative small mL (e.g., 8 and 16).
Cross-attention vs. heuristics for integrating long- and short-term memories. We explore integrating the long- and short-term memories by using average pooling and concatenation. Specifically, the encoded long-term features of size n1×C is converted to a vector of C elements by channel-wise averaging, and the short-term memory of size mS ×C is encoded by `dec Transformer encoder units. Then, each slot of the short-term features is either averaged or concatenated with the long-term feature vector for action classification. Note that these models still benefit from LSTR’s effectiveness for long-term modeling. Table 5 shows that using average pooling and concatenation obtain comparable results, but LSTR with cross-attention stably outperforms these baselines.

heading: 4.6 Runtime
text: We report LSTR’s runtime in frames per second (FPS) on a system with a single V100 GPU, and use the videos from THUMOS’14 dataset. The results are shown in Table 6.
We start by comparing the runtime between LSTR’s different design choices without considering the pre-processing (e.g., feature extraction). First, LSTR runs at 91.6 FPS using our two-stage memory compression (row 4), whereas using the one-stage design runs at a slower 59.5 FPS (row 3). Our two-stage design is more efficient because it does not need to reference information from the long-term memory multiple times, and can be further accelerated during online inference (see Sec. 3.6). Second, we test the LSTR encoder using self-attention mechanisms (row 2). This design does not compress the long-term memory, thus increasing the computational cost of both the LSTR encoder and decoder, leading to a slower speed of 50.2 FPS. Third, we test the standard Transformer Encoder [61] (row 1), whose runtime speed, 43.2 FPS, is about 2× slower than LSTR. We also compare LSTR with state-of-the-art recurrent models. As we are not aware of any prior work that reports their runtime, we test TRN [72] using their official open-source code [2]. The result shows that TRN runs at 123.3 FPS, which is faster than LSTR. This is because recurrent models abstract the entire history as a compact representation but LSTR needs to process much more information. On the other hand, LSTR achieves much higher performance, outperforming TRN by about 7.5% in mAP on THUMOS’14 and about 4.5% in cAP on TVSeries.
For end-to-end online inference, we follow the state-of-the-art methods [72, 19, 25] and build LSTR on two-stream features [62]. LSTR together with pre-processing techniques run at 4.6 FPS. Table 6 shows that the speed bottleneck is the motion feature extraction — it accounts for about 90% of the total runtime including the optical flow computation with DenseFlow [63]. One can improve the efficiency largely by using real-time optical flow extractors (e.g., PWC-Net [55]) or using only visual features extracted by a light-weight backbone (e.g., MobileNet [29] and FBNet [67]).

heading: 4.7 Error Analysis
text: In Table 7, we list the action classes from THUMOS’14 where LSTR gets the highest (color green) and the lowest (color red) per-frame APs. In Fig. 5, we illustrate four sample frames with incorrect predictions. More visualizations are included in the Supplementary Material. We observe that LSTR sees a decrease in detection accuracy when the action incurs only tiny motion or the subject is very far away from the camera, but excels at recognizing actions with long temporal span and multiple stages, such as “PoleVault” and “Long Jump”. This suggests we may explore extending the temporal modeling capability of LSTR to both spatial and temporal domains.

heading: 5 Conclusion
text: We present LSTR which captures both long- and short-term correlations in past observations of a time series by compressing long-term memory into encoded latent features and referencing related temporal context from them with short-term memory. This demonstrates the importance of separately modeling long- and short-term information and then integrating them for online inference tasks. Experiments on multiple datasets and ablation studies validate the effectiveness and efficiency of the LSTR design in dealing with prolonged video sequences. However, we note that LSTR is operating only on the temporal dimension. An end-to-end video understanding system requires simultaneous spatial and temporal modeling for optimal results. Therefore extending the idea of LSTR to spatio-temporal modeling remains an open yet challenging problem.

heading: 6 Acknowledgments and Disclosure of Funding
text: We thank the anonymous reviewers for their helpful suggestions. This work was funded by Amazon.
References [1] http://pytorch.org/.
[2] https://github.com/xumingze0308/TRN.pytorch.
[3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv:2103.15691, 2021.
[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv:2102.05095, 2021.
[5] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. SST: Single-stream temporal action proposals. In CVPR, 2017.
[6] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv:2006.11527, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In CVPR, 2017.
[9] Marvin M Chun. Visual working memory as visual attention sustained internally over time. Neuropsychologia, 2011.
[10] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014.
[11] MMAction2 Contributors. Openmmlab’s next generation video understanding toolbox and benchmark. https://github.com/open-mmlab/mmaction2, 2020.
[12] Nelson Cowan. Attention and memory: An integrated framework. Oxford University Press, 1998.
[13] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. ACL, 2019.
[14] Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In ECCV, 2016.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
[16] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[18] Jeffrey L Elman. Finding structure in time. Cognitive science, 1990.
[19] Hyunjun Eun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Learning to discriminate information for online action detection. In CVPR, 2020.
[20] Hyunjun Eun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Temporal filtering networks for online action detection. Pattern Recognition, 2021.
[21] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recognition. In CVPR, 2017.
[22] Jiyang Gao, Zhenheng Yang, and Ram Nevatia. RED: Reinforced encoder-decoder networks for action anticipation. In BMVC, 2017.
[23] Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, and Ram Nevatia. TURN TAP: Temporal unit regression network for temporal action proposals. In ICCV, 2017.
[24] Mingfei Gao, Mingze Xu, Larry S Davis, Richard Socher, and Caiming Xiong. Startnet: Online detection of action start in untrimmed videos. In ICCV, 2019.
[25] Mingfei Gao, Yingbo Zhou, Ran Xu, Richard Socher, and Caiming Xiong. WOAD: Weakly supervised online action detection in untrimmed videos. In CVPR, 2021.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Minh Hoai and Fernando De la Torre. Max-margin early event detectors. In IJCV, 2014.
[28] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[29] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017.
[30] Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos “in the wild”. CVIU, 2017.
[31] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[32] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv:2103.03206, 2021.
[33] Young Hwi Kim, Seonghyeon Nam, and Seon Joo Kim. Temporally smooth online action detection using cycle-consistent future anticipation. Pattern Recognition, 2021.
[34] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[35] Anastasia Kiyonaga and Tobias Egner. Working memory as internal attention: Toward an integrative account of internal and external selection processes. Psychonomic bulletin & review, 2013.
[36] Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. arXiv:2104.11746, 2021.
[37] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: Boundary sensitive network for temporal action proposal generation. In ECCV, 2018.
[38] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. BMN: Boundary-matching network for temporal action proposal generation. In ICCV, 2019.
[39] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multi-granularity generator for temporal action proposal. In CVPR, 2019.
[40] Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in lstms for activity detection and early detection. In CVPR, 2016.
[41] Antoine Miech, Ivan Laptev, and Josef Sivic. Learnable pooling with context gating for video classification. arXiv:1706.06905, 2017.
[42] Megha Nawhal and Greg Mori. Activity graph transformer for temporal action localization. arXiv:2101.08540, 2021.
[43] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. arXiv:2102.00719, 2021.
[44] Klaus Oberauer. Working memory and attention–a conceptual analysis and review. Journal of cognition, 2019.
[45] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In CVPR, 2019.
[46] Sanqing Qu, Guang Chen, Dan Xu, Jinhu Dong, Fan Lu, and Alois Knoll. LAP-Net: Adaptive features sampling via learning action progression for online action detection. arXiv:2011.07915, 2020.
[47] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[48] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. ICLR, 2020.
[49] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 1986.
[50] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. ITSS, 1997.
[51] Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manor. An image is worth 16x16 words, what is a video worth? arXiv:2103.13915, 2021.
[52] Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In CVPR, 2016.
[53] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. CDC: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. In CVPR, 2017.
[54] Zheng Shou, Junting Pan, Jonathan Chan, Kazuyuki Miyazawa, Hassan Mansour, Anthony Vetro, Xavier Giro-i Nieto, and Shih-Fu Chang. Online action detection in untrimmed, streaming videos-modeling and evaluation. In ECCV, 2018.
[55] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. PWC-Net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[56] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Relaxed transformer decoders for direct action proposal generation. arXiv:2102.01894, 2021.
[57] Yongyi Tang, Xing Zhang, Lin Ma, Jingwen Wang, Shaoxiang Chen, and Yu-Gang Jiang. Non-local netvlad encoding for video classification. In ECCVW, 2018.
[58] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012.12877, 2020.
[59] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[60] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv:1706.03762, 2017.
[62] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
[63] Shiguang Wang*, Zhizhong Li*, Yue Zhao, Yuanjun Xiong, Limin Wang, and Dahua Lin. denseflow. https://github.com/open-mmlab/denseflow, 2020.
[64] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv:2006.04768, 2020.
[65] Wen Wang, Xiaojiang Peng, Yanzhou Su, Yu Qiao, and Jian Cheng. TTPP: Temporal transformer with progressive prediction for efficient action anticipation. Neurocomputing, 2021.
[66] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[67] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019.
[68] Chao-Yuan Wu and Philipp Krähenbühl. Towards long-form video understanding. In CVPR, 2021.
[69] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, 2019.
[70] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In CVPR, 2017.
[71] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: Region convolutional 3d network for temporal activity detection. In ICCV, 2017.
[72] Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. In ICCV, 2019.
[73] Yu Yao, Mingze Xu, Yuchen Wang, David J Crandall, and Ella M Atkins. Unsupervised traffic accident detection in first-person videos. In IROS, 2019.
[74] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
[75] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
[76] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In ICCV, 2019.
[77] Peisen Zhao, Jiajie Wang, Lingxi Xie, Ya Zhang, Yanfeng Wang, and Qi Tian. Privileged knowledge distillation for online action detection. arXiv:2011.09158, 2020.
[78] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, 2017.

Review summaries:
This paper proposed a transformer-based architecture for online action detection. The architecture contains a long-term memory that stores information from the past 8 minutes and a short-term memory that processes the recent 8 seconds. Some techniques are used to reduce the complexity of transformer operations. The proposed method gets good results on THUMOS’14, TVSeries and HACS datasets. 

This work proposes a two-stage processing of video frames by Transformer decoders for video classification.  In the first stage a set of learnt queries “decode” a sequence of fixed-number of earlier video frames. Then in the second stage, features from recent frames act as queries and further “decode” the vectors output in the first stage. At the end of the second stage, final classification is carried out.   The main contribution is this two-stage design of the video classification model. The fixed-length “long-term” context of video frames is shown to be useful for “online” frame classification (i.e., w/o access to the future frames).  This work presents a set of experiments for different design choices, which is also informative.  The key experiments are online video classification on — THUMOS’14, TVSeries, and HACS datasets, where the proposed model outperforms the baselines reported in this work. 

The paper addresses an interesting and practical problem in videos – online detection of actions by examining only frames available till the present time and without any access to future frames. The problem is not only challenging but has practical significance too as online action detection is important in autonomous driving or surveillance scenarios and handling this with fast and light memory-weight training and inference enables one to address the problem real-time. The solution approach is based on the intuition that recent frames are more instrumental about the ongoing action while frames from extended past offer potential context to the present action. While differential exploitation of recent and extended past can definitely help (compared to using frames only from the recent past), it comes with extra computation and memory overhead. Based on the recently popular transformer philosophy, the authors came up with an encoder-decoder architecture with short and long term memories for recent and extended memories respectively. While related works have explored long and short term memories, the integration of the two outputs from them is not explicit and rather ad-hoc (pooling, concatenation etc.). The proposed work, on the other hand, made intelligent use of transformers with a two-stage memory compression design based on transformer decoders. Such a design is shown to be not only effective but also both memory and compute efficient. The experiments are well-designed. Extensive comparison with SOTA approaches and well-thought ablations show its superiority and robustness of design choices. The writing is crisp and adequate with insightful experimental analysis. The figures are also self-explanatory with descriptive captions. The related works are also covered good.

This paper presents a novel long video model named long short term transformer, with online action detection as the study case. The proposed transformer uses the encoder to model long-term historical information and uses the decoder to model the short-term information i.e. a few frames right before the current time. Experiments have been done on three standard benchmarks (THUMOS, TV, HACS) and the performance gains are clear compared to SOTA. Detailed analysis has been conducted as well.