Long Short-Term Transformer for Online Action Detection

1 Introduction . Given an incoming stream of video frames , online action detection [ 14 ] is concerned with the task of classifying what is happening at each frame without seeing the future . Unlike offline methods that assume the entire video is available , online methods process the data causally , up to the current time . In this paper , we present an online temporal modeling algorithm capable of capturing temporal relations on prolonged sequences up to 8 minutes long , while retaining fine granularity of the event in the representation . This is achieved by modeling activities at different temporal scales , so as to capture a variety of events ranging from bursts to slow trends . Specifically , we propose a method , named Long Short-term TRansformer ( LSTR ) , to jointly model long- and short-term temporal dependencies . LSTR has two main advantages over prior work . 1 ) It stores the history directly thus avoiding the pitfalls of recurrent models [ 18 , 50 , 28 , 10 ] . Backpropagation through time , BPTT , is not needed as the model can directly attend to any useful frames from memory . 2 ) It separates long- and short-term memories , which allows modeling short-term context while extracting useful correlations from the long-term history . This allows us to compress the long-term history without losing important fine-scale information . As shown in Fig . 1 , we explicitly divide the entire history into the long- and short-term memories and build our model with an encoder-decoder architecture . Specifically , the LSTR encoder compresses and abstracts the long-term memory into a latent representation of fixed length , and the LSTR decoder uses a short window of transient frames to perform self-attention and cross-attention operations on the extracted token embeddings from the LSTR encoder . In the LSTR encoder , an extended temporal support becomes beneficial in dealing with untrimmed , streaming videos by devising twostage memory compression , which is shown to be computationally efficient in both training and inference . Our overall long short-term Transformer architecture gives rise to an effective and efficient representation for modeling prolonged sequence data . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We validate LSTR on standard benchmark datasets ( THUMOS ’ 14 [ 30 ] , TVSeries [ 14 ] , and HACS Segment [ 76 ] ) . These have distinct characteristics such as video length spanning from a few seconds to tens of minutes . Experimental results establish LSTR as the state-of-the-art for online action detection . Ablation studies further showcase LSTR ’ s abilities in modeling long video sequences . 2 Related Work . Online Action Detection . Temporal action localization aims to detect the onset and termination of action instances after observing the entire video [ 52 , 71 , 23 , 53 , 78 , 5 , 37 , 39 , 38 ] . Embodied perception , however , requires causal processing [ 73 ] , where we can only process the data up to the present . Online action detection focuses on this setting [ 14 ] . RED [ 22 ] uses a reinforcement loss to encourage recognizing actions as early as possible . TRN [ 72 ] models greater temporal context by simultaneously performing online action detection and anticipation . IDN [ 19 ] learns discriminative features and accumulates only relevant information for the present . LAP-Net [ 46 ] proposes an adaptive sampling strategy to obtain optimal features . PKD [ 77 ] transfers knowledge from offline to online models using curriculum learning . As with early action detection [ 27 , 40 ] , Shou et al . [ 54 ] focus on online detection of action start ( ODAS ) . StartNet [ 24 ] decomposes ODAS into two stages and learns with policy gradient . WOAD [ 25 ] uses weakly-supervised learning with video-level labels . Temporal/Sequence Modeling . Causal time series analysis has traditionally assumed the existence of a latent “ state ” variable that captures all information in past data , and is updated using only the current datum [ 49 , 28 , 10 ] . While the Separation Principle ensures that such a state exists for linearGaussian time series , in general it is not possible to summarize all past history of complex data in a finite-dimensional sufficient statistic . Therefore , we directly model the history , in accordance with other work on video understanding [ 69 , 45 , 68 ] . Earlier work on action recognition usually relies on heuristic sub-sampling ( typically 3 to 7 video frames ) for more feasible training [ 74 , 62 , 21 , 41 , 57 ] . 3D ConvNets [ 59 , 8 , 60 ] are used to perform spatio-temporal feature modeling on more frames , but they fail to capture temporal correlations beyond their receptive field . Recently , Wu et al . [ 69 ] propose long-term feature banks to capture objects and scene features , but discarding their temporal order which is clearly informative . Most of work above does not explicitly separate the long- and short-term context modeling , but instead integrates all observed features with simple mechanisms such as pooling or concatenation . We are motivated by work in Cognitive Science [ 44 , 12 , 9 , 35 ] that has shed light on the design principles for modeling long-term dependencies with attention mechanism [ 66 , 13 , 48 , 75 ] . Transformers for Action Understanding . Transformers have achieved breakthrough success in NLP [ 47 , 15 ] and are adopted in computer vision for image recognition [ 17 , 58 ] and object detection [ 7 ] . Recent papers exploit Transformers for temporal modeling tasks in videos , such as action recognition [ 43 , 51 , 36 , 4 , 3 ] and temporal action localization [ 42 , 56 ] , and achieve promising results . However , computational and memory demands result in most work being limited to short video clips , with few exceptions [ 13 , 6 ] that focuses on designing Transformers to model long-range context . The mechanism for aggregating long- and short-term information is relatively unexplored [ 32 ] . 3 Long Short-Term Transformer . Given a live streaming video , our goal is to identify the actions performed in each video frame using only past and current observations . Future information is not accessible during inference . Formally , a streaming video at time t is represented by a batch of τ past frames It = { It−τ , · · · , It } , which reads “ I up to time t. ” The online action detection system receives It as input , and classifies the action category ŷt belonging to one of ( K + 1 ) classes , ŷt ∈ { 0 , 1 , · · · , K } , ideally using the posterior probability P ( ŷt = k|It ) , where k = 0 denotes the probability that no event is occurring at frame t. We design our method by assuming that there is a pretrained feature extractor [ 62 ] that processes each video frame It into a feature vector ft ∈ RC of C dimensions1 . These vectors form a ( τ × C ) -dimensional temporal sequence that serves as the input of our method . 3.1 Overview . Our method is based on the intuition that frames observed recently provide precise information about the ongoing action instance , while frames over an extended period offer contextual references for actions that are potentially happening right now . We propose Long Short-term TRansformer ( LSTR ) in an explicit encoder-decoder manner , as shown in Fig . 2 . In particular , the feature vectors of mL frames in the distant past are stored in a long-term memory , and a short-term memory stores the features of mS recent frames . The LSTR encoder compresses and abstracts features in long-term memory to an encoded latent representation of n1 vectors . The LSTR decoder queries the encoded long-term memory with the short-term memory for decoding , leading to the action prediction ŷt . This design follows the line of thought in combining long- and short-term information for action understanding [ 16 , 62 , 69 ] , but addresses several key challenges to efficiently achieve this goal , exploiting the flexibility of Transformers [ 61 ] . 3.2 Long- and Short-Term Memories . We store the streaming input of feature vectors into two consecutive memories . The first is the short-term memory which stores only a small number of frames that are recently observed . We implement it with a first-in-first-out ( FIFO ) queue of mS slots . At time T , it stores the feature vectors as MS = { fT , · · · , fT−mS+1 } . When a frame becomes “ older ” than mS time steps , it graduates from MS and enters into the long-term memory , which is implemented with another FIFO queue of mL slots . The long term memory stores ML = { fT−mS , · · · , ft−mS−mL+1 } . The long-term memory serves as the input memory to the LSTR encoder and the short-term memory serves as the queries for the LSTR decoder . In practice , the long-term memory stores a much longer time span than the short-term memory ( mS mL ) . A typical choice is mL = 2048 , which represents 512 seconds worth of video contents with 4 frames per second ( FPS ) sampling rate , and mS = 32 representing 1In practice , some feature extractors [ 59 ] take consecutive frames to produce one feature vector . Nonetheless , it is still temporally “ centered ” on a single frame . Thus we use the single frame notation here for simplicity . 8 seconds . We add a sinusoidal positional encoding s [ 61 ] to each frame feature in the memories relative to current time T ( i.e. , the frame at T − τ receives a positional embedding of sτ ) . 3.3 LSTR Encoder . The LSTR encoder aims at encoding the long-term memory of mL feature vectors into a latent representation that LSTR can use for decoding useful temporal context . This task requires a large capacity in capturing the relations and temporal context across a span of hundreds or even thousands of frames . Prior work on modeling long-term context for action understanding relies on heuristic temporal sub-sampling [ 69 , 62 ] or recurrent networks [ 16 ] to make training feasible , at the cost of losing specific information of each time step . Attention-based architectures , such as Transformer [ 61 ] , have recently been shown promising for similar tasks that require long-range temporal modeling [ 51 ] . A straightforward choice for LSTR encoder would be to use a Transformer encoder based on self-attention . However , its time complexity , O ( m2LC ) , grows quadratically with the memory sequence length mL . This limits our ability to model long-term memory with sufficient length to cover long videos . Though recent work [ 64 ] has explored self-attention with linear complexity , repeatedly referencing information from the long-term memory with multi-layer Transformers is still computationally heavy . In LSTR , we propose to use the two-stage memory compression mechanism based on Transformer decoder units [ 61 ] to achieve more effective memory encoding . The Transformer decoder unit [ 61 ] takes two sets of inputs . The first set includes a fixed number of n learnable output tokens λ ∈ Rn×C , where C is the embedding dimension . The second set includes another m input tokens θ ∈ Rm×C , where m can be a rather large number . It first applies one layer of multi-head self-attention on λ . The outputs λ′ are then used as queries in an “ QKV cross-attention ” operation and the input embeddings θ serve as key and value . The two steps can be written as λ′ = SelfAttn ( λ ) = Softmax ( λ · λT√ C ) λ and CrossAttn ( σ ( λ′ ) , θ ) = Softmax ( σ ( λ′ ) · θT√ C ) θ , where σ : Rn×C → Rn×C denotes the intermediate layers between the two attention operations . One appealing property of this design is that it transforms the m× C dimensional input tokens into output tokens of n× C dimensions in O ( n2C + nmC ) time complexity . When n m , the time complexity becomes linear to m , making it an ideal candidate for compressing long-term memory . This property is also utilized in [ 32 ] to efficiently process large volume inputs , such as image pixels . Two-Stage Memory Compression . Stacking multiple Transformer decoder units on the long-term memory , as in [ 61 ] , can form a memory encoder with linear complexity with respect to the memory size mL . However , running the encoder at each time step can still be time consuming . We can further reduce the time complexity with a two-stage memory compression design . The first stage has one Transformer decoder unit with n0 output tokens . Its input tokens are the entire long-term memory of size mL . The outputs of the first stage are used as the input tokens to the second stage , which has ` enc stacked Transformer decoder units and n1 output tokens . Then , the long-term memory of size mL × C is compressed into a latent representation of size n1 × C , which can then be efficiently queried in the LSTR decoder later . This two-stage memory compression design is illustrated in Fig . 2 . Compared to an ( 1 + ` enc ) -layer Transformer encoder with O ( m2L ( 1 + ` enc ) C ) time complexity or stacked Transformer decoder units with n output-tokens having O ( ( n2 + nmL ) ( 1 + ` enc ) C ) time complexity , the proposed LSTR encoder has complexity of O ( n20C+n0mLC+ ( n 2 1+n1n0 ) ` encC ) . Because both n0 and n1 are much smaller than mL , and ` enc is usually larger than 1 , using two-stage memory compression could be more efficient . In Sec . 3.6 , we will show that , during online inference , it further enables us to reduce the runtime of the Transformer decoder unit of the first stage . In Sec . 4.5 , we empirically found this design also leads to better performance for online action detection .

Summary: This paper presents a novel long video model named long short term transformer, with online action detection as the study case. The proposed transformer uses the encoder to model long-term historical information and uses the decoder to model the short-term information i.e. a few frames right before the current time. Experiments have been done on three standard benchmarks (THUMOS, TV, HACS) and the performance gains are clear compared to SOTA. Detailed analysis has been conducted as well.