Long Short-Term Transformer for Online Action Detection

1 Introduction . Given an incoming stream of video frames , online action detection [ 14 ] is concerned with the task of classifying what is happening at each frame without seeing the future . Unlike offline methods that assume the entire video is available , online methods process the data causally , up to the current time . In this paper , we present an online temporal modeling algorithm capable of capturing temporal relations on prolonged sequences up to 8 minutes long , while retaining fine granularity of the event in the representation . This is achieved by modeling activities at different temporal scales , so as to capture a variety of events ranging from bursts to slow trends . Specifically , we propose a method , named Long Short-term TRansformer ( LSTR ) , to jointly model long- and short-term temporal dependencies . LSTR has two main advantages over prior work . 1 ) It stores the history directly thus avoiding the pitfalls of recurrent models [ 18 , 50 , 28 , 10 ] . Backpropagation through time , BPTT , is not needed as the model can directly attend to any useful frames from memory . 2 ) It separates long- and short-term memories , which allows modeling short-term context while extracting useful correlations from the long-term history . This allows us to compress the long-term history without losing important fine-scale information . As shown in Fig . 1 , we explicitly divide the entire history into the long- and short-term memories and build our model with an encoder-decoder architecture . Specifically , the LSTR encoder compresses and abstracts the long-term memory into a latent representation of fixed length , and the LSTR decoder uses a short window of transient frames to perform self-attention and cross-attention operations on the extracted token embeddings from the LSTR encoder . In the LSTR encoder , an extended temporal support becomes beneficial in dealing with untrimmed , streaming videos by devising twostage memory compression , which is shown to be computationally efficient in both training and inference . Our overall long short-term Transformer architecture gives rise to an effective and efficient representation for modeling prolonged sequence data . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We validate LSTR on standard benchmark datasets ( THUMOS ’ 14 [ 30 ] , TVSeries [ 14 ] , and HACS Segment [ 76 ] ) . These have distinct characteristics such as video length spanning from a few seconds to tens of minutes . Experimental results establish LSTR as the state-of-the-art for online action detection . Ablation studies further showcase LSTR ’ s abilities in modeling long video sequences . 2 Related Work . Online Action Detection . Temporal action localization aims to detect the onset and termination of action instances after observing the entire video [ 52 , 71 , 23 , 53 , 78 , 5 , 37 , 39 , 38 ] . Embodied perception , however , requires causal processing [ 73 ] , where we can only process the data up to the present . Online action detection focuses on this setting [ 14 ] . RED [ 22 ] uses a reinforcement loss to encourage recognizing actions as early as possible . TRN [ 72 ] models greater temporal context by simultaneously performing online action detection and anticipation . IDN [ 19 ] learns discriminative features and accumulates only relevant information for the present . LAP-Net [ 46 ] proposes an adaptive sampling strategy to obtain optimal features . PKD [ 77 ] transfers knowledge from offline to online models using curriculum learning . As with early action detection [ 27 , 40 ] , Shou et al . [ 54 ] focus on online detection of action start ( ODAS ) . StartNet [ 24 ] decomposes ODAS into two stages and learns with policy gradient . WOAD [ 25 ] uses weakly-supervised learning with video-level labels . Temporal/Sequence Modeling . Causal time series analysis has traditionally assumed the existence of a latent “ state ” variable that captures all information in past data , and is updated using only the current datum [ 49 , 28 , 10 ] . While the Separation Principle ensures that such a state exists for linearGaussian time series , in general it is not possible to summarize all past history of complex data in a finite-dimensional sufficient statistic . Therefore , we directly model the history , in accordance with other work on video understanding [ 69 , 45 , 68 ] . Earlier work on action recognition usually relies on heuristic sub-sampling ( typically 3 to 7 video frames ) for more feasible training [ 74 , 62 , 21 , 41 , 57 ] . 3D ConvNets [ 59 , 8 , 60 ] are used to perform spatio-temporal feature modeling on more frames , but they fail to capture temporal correlations beyond their receptive field . Recently , Wu et al . [ 69 ] propose long-term feature banks to capture objects and scene features , but discarding their temporal order which is clearly informative . Most of work above does not explicitly separate the long- and short-term context modeling , but instead integrates all observed features with simple mechanisms such as pooling or concatenation . We are motivated by work in Cognitive Science [ 44 , 12 , 9 , 35 ] that has shed light on the design principles for modeling long-term dependencies with attention mechanism [ 66 , 13 , 48 , 75 ] . Transformers for Action Understanding . Transformers have achieved breakthrough success in NLP [ 47 , 15 ] and are adopted in computer vision for image recognition [ 17 , 58 ] and object detection [ 7 ] . Recent papers exploit Transformers for temporal modeling tasks in videos , such as action recognition [ 43 , 51 , 36 , 4 , 3 ] and temporal action localization [ 42 , 56 ] , and achieve promising results . However , computational and memory demands result in most work being limited to short video clips , with few exceptions [ 13 , 6 ] that focuses on designing Transformers to model long-range context . The mechanism for aggregating long- and short-term information is relatively unexplored [ 32 ] . 3 Long Short-Term Transformer . Given a live streaming video , our goal is to identify the actions performed in each video frame using only past and current observations . Future information is not accessible during inference . Formally , a streaming video at time t is represented by a batch of τ past frames It = { It−τ , · · · , It } , which reads “ I up to time t. ” The online action detection system receives It as input , and classifies the action category ŷt belonging to one of ( K + 1 ) classes , ŷt ∈ { 0 , 1 , · · · , K } , ideally using the posterior probability P ( ŷt = k|It ) , where k = 0 denotes the probability that no event is occurring at frame t. We design our method by assuming that there is a pretrained feature extractor [ 62 ] that processes each video frame It into a feature vector ft ∈ RC of C dimensions1 . These vectors form a ( τ × C ) -dimensional temporal sequence that serves as the input of our method . 3.1 Overview . Our method is based on the intuition that frames observed recently provide precise information about the ongoing action instance , while frames over an extended period offer contextual references for actions that are potentially happening right now . We propose Long Short-term TRansformer ( LSTR ) in an explicit encoder-decoder manner , as shown in Fig . 2 . In particular , the feature vectors of mL frames in the distant past are stored in a long-term memory , and a short-term memory stores the features of mS recent frames . The LSTR encoder compresses and abstracts features in long-term memory to an encoded latent representation of n1 vectors . The LSTR decoder queries the encoded long-term memory with the short-term memory for decoding , leading to the action prediction ŷt . This design follows the line of thought in combining long- and short-term information for action understanding [ 16 , 62 , 69 ] , but addresses several key challenges to efficiently achieve this goal , exploiting the flexibility of Transformers [ 61 ] . 3.2 Long- and Short-Term Memories . We store the streaming input of feature vectors into two consecutive memories . The first is the short-term memory which stores only a small number of frames that are recently observed . We implement it with a first-in-first-out ( FIFO ) queue of mS slots . At time T , it stores the feature vectors as MS = { fT , · · · , fT−mS+1 } . When a frame becomes “ older ” than mS time steps , it graduates from MS and enters into the long-term memory , which is implemented with another FIFO queue of mL slots . The long term memory stores ML = { fT−mS , · · · , ft−mS−mL+1 } . The long-term memory serves as the input memory to the LSTR encoder and the short-term memory serves as the queries for the LSTR decoder . In practice , the long-term memory stores a much longer time span than the short-term memory ( mS mL ) . A typical choice is mL = 2048 , which represents 512 seconds worth of video contents with 4 frames per second ( FPS ) sampling rate , and mS = 32 representing 1In practice , some feature extractors [ 59 ] take consecutive frames to produce one feature vector . Nonetheless , it is still temporally “ centered ” on a single frame . Thus we use the single frame notation here for simplicity . 8 seconds . We add a sinusoidal positional encoding s [ 61 ] to each frame feature in the memories relative to current time T ( i.e. , the frame at T − τ receives a positional embedding of sτ ) . 3.3 LSTR Encoder . The LSTR encoder aims at encoding the long-term memory of mL feature vectors into a latent representation that LSTR can use for decoding useful temporal context . This task requires a large capacity in capturing the relations and temporal context across a span of hundreds or even thousands of frames . Prior work on modeling long-term context for action understanding relies on heuristic temporal sub-sampling [ 69 , 62 ] or recurrent networks [ 16 ] to make training feasible , at the cost of losing specific information of each time step . Attention-based architectures , such as Transformer [ 61 ] , have recently been shown promising for similar tasks that require long-range temporal modeling [ 51 ] . A straightforward choice for LSTR encoder would be to use a Transformer encoder based on self-attention . However , its time complexity , O ( m2LC ) , grows quadratically with the memory sequence length mL . This limits our ability to model long-term memory with sufficient length to cover long videos . Though recent work [ 64 ] has explored self-attention with linear complexity , repeatedly referencing information from the long-term memory with multi-layer Transformers is still computationally heavy . In LSTR , we propose to use the two-stage memory compression mechanism based on Transformer decoder units [ 61 ] to achieve more effective memory encoding . The Transformer decoder unit [ 61 ] takes two sets of inputs . The first set includes a fixed number of n learnable output tokens λ ∈ Rn×C , where C is the embedding dimension . The second set includes another m input tokens θ ∈ Rm×C , where m can be a rather large number . It first applies one layer of multi-head self-attention on λ . The outputs λ′ are then used as queries in an “ QKV cross-attention ” operation and the input embeddings θ serve as key and value . The two steps can be written as λ′ = SelfAttn ( λ ) = Softmax ( λ · λT√ C ) λ and CrossAttn ( σ ( λ′ ) , θ ) = Softmax ( σ ( λ′ ) · θT√ C ) θ , where σ : Rn×C → Rn×C denotes the intermediate layers between the two attention operations . One appealing property of this design is that it transforms the m× C dimensional input tokens into output tokens of n× C dimensions in O ( n2C + nmC ) time complexity . When n m , the time complexity becomes linear to m , making it an ideal candidate for compressing long-term memory . This property is also utilized in [ 32 ] to efficiently process large volume inputs , such as image pixels . Two-Stage Memory Compression . Stacking multiple Transformer decoder units on the long-term memory , as in [ 61 ] , can form a memory encoder with linear complexity with respect to the memory size mL . However , running the encoder at each time step can still be time consuming . We can further reduce the time complexity with a two-stage memory compression design . The first stage has one Transformer decoder unit with n0 output tokens . Its input tokens are the entire long-term memory of size mL . The outputs of the first stage are used as the input tokens to the second stage , which has ` enc stacked Transformer decoder units and n1 output tokens . Then , the long-term memory of size mL × C is compressed into a latent representation of size n1 × C , which can then be efficiently queried in the LSTR decoder later . This two-stage memory compression design is illustrated in Fig . 2 . Compared to an ( 1 + ` enc ) -layer Transformer encoder with O ( m2L ( 1 + ` enc ) C ) time complexity or stacked Transformer decoder units with n output-tokens having O ( ( n2 + nmL ) ( 1 + ` enc ) C ) time complexity , the proposed LSTR encoder has complexity of O ( n20C+n0mLC+ ( n 2 1+n1n0 ) ` encC ) . Because both n0 and n1 are much smaller than mL , and ` enc is usually larger than 1 , using two-stage memory compression could be more efficient . In Sec . 3.6 , we will show that , during online inference , it further enables us to reduce the runtime of the Transformer decoder unit of the first stage . In Sec . 4.5 , we empirically found this design also leads to better performance for online action detection .

Summary: This paper proposed a transformer-based architecture for online action detection. The architecture contains a long-term memory that stores information from the past 8 minutes and a short-term memory that processes the recent 8 seconds. Some techniques are used to reduce the complexity of transformer operations. The proposed method gets good results on THUMOS’14, TVSeries and HACS datasets.