Unified Recurrence Modeling for Video Action Anticipation

1 INTRODUCTION . Video action recognition is a long-standing problem in computer vision . It predicts the action from experienced frames by understanding the contexts in the observations . However , having the prediction after the whole observation of action is insufficient to evaluate the outcome of any decision making . Predicting future human activities and interactions of objects are receiving research interests in these years , in assisting navigation systems ( OhnBar et al. , 2018 ) , robotics ( Park et al. , 2016 ) , entertainment ( Liang et al. , 2015 ; Taylor et al. , 2020 ) and autonomous vehicles ( Hirakawa et al. , 2018 ) . Different from action recognition , the video action anticipation is required to give the predictions strictly before the action frames being observed . Models , in this case , are expected to collect the subtle evidence in the preamble of target actions ( Furnari et al. , 2018 ; Furnari & Farinella , 2020 ; Rodin et al. , 2021 ) . In the high-level viewpoint , the objective of anticipating is not only to understand the actual content in the observations but seeking to associate the minor details to the next possible action . Figure 1 illustrates the definition of the video anticipation problem . In the absence of target action frames , pretrained models for the robust feature extraction from RGB inputs are often deployed on video anticipation , either pretrained from image classification or video action recognition . On top of the pretrained models , recurrent neural networks are the widely adopted options to model the temporal relationship in anticipation problems and lead to better performance than clip based modeling methods . Unlike the action recognition settings , where the relevant frames associated with the target action are not necessary in the end of observation and may locate in any action duration . video anticipation predicts the action that always performs in the future of observations , making the accumulated information gains aligned with the temporal orders . To this end , we propose a unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from a sequence into graph representation via message passing . We utilize self-attention as the universal building block in the graph modeling . In the graph point of view , selfattention can be treated as the information routing between vertices . The attention weight derived by the scaled dot-product , which computes the correlation of vertices , can be explained as an adjacency estimation implicitly . However , in this way , the representation of edges is limited and purely based on the similarity in input tensors . Therefore , we propose three explicitly edge learning strategies trying to escape to the trivial estimation by bringing the flexibility of edge representations . They are end-to-end optimized and can be jointly trained with the main task . In the experimental results , we show our proposed unified recurrent modeling outperforms several state-of-the-art methods on largescale egocentric video dataset EPIC-Kitchen on video anticipation . Combining with edge learning strategies , the performance can even boost to the next level . 2 RELATED WORKS . 2.1 VIDEO ACTION ANTICIPATION . Early works in video anticipation model the problem with recurrent neural networks ( Gao et al. , 2017 ; Farha et al. , 2018 ; Miech et al. , 2019 ) . Some prior works also leverage the future frames for learning the representations ( Vondrick et al. , 2016 ; Fernando & Herath , 2021 ) . A self-regulated learning framework for action anticipation in the egocentric video is presented in ( Qi et al. , 2021 ) . RU-LSTM ( Furnari & Farinella , 2019 ) deploys two LSTMs and behaves like an encoder-decoder , where the first progressively summarizes the observed together with the second that unrolls over future predictions without observing . ( Osman et al. , 2021 ; Tai et al. , 2021 ) both integrate unrolling mechanism as RU-LSTM does , but replace the rolling part in RU-LSTM with SlowFast ( Feichtenhofer et al. , 2019 ) and higher-order recurrent networks , respectively . ( Sener et al. , 2020 ) aggregates the predictions by pooling over different granularity of temporal segments . ( Girdhar & Grauman , 2021 ) combine causal self-attention with the look ahead prediction on successive frames . 2.2 MESSAGE PASSING NEURAL NETWORK . The concept of message passing is first published in ( Gilmer et al. , 2017 ) , where it was originally designed for molecular property prediction . It assumes an undirected graph structure with dataindependent , equal edge contributions . To address this limitation , ( Ma et al. , 2020 ) deploys two encoders separately for vertex and edge estimation and aggregates them by a self-attention readout . Some prior works use attention or dedicated network design to learn the directed edge representations to improve the model capability ( Jørgensen et al. , 2018 ; Withnall et al. , 2020 ; Gong & Cheng , 2019 ) . Recently , ( Arnab et al. , 2021b ) express the non-local attention ( Wang et al. , 2018 ) and GAT ( Veličković et al. , 2017 ) as message passing functions and apply them to video understanding task . Differently , we view the message passing framework as the generalized recurrent models and specialize it for edge representations learning in this work . 2.3 SELF-ATTENTION . ( Vaswani et al. , 2017 ) first propose a recurrence free sequence learning architecture by stacking several self-attention layers , which can achieve remarkable performance in the NLP domain . ( Dehghani et al. , 2018 ) demonstrates that the self-attention can be treated as the recurrent unit which unfolds to input sequences to processes with shared weights . On the other hand , ( Dosovitskiy et al. , 2020 ) propose Vision Transformer ( ViT ) , an architecture with only self-attention for image classification . ( Zhou et al. , 2021 ) improves ViT by re-attending the multi-heads information in the post-softmax step to enable the deeper configuration . Some recent studies explore ViT based models on video action recognition ( Arnab et al. , 2021a ; Bertasius et al. , 2021 ) , and also video anticipation ( Girdhar & Grauman , 2021 ) . Unlike these prior works , our proposed model processes the video in a flexible graph representation and fits into the message passing framework . It is lightweight and only contains few self-attention layers which sequentially process each timestep . 3 METHODOLOGY . 3.1 BACKGROUNDS . 3.1.1 MESSAGE PASSING . Given an undirected graph G , the Message Passing algorithm involves two-phase forwarding processes , message passing phase with message function M and update function U , and readout phase with readout function R. The message passing phase can run arbitrary T step with the following definition : mt+1v = ∑ w∈N ( v ) Mt ( h t v , h t w , avw ) ( 1 ) ht+1v = Ut ( h t v , m t+1 v ) ( 2 ) where v is the vertex in G , and N ( v ) defines the neighbors of v. avw is the connection strength bonding between vertex v and w. The readout phase then extract the features of the whole graph , abstracted from the message passing phase , at time T , ŷ = R ( hTv |v ∈ G ) . ( 3 ) Our proposed method inherits these three core functions ( e.g. , message , update , and readout ) . However , the most notable difference is that we treat the whole anticipation predicting as a chain of messages conveyed in space-time , with the constant strengthening of the signal by the RGB inputs in each time-step . The readout function is called when the prediction is required at any time t . 3.1.2 SELF-ATTENTION . Self-Attention ( SA ) forms theQ , K , V tokens ( for query , key , and value ) from the same source input x1 . The output of the attention is the linear combination of the value , based on the attention weight computed by the scaled dot-product between the query and key followed by a softmax . Q , K , V = xW iQ , xW i K , xW i V ( 4 ) SAi ( x ) = softmax ( QTK√ D ) V , ( 5 ) where scaled factor D default to the input feature dimension , W iQ , W i K , W i V are the trainable embeddings . We use superscript i to note the embeddings are associated to each self-attention layer . Multi-Head Self Attention ( MHSA ) performs n-way self-attention in parallel , where n is the total number of heads . An additional aggregation function , with parameters Wagg , is adopted to fuse the information computed from each head . MHSA ( x ) = [ H1 , . . . , Hn ] Wagg , ( 6 ) where Hi = SAi ( x ) 1Some literature uses self-attention no matter the inputs Q , K , V comes from the same source . We use the term , self-attention , strictly when Q , K , V is formed by the same source input . where the [ . , . ] presents the concatenation . Following the transformer-style architecture , a Feed-Forward Network ( FFN ) performs after each attention layer to project the attention output and bring the non-linearity . The FFN computes : FFN ( x ) = σ ( xW1 + b1 ) W2 + b2 ( 7 ) where σ can be any arbitrary nonlinear function . Our proposed model utilizes multi-head self-attention and deeply integrates into the message passing design . We use multi-head self-attention as the information routing between vertices of a graph . Note in this case , the resulting graph is bi-directed , because the dot-product between Q and K is not symmetric . 3.2 UNIFIED RECURRENT MODELING . Self-Attention Block : We implement the self-attention building block in prenorm style ( Xiong et al. , 2020 ) , as implemented in many transformer-style architecture , where the normalization layer is placed before MHSA and FFN and not counts in shortcuts . Figure 2 left shows the self-attention block fMHSA ( x ) = x+MHSA ( LayerNorm ( x ) ) ( 8 ) fFFN ( x ) = x+ FFN ( LayerNorm ( x ) ) ( 9 ) SABlock ( x ) = fFFN ( fMHSA ( x ) ) ( 10 ) We can optionally expose the edge information , in matrix A , into SABlock ( . ; A ) . In this case , the extension definesMHSA ( x ; A ) where the matrixA is fused into the step after the softmax of scaled dot-product computation . We rewrite the equation 5 and equation 6 to MHSA ( x ; A ) = [ H1 , . . . , Hn ] Wagg , ( 11 ) where Hi = SAi ( x ; A ) SAi ( x ; A ) = ( softmax ( A ) + softmax ( ( xW iQ ) T ( xW iK ) √ D ) ) ( xW iV ) ( 12 ) NoteA is unique and shared in multi-heads self-attention . Although the probability comes with postsoftmax becomes twice larger , such difference may not cause an issue because trainable weights , Wagg , WV , are able to absorb the impact . Recurrent Cell : Figure 2 shows the overall architecture . Given the frame features xt at time t , with shape ( H , W , C ) , which is transformed by a backbone model , we define the vertices etv by taking a nonlinear transformation of xt and flatten into 1D , resulting in the shape ( HW , C ) . The position encoding Wpe is added on top of etv . We then leverage SABlock ( . ; A ) to message function M with optionally matrix A , and SABlock ( . ) to update function U for computing the hidden states htv , etv = fse ( σ ( x tWx + b ) ) +Wpe ( 13 ) gtv = fh ( [ e t v , h t−1 v ] ) ( 14 ) mtv = SABlock ( g t v ; A t ) ( 15 ) htv = tanh ( SABlock ( [ e t v , m t v ] ) ) ( 16 ) where fse is a Squeeze-and-Excitation ( SE ) module ( Hu et al. , 2018 ) performs on vertices dimension , and fh is a linear transformer to reduce the feature size from concatenated 2C into C. Note we bound the hidden states by the tanh to ensure the value stability in temporal propagation . For the readout function , we leverage SABlock ( . ) to abstract the information from the hidden state representation htv , yt = SABlock ( fscale ( h t v ) ) . ( 17 ) where fscale is a fully-connected transformation to scale the values of htv . We assume every vertex in the graph G is accessible by any other vertex to maintain the maximal possibility without prior . In this case , N ( v ) ≡ v in equation 1 . The edges , presents by adjacency matrix A = { avw ; ∀w ∈ N ( v ) , v ∈ G } , can be optionally provided , explicitly , else implicitly derived in self-attention . Implicit Edge Estimation : The scaled dot-product original in the self-attention operator can be treated as implicit estimation of edges , which computes the pairwise similarity by correlation measurement of vertices on-the-fly with the parametric trainable embeddings of inputs . Explicit Edge Estimation : On the other hand , edge information can also be provided by feeding a adjacency matrix A during the attention computation as discussed in equation 11 and equation 12 . All edge learning strategies introduced in the following all belong to explicit edge estimation category .

Summary: The paper proposes a method for action anticipation based on a graph representation of the temporal structure of video. Specifically a graph is built using semantic representations of video frames extracted with a 2D CNN backbone as vertices. Temporal reasoning is hence achieved modeling spatiotemporal information flow thorugh message passing across vertices. The approach is based on a message function, an unpdate function and a readout function, all based o a multi-head self-attention block. Experiments compare the proposed approach with respect to other approaches suing RGB inputs on EPIC-KITCHENS-55.