Unified Recurrence Modeling for Video Action Anticipation

1 INTRODUCTION . Video action recognition is a long-standing problem in computer vision . It predicts the action from experienced frames by understanding the contexts in the observations . However , having the prediction after the whole observation of action is insufficient to evaluate the outcome of any decision making . Predicting future human activities and interactions of objects are receiving research interests in these years , in assisting navigation systems ( OhnBar et al. , 2018 ) , robotics ( Park et al. , 2016 ) , entertainment ( Liang et al. , 2015 ; Taylor et al. , 2020 ) and autonomous vehicles ( Hirakawa et al. , 2018 ) . Different from action recognition , the video action anticipation is required to give the predictions strictly before the action frames being observed . Models , in this case , are expected to collect the subtle evidence in the preamble of target actions ( Furnari et al. , 2018 ; Furnari & Farinella , 2020 ; Rodin et al. , 2021 ) . In the high-level viewpoint , the objective of anticipating is not only to understand the actual content in the observations but seeking to associate the minor details to the next possible action . Figure 1 illustrates the definition of the video anticipation problem . In the absence of target action frames , pretrained models for the robust feature extraction from RGB inputs are often deployed on video anticipation , either pretrained from image classification or video action recognition . On top of the pretrained models , recurrent neural networks are the widely adopted options to model the temporal relationship in anticipation problems and lead to better performance than clip based modeling methods . Unlike the action recognition settings , where the relevant frames associated with the target action are not necessary in the end of observation and may locate in any action duration . video anticipation predicts the action that always performs in the future of observations , making the accumulated information gains aligned with the temporal orders . To this end , we propose a unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from a sequence into graph representation via message passing . We utilize self-attention as the universal building block in the graph modeling . In the graph point of view , selfattention can be treated as the information routing between vertices . The attention weight derived by the scaled dot-product , which computes the correlation of vertices , can be explained as an adjacency estimation implicitly . However , in this way , the representation of edges is limited and purely based on the similarity in input tensors . Therefore , we propose three explicitly edge learning strategies trying to escape to the trivial estimation by bringing the flexibility of edge representations . They are end-to-end optimized and can be jointly trained with the main task . In the experimental results , we show our proposed unified recurrent modeling outperforms several state-of-the-art methods on largescale egocentric video dataset EPIC-Kitchen on video anticipation . Combining with edge learning strategies , the performance can even boost to the next level . 2 RELATED WORKS . 2.1 VIDEO ACTION ANTICIPATION . Early works in video anticipation model the problem with recurrent neural networks ( Gao et al. , 2017 ; Farha et al. , 2018 ; Miech et al. , 2019 ) . Some prior works also leverage the future frames for learning the representations ( Vondrick et al. , 2016 ; Fernando & Herath , 2021 ) . A self-regulated learning framework for action anticipation in the egocentric video is presented in ( Qi et al. , 2021 ) . RU-LSTM ( Furnari & Farinella , 2019 ) deploys two LSTMs and behaves like an encoder-decoder , where the first progressively summarizes the observed together with the second that unrolls over future predictions without observing . ( Osman et al. , 2021 ; Tai et al. , 2021 ) both integrate unrolling mechanism as RU-LSTM does , but replace the rolling part in RU-LSTM with SlowFast ( Feichtenhofer et al. , 2019 ) and higher-order recurrent networks , respectively . ( Sener et al. , 2020 ) aggregates the predictions by pooling over different granularity of temporal segments . ( Girdhar & Grauman , 2021 ) combine causal self-attention with the look ahead prediction on successive frames . 2.2 MESSAGE PASSING NEURAL NETWORK . The concept of message passing is first published in ( Gilmer et al. , 2017 ) , where it was originally designed for molecular property prediction . It assumes an undirected graph structure with dataindependent , equal edge contributions . To address this limitation , ( Ma et al. , 2020 ) deploys two encoders separately for vertex and edge estimation and aggregates them by a self-attention readout . Some prior works use attention or dedicated network design to learn the directed edge representations to improve the model capability ( Jørgensen et al. , 2018 ; Withnall et al. , 2020 ; Gong & Cheng , 2019 ) . Recently , ( Arnab et al. , 2021b ) express the non-local attention ( Wang et al. , 2018 ) and GAT ( Veličković et al. , 2017 ) as message passing functions and apply them to video understanding task . Differently , we view the message passing framework as the generalized recurrent models and specialize it for edge representations learning in this work . 2.3 SELF-ATTENTION . ( Vaswani et al. , 2017 ) first propose a recurrence free sequence learning architecture by stacking several self-attention layers , which can achieve remarkable performance in the NLP domain . ( Dehghani et al. , 2018 ) demonstrates that the self-attention can be treated as the recurrent unit which unfolds to input sequences to processes with shared weights . On the other hand , ( Dosovitskiy et al. , 2020 ) propose Vision Transformer ( ViT ) , an architecture with only self-attention for image classification . ( Zhou et al. , 2021 ) improves ViT by re-attending the multi-heads information in the post-softmax step to enable the deeper configuration . Some recent studies explore ViT based models on video action recognition ( Arnab et al. , 2021a ; Bertasius et al. , 2021 ) , and also video anticipation ( Girdhar & Grauman , 2021 ) . Unlike these prior works , our proposed model processes the video in a flexible graph representation and fits into the message passing framework . It is lightweight and only contains few self-attention layers which sequentially process each timestep . 3 METHODOLOGY . 3.1 BACKGROUNDS . 3.1.1 MESSAGE PASSING . Given an undirected graph G , the Message Passing algorithm involves two-phase forwarding processes , message passing phase with message function M and update function U , and readout phase with readout function R. The message passing phase can run arbitrary T step with the following definition : mt+1v = ∑ w∈N ( v ) Mt ( h t v , h t w , avw ) ( 1 ) ht+1v = Ut ( h t v , m t+1 v ) ( 2 ) where v is the vertex in G , and N ( v ) defines the neighbors of v. avw is the connection strength bonding between vertex v and w. The readout phase then extract the features of the whole graph , abstracted from the message passing phase , at time T , ŷ = R ( hTv |v ∈ G ) . ( 3 ) Our proposed method inherits these three core functions ( e.g. , message , update , and readout ) . However , the most notable difference is that we treat the whole anticipation predicting as a chain of messages conveyed in space-time , with the constant strengthening of the signal by the RGB inputs in each time-step . The readout function is called when the prediction is required at any time t . 3.1.2 SELF-ATTENTION . Self-Attention ( SA ) forms theQ , K , V tokens ( for query , key , and value ) from the same source input x1 . The output of the attention is the linear combination of the value , based on the attention weight computed by the scaled dot-product between the query and key followed by a softmax . Q , K , V = xW iQ , xW i K , xW i V ( 4 ) SAi ( x ) = softmax ( QTK√ D ) V , ( 5 ) where scaled factor D default to the input feature dimension , W iQ , W i K , W i V are the trainable embeddings . We use superscript i to note the embeddings are associated to each self-attention layer . Multi-Head Self Attention ( MHSA ) performs n-way self-attention in parallel , where n is the total number of heads . An additional aggregation function , with parameters Wagg , is adopted to fuse the information computed from each head . MHSA ( x ) = [ H1 , . . . , Hn ] Wagg , ( 6 ) where Hi = SAi ( x ) 1Some literature uses self-attention no matter the inputs Q , K , V comes from the same source . We use the term , self-attention , strictly when Q , K , V is formed by the same source input . where the [ . , . ] presents the concatenation . Following the transformer-style architecture , a Feed-Forward Network ( FFN ) performs after each attention layer to project the attention output and bring the non-linearity . The FFN computes : FFN ( x ) = σ ( xW1 + b1 ) W2 + b2 ( 7 ) where σ can be any arbitrary nonlinear function . Our proposed model utilizes multi-head self-attention and deeply integrates into the message passing design . We use multi-head self-attention as the information routing between vertices of a graph . Note in this case , the resulting graph is bi-directed , because the dot-product between Q and K is not symmetric . 3.2 UNIFIED RECURRENT MODELING . Self-Attention Block : We implement the self-attention building block in prenorm style ( Xiong et al. , 2020 ) , as implemented in many transformer-style architecture , where the normalization layer is placed before MHSA and FFN and not counts in shortcuts . Figure 2 left shows the self-attention block fMHSA ( x ) = x+MHSA ( LayerNorm ( x ) ) ( 8 ) fFFN ( x ) = x+ FFN ( LayerNorm ( x ) ) ( 9 ) SABlock ( x ) = fFFN ( fMHSA ( x ) ) ( 10 ) We can optionally expose the edge information , in matrix A , into SABlock ( . ; A ) . In this case , the extension definesMHSA ( x ; A ) where the matrixA is fused into the step after the softmax of scaled dot-product computation . We rewrite the equation 5 and equation 6 to MHSA ( x ; A ) = [ H1 , . . . , Hn ] Wagg , ( 11 ) where Hi = SAi ( x ; A ) SAi ( x ; A ) = ( softmax ( A ) + softmax ( ( xW iQ ) T ( xW iK ) √ D ) ) ( xW iV ) ( 12 ) NoteA is unique and shared in multi-heads self-attention . Although the probability comes with postsoftmax becomes twice larger , such difference may not cause an issue because trainable weights , Wagg , WV , are able to absorb the impact . Recurrent Cell : Figure 2 shows the overall architecture . Given the frame features xt at time t , with shape ( H , W , C ) , which is transformed by a backbone model , we define the vertices etv by taking a nonlinear transformation of xt and flatten into 1D , resulting in the shape ( HW , C ) . The position encoding Wpe is added on top of etv . We then leverage SABlock ( . ; A ) to message function M with optionally matrix A , and SABlock ( . ) to update function U for computing the hidden states htv , etv = fse ( σ ( x tWx + b ) ) +Wpe ( 13 ) gtv = fh ( [ e t v , h t−1 v ] ) ( 14 ) mtv = SABlock ( g t v ; A t ) ( 15 ) htv = tanh ( SABlock ( [ e t v , m t v ] ) ) ( 16 ) where fse is a Squeeze-and-Excitation ( SE ) module ( Hu et al. , 2018 ) performs on vertices dimension , and fh is a linear transformer to reduce the feature size from concatenated 2C into C. Note we bound the hidden states by the tanh to ensure the value stability in temporal propagation . For the readout function , we leverage SABlock ( . ) to abstract the information from the hidden state representation htv , yt = SABlock ( fscale ( h t v ) ) . ( 17 ) where fscale is a fully-connected transformation to scale the values of htv . We assume every vertex in the graph G is accessible by any other vertex to maintain the maximal possibility without prior . In this case , N ( v ) ≡ v in equation 1 . The edges , presents by adjacency matrix A = { avw ; ∀w ∈ N ( v ) , v ∈ G } , can be optionally provided , explicitly , else implicitly derived in self-attention . Implicit Edge Estimation : The scaled dot-product original in the self-attention operator can be treated as implicit estimation of edges , which computes the pairwise similarity by correlation measurement of vertices on-the-fly with the parametric trainable embeddings of inputs . Explicit Edge Estimation : On the other hand , edge information can also be provided by feeding a adjacency matrix A during the attention computation as discussed in equation 11 and equation 12 . All edge learning strategies introduced in the following all belong to explicit edge estimation category .

Summary: The authors propose a framework for video action anticipation, where the task is to observe input frames and predict the action label after an anticipation period (not observed). The proposed architecture builds on top of message passing terminology (message, update, and readout functions) to create a recurrent transformer model that uses self-attention to capture spatial dependencies between frame patches (similar to ViT w/o [cls]) and recurrent module to capture the temporal dependencies across frames. Edge affinity matrix is learned and introduced into the MHSA block (Eqn. 12) as a prior to influence the attention matrix at every layer. Three different explicit edge learning approaches are proposed, in addition to implicit edge learning. The model is evaluated on the Epic Kitchen 55 dataset.