Year: 2021

Abstract:
Forecasting future events based on evidence of current conditions is an innate skill of human beings, and key for predicting the outcome of any decision making. In artificial vision for example, we would like to predict the next human action before it actually happens, without observing the future video frames associated to it. Computer vision models for action anticipation are expected to collect the subtle evidence in the preamble of the target actions. In prior studies recurrence modeling often leads to better performance, and the strong temporal inference is assumed to be a key element for reasonable prediction. To this end, we propose a unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from sequence into graph representation via message passing. The information flow in space-time can be described by the interaction between vertices and edges, and the changes of vertices for each incoming frame reflects the underlying dynamics. Our model leverages self-attention for all building blocks in the graph modeling, and we introduce different edge learning strategies that can be end-to-end optimized while updating the vertices. Our experimental results demonstrate that our modeling method is light-weight, efficient, and outperforms all previous works using the large-scale EPIC-Kitchen dataset.

heading: 1 INTRODUCTION
text: Video action recognition is a long-standing problem in computer vision. It predicts the action from experienced frames by understanding the contexts in the observations. However, having the prediction after the whole observation of action is insufficient to evaluate the outcome of any decision making. Predicting future human activities and interactions of objects are receiving research interests in these years, in assisting navigation systems (OhnBar et al., 2018), robotics (Park et al., 2016), entertainment (Liang et al., 2015; Taylor et al., 2020) and autonomous vehicles (Hirakawa et al., 2018).
Different from action recognition, the video action anticipation is required to give the predictions strictly before the action frames being observed. Models, in this case, are expected to collect the subtle evidence in the preamble of target actions (Furnari et al., 2018; Furnari & Farinella, 2020; Rodin et al., 2021). In the high-level viewpoint, the objective of anticipating is not only to understand the actual content in the observations but seeking to associate the minor details to the next possible action. Figure 1 illustrates the definition of the video anticipation problem.
In the absence of target action frames, pretrained models for the robust feature extraction from RGB inputs are often deployed on video anticipation, either pretrained from image classification or video action recognition. On top of the pretrained models, recurrent neural networks are the widely adopted options to model the temporal relationship in anticipation problems and lead to better performance than clip based modeling methods. Unlike the action recognition settings, where the relevant frames associated with the target action are not necessary in the end of observation and may locate in any action duration. video anticipation predicts the action that always performs in the future of observations, making the accumulated information gains aligned with the temporal orders. To this end, we propose a unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from a sequence into graph representation via message passing. We utilize self-attention as the universal building block in the graph modeling. In the graph point of view, selfattention can be treated as the information routing between vertices. The attention weight derived by
the scaled dot-product, which computes the correlation of vertices, can be explained as an adjacency estimation implicitly. However, in this way, the representation of edges is limited and purely based on the similarity in input tensors. Therefore, we propose three explicitly edge learning strategies trying to escape to the trivial estimation by bringing the flexibility of edge representations. They are end-to-end optimized and can be jointly trained with the main task. In the experimental results, we show our proposed unified recurrent modeling outperforms several state-of-the-art methods on largescale egocentric video dataset EPIC-Kitchen on video anticipation. Combining with edge learning strategies, the performance can even boost to the next level.

heading: 2 RELATED WORKS

heading: 2.1 VIDEO ACTION ANTICIPATION
text: Early works in video anticipation model the problem with recurrent neural networks (Gao et al., 2017; Farha et al., 2018; Miech et al., 2019). Some prior works also leverage the future frames for learning the representations (Vondrick et al., 2016; Fernando & Herath, 2021). A self-regulated learning framework for action anticipation in the egocentric video is presented in (Qi et al., 2021). RU-LSTM (Furnari & Farinella, 2019) deploys two LSTMs and behaves like an encoder-decoder, where the first progressively summarizes the observed together with the second that unrolls over future predictions without observing. (Osman et al., 2021; Tai et al., 2021) both integrate unrolling mechanism as RU-LSTM does, but replace the rolling part in RU-LSTM with SlowFast (Feichtenhofer et al., 2019) and higher-order recurrent networks, respectively. (Sener et al., 2020) aggregates the predictions by pooling over different granularity of temporal segments. (Girdhar & Grauman, 2021) combine causal self-attention with the look ahead prediction on successive frames.

heading: 2.2 MESSAGE PASSING NEURAL NETWORK
text: The concept of message passing is first published in (Gilmer et al., 2017), where it was originally designed for molecular property prediction. It assumes an undirected graph structure with dataindependent, equal edge contributions. To address this limitation, (Ma et al., 2020) deploys two encoders separately for vertex and edge estimation and aggregates them by a self-attention readout. Some prior works use attention or dedicated network design to learn the directed edge representations to improve the model capability (Jørgensen et al., 2018; Withnall et al., 2020; Gong & Cheng, 2019). Recently, (Arnab et al., 2021b) express the non-local attention (Wang et al., 2018) and GAT (Veličković et al., 2017) as message passing functions and apply them to video understanding task. Differently, we view the message passing framework as the generalized recurrent models and specialize it for edge representations learning in this work.

heading: 2.3 SELF-ATTENTION
text: (Vaswani et al., 2017) first propose a recurrence free sequence learning architecture by stacking several self-attention layers, which can achieve remarkable performance in the NLP domain. (Dehghani et al., 2018) demonstrates that the self-attention can be treated as the recurrent unit which unfolds to input sequences to processes with shared weights. On the other hand, (Dosovitskiy et al., 2020) propose Vision Transformer (ViT), an architecture with only self-attention for image classification. (Zhou et al., 2021) improves ViT by re-attending the multi-heads information in the post-softmax step to enable the deeper configuration. Some recent studies explore ViT based models on video action recognition (Arnab et al., 2021a; Bertasius et al., 2021), and also video anticipation (Girdhar & Grauman, 2021). Unlike these prior works, our proposed model processes the video in a flexible graph representation and fits into the message passing framework. It is lightweight and only contains few self-attention layers which sequentially process each timestep.

heading: 3 METHODOLOGY

heading: 3.1 BACKGROUNDS

heading: 3.1.1 MESSAGE PASSING
text: Given an undirected graph G, the Message Passing algorithm involves two-phase forwarding processes, message passing phase with message function M and update function U , and readout phase with readout function R. The message passing phase can run arbitrary T step with the following definition:
mt+1v = ∑
w∈N(v)
Mt(h t v, h t w, avw) (1)
ht+1v = Ut(h t v,m t+1 v ) (2)
where v is the vertex in G, and N(v) defines the neighbors of v. avw is the connection strength bonding between vertex v and w. The readout phase then extract the features of the whole graph, abstracted from the message passing phase, at time T ,
ŷ = R(hTv |v ∈ G). (3) Our proposed method inherits these three core functions (e.g., message, update, and readout). However, the most notable difference is that we treat the whole anticipation predicting as a chain of messages conveyed in space-time, with the constant strengthening of the signal by the RGB inputs in each time-step. The readout function is called when the prediction is required at any time t.

heading: 3.1.2 SELF-ATTENTION
text: Self-Attention (SA) forms theQ,K, V tokens (for query, key, and value) from the same source input x1. The output of the attention is the linear combination of the value, based on the attention weight computed by the scaled dot-product between the query and key followed by a softmax.
Q,K, V = xW iQ, xW i K , xW i V (4)
SAi(x) = softmax( QTK√ D )V, (5)
where scaled factor D default to the input feature dimension, W iQ,W i K ,W i V are the trainable embeddings. We use superscript i to note the embeddings are associated to each self-attention layer.
Multi-Head Self Attention (MHSA) performs n-way self-attention in parallel, where n is the total number of heads. An additional aggregation function, with parameters Wagg , is adopted to fuse the information computed from each head.
MHSA(x) = [H1, . . . ,Hn]Wagg, (6) where Hi = SAi(x)
1Some literature uses self-attention no matter the inputs Q,K, V comes from the same source. We use the term, self-attention, strictly when Q,K, V is formed by the same source input.
where the [., .] presents the concatenation.
Following the transformer-style architecture, a Feed-Forward Network (FFN) performs after each attention layer to project the attention output and bring the non-linearity. The FFN computes:
FFN(x) = σ(xW1 + b1)W2 + b2 (7) where σ can be any arbitrary nonlinear function.
Our proposed model utilizes multi-head self-attention and deeply integrates into the message passing design. We use multi-head self-attention as the information routing between vertices of a graph. Note in this case, the resulting graph is bi-directed, because the dot-product between Q and K is not symmetric.

heading: 3.2 UNIFIED RECURRENT MODELING
text: Self-Attention Block: We implement the self-attention building block in prenorm style (Xiong et al., 2020), as implemented in many transformer-style architecture, where the normalization layer is placed before MHSA and FFN and not counts in shortcuts. Figure 2 left shows the self-attention block
fMHSA(x) = x+MHSA(LayerNorm(x)) (8) fFFN (x) = x+ FFN(LayerNorm(x)) (9)
SABlock(x) = fFFN (fMHSA(x)) (10) We can optionally expose the edge information, in matrix A, into SABlock(.;A). In this case, the extension definesMHSA(x;A) where the matrixA is fused into the step after the softmax of scaled dot-product computation. We rewrite the equation 5 and equation 6 to
MHSA(x;A) = [H1, . . . ,Hn]Wagg, (11) where Hi = SAi(x;A)
SAi(x;A) =
( softmax(A) + softmax(
(xW iQ) T (xW iK)√ D )
) (xW iV ) (12)
NoteA is unique and shared in multi-heads self-attention. Although the probability comes with postsoftmax becomes twice larger, such difference may not cause an issue because trainable weights, Wagg,WV , are able to absorb the impact.
Recurrent Cell: Figure 2 shows the overall architecture. Given the frame features xt at time t, with shape (H , W , C), which is transformed by a backbone model, we define the vertices etv by taking a nonlinear transformation of xt and flatten into 1D, resulting in the shape (HW , C). The position encoding Wpe is added on top of etv . We then leverage SABlock(.;A) to message function M with optionally matrix A, and SABlock(.) to update function U for computing the hidden states htv ,
etv = fse(σ(x tWx + b)) +Wpe (13) gtv = fh([e t v, h t−1 v ]) (14)
mtv = SABlock(g t v;A t) (15) htv = tanh(SABlock([e t v,m t v])) (16)
where fse is a Squeeze-and-Excitation (SE) module (Hu et al., 2018) performs on vertices dimension, and fh is a linear transformer to reduce the feature size from concatenated 2C into C. Note we bound the hidden states by the tanh to ensure the value stability in temporal propagation. For the readout function, we leverage SABlock(.) to abstract the information from the hidden state representation htv ,
yt = SABlock(fscale(h t v)). (17)
where fscale is a fully-connected transformation to scale the values of htv . We assume every vertex in the graph G is accessible by any other vertex to maintain the maximal possibility without prior. In this case, N(v) ≡ v in equation 1. The edges, presents by adjacency matrix A = {avw;∀w ∈ N(v), v ∈ G}, can be optionally provided, explicitly, else implicitly derived in self-attention. Implicit Edge Estimation: The scaled dot-product original in the self-attention operator can be treated as implicit estimation of edges, which computes the pairwise similarity by correlation measurement of vertices on-the-fly with the parametric trainable embeddings of inputs.
Explicit Edge Estimation: On the other hand, edge information can also be provided by feeding a adjacency matrix A during the attention computation as discussed in equation 11 and equation 12. All edge learning strategies introduced in the following all belong to explicit edge estimation category.

heading: 3.3 EDGE LEARNING
text: We propose three different learning strategies to construct the edge information (see Figure 3). Edge attention decouples the attention operator in message function into vertex and edge estimation separately. Class token projection performs the outer-product of a trainable vector with supervision signal from class labels, and template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.

heading: 3.3.1 EDGE ATTENTION
text: We can model the adjacency matrix by deploying a dedicated SABlock for edge matrix. In this case, we put two separate self-attention layers in parallel, one for vertex estimation (with subscript v) and another one for edge estimation (with subscript e). Applying to equation 15, we have
Ât = SABlocke(g t v)) (18)
mtv = SABlockv(g t v; Â t)) (19)
where the Ât is the estimated edge information used in the message function.

heading: 3.3.2 CLASS TOKEN PROJECTION
text: With class token accompanied with supervision signals received, we can span an edge matrix by applying outer-product of class token. Consider a class token, xcls, with dimension (1, C), the projection can be done by a linear transformation, fproj , from C dimension toN . The outer-product of the token then spans it into a matrix with shape (N , N ), which serves as the edge matrix.
P = fproj(xcls) (20)
Ât = P ⊗ P, (21) where the ⊗ is the outer-product operator.

heading: 3.3.3 TEMPLATE BANK
text: We define a template bank B with bank size S indicates the number of templates contained. Each template in the bank is the map with the shape (N , N ) where N is the number of vertices. During the forward process, the edge information is formed by weighted sum of S templates by a selecting module, fselect(.), a Multi-Layer Perception (MLP) followed by a sigmoid to produce the selecting ratio conditioned on inputs and apply to the bank:
etv = 1
N N∑ i=0 etv,i (22)
I = sigmoid(fselect(etv)) (23)
Ât = S∑ i=0 I:,:,i ·Bi,:,: (24)
where etv is the average of e t v over the vertex dimension. The edge estimation Â t, in the combination of the template bank B, is then fed into the message function the equation 15.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION DETAILS
text: We based the pretrained BN-Inception model of (Furnari & Farinella, 2019) as backbone and kept it frozen during training. All input frames are resized to 256x454 and fed through a proposed model followed by an unrolling classifier forming the verb, noun, and action prediction. The unrolling classifier is a widely adopted design in action anticipation (Furnari & Farinella, 2019; Osman et al., 2021; Tai et al., 2021), which unfolds the final output over the unobserved interval with a LSTM, till the moment where the action is expected to start, and then computes the predictions. The outputs are supervised by comparing predictions with ground-truth labels with cross-entropy in the last 8 anticipate interval for each individual verb, noun, and action. Note in the class token projection scheme, the class token for action is used, otherwise mean-pool over the output from readout function is performed. For training details, RandAugment (Cubuk et al., 2020) augmentation scheme is applied in all experiments. AdaBelief (Zhuang et al., 2020) in combination with look-ahead optimizer (Zhang et al., 2019) is adopted. Weight decay is set to 0.001. Learning rate is initially set to 1e-4 and cosine annealed to 1e-7 on the last 25% of epochs. We use 4 × NVIDIA V100 32GB GPUs for training, with batch size set to 32 and run in total 50 epochs.

heading: 4.2 EPIC-KITCHEN DATASET
text: EPIC-Kitchen 55 (EK55) (Damen et al., 2018) is a large scale egocentric video dataset, captured by 32 subjects in 32 kitchens. The data split scheme is inherited from (Furnari & Farinella, 2019) for action anticipation tasks, resulting in 23492 action segments for training and 4979 for validation. All unique verb-noun pairs, of 125 verbs and 352 nouns. define 2513 action categories. The top-1 and top-5 accuracy at 1s are the major evaluation metrics used on EK55. We also include the specific performance metrics at 1s, with additional mean top-5 recall reported. We sample 14 frames from each clip with a fixed stride of αs = 0.25 (4 fps), resulting in 3.5s context in each sample. We use τa to indicate anticipation time.

heading: 4.3 EXPERIMENTAL RESULTS
text: Baselines. We consider the baselines methods, all in RGB inputs, including Deep Multimodal Regressor (DMR), (Vondrick et al., 2016), TSN-based models MCE (Furnari et al., 2018) and ATSN (Damen et al., 2018), deep network trained with top-k classifier (SVM-Top3/5) (Berrada et al., 2018), Verb-Noun Marginal Cross Entropy (VNMCE) (Furnari et al., 2018), and several LSTM variants, Encoder-Decoder LSTM (ED) (Gao et al., 2017), Feedback Network LSTM (FN) (Geest & Tuytelaars, 2018), LSTM with Ranking Loss (RL) (Ma et al., 2016), and LSTM with Exponential Anticipation Loss (EL) (Jain et al., 2016). We also compare the state-of-the-art RollingUnrolling LSTM (RU) (Furnari & Farinella, 2020), ImagineRNN by predicting future feature (Wu et al., 2020), Higher-Order Recurrent Space-Time Transformer (HORST) (Tai et al., 2021), SelfRegulation Learning (SRL) (Qi et al., 2021), SlowFast with unrolling classifier (SF-RU) (Osman et al., 2021), and previous winners in EPIC-Kitchens anticipation challenge (in Table 3), ActionBanks in 2020 challenge (Sener et al., 2020) and Anticipative Video Transformer (AVT) in 2021 (Girdhar & Grauman, 2021). Some methods cannot query the accuracy at any time (i.e., DMR, ATSN, MCE, VN-CE, SVM-Top3/5, VNMCE) and some are constraint by the specific sampling rates (i.e., SF-RU, AVT). The αs in SF-RU presents the data sampling rate used in the algorithm.
Results. From Table 1, we can observe that our proposed method with implicit edge estimation already performs strongly compared to previous methods on EK55. Using edge learning strategies discussed in section 3.3, we found template bank and class token projection methods with significant improvements. Table 2 shows the specific performance at 1s, which is the widely used evaluation criteria in the EPIC-Kitchen, we provide the top-5 accuracy, and also top-5 recall, for each individual verb, noun, and action. We can clearly see our proposed unified recurrent modeling outperforms previous methods, not only on the accuracy but also on the recall. Equipped with the explicit edge
estimation, template bank and class token projection even shows significant improvements. Table 3 reveals top-1 accuracy at 1s and compared with additional candidates who won the EPIC-Kitchen official challenge in recent two years (ActionBanks and AVT). Despite the AVT trained with advanced backbone deployed, and with different sampling rate αs = 1 (1 fps) for 10s for each sample, our proposed methods are with superior accuracy in both top-1 and top-5 accuracy over all the methods using the same BN-Inception backbone. It is worth noting although the edge attention is not as strong as template bank and class token projection proposals in top-5 comparison, it gets the best amongst all top-1 accuracy.

heading: 4.4 ABLATION STUDY
text: From the previous section, we found edge learning with template bank achieved overall best according to accuracy and recall among all experiments. In this section we compare and discuss the two major hyper-parameters set in this model, the bank size and the feature dimension.

heading: 4.4.1 BANK SIZE
text: Table 4 shows the experimental results on setting different bank sizes for the template bank edge learning. We manipulate the bank size ranges from 1 to 2048. In the case of bank size being equal to 1, a unique edge matrix is shared globally across all samples and each timestep. It can be explained as placing a strong regularization to the intermediate representations before feeding into the message function, forcing the vertex constrained to the specific representation that is able to co-work with the
global edge matrix. When the bank size is larger than 1, the constraint becomes weaker, bringing additional flexibility to the representations. We can see when the bank size is greater than 64, top1/5 accuracy and top-5 recall are showing better results than the global template used. The peak performance is at bank size 512, and showing no improvement with more templates.

heading: 4.4.2 FEATURE DIMENSIONS
text: Table 5 shows the performance difference while varying the width of the proposed model. By default, the feature dimension is set to 1024, which is aligned with the output dimension of BNInception backbone. We adjust the feature dimension from 0.25x to 4x. This adjustment also affects the feature dimension used in multi-head self-attentions. From the table results, we can observe the best configuration falls in the 1024 feature dimension, which is equivalent to setting each attention head to 128 feature dimensions. This ablation suggests setting the appropriate feature dimension is needed for a specific task.

heading: 4.5 PARAMETERS AND COMPUTATION EFFICIENCY
text: Our model contains only three multi-head self-attention layers with linear transformations for dimension reduction or selection. Compared to other transformer-based models, which often stack multiple self-attention layers, our modeling is lightweight and efficient. The total parameters in our model, excluding the backbone, classifier, and normalization layers, are only about 21C2 (21M when C=1024). The template bank brings about additional 6.5M parameters, including a selector with parameters 14C(S + C), and templates that occupy SN
2. In the EPIC-Kitchen dataset with (256, 454) input dimension based on BN-Inception backbone, N is 112. The total computation is about 40 GFLOPs per timestep.

heading: 5 CONCLUSION
text: In this work we present a unified recurrent modeling which only uses self-attention as building block to represent features through graph, and which leverages edge learning to estimate vertex semantics. On the large-scale egocentric EPIC-Kitchen dataset for the video action anticipation, we surpass the current state-of-the-art in action anticipation. Proposed model is simple, lightweight, and thus embodies the flexibility for further extensions.

Review summaries:
This paper addressed an interesting problem of action anticipation. To this end, the authors proposed a unified recurrence model that generates the graph representation of the video sequence. Extensive experimental results have shown the effectiveness of the proposed method.

The authors propose a framework for video action anticipation, where the task is to observe input frames and predict the action label after an anticipation period (not observed). The proposed architecture builds on top of message passing terminology (message, update, and readout functions) to create a recurrent transformer model that uses self-attention to capture spatial dependencies between frame patches (similar to ViT w/o [cls]) and recurrent module to capture the temporal dependencies across frames. Edge affinity matrix is learned and introduced into the MHSA block (Eqn. 12) as a prior to influence the attention matrix at every layer. Three different explicit edge learning approaches are proposed, in addition to implicit edge learning. The model is evaluated on the Epic Kitchen 55 dataset.

Paper proposes a novel architecture for video action anticipation task. The proposed method used a graph representation via message passing. The key contribution lies in the three explicitly edge learning strategies. These strategies to escape to the trivial estimation which is  purely based on the similarity in input tensors, by bringing the flexibility of edge representations.   1. Edge attention decouples the attention operator in message function into vertex and edge estimation separately. 2. Class token projection performs the outer-product of a trainable vector with supervision signal from class labels. 3. Template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.  Experiment on the EK55 dataset outperforms the state-of-the-arts algorithms.

The paper proposes a method for action anticipation based on a graph representation of the temporal structure of video. Specifically a graph is built using semantic representations of video frames extracted with a 2D CNN backbone as vertices. Temporal reasoning is hence achieved modeling spatiotemporal information flow thorugh message passing across vertices. The approach is based on a message function, an unpdate function and a readout function, all based o a multi-head self-attention block. Experiments compare the proposed approach with respect to other approaches suing RGB inputs on EPIC-KITCHENS-55.