Unified Recurrence Modeling for Video Action Anticipation
Authors: Tsung-Ming Tai
Giuseppe Fiameni
Cheng-Kuang Lee
Simon See
Oswald Lanz
Venue: ICLR 2022 Submitted (2022)

Abstract: Forecasting future events based on evidence of current conditions is an innate skill of human beings, and key for predicting the outcome of any decision making. In artificial vision for example, we would like to predict the next human action before it is actually performed, without observing the future video frames associated to it. Computer vision models for action anticipation are expected to collect the subtle evidence in the preamble of the target actions. In prior studies recurrence modeling often leads to better performance, and the strong temporal inference is assumed to be a key element for reasonable prediction. To this end, we propose a unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from sequence into graph representation via message passing. The information flow in space-time can be described by the interaction between vertices and edges, and the changes of vertices for each incoming frame reflects the underlying dynamics. Our model leverages self-attention for all building blocks in the graph modeling, and we introduce different edge learning strategies can be end-to-end optimized while updating the vertices. Our experimental results demonstrate that our modeling method is light-weight, efficient, and outperforms all previous works on the large-scale EPIC-Kitchen dataset. 

Under review as a conference paper at ICLR 2022
UNIFIED RECURRENCE MODELING FOR VIDEO AC- TION ANTICIPATION
Anonymous authors Paper under double-blind review
ABSTRACT
Forecasting future events based on evidence of current conditions is an innate skill of human beings, and key for predicting the outcome of any decision making. In artificial vision for example, we would like to predict the next human action be- fore it actually happens, without observing the future video frames associated expected collect the
it. Computer vision models for action anticipation subtle evidence in the preamble of the target actions. In prior studies recurrence modeling often leads better performance, and the strong temporal inference is assumed to be a key element for reasonable prediction. To this end, we pro- pose unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from sequence into graph representation via message passing. The information flow in space-time can be described by the interac- tion between vertices and edges, and the changes of vertices for each incoming frame reflects the underlying dynamics. Our model leverages self-attention for all building blocks in the graph modeling, and we introduce different edge learning strategies that can be end-to-end optimized while updating the vertices. Our ex- perimental results demonstrate that modeling method is light-weight, efficient, and outperforms previous works using the large-scale PPtC-Kitchen dataset.
INTRODUCTION
Video action recogmition is a stag-standing problem in computer vision. It predicts the action from experienced frames by understanding the contexts in the observations. However, having the pre- diction after the whole observation of action is insufficient to evaluate the outcome of any decision making. Predicting future human activities and interactions of objects are receiving research inter- ests in these years, in assisting navigation systems (OhnBar et al. 2018),, robotics (Park et al. 2016). entertainment (Liang et al. 2015 Taylor et al. 2020) and autonomous vehicles (Hirakawa et al. 2018).
Different from action recognition, the video action anticipation is required to give the predictions strictly before the action frames being observed. Models, in this case, are expected to collect the subtle evidence in the preamble of target actions (Furnari et al.] 2018 Furnari & Farinella 2020 Rodin et al. 2021). In the high-level viewpoint, the objective of anticipating is not only to understand the actual content in the observations but seeking to associate the minor details the next possible action. Figure illustrates the definition of the video anticipation problem.
In the absence of target action frames, pretrained models for the robust feature extraction from RGB inputs are often deployed on video anticipation, either pretrained from image classification or video action recognition. On top of the pretrained models. recurrent neural networks are the widely adopted options to model the temporal relationship in anticipation problems and lead to better performance than clip based modeling methods. Unlike the action recognition settings, where the relevant frames associated with the target action are not necessary in the end of observation and may locate in any action duration. video anticipation predicts the action that always performs in the future of observations, making the accumulated information gains aligned with the temporal orders. To this end, we propose a unified recurrence modeling for video action anticipation by generalizing the recurrence mechanism from sequence into graph representation via message passing. We utilize self-attention as the universal building block in the graph modeling. In the graph point of view, self- attention can be treated as the information routing between vertices. The attention weight derived by

---

Under review as a conference paper at ICLR 2022
Preamble
Action
Time
Observation
Figure 1: Illustration of video action anticipation problem. The context of action may be different than the preamble of action. Models can only observe some frames before action actually starts, which strictly ensured by an inaccessible "Skip" period, and based on the evidence collected to predict the following action.
the scaled dot-product, which computes the correlation of vertices, can be explained an adjacency estimation implicitly. However, in this way, the representation of edges is limited and purely based on the similarity in input tensors. Therefore, we propose three explicitly edge learning strategies trying to escape to the trivial estimation by bringing the flexibility of edge representations. They are end-to-end optimized and can be jointly trained with the main task, In the experimental results, we show our proposed unified recurrent modeling outperforms several state-of-the-at methods on large- scale egocentric video dataset Kitchen on video anticipation. Combining with edge learning strategies, the performance can even boost to the next level.
RELATED WORKS
2.1 VIDEO ACTION ANTICIPATION
Early works in video anticipation model the problem with recurrent neural networks (Gao et al. 2017 Farha et al. 2018: Miech et al., 2019). Some prior works also leverage the future frames for learning the representations (Vondrick al. 2016 Fernando & Herath 2021). A self-regulated learning framework for action anticipation in the egocentric video is presented in (Qi et al. 2021). RU-LSTM (Furnari & Farinella, 2019) deploys LSTMs and behaves like an encoder-decoder. where the first progressively summarizes the observed together with the second that unrolls over future predictions without observing. (Osman al. 2021 Tai al.) (202T) both integrate unrolling mechanism as RU-LSTM does, but replace the rolling part in RU-LSTM with SlowFast (Feichten hofer et al., 2019) and higher-order recurrent networks, respectively. (Sener et al. 2020)) aggregates the predictions by pooling over different granularity of temporal segments. (Girdhar & Grauman, 2021) combine causal self-attention with the look ahead prediction on successive frames.
2.2 MESSAGE PASSING NEURAL NETWORK
The concept of message passing first published in (Gilmer et 2017 where it was originally designed for molecular property prediction. It assumes an undirected graph structure with data- independent, equal edge contributions. To address this limitation, (Ma al., 2020) deploys two encoders separately for vertex and edge estimation and aggregates them by self-attention readout. Some prior works use attention or dedicated network design to learn the directed edge representa- tions to improve the model capability (Jørgensen et al.) 2018 Withnall et al. 2020 Gong & Cheng 2019). Recently, (Arnab et al., (0021) express the non- nocalocal attention (Wang et al. 2018) and GAT VVelickové et al. 2017) ) message passing functions and apply them to video understanding task. Differently, we view the message passing framework as the generalized recurrent models and spe- cialize it for edge representations learning in this work.
2

---

Under review as a conference paper at ICLR 2022
2.3 SELF-ATTENTION
Vaswani et al. 2017) first propose recurrence free sequence learning architecture by stacking sev- eral self-attention layers, which achieve remarkable performance in the NLP domain. (Dehghani et al. 2018) demonstrates that the self-attention can be treated as the recurrent unit which unfolds to ipput sequences processes with shared weights. On the other hand, (Dosovitskiy et 2020) propose Vision Transformer (ViT), an architecture with only self-attention for image classification. (Zhou al., 202T) improves ViT by re-attending the multi-heads information in the post-softmax step to enable the deeper configuration. Some recent studies explore ViT based models on video action recognition (Arnab et al., 2021a; Bertasius et al. 2021), and also video anticipation (Girdhar & Grauman 2021). Unlike these prior works, our proposed model processes the video in flexible graph representation and fits into the message passing framework. is lightweight and only contains few self-attention layers which sequentially process each timestep.
3
METHODOLOGY
3.1 BACKGROUNDS
3.1.1 MESSAGE PASSING
Given an undirected graph G, the Message Passing algorithm involves two-phase forwarding pro- cesses, message passing phase with message function M and update function U, and readout phase with readout function R. The message passing phase can run arbitrary T step with the following definition:
m
++
Σ wEN(v)
Mc(hin Wi aew)
h+
Uh,,, 1)
where e is the vertex in G. and ((v) defines the neighbors of v. Oew is the connection strength bonding between vertex and The readout phase then extract the features of the whole graph, abstracted from the message passing phase, at time T,
R(bTlue G).
(3)
Our proposed method inherits these three core functions (e.g., message, update, and readout). How- the most notable difference is that treat the whole anticipation predicting messages conveyed in space-time, with the constant strengthening of the signal by the RGB inputs in each time-step. The readout function is called when the prediction is required any time t.
chain of
3.1.2 STLE-ATTENTOON
Self- Seff-Attention (SA) forms the Q, K, V tokens (for query, key, and value) from the same source input The output of the attention is the linear combination of the value, based on the attention weight computed by the scaled dot-produc, between the query and key followed by softmax.
Q,K.V V
zW xW K zWi
(4)
QT
K
SA.(z) softmax(
V
(5)
VD
where scaled factor D default the input feature dimension, W Wi Wi the trainable beddings. We use superscript to note the embeddings are associated to each self-attention layer.
Multi-Head Self Attention (MHSA) performs n-way self-attention in parallel, where is the total number of heads. An additional aggregation function, with parameters W Wagg is adopted to fuse the information computed from each head.
MHSA(a)) = [H [ Hn) agg where H; = SA,(z)
'Some literature uses self-attention matter the inputs Q. K, V comes from the same source. We use the term, self-attention, strictly when Q, K, V formed by the same source input.
3

---

Under review as a conference paper at ICLR 2022
Self- Aeff-Attention Block
Unrolling Classifier
Readout Function
FeedForward
LayerNorm
Cell
Next vertices A
Multi-lledd E-Attention
Updata Functionn
Edges (Optional)
LayerNorm
Massage
A
Encoded vertices
Backbone
Figure 2: Overview of proposed unified recurrent model. The message function, update function, and readout function leverage multi-head self-attention. Our proposed message function is designed be flexible work in conjugation with explicit edges information provided.
where the J...] presents the concatenation.
Following the transformer-style architecture, Feed-Forwart Network (FFN) performs after each attention layer to project the attention output and bring the non-lincarity. The FFN computes:
FFN(x) = o< + W2 + b2
where a can any arbitrary nonlinear function.
Our proposed model utilizes multi-head self-attention and deeply integrates into the message passing of graph. Note in this case, the resulting graph is bi-directed, because the dot- dot-product between Q and K is not symmetric.
design. We use multi-head self-attention as the information routing between vertices
3.2 UNIFIED RECURRENT MODELING
We implement the self-attention building block in prenorm style (Xiong (ct al. 2020), as implemented in many transformer-style architecture, where the normalization layer placed before MHSA and FFN and not counts in shorteuts. Figure left shows the self-attention block
Self-Attention Block:
fMuSA(x) x (ayeroon))))))(( fFrN(E) x- (=N[(ayerNNr(((zz
(8)
(9)
SABlock(s)
(FPx((MusA(tt)
(10)
We can optionally expose the edge information, in matrix A, into SABlock(,; A). In this case, the extension defines M HHSA(a; A) where the matrix is fused into the step after the softmax of scaled dot- dot-product computation. We rewrite the equation 5 and equation
MHSA(<; A) where H: = (A,(x;A)
[....., H,, Wagg.
(11)
(x (x ( (xW+)
SA,(;;)
sootmaa((A) + softmax
(12)
VD
Note A is unique and shared in multi-heads self-attention. Although the probability comes with post- softmax becomes twice larger, such difference may not cause an issue because trainable weights, W Wagg, Wy, are able to absorb the impact.
4

---

Under review as a conference paper at ICLR 2022
Recurrent Cell: Figure shows the overall architecture. Given the frame features x' time t, with shape (H, W. C), which is transformed by a backbone model, we define the vertices e by taking nonlinear transformation of z' and flatten into ID, resulting in the shape (HW, C). The position encoding pe added on top of ey. We then leverage S A Alooc((: A) to message function M with optionally matrix A, and S S.Aokkkk())
update function U for computing the hidden states ny
et fae (a 6)) Wpe
(13)
g
(14)
m. = (s::
(15)
hy = Canh(SAAlocck(ed... m())
(16)
where Sae is suueere-and-Excitatie (SE) module (Hu et al. 2018) performs on vertices dimen-
sion, and fn. a linear transformer to reduce the feature size from concatenated 2C into C. Note we bound the hidden states by the tanh to ensure the value stability in temporal propagation. For
the readout function, we leverage SABlock(.) to abstract the information from the hidden state representation hr
o¹ = SABlock(Jscatee
(17)
where fecate is connected transformation to scale the values of h. We assume every vertex in the graph G accessible by any other vertex to maintain the maximal possibility without prior. In this case, (v) v in equation [ The edges, presents by adjacency matrix A = {a.mi Ve E (e)), E G}, can be optionally provided, explicitly, else implicitly derived in self-attention.
Implicit Edge Estimation:
The scaled dot-product original in the self-attention operator can be treated as implicit estimation of edges, which computes the pairwise similarity by correlation mea- surement of vertices on-the-fly with the parametric trainable embeddings of inputs.
Explicit Edge Estimation: On the other hand, edge information can also be provided by feeding a adjacency matrix A during the attention computation as discussed in equation aadd equation [2] All edge learning strategies introduced in the following all belong to explicit edge estimation cate- gory.
3.3 EDGE LEARNING
We propose three different learning strategies to construct the edge information (see Figure 3 Edge atlention decouples the attention operator in message function into vertex and edge estimation sep- arately. Class token projection performs the outer-product of a trainable vector with supervision signal from class labels, and template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.
3.3. EDGE ATTENTION
We can model the adjacency matrix by deploying a dedicated SABlock for edge matrix. In this case, we put two separate self-attention layers in parallel, one for vertex estimation (with subscript v) and another one for edge estimation (with subscript e). Applying to equation 13 we have
Á'
(ABloock(gl))
(18)
m
SABlock, (g': A²)
(19)
where the A' is the estimated edge information used in the message function.
3.3.2 CLASS TOKEN PROJECTION
With class token accompanied with supervision signals received, we can span an edge matrix by applying outer-product of class token. Consider class token, Vets with dimension (1, C),, the projection can be done by a linear transformation, fproj. from C' dimension to N. The outdr-product of the token then spans it into a matrix with shape (N, N), which serves the edge matrix.
P
Sproj (Fess)
(20)
P&P
(21)
where the 6 is the outer-product operator.
5

---

Under review as a conference paper at ICLR 2022
FoodForward
fesdforward
FoodForword
Bank
LaverNorm
Verter Multi-Hodd
Seli-Attuntion
Sell Ael--iitntioon
Malti-eed
Multi Self Attention
Outer Product
Muti-Head Sel drit-ittettonn
Selector
Layerkiom
Laye Norm
Layerfoom
citc 104M1
Figure 3: Propose edge learning extensions to the multi-head self-attention layer. The three block diagrams present the (left) edge attention, (middle) class token projection, and (right) template bank, respectively.
3.3.3 TEMPLATE BANK
We define a template bank B with bank size S indicates the number of templates contained. Each template in the bank is the map with the shape (N, N) where N is the number of vertices. During the forward process, the edge information is formed by weighted sum S templates by selecting module, feelect(.) (-). Multi-Layer Perception (MLP) followed by a sigmoid to produce the selecting ratio conditioned on inputs and apply the bank:
T
Σ
(22)
N
I sigmoid( foetect (ct))
(23)
A'
Σ
Ia B
(24)
where et, is the average of e over the vertex dimension. The edge estimation A', in the combination of the template bank B, is then fed into the message function the equation 15
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We based the pretrained BN-Inception model of (Furnari & Farinella] 2019) backbone and kept it frozen during training. All input frames are resized to 256x454 and fed through a proposed model followed by an unrolling classifier forming the verb, noun, and action prediction. The unrolling classifier is a widely adopted design in action anticipation (Furnari & Farinella, 2019 Osman et al. 2021 Tai the moment where the action is expected to start, and then computes the predictions. The outputs are supervised by comparing predictions with ground-truth labels with cross-entropy the last antic- ipate interval for each individual verb, noun, and action. Note in the class token projection scheme, the class token for action is used, otherwise mean-poo! over the output from readout function is per- formed. For training details, RandAugment (Cubuk et al. 2020) augmentation scheme is applied in all experiments. AdaBelief (Zhuang et al., 2020) in combination with book-ahead optimizer (Zhang let al. 2019) is adopted. Weight decay set to 0.001. Learning rate is initially set to le-4 and cosine annealed to 1c-7 on the last 25% of epochs. We use 4 x NVIDIA V100 32GB GPUs for training, with batch size set to 32 and run in total 50 epochs.
2021). which unfolds the final output over the unobserved interval with LSTM, till

---

Under review as a conference paper at ICLR 2022
Table 1: EK55 Action Anticipation validation results using RGB different To in Top-5 action
accuracy.
TOP-5 ACCURACY (8) AT DIFFERENT
METHODS
1.25
DMR
16.86
ATSN
16,29
26.11
VN-CE CB
17.31
SVM-TOPS
24.46
VNMCE+13
25.95
VNMCE+TS
26.01
ED
21.53
22.22
23.20
24.78
25.75
26.69
27.66
29.74
RL
25.95
26.49
27.15
28.48
29.61
30.81
31.86
32.84
24.68
28.56
30.27
31.50
RU-RGB
25.44
26.89
28.32
29.42
30.83
32,00
33.31
34.47
HORST
25.38
26.37
27.82
29.16
30.69
31.54
32.52
33.45
UORST--RL
25.95
27.03
28.24
29.81
31.58
32.68
34.21
35.56
SRL
25.82
27.21
28.52
29.81
31.68
33.11
34.75
36.89
SF-RU (0.mg))
24.53
25.63
27.30
28.97
30.96
32.23
33.49
35.02
SE RU (o.=
26.39
28.40
30.94
32.87
SF-RU (a. (a.=2) )
26.78
29.25
32.05
34.34
OURS (IMPLICIT)
27.25
27.76
29.36
30.63
31,68
32.76
34.41
OURS (EDGE ATTENTION)
27.25
27.98
29.24
30.29
31.52
32.92
34.77
36.95
OURS (TEMPLATE BANK)
26.67
27.76
29.32
30.49
32.02
33.47
34.71
36.85
OURS (CLS PROJECTION)
26.87
27.90
29.44
30.63
31,96
33.19
34.92
37.05
4.2 EPIC-KITCHEN DATASET
EPIC-Kitchen 55 (EK55) (Damen al. 2018) is large scale egocentric video dataset, captured by 32 subjects in 32 kitchens. The data split scheme inherited from (Furnari & Farinella (019) for action anticipation tasks, resulting in 23492 action segments for training and 4979 for validation. All unique verb-noun pairs, of 125 verbs and 352 nouns. define 2513 action categories. The top-1 and top-5 accuracy at are the major evaluation metrics used on EK55. We also include the specific performance metrics at 1s, with additional mean top-5 recall reported. We sample 14 frames from each clip with fixed stride of a, 0.25 (4 fps). resulting in 3.5s context in each sample. We use Ta to indicate anticipation time.
4.3 EXPERIMENTAL RESULTS
Baselines. We consider the baselines methods, all in RGB inputs, including Deep Multimodal Re- gressor (DMR), (Vondrick et al., 2016). TSN-based models MCE (Furnari et al.) 2018) and ATSN DDamen et al. 2018), deep network trained with top-k classifier (SVM-Top3/5) (Berrada et al. 2018). Verb-Noun Marginal Cross Entropy (VNMCE) (Furnari et al., 2018), and several LSTM variants, Encoder- Decoder LSTM (ED) (Gao al. 2017). Feedback Network LSTM (FN) (Geest & Tuytellars 2018). LSTM with Ranking Loss (RL) (Ma et al.) 2016). and LSTM with Expo- nential Anticipation Loss (EL.) (Jain al., 2016) We also compare the state-of-the-art Rolling-
Unrolling LSTM (RU) (Furnari Farinella, 2020). ImagineKNN by predicting future feature (Wul
al. 2020), Higher-Order Recurrent -pace-Time Transformer (HORST) (Tai al.) (021). Self- Regulation Learning (SRL) (Qi et al.) 2021). SlowFast with unrolling classifier (SF-RU) (Osman) al. 202T), and previous winners EPIC-Kitchen anticipation challenge (in Table 3). Action- Banks in 2020 challenge (Sener et al,, 2020 and Anticipative Video Transformer (AVT) in 2021 (Girdhar & GGauman 2021). Some methods cannot query the accuracy at any time (i.e., DMR, ATSN, MCE, VN-CE, SVM-Top3/5, VNMCE) and some are constraint by the specific sampling rates (i.e., SF-RU, AVT). The as SF-RU presents the data sampling rate used in the algorithm.
Results. From Table 1 we can observe that our proposed method with implicit edge estimation already performs strongly compared to previous methods on EK55, Using edge learning strategies discussed in section 3.3 we found template bank and class token projection methods with significant improvements. Table shows the specific performance at Is, which the widely used evaluation provide the top-5 accuracy, and also top-5 recall, for each individ-
criteria in the EPIC-Kitchen, ual verb, noun, and action. We can clearly see our proposed unified recurrent modeling outperforms previous methods, not only the accuracy but also the recall. Equipped with the explicit edge

---

Under review as a conference paper at ICLR 2022
Table 2: EK55 Action Anticipation validation results using RGB at 1s in Top-5 accuracy and recall for noun, verb, and action.
METHODS
ToP-5
ACC. (%)
MEAN
ToP-5 5 REC. (S) @ 1s
VERB
NoUN
ACTION
VERB
NOUN
ACTION
DMR
73.66
29.99
16.86
24.50
20.89
03.23
ATSN
77.30
39.93
16.29
33.08
32.77
MCE
73.35
38.86
26.11
34,62
32.59
06.50
VN VN-CE CE
77.67
39.50
17.31
34.05
34.50
07.73
SVM-TOP3
72.70
28.41
25.42
41.90
34.69
05.32
SVM-TOPS
69.17
36.66
24.46
40.27
32.69
05.23
VNMCE+13
25.95
VNMCE+TS
74.07
39.10
26.01
41.62
35.49
05.78
ED
75.46
42.96
25.75
41.77
42.59
10.97
FN
74.84
40.87
26.27
35.30
37.77
06.64
RL
76.79
44.53
29.61
40, 80
40.87
10.64
EL
75.66
43.72
28.56
38.70
40.32
08.62
RU-RGB
30.83
HORST-URL
78.80
46.54
31,58
42.62
45.68
12.18
SRL
78.90
47.65
31.68
42.83
47.64
13.24
SP-RU (O.
30,96
SP-RU (0.
30.94
SE-RU (o.
32.05
(IMPLICIT)
78.66
31.68
47.93
OURS (EDGE ATTENTION)
78.54
47.91
31.52
44.57
46.87
12.84
(TEMPLATE Bank)
32.02
OURS (CLS PROJECTION)
78.74
47.59
31,96
44.96
47.19
13.61
Table 3: EK55 Action Anticipation validation results using RGB with top-1 and top-5 action accu- racy at Ta = Is.
METHON
BACKRONE
PRETRAIN
Top-1 (%)
ToP-5 (%)
RGB
BNINC
INIK
13.1
30.8
ACTIONBANKS
BNINC
INIK
28.5
IMAGINERNN
BNINC
IN1K
13.7
31.6
AVT-H
AVT-8
IN21+IK
12.5
30.
AVT-II
IRCSN152
IG65M
14.4
31.7
HORST
BNINC
INIK
12.6
30.7
HORRST-URL
BNINC
INIK
12.8
31.6
(IMPLICIT)
INIK
31.7
OURS (EDGE ATTENTION)
BNINC
IN
13.9
31.5
OURS (TEMPLATE BANK)
BNINC
INIK
13.8
32.0
OURS (CLS PROJECTION)
BNINC
INIK
13.6
32.0
estimation, template bank and class token projection even shows significant improvements. Table reveals top-1 accuracy at and compared with additional candidates who won the EPIC-Kitchen official challenge in recent two years (ActionBanks and AVT). Despite the AVT trained with ad- vanced backbone deployed, and with different sampling rate a. = 1 (1 fps) for 10s for each sample, our proposed methods are with superior accuracy in both top-1 and top-5 accuracy over all the meth- ods using the same BN-Inception backbone. worth noting although the edge attention is not as strong template bank and class token projection proposals in top-5 comparison, it gets the best amongst all top-1 accuracy.
4.4 ABLATION STUDY
From the previous section, we found edge learning with template bank achieved overall best accord- ing to accuracy and recall among all experiments. In this section we compare and discuss the two major syper-parameter set this model, the bank size and the feature dimension.
4.4.1
BANK SIZE
Table shows the experimental results on setting different bank sizes for the template bank edge learning. We manipulate the bank size ranges from to 2048. In the case bank size being equal to unique edge matrix is shared globally across all samples and each timestep. It can be explained as placing a strong regularization to the intermediate representations before feeding into the message function, forcing the vertex constrained to the specific representation that able co-work with the
8

---

Under review as a conference paper at ICLR 2022
Table 4: Ablation different bank sizes is EK55. All the numbers are in % and 1s anticipate interval.
ToP-1 Acc.
TOP-5 Acc.
MEAN ToP-5 REC.
BANK SIZE
VERB
13.22
78.34
47.39
31.38
44.07
45.86
12.99
32
12.99
78.96
47.49
31.68
43.60
46.43
12.62
128
13.37
79.04
48.17
31.84
44.70
47.15
13.07
78.80
31.98
512
13.84
78.60
46.86
32.02
43.63
46.86
13,58
1024
13.31
78.38
48.23
32.08
42.76
47.75
13.34
2048
12.89
78.98
47.18
31.23
42.35
45.76
12.63
Table 5: Ablation various feature dimensions set in model on EK55. All the numbers in % and at Is anticipate interval.
Top-1 Acc.
ToP-5 Acc.
MEAN ToP-5 REC.
PENT. DIM
VERB
NoUN
HEAD DIM=32 (0.25x)
11.14
78.10
45.25
28.80
38.76
43.79
09.28
HEAD DIM=64 (0.50x)
12.57
78.74
47.28
30.85
41.33
47.28
11.83
DIM 128 (1.00%)
13.84
78.60
46.86
32.02
43.63
46.86
13.58
HEAD DIM=256 (2.00x)
12.49
78.78
47.47
30.95
44.08
47.18
12.60
(4.00x)
78.34
global edge matrix. When the bank size is larger than the constraint becomes weaker, bringing additional flexibility to the representations. We can see when the bank size is greater than 64, top- 1/5 accuracy and top-5 recall are showing better results than the global template used. The peak performance bank size 512, and showing no improvement with more templates.
4.4.2 FEATURE DIMENSIONS
Table 5 shows the performance difference while varying the width of the proposed model. By default, the feature dimension is set to 1024, which is aligned with the output dimension BN- Inception backbone. We adjust the feature dimension from 0.25x to 4x. This adjustment also affects the feature dimension used in multi-head self-attentions. From the table results, we can observe the best configuration falls in the 1024 feature dimension, which is equivalent to setting each attention head 128 feature dimensions. This ablation suggests setting the appropriate feature dimension needed for specific task.
4.5 PARAMETERS AND COMPUTATION EFFICIENCY
Our model contains only three multi-head self-attention layers with linear transformations for di- mension reduction or selection. Compared to other transformer- models, which often stack multiple self-attention layers, our modeling is lightweight and efficient. The total parameters in our model, excluding the backbone, classifter, and normalization layers, are only about 21 (21M when C=1024). The template bank brings about additional 6.5M parameters, including selector with parameters C + C), and templates that occupy SN2, In the EPIC-Kitchen dataset with (256, 454) input dimension based on BN-Inception backbone, N 112. The total computation about 40 GFLOPs per timestep.
CONCLUSION
In this work present unified recurrent modeling which only uses self-attention as building block to represent features through graph, and which leverages edge learning to estimate vertex semantics. On the large-scale egocentric EPIC-Kitchen dataset for the video action anticipation, surpass the current state-of-the-an in action anticipation. Proposed model simple, lightweight, and thus embodies the flexibility for further extensions.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. ViVIT: video vision transformer. CoRR, abs/2103.15691 2021a.
Anurag Arnab, Chen Sun, and Cordelia Schmid. Unified graph structured models for video under- standing. arXiv preprint 1:3.::2103.15662, 2021b.
Leonard Berrada, Andrew Zisserman, and M Pawan Kumar. Smooth loss functions for deep top-k classification. In ICLR, 2018.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? CoRR, abs/2102.05095, 2021.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, pp. 702-703, 2020.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. ECCV, pp. 720-736, 2018.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint 1rxivv1807.038811 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2020.
Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what? anticipating temporal occurrences of activities. CVPR, pp. 5343-5352, 2018. doi: 10.1109/CVPR.2015 00560.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recogmition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211, 2019.
Basura Fernando and Samitha Herath. Anticipating human actions by correlating past with the future with jaccard similarity measures. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13224-13233. 2021.
Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with colling-unrollingg Istms and modality attention. In ICCV, pp. 6251 6260, 2019. doi: 10. 10.1109/CCC..221990006
Antonino Furnari and Giovanni Maria Farinella. Rolling-unrolling Istms for action anticipation from first-person video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Antonino Furnari, Sebastiano Battiato, and Giovanni Maria Farinella. Leveraging uncertainty to rethink loss functions and evaluation measures for egocentric action anticipation. In ECCV Work- shops, 2018
Jiyang Gao, Zhenheng Yang, and Ram Nevatia. RED: reinforced encoder-decoder networks for action anticipation. In BMVC, 2017.
Roeland De Geest and Tinne Tuytelaars. Modeling temporal structure with LSTM for online action detection. In WACV, pp. 1549-1557, 2018. doi: 10. 10.10009A
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMLR, 2017.
Rohit Girdhar and Kristen Grauman. 10Xiv:2/06.02088 2021.
Anticipative video transformer.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2022
Liyu Gong and Qiang Cheng. Exploiting edge features for graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9211-9219, 2019.
Tsubasa Hirakawa, Takayoshi Yamashita, Toru Tamaki, and Hironobu Fujiyoshi. Survey on vision- based path prediction. In International Conference Distributed, Ambient, and Pervasive Inter- actions, pp. 48-64. Springer, 2018.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Ashesh Jain, Avi Singh, Hema Swetha Koppula, Shane Soh, and Ashutosh Saxena. Recurrent neural networks for driver activity anticipation via sensory- fusion architecture. In ICRA, pp- 3118-3125, 2016. doi: 10. 10.11100 00.1109/RRA22211..7447474
Peter Bjørn Jørgensen, Karsten Wedel Jacobsen, and Mikkel Nørgaard Schmidt. Neural message passing with edge updates for predicting properties of molecules and materials. In 32nd Confer- ence on Neural Information Processing Systems, 2018.
Hui Liang, Junsong Yuan, Daniel Thalmann, and Nadia Magnenat Thalmann. Ar in hand: Egocen- tric palm pose tracking and gesture recognition for augmented reality applications. In Proceedings of the 23rd ACM international conference on Multimedia, pp.
2015.
Hehuan Ma, Yatao Bian, Yu Rong, Wenbing Huang, Tingyang Xu, Weiyang Xie, Geyan Ye, and Junzhou Huang. Dual message passing neural network for molecular property prediction. 2020.
Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in Istms for activity detection and early detection. In CVPR, pp. 1942-1950, 2016. doi: (0.1109/CVPR.20166
Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran. Leveraging the present to anticipate the future in videos. In CVPR Workshops, pp. 1915-2922 2019. doi: 10. 10.11099CVP.
Eshed OhnBar, Kris Kitani, and Chicko Asakawa. Personalized dynamics models for adaptive assistive navigation systems. In Conference on Robot Learning, pp. 19.39. PMLR, 2018.
Nada Osman, Guglielmo Camporese, Pasquale Coscia, and Lamberto Ballan. Slowfast rolling- unrolling Istms for action anticipation in egocentric videos. arXiv preprint urXiv:2109.00829 2021.
Hyun Soo Park, Jyh-Jing Hwang, Yedong Niu, and Jianbo Shi. Egocentric future localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4697- 4705, 2016.
Z. Qi, S. Wang. C. Su, L. Su, Q. Huang, and Q. Tian. Self-regulated learning for egocentric video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, doi: 10. 1..1109/TAAMMI.22133000000
Ivan Rodin, Antonino Furnari. Dimitrios Mavroeidis, and Giovanni Maria Farinella. Predicting the future from first person (egocentric) vision: survey. Computer Vision and Image Understanding, 211: 2021.
Fadime Sener, Dipika Singhania, and Angela Yao. Temporal aggregate representations for long- range video understanding. In European Conference on Computer Vision, pp. 154-171. Springer, 2020.
Tsung-Ming Tai, Giuseppe Fiameni, Cheng-Kuang Lee, and Oswald Lanz. Higher order recurrent space-time transformer. arXiv preprint arXiv:2104.0866 2021.
Catherine Taylor, Robin McNicholas, and Darren Cosker. Towards egocentric framework for rigid and articulated object tracking in virtual reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 354-359. IEEE, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017.
11

---

Under review as a conference paper at ICLR 2022
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:/710.10903, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, pp. 98-106, 2016. doi: 10.1 N0.1109/CVR.2201.1188
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pp. 7794-7803, 2018. doi: 10.1 10.1 8.00813..
Michael Withnall, Edvard Lindelöf, Ola Engkvist, and Hongming Chen. Building attention and edge message passing neural networks for bioactivity and physical-chemicee property prediction. Journal of cheminformatics, 12(1):1-18, 2020.
Yu Wu, Linchao Zhu, Xiaohan Wang, Yi Yang, and Fei Wu. Learning anticipate egocentric actions by imagination. IEEE Transactions on Image Processing, 30: 143-1152, 2020.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng. Shuxin Zheng, Chen Xing. Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, step back. In NeurlPS, volume 32, 2019.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer, 2021.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C. Tatikonda, Nicha C. Dvoreek, Xenophon Papademetris, and James S. Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In NeurIPS, 2020.
12


Reviews:
[{"review_id": "cDyBSBdlWjq", "reviewer": "Reviewer_yqiW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\t\t\na) The proposed approach provides a transformer-based method to encode Spatio-temporal features from a graph or RGB inputs. Adding the edge affinity matrix to the attention map allows the model to make use of edge affinity information in a self-attention encoder block.\n\nb) The task of video action anticipation can arguably aid in learning more robust and transferable representations of inputs by forcing the model to predict further away actions.\n\nc) Unlike other methods (Furnari et al.), this method does not require optical flow or object detections as inputs \nThe paper is well written, and claims are supported by empirical results (Tables 1,2, and 3).\n\nWeaknesses:\n\na) The authors present the architecture using GNN terminology (message, update, readout functions). Generally, GNNs are useful in the following scenarios where:\n\n1) The input graphs are not fully connected, so the adjacency (or affinity) matrix introduces an inductive bias on the data forcing the nodes to pass messages in a certain manner, and inhibit message passing for disconnected nodes.\n\n2) The input graphs are fully connected, but affinity matrix (or edge embeddings) influence the message passing function.\n\n3) Permutation invariance is needed.\n\nIn the proposed approach, the authors use a transformer architecture to process the image patches. The vertices (image patches) of the graph are flattened into a 1D transformer sequence. These vertices are updated with every new observation using a recurrent model (similar to LSTM) where the input, update, and output functions (gates) are implemented with self-attention blocks. The input data contains no edge information between image patches. And positional encoding is added to the input, allowing the network to be aware of the positional information of patches. It becomes hard to convince the readers that GNNs terminology and edge learning is necessary unless it is well supported by empirical results. However, the results of explicit edge learning do not outperform implicit edge learning through attention, because the attention operation is capable of capturing the spatial dependencies between image patches (as implied by the authors). Introducing the edge affinity matrix into the attention block will only be useful if it is part of the input; it can be used as a prior to influence the attention maps. This framework can be more useful for different tasks with edges defined as part of the input.\n\nb) More empirical results can be provided to discuss Tail classes (similar to table 3 in AVT paper) and more results on EK100 or a different dataset. Providing results on multiple datasets helps prove the efficacy of the approach in different settings (egocentric vs. third-person, indoor vs. outdoor, etc.). \n\nc) The claim that edge learning with template bank achieved the best results is not well supported by quantitative results. Ablations on the main architecture (instead of template bank size) would be more useful, i.e. different backbones, removing squeeze and excitation, etc. Ideally, this can be fixed with an additional table showing the Top-1/Top-5 performance with different backbone variations and initializations. This will allow the reader to understand which part of the main architecture causes the model to perform better than the previous approaches.\n\nd) The authors claim to surpass current SOTA approaches but based on the reported results, improvements are not significant (and not consistent).\n\n\nMinor Issues:\n\na) Please provide an explanation for how the output of Eqn. 18 can be inserted in Eqn. 19. The output of SABlock is an (Nxd) tensor and we need an NxN attention matrix in SABlock_v. One option is to extract the attention from SABlock_e before multiplying by the Value matrix but it is not clear in the text.\n\nb) It would be a good idea to create a separate section for the decoder (unrolling classifier) and the loss function instead of briefly mentioning them in the implementation details. The background section can be shortened to save space.\n\nc) Some implementation details are missing, such as:\n1) Is the backbone frozen or just pretrained?\n2) How many blocks of SA are used? just one for every function?\n3) What is the number of patches (HW) used as input to the transformer?\n4) What initialization is used for node states, template bank, etc.?\n5) Any data augmentation applied to the dataset? horizontal flipping seems like a good fit for this data.\n6) What is the function f_{scale} scaling the values of h_{v}^{t} to?\n7) Minor typos are present (i.e., Section 4.1:L2)\n\n\nIdeas that are not required for publication, but would make the paper stronger:\n\na) It would be an interesting idea to test the learned features on the action recognition task. Using this architecture as a pretrained encoder for an action recognition task may outperform other approaches with naive initializations. The would be a very strong section in the evaluation.\n\nb) Other papers (Furnari et al.) use object detections and optical flow features as part of the input. It would be interesting to see the effect of including these features as input to the model.\n", "summary_of_the_paper": "The authors propose a framework for video action anticipation, where the task is to observe input frames and predict the action label after an anticipation period (not observed). The proposed architecture builds on top of message passing terminology (message, update, and readout functions) to create a recurrent transformer model that uses self-attention to capture spatial dependencies between frame patches (similar to ViT w/o [cls]) and recurrent module to capture the temporal dependencies across frames. Edge affinity matrix is learned and introduced into the MHSA block (Eqn. 12) as a prior to influence the attention matrix at every layer. Three different explicit edge learning approaches are proposed, in addition to implicit edge learning. The model is evaluated on the Epic Kitchen 55 dataset.", "summary_of_the_review": "The authors propose an interesting architecture for encoding RGB inputs and predicting actions after the anticipation period. The method has the capacity to encode edge information of graphs in the transformer model, but the type of data/task does not make use of this advantage. Other approaches (AVT) provide very similar solutions by using a temporal transformer, instead of a recurrent model, to capture the temporal dependencies. The GNN analogy seems weak and distracts the reader from the main approach (recurrent transformer). It can be fixed by simply removing the GNN and edge related sections, this will save space for additional ablations and quantitative results on other datasets.\n\nThe claim that this approach surpasses the SOTA approaches is weakly supported by quantitative results. The slight overall increase in performance cannot be attributed to explicit edge learning (as can be seen in the results). It is unclear which component(s) of the proposed architecture is the cause; it could be the added squeeze and excitation block, or bypassing the bottleneck representation (by not using a [CLS] token). The paper is not ready for publication in its current state; however, additional ablation studies, experiments on additional datasets, and removing unnecessary sections can significantly increase the paper's chances for publication.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NYPb20Wlq7a", "reviewer": "Reviewer_iyEe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n+The message passing method for learning graph representation under the video anticipation setting is novel to my knowledge\n\n+The idea of using self-attention learning the correlation between vertices and further building the graph representation is interesting.\n\n+Different edge learning methods are proposed and properly evaluated in the ablation study\n\n+The experiment results seem to be promising\n\n+The paper is overall well-written\n\n\nWeakness (sorted by priority):\n\n-The major concern is the effectiveness of the proposed three edge learning strategies. According to Table1-3, those 3 strategies have minor improvement on the overall performance.\n\n-The method section is not well-structured. It will be nice to have an overview of how to build the graph of the video sequence with proposed operations\n\n-It would be great to compare with previous methods on the efficiency.\n\n-Some visualization of what has been learned by the self-attention would be interesting, and may potentially improve the paper quality/\n\n-Only results on EPIC-Kitchens are shown. Many baseline methods also conduct experiments on the EGTEA Gaze+ dataset. \n\n-missing reference:\n\nEGO-TOPO: Environment Affordances from Egocentric Video CVPR 2020\n\nForecasting Human Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Vision. ECCV2020\n\n", "summary_of_the_paper": "This paper addressed an interesting problem of action anticipation. To this end, the authors proposed a unified recurrence model that generates the graph representation of the video sequence. Extensive experimental results have shown the effectiveness of the proposed method.", "summary_of_the_review": "The overall idea of using message passing for constructing graph representation for video action anticipation. But the proposed edge learning method has a minor influence on the model performance. Therefore some of the arguments made by the paper are not fully supported.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FABoBHe7u7k", "reviewer": "Reviewer_hNXw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Novel approach which solves an important vision task: video action anticipation.\n2. Method is clearly and methodically described.\n3. Experimental results are strong on the challenging EK55 dataset. The approach is compared against many state-of-the-arts algorithms.\n\nWeaknesses\n1. Paper's approach is focus on the specific task of video action anticipation task. While it is an important task in vision, it is not clear if it is of interests to the wider audience in the ICLR community.\n2. Minor. Paper's main technical contributions are specific to the task and may not generalise to other problems.", "summary_of_the_paper": "Paper proposes a novel architecture for video action anticipation task. The proposed method used a graph representation via message passing. The key contribution lies in the three explicitly edge learning strategies. These strategies to escape to the trivial estimation which is  purely based on the similarity in input tensors, by bringing the flexibility of edge representations. \n\n1. Edge attention decouples the attention operator in message function into vertex and edge estimation separately.\n2. Class token projection performs the outer-product of a trainable vector with supervision signal from class labels.\n3. Template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.\n\nExperiment on the EK55 dataset outperforms the state-of-the-arts algorithms.", "summary_of_the_review": "I recommend to accept this paper as the technical contributions are quite strong. However, its appeal to the wider audience in ICLR is a minor concern.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3EIYxOBuC_m", "reviewer": "Reviewer_HJA6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considers a topic which has attracted interest lately. Anticipation models are known to be hard to design and optimize, and learning representations good for anticipation is not trivial, so having discussion about this topic at this conference can be of interest.\nAs another plus, the proposed approach is compared with several other methods, which is not that common, especially for a task which has been tackled mostly recently. This gives a good basis for other works to compare with the state of the art.\n\nNevertheless, I think the paper has several flaws would be ideal to address in order to make it suitable for publication in this venue.\n\n1) The main novelty of the paper seems to be the proposal of a method for action anticipation based on a graph representation and message passing. While these kind of approaches have not been much investigated for this problem, there is at least one work which the authors should compare with in order to put the scope of their work in the right context [A]. Apart from this missed comparison, the paper lacks an in-depth discussion on the main relation between the proposed work and the current art. In particular, it would be interesting to answer the following questions: In which ways the proposed approach is different from the current ones? What are the main advantages of the proposed approach? I would suggest revising the paper to make these points more explicit and put the proposed work in the right context, by explaining how it advances our current understanding of the problem at hand.\n\n 2) Another major flaw of the paper is its lack of clarity in several points. I found the description of the method often hard to follow, partially due to the presence of unclear statements. Some of these statements are hard to grasp, while others reduce readability. I would suggest revising and proof readning. Some examples are reported in the following:\n - Page 1: \"making the accumulated information gains aligned with the temporal orders\" -> what is it meant by this statement? Is there a proof in the paper that this actually happens? What is it meant by aligned? How can it be measured?\n - Page 4: \"The message function, update function, and redout funciton are leveraged multi-head self-attention\" -> please revise (maybe \"leverage multi-head self-attention\"?)\n - Page 5: \"Given the frame features $x^t$ at time $t$, with shape $(H,W,C)$, which is transformed by a backbone model.\" -> this sentence is incomplete, please revise\n - Page 5: \"In this case where $N(v) \\equiv v$ in equation 1.\" -> this sentence is also incomplete.\n - Page 5: \"All edge learning strategies introduced following are all belong to explicit edge estimation category\" -> please revise (maybe \"introduced in the following all belong...\")\n\n3) The paper reports experiments on EPIC-KITCHENS-55, while an extension to the dataset, EPIC-KITCHENS-100 [B] has been introduced. Previous works also reported results on EGTEA-GAZE+ [C]. I think it is hard to draw general conclusions from experimenting on a single dataset, and it would have been interesting seeing experiments on these other datasets as well for completeness.\n \n4) While experiments include comparisons with several other approaches. All results are restricted to the case in which only RGB inputs are processed. This is odd, as previous works have been demonstrated to be able to leverage multimodal information. For instance, RULSTM and Action Banks both use RGB, Flow and object inputs. I would have expected the paper to report comparisons with methods taking these inputs as well (I believe these would outperform the results currently reported in the paper). Additionally, it would have been interesting to study whether the proposed approach can work well also on the other representations apart from RGB. This would be natural to assess, as also Flow features can be obtained with a 2D CNN backbone and object features are proposed in RULSTM as 1D vectors, which are inputs compatible with the proposed approach.\n\n[A] E. Dessalene, C. Devaraj, M. Maynord, C. Fermuller and Y. Aloimonos, \"Forecasting Action through Contact Representations from First Person Video,\" in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2021.3055233.\n\n[B] Damen, Doughty, Farinella, Furnari, Kazakos, Moltisanti, Munro, Price, Wray (2021). Rescaling Egocentric Vision . International Journal on Computer Vision (IJCV), abs/2006.13256 .\n\n[C] Li, Yin, Miao Liu, and James M. Rehg. \"In the eye of beholder: Joint learning of gaze and actions in first person video.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.", "summary_of_the_paper": "The paper proposes a method for action anticipation based on a graph representation of the temporal structure of video. Specifically a graph is built using semantic representations of video frames extracted with a 2D CNN backbone as vertices. Temporal reasoning is hence achieved modeling spatiotemporal information flow thorugh message passing across vertices. The approach is based on a message function, an unpdate function and a readout function, all based o a multi-head self-attention block. Experiments compare the proposed approach with respect to other approaches suing RGB inputs on EPIC-KITCHENS-55.", "summary_of_the_review": "All in all I think the paper tackles an interesting problem, but its presentation lacks clarity, which makes the paper hard to follow at times and difficult to compare to current art. Also, the experimental validation is lacking in some aspects, which should be fixed to make the paper a solid contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on action anticipation.  The reviewers appreciated the message passing based method.  However, concerns were raised regarding novelty, effectiveness, presentation, empirical results, and magnitude of impact for ICLR.  The reviewers considered the authors' response in their subsequent discussions but felt the concerns were not adequately addressed.  Based on this feedback the paper is not yet ready for publication in ICLR.
Decision: Reject