Action-Guided Attention for Video Action Anticipation
Authors: Tsung-Ming Tai
Sofia Casarin
Andrea Pilzer
Werner Nutt
Oswald Lanz
Venue: ICLR 2026 Poster (2026)
Keywords: video action anticipation
video understanding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Anticipating future actions in videos is challenging, as the observed frames provide only evidence of past activities, requiring the inference of latent intentions to predict upcoming actions. Existing transformer-based approaches, which rely on dot-product attention over pixel representations, often lack the high-level semantics necessary to model video sequences for effective action anticipation. As a result, these methods tend to overfit to explicit visual cues present in the past frames, limiting their ability to capture underlying intentions and degrading generalization to unseen samples. To address this, we propose Action-Guided Attention (AGA), an attention mechanism that explicitly leverages predicted action sequences as queries and keys to guide sequence modeling. Our approach fosters the attention module to emphasize relevant moments from the past based on the upcoming activity and combine this information with the current frame embedding via a dedicated gating function. The design of AGA enables post-training analysis of the knowledge discovered from the training set. Experiments on the widely adopted EPIC-Kitchens-100 benchmark demonstrate that AGA generalizes well from validation to unseen test sets. Post-training analysis can further examine the action dependencies captured by the model and the counterfactual evidence it has internalized, offering transparent and interpretable insights into its anticipative predictions.

Published as a conference paper at ICLR 2026
ACTION-GUIDED ATTENTION FOR VIDEO ACTION ANTICIPATION
Tsung-Ming Tai Tai Sofia Casarin' Andrea Pilzer?, Werner Nutt', Oswald Lanz' 1 Free University of Bozen-Bolzano
2 NVIDIA
(ntai, apilee:/vdize
fcasarin, nutt, 1anz
(aann220a0 @iinf. unib2.it
ABSTRACT
Anticipating future actions in videos is challenging, as the observed frames pro- vide only evidence of past activities, requiring the inference latent intentions to predict upcoming actions. Existing transformer-based approaches, which rely on dot-product attention over pixel representations, often lack the high-level se- mantics necessary model video sequences for effective action anticipation. As a result, these methods tend to overfit to explicit visual cues present in the past frames, limiting their ability capture underlying intentions and degrading gen- eralization to unseen samples. To address this, we propose ection-Guided At- tention (AGA), attention mechanism that explicitly leverages predicted action sequences as queries and keys to guide sequence modeling. Our approach fosters the attention module to emphasize relevant moments from the past based on the upcoming activity and combine this information with the current frame embed- ding via dedicated gating function. The design of AGA enables post-training analysis of the knowledge discovered from the training set. Experiments on the widely adopted EPIC-Kitchens 100 benchmark demonstrate that AGA general- izes well from validation unseen test sets. Post-training analysis can further examine the action dependencies captured by the model and the counterfactual evidence it has internalized, offering transparent and interpretable insights into its anticipative predictions.
INTRODUCTION
Anticipating future actions in videos is challenging in computer vision, with broad implications for assistive systems, robotics, autonomous vehicles, and interactive entertainment. The core difficulty stems from the need to predict upcoming actions based solely observed video frames, which provide only indirect evidence of subtle human intentions. Unlike action recognition, where its an- notations are present and synchronized with observed frames, action anticipation, the other hand, requires inferring future action labels from up-to-current observations. The observed frames often contain merely partially revealed and ambiguous visual cues, and the fact that the same past obser- vations could lead to multiple possible future outcomes further makes the task inherently uncertain.
The emergence of the vision transformer has appeared dominant paradigm in video action antic- ipation modeling. However, existing vision transformer approaches typically rely conventional self-attention over the token representation transformed from the video pixels. This design principle could have advantages in the action recognition task, as self-attention can effectively construct the vi- sual patterns and match their synchronized annotations. Nonetheless, the inherent non-deterministic nature of the future makes action anticipation more complex task than pattern recognition. Con- ventional self-attention can therefore be misled by visual clutter, potentially leading to overfitting.
To address this problem, propose leveraging action predictions sequentially guide the focus of attention. We call our method Action-Guide Attention (AGA), where both queries and keys are represented by action probabilities. The design builds upon the fundamentals of dot-product atten- tion, which attends to values through correlations between query and key representations. Instead of relying on pixel-level features, AGA uses predicted actions as high-level semantic guidance. This

---

Published as a conference paper at ICLR 2026
formulation explicitly models the idea of predicting the next action conditioned previously pre- dicted actions. learning dependencies among actions to improve the anticipation of future events. The output of the AGA aggregates relevant past moments and further adaptively mixes the informa- tion derived from the current frame inputs through a dedicated gating function to balance between history context and current evidence.
On SPICC-Kicceens1000 100, AGA generalizes effectively from validation to unseen test sets, with a consistently narrow gap that suggests resistance overfitting on partially observed video inputs. Its robustness is further validated on EPIC-Kitchens-55 and the sparsely annotated EGTEA Gaze+.
Moreover, the design of AGA also enables post-training analysis to uncover the knowledge learned during training. Through forward analysis and backward analysis, we can examine the action de- pendencies captured by the model and the counterfactual evidence it has internalized, offering trans- parent and interpretable insights into its anticipative predictions.
2 RELATED WORK
A broad spectrum of anticipation methods have been proposed over the past five years, Sequence modeling approaches, primarily based on recurrent networks (e.g., RNN, GRU, LSTM), were ini- tially explored to map observed features to future actions. RULSTM (Furnari & Farinella, (202)) employs two LSTMs, the first summarizes the observed video sequence, while the second unrolls predictions according to the temporal distance to the start of the future action. By explicitly in- corporating the anticipation interval and unrolling toward the end of the interval, this architecture established strong baseline on large-scale video action anticipation benchmarks such as EPIC- Kitchens-55 55 (Damen et al.) 2018) and PetCC-itchens-t 100 (Damen et al. 2022). However, sequential recurrent models are prone to error accumulation as time progresses. To address this, SRI. (Qi al., 202T] introduces regulation mechanism that emphasizes novel information at each timestamp, con- trasting it with previously observed content and modeling its correlation with past frames, thereby improving upon the RULSTM baseline. Building on this line, ImagineRNN (Wu et al. (0200) further enhances unrolling by learning to explicitly predict frame-wise differences within the anticipation interval in a contrastive manner.
More recently, transformer- based architectures have emerged. delivering significant improvements in action recognition and anticipation. 'The Anticipative Visual Transformer (AVT) (Girdhar & Grauman 202T) introduced causal attention decoder on top of the standard Vision Transformer framework for action anticipation, achieving state-of-the-ee performance at its time of publication. MemViT (Wu et al. 2022) extended AVT by storing longer historical context within the attention keys and values through token compression techniques. Later, RaftFormer (Girase et al. 2023) ad- dressed the high computational cost of attention by optimizing the model design for efficiency and faster inference.
Besides, several works have explored multi- integration to enhance prediction accuracy. S-GEAR (Diko al. 2024 emphasized semantic representations by jointly supervising visual and language branches. AFFT (Zhong et al.) (203) adapted the GPT-2 architecture and proposed an efficient fusion mechanism capable of incorporating additional modalities such as audio and optical flow. InAViT (Roy et al. 2024) leveraged prior information in the form of hand masks to disentangle human interactions from environmental clutter. More recent directions investigate the use of vision-
language models and large language models for anticipation (Zhang et al. 2023 Mittal et al. 2024
Wang et al. 2025). Other extensions include semantic action augmentation (Qiu & Rajan 2025)
and long-term anticipation (Zhao et al. 2023 Zatsarynna et al. 2025). further broadening the scope of multi-modal anticipation research.
Prior work also explores the semantic label space in anticipation, but unlike AGA they do not feed
predictions back into the attention mechanism to reweight the visual representation. Abu Farha
et al. 2018)) use RNN/CNN predictors to decode future actions in the label space. Ke et al. (2019)
refine predictions by injecting temporal features into an attention module. Zhao & Wilde (202)) condition the initial visual observation
action-time sequence through conditional GAN to
generate diverse futures. However, in all these methods the predictions remain separate from feature
extraction, they do not guide the attention over visual tokens. AGA is distinct in that it uses the
2

---

Published as a conference paper at ICLR 2026
S1-84
8--s.
91-1-1>
8-3
8e-2-
8e-1
MLP
Muad-HHad Attention
Past
Current
EMA
Ee
Action Predictions
a
K
YOYO
FFN
Ex
MLP
e
ht
History Context
Ey
Frame Embeddlings
V
Adaptive Gating
Action-Guide Attention
e
Ct-(S-1)
e
2
9-1
C2
Figure 1: Architecture Overview. The model consists of two modules. The Action-Guided Atten- tion uses the most recent S action predictions as keys, the exponential moving average (EMA) of all predicted actions the query, and S frame embeddings values generate history context he The Adaptive Gating then integrates this history context with the current frame embedding e to produce a fused representation, which is mapped to the new prediction jt.
model's own evolving action predictions to dynamically focus attention semantically meaningful visual cucs.
In this work, we focus fundamental challenge in video action anticipation, which causes overfit- ting due to incomplete and uncertain observations associated with the future action. We propose an alternative attention design, AGA, that incorporates high-level, task-guides action representations queries and keys, thereby mitigating attention to over-reliance on unreliable visual cues from partial video evidence. Our study considers explicitly anticipation methods that RGB video frames as input, excluding those that integrate multi-modal or auxiliary knowledge sources such as text.
3 METHOD
Let (zx)220 be a sequence of video frames sampled every A/ > 0 seconds, where Dy E B For indices s, we write Zr:s to denote the subsequence (Imax(0.r):... 22), that is, values with r < 0 are clipped at zero; if s exceeds the total sequence length then set it to the last index, and s 2 0. Here, denotes the discrete frame index, corresponding to real time At. Given the observed frames Torr, the goal is to predict the future action +r+ E NR represented as a one-hot vector over N, classes, which occurs la At seconds after the last observed frame. For each index 0 < 1 < T, the model outputs probability distribution [0, 11 as an estimate of the true future action litt..
3.1 ACTION-GUIDED ATTENTION
We introduce conditional attention mechanism that extends the standard formulation of Vaswani al. (2017) by conditioning queries (Q) and keys (K) on semantic-level action predictions, while using frame-encoded video features values (V). Ideally, the ground-truth action label yı would provide the most reliable signal for guiding the estimation of future actions. However, because the ground-truth is unavailable at inference time, we instead rely on the self- self-predicted distribution Ộr as an approximate guide. In dot-product attention, correlations between query and key yield weights that select relevant values. Our design represents the features of video frame by its corresponding action prediction, and then utilizes the dot-produc between prediction embeddings to aggregate rel- evant frame features forecast the anticipated action. The overall architecture is shown in Figure T.
3

---

Published as a conference paper at ICLR 2026
At each timestep t, the input video frame 2; is first processed by trame-based backbone fbacktone (-), followed by an trainable encoder fx() that extracts the frame features ez:
et fu (Tonckoooon (I+)).
The trainable function (.).) finetunes features from frozen backbone, keeping the backbone mod- ular and interchangeable for any image architecture.
These features are subsequently stored in a first-in, first-out (FIFO) queue of size S, a hyperparame- ter that determines the temporal window over which the model can reference past information. The number the keys and values thus dependent on S.
At each timestep, the frame features stored in the queue serve as the values, while the predictions from the same time, which can be viewed as semantic tags summarizing the information up to the current moment, features e, we employ the past predictions ý to represent high-level semantics in the attention mech- anism. Formally, we define:
used to construct the keys. Specifically, instead of using the perceptual
K, (8(
V (vie--::-1)).
where (-(-) and (-) are implemented as multilayer perceptrons (MLP).
Correspondingly, the query also leverages the sequence of action predictions to match the semantic level with the keys. To effectively incorporate longer temporal dependencies, we apply an exponen- tial moving average (EMA) over the past action predictions to form the query:
Tt = - + (1 - a))x-1
Q: = EQ))),
where To initialized as the zero vector and a empirically set to 0.8 (see Section 4.3 and Ap- pendix 1.2 for experiments the choice of The function E-(-) consists of MLP. Note that Ko and Vo are undefined and are never used for attention. When the queue is empty, the first input frame, the attention is bypassed and its output initialized to zero vector. Starting from the second frame, the queue contains at least one timestep for both Q and K, allowing the attention mechanism to begin processing information even if the queue is not yet full.
Then, with Qt, Kr, and V, defined, the multi head dot-produce attention is applied:
h MultiHead( Qt, Ke. V/) = (head, || . (head,)
where
(K, (V.W). Vd
head; Softmax
(3)
Here, ho is initially set to a zero-vector, W W WY are the trainable parameters for multi-heads, and W is the output weight for aggregating the multi-head attention outputs.
As usual, the output of the attention module is further transformed by feedforward network (FFN), implemented MLP, with residual connection. The FFN complements the attention mecha- nism by introducing non-linear transformations, while the learnable parameters within the attention confined the input and output embedding projections:
The h4 + FF.()
For both multi-head attention and FFN the normalization performed in PreNorm style (Xiong al. 2020) using the RMSNorm layer (Zhang & Sennrich 2019). The outputs of the meldi-head attention and the FFN are then fused with e, through the Adaptive Gating.
3.2 ADAPTIVE GATING
The relevance of the history context and the current visual evidence vary over time. Therefore, we componentwise fuse the history h and evidence e, using a gating vector ge E [0, 1]d that has the same dimension as h, and et, resulting in the output
0r = gt. hhe (1 gt) et.
4

---

Published as a conference paper at ICLR 2026
where @ denotes componentwise multiplicatiin. The gating vector itself is computed as Sr (MLP(h; || Et)), where the MLP is composed of two linear layers and ReLU activation func- tion while (.) denotes the sigmoid function. Adaptive Gating is illustrated in Figure
Entries of 9i close to I emphasize the history in those components, while entries near 0 favor the current visual evidence. This adaptive mixing lets the model rely on past or presence as needed.
Finally, the new prediction ÿt is obtained from the Adaptive Gating output o
ft (efttmax(MIP(oo))
by applying an MLP, consisting of two linear layers with a ReLU activation to produce the logits, followed by Softmax.
3.3 FORWARD ANALYSIS
The design of AGA gives us insights into how a trained model works. Specifically, we can answer questions of the kind: Given a currently observed action, which past actions does the model consider as relevant for predicting the next action? We refer to this as forward analysis.
For example consider the actions oc = open cupboard and cc = close cupboard. Suppose that in the current frame the cupboard is opened and the history contains both, frames where the cupboard is opened and where it is being closed. Which of those more relevant? To find out we create a query for the current observation, a key for each candidate action, and then compute the weights that the attention mechanism gives to each candidate with regard to the query.
More precisely, for each action a, let ya be the one-hot distribution with (a(a) = 1 and (a(c) = for all action classes # a. In our example, create for the query embedding Que EQ(Uor) and for oc and cc together the key embedding Kac,ee = Ex (Yoe ll Mec)) Using the Softmax expression in Equation we compute for each head a vector w (wac
(a) weee) of attention weights and then take the average ú = 1/4 Σk_1 ((( of these vectors. The component Woe reflects the average relevance given frames where the cupboard is opened and W.e where it is closed. Figure 2 shows actual weights for this example obtained from an AGA model trained on PPIC--itchens-- 100 dataset.
3.4 BACKWARD ANALYSIS
While forward analysis explains which past predictions the model actually relies on when producing its output, backward analysis instead asks a counterfactual question: If the next action were a, what changes in the past predictions would make the model assign high probability to a?
Let Y = (Ji-s..... 0t-1) be the sequence of past predictions and let Ôc Xx) denote the predicted distribution at time given the sequence of frames X Xit During training, each 9. is probability distribution, but as function fo is defined for arbitrary real vectors Y. Hence, for fixed X, the following expressions are well defined for any Y.
Suppose the model has produced the prediction Gt. while the correct next action is a, represented by the one-hot distribution ya- The discrepancy between prediction and target is measured by the cross-entropy
Hyya in) - E. log
log (c(a).
where the sum is taken over all class labels c. Since ÿt depends on the past predictions Y, we can ask how Y can be changed that the resulting prediction comes close to ya- To capture this, define
L(Y)- H (ya. fo( log (Y, X)(a))
Minimizing L(Y) amounts to finding past predictions that make the target action more probable for the given model. We can find an approximate local minimum by gradient descent, choosing a step size n and starting from the original predictions ( iterating through
yū+l) = y()) L(Y()))
until the loss function plateaus, that is,
[ -((YS) <
5

---

Published as a conference paper at ICLR 2026
Overall Classes
Unseen Classes
Tail Classes
Methods
Modality
Action
Verb
Noun
Action
Verb
Noun
Action
Verb
Noun
RULSTM Furnani Farinella 2020
RGB, Flow, Obj
11.2
25.3
26.7
9.7
19.4
26.9
7.9
17.6
16.0
AVT+ (Girana Graumaa 2021
RGB. Obj
12.6
25.6
28.8
8.8
20.9
22.3
10.1
19.0
22.0
AVT Gironir Galinn AUM
Flow; Obj
16.7
32.3
12.9
21.0
27.6
13.8
19.3
24.0
AFFT-TNN+ Zhving el N144
RGB. Flow, Obj, Audio
13.4
19.4
28.3
9.9
14.0
24.2
10.9
12.0
19.5
AFFT-Swin+ Zhivnt Ml
Flow, Obj.
12.1
27.7
RAFTiomer (Uiiee 0E KI5S
RGB
14.7
27.4
34.0
RAFTforma-228
RGB
15.4
34.1
S-GRAR (Dika 2074
RGB. Ohj
14.7
32.0
25.9
S S-GEAR 2B (Uis 2004
RGB, Obj
15.3
25.5
31.7
S-GEAR-IB [Diso al 2029
RGB, Obj
15.5
26.6
32.6
AGA (Ours, Swm-B)
RGB
16.9
30.8
36.4
13.5
22.3
30.0
14.9
25.8
30.0
Table 1: EK100 Test Results. Evaluation on unseen test set via the official challenge server.
for threshold when the maximum number of iteration steps iter has been exceeded, that is, j 2 iter.
The resulting Y* need not be itself a valid sequence of probability distributions, but it can still be decomposed into vectors as long as distributions and we can compare it with the actual distributions in (0), In the difference AY Y y (0) positive entries indicate supportive changes and negative entries indicate blocking changes for the target action This provides a counterfactual perspective on the dependencies captured by the model, complementary to forward analysis.
Qualitative results of both forward and backward analyses are presented in the experiments section, where they provide concrete insights into the model decision madiiiinmmmaingggg process and the knowledge that our model can explore in the training dataset.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUP
We evaluated our proposed method using the TSN (Wang et al. 2016) and Swin-B (Liu et al. 2021 backbones, with AGA applied on frozen features as single attention layer with hidden size 2048 and 16 heads. The queue size S is set to 16 (see Appendix [B..I] for experiments on the choice of M).Models were trained for 50 epochs with a batch size of I28, using AdamW with a learn- ing rate of 2 10-4, weight decay of 0.01, and a cosine schedule. All input frames were resized to 224x 224. Performance was measured using Mean Top-5 Recall (MTSR), which averages re- call across classes mitigate the dominance of frequent actions. Following the training protocol of (Girdhar& & Grauman] 2021). we compute class-reweighting factors with the inverted class fre- quency, and apply these weights when supervising the anticipation predictions with the ground-truth iicootationsRRssule were reported at the dataset-specific anticipation interval Ta We converted Ta (provided in seconds) into the corresponding anticipation index ta (used by our method) as the number of time units At to reach T., that is, Ta = fa At.
Experiments were conducted on three benchmarks. EPICC-Kitchens-100 (EK100) DDamen et al. 2022) contains 100 hours of video with 3,806 actions, including 67,217 training and 9,668 validation segments, and was evaluated тa 1s and At E.PICC-itchhen-555 (EK55) (Damen al.) 2018) contains 55 hours of video with 2.513 actions, including 23,492 training and 4,979 validation segments, and was evaluated using Top-1 and Top-5 accuracy as well as MT5R at тa = 1s and At = 1s. EGGEA Gaze+ (Li et al. 2018) comprises 28 hours of egocentric video with 106 actions, including 8,299 training and 2,022 validation clips, where we followed the protocol in (Girdhar & Grauman 2021) and reported Top-1 accuracy and mean Top-1 recall at Ta 0.5s and At 0.5s.
All the benchmark scores are evaluated on the time point At + Te, where T is the sequence length.
4.2 RESULTS
Tables and present the performance of the proposed AGA in comparison with prior methods on 100 for the unseen test set and the validation set, respectively. On the validation

---

Published as a conference paper at ICLR 2026
Overall Classes
Unseen Classes
Tail Classes
Methods
Backhone
Action
Verh
Noun
Action
Verb
Noun
Action
Verb
Noun
RUT STM
Furnari & Farinella
2020
TSN
13.3
27.5
29.0
AVT-h Girdhar Grauman
2021
27.2
30.7
AVT-h Girdhar Grauman
2021
RCSN152
12.8
25.5
28.1
AVT-b Girdhar Graunail
4021
AVT
14.9
30.2
31.7
AFFT Zhhing
2125
TSN
16.4
21.3
32.7
13.6
24.1
25.5
14.3
13.2
25.8
AFFT ZZhngg al 2075
Swin-B
17.6
23.4
33.7
15.2
24.5
25.4
15.3
15.6
26.5
McMVIT
2022
MVITv2-16
33.2
27.5
21.7
13.2
26.3
27.4
McMViT 32x3 Ww al 2124
MViIv2-24
17.7
32.2
37.0
15.2
28.6
27.4
15.5
25.3
31.0
RAFTiormer (Girase
MVTTv2-16
35.5
RAFTformer- Cairase
20255
MVITv2-16 24
19.1
33.8
37.9
29.8
S-GEAR Di 2026
VIT-B
18.3
31.1
37.3
S-GEAR-2B DiKO of
2024
VIT-B x2
32.7
37.9
AGA (Ours)
TSN
17.5
32.2
35.7
11.9
31.4
24.9
16.8
26.9
31.8
AGA (Ours)
Swin-B
18.8
32.5
38.7
16.3
34.4
28.5
18.4
27.4
35.0
Table EK100 Validation Results. Comparison of methods on the validation set using only RGB input. Methods highlighted in gray the same TSN backbone weights.
Top-1 Acc
Top-5 Acc
Methods
Backhone
MTSR
RULSTM Furnari & Farinella
2020
TSN
13.1
30.8
12.5
TempAgg (Snner 2020
TSN
12.3
28.5
13.1
ImagineRNN Wu al 2020
TSN
13.7
31.6
SRL Qi al. 2021
TSN
31.7
13.2
AVT-h Girdhar & Grauran
2021
TSN
13.1
28.1
13.5
AVT-h Girdhar Grauman
2021
AVT-b
12.5
30.1
13.6
AVT-h Curdhar
2021
iCSNN152
14.4
31.7
13.2
DCR Xu
2022
TSN
13.6
30.8
DCR
Xu el al
2022
irCNN152
15,1
34,0
DCR Xu el al.
2022
TSM
16.1
33.1
RAFThomer (irase al.
2023
TSN
13.8
S-GEAR DDiko al. 2024
15.6
32.8
S-GEAR Diko al.
iCCNN152
16.2
33.1
S-GBAR Diko
2024
VIT-B
15.8
34.5
AGA (Ours)
ISN
13.5
32.1
14.3
AGA (Ours)
Swin-B
16.3
37.4
16.6
Table 3: EK55 Validation Results. Top- 1/Tp-5 action accuracy and MT5R at Methoss highlighted in gray use the same TSN backbone weights.
set, all methods are evaluated using the RGB modality only, and approaches that exploit additional modalities external knowledge excluded to ensure fair comparison. With the TSN backbone (highlighted gray background colors), AGA achieves 17.5% accuracy, outperforming AFFT at 16.4%, S-GEAR at 14.9%, and other baselines. Replacing TSN with the stronger Swin-B backbone further improves performance from 17.5% to 18.8%, with noticeable gains observed on unseen and tail classes. Methods that ensemble features from multiple backbones highlighted in gray.
On the unseen test set, evaluated through the official challenge server, AGA maintains consistent performance, demonstrating strong generalization ability. Many methods listed in Table [ are pri- marily competition submissions that rely on multi-modal inputs or ensemble strategies, whereas AGA achieves competitive scores without such enhancements. The gap between validation and test is consistently small, indicating that AGA generalizes effectively beyond the validation set com- pared with other methods. Leveraging action predictions to guide attention prevents the model from over-emphasizing the visual clutters, which is especially important in action anticipation where un- certainty high and observations are less reliable than in other video recognition tasks.
Tables 3 and compare the performance of AGA on two smaller-scale datasets, EK55 and EGTEA Gaze+. EGTEA Gaze+ provides sparse annotations only for the target action, without detailed labels for the preceding frames in the pre-action observation window. Accordingly, model must rely entirely on its own predictions of prior activities without supervision. Nevertheless, empirical under constrained annotation availability
results show that AGA maintains strong performance, and sparse supervision. On EK55, S-GEAR reports higher accuracy than AGA under the same TSN backbone, likely due its use of higher-resolution inputs 388 compared to 224 x24x224).
7

---

Published as a conference paper at ICLR 2026
Top-1 Ace
Mean
Top- Recall
Methods
Action
Verb
Noun
Action
Verb
Noun
13D RID-Res50 Carreira & Zisserman
2017
34,8
48.0
42.1
23.2
31.3
30.0
FHOI (Liu el 2020
36.6
49.0
45.5
32.5
32.7
25.3
TSN-AVIS (Girhaa & Grauman
2021
39.8
51.7
50.3
28.3
41.2
41.4
AVT (Girdhar Grauman
A021
52.2
48.3
TSN-APFT 1Zhong 2023
42.5
53.4
50.4
35.2
42.4
44.5
AGA (Ours, TSN]
43.5
54.3
52.2
35.5
43.8
46.6
AGA (Ours, Swin-B)
45,4
55.9
54.3
37.4
46.5
49.3
Table 4: EGTEA Gaze+ Validation Results. Top-1 accuracies and Mean Top-1 Recall at To 0.5s.
Methods highlighted in gray use the same TSN backbone weights.
Configuration
MT5R
EMA
MT5R
0 0.0
17.4
Reference (LSTM)
14.5
0
18.7
Baseline (Causal Attention)
15.9
0
18.4
Action-Guided Attention
18.2
0.6
18.5
0 0.8
18.8
Action-Guided Attention + Adaptive Gating
18.8
0= 18.2
Table Ablation study of proposed Action Action-Guided Attention.
Table 6: Selection of EMA a for Q.
Moreover, EK55 evaluation is majorly reported overall accuracy rather than in class- mean recall, which biases results toward common classes and obscures performance actions.
4.3 ABLATION STUDY
Table presents the ablation study the proposed model. The analysis begins with a baseline model. causal attention (c.g., AVT). Replacing the standard dot-product attention with queries and keys defined by action predictions yields substantial improvement, raising accuracy from 15.9% 18.2%. this setting. adaptive gating is disabled, so the attention output is directly added to the current frame input without weighting past and current features. Incorporating adaptive gating provides additional gains, further increasing accuracy from 18.2% 18.8%. For reference, an LSTM baseline trained under the same conditions is also reported.
Table 6 examines the impact of different a values the exponential moving average used to form the query. Larger values of a make the query depend more heavily on the most recent prediction. When a = 0, the query collapses to a constant (projected from the zero vector) across all timesteps, leaving attention unconditioned and driven solely by the keys, which results in poor anticipation accuracy. For 0.2 < a S 0.8, the query aggregates action dynamics over time, providing a coherent action-guided signal and improving performance. When the query depends only on the latest prediction. Based on empirical results, we adopt = 0.8 in our model. All ablations reported here were conducted on the EK100 validation and evaluated using MT5R. Appendix B.1.2 informs about studies on the other two datasets used in the experiments.
1, performance again drops because
4.4 FORWARD ANALYSIS
Fillustrates the attention focus on past actions given different queries. Since AGA treats past actions (keys) as an unordered set, the order shown along the x-axis is not indicative. Four examples are analyzed using open/close cupboard and open/close fridge as queries to examine how the model allocates attention across past actions.
Figure 2
In the first two examples, where the queries are open cupboard and close cupboard, the model con- sistently attends more on earlier open cupboard rather than close cupboard. This behavior suggests that the model learns to revisit earlier events and search for evidence that indicates the duration of an open cupboard state, which may not be visible when the cupboard is closed. Moreover, the attention distribution conditioned on open cupboard is more uniform. implying that the next action following open event is less certain and can be supported by multiple cues. By contrast, conditioning
8

---

Published as a conference paper at ICLR 2026
close cupboard produces relatively sharper distribution, reflecting substantial confidence that the next action is linked to objects accessed during the preceding open period.
In the third and fourth examples, where the queries open fridge and close fridge, the model shows a similar pattern. Conditioning on open fridge yields a more uniform attention across past actions, though food-related interactions receive higher weights. When conditioned the attention becomes more concentrated, focusing primarily on take food, indicating that the model leverages this evidence predict subsequent actions involving the food that has been taken before. Overall, these examples suggest that AGA captured plausible dependencies and learnt meaningful associations between actions focus on scenes in the past relevant for the immediate future.
close fridge,
4.5 BACKWARD ANALYSIS
Figure presents the backward analysis of the model trained on the EK100 dataset.
The example is taken from the EK 100 validation set. The original top-5 predicted actions are shown along with the counterfactual top-5 supportive actions via proposed backward analysis optimized toward the target action take pan (which originally ranked 5th). Each column corresponds timestep, with the final column (gray background) representing the anticipated output of the video clip. Actions marked red indicate those suppressed by backward analysis, while those in green indicate promoted actions.
In the original t predictions, the model prioritized actions such as eat squash, dry ladle, and close cupboard, with take pan largely absent. This pattern reveals tendency to favor immediate, contextually dominant actions, as reflected in the diversity verbs and nouns. Backward analysis conditioned the counterfactual target take par discovers alternative reasoning path. Supportive actions involving containers and utensils, such as take spatula, put pan, insert pan, and take ladle, forming semantically coherent trajectory toward the target. These results, obtained with model weights frozen, suggest that the model has already encoded multiple plausible futures. However, when faced with broad uncertainty, it originally distributed focus across diverse environmental cues. Backward analysis highlights counterfactual path that converges on utensils, offering insights into both the model inference process and the latent knowledge has acquired.
Backward analysis in the paper was conducted using a stopping criterion of = 1e-6, step size of ?) le2, and a maximum of iter 5000. Under this configuration, the optimization consistently identified counterfactual actions within the top-10 predictions across 30 validation samples drawn from EK 100. When further varying the step size from le-1 to 1e5, found that the same behavior remained stable for step sizes in the range lel to le4, using the same stopping criterion.
the top-1 choice, validated
4.6 ADAPTIVE GATING
Figure shows two action sequences together with their average gating ratios, retrieved from the adaptive gating module during inference. Each sequence contains colored segments representing
Query: Query: open cupboard close cupboard
Query: open fridge
Query: close fridge
open cupboard
close cupboard
open cupboard
close cupboard
take board:
chopping
board: chopping
take
food
put
food
close
drawer
open drawer
take board: chopping
board:
take hropping
food
put
food
close drawer
open drawer
put
aut
Figure 2: Forward Analysis identifies which past actions the model attends to when predicting its
next action in response query. This analysis conducted the model trained with EK100.

---

Published as a conference paper at ICLR 2026
f-3
Video observation
dry ladie close oupboard eat squash move tray take tongs
ladle eat squash take pan close cupboard take tongs
close cupboard search container dry ladle take pen move tray
close cupboard dry ladle eat squash insert grater
Original t anticipation
take pan
Backward analysis (Target "take pan")
close cupboard
take pan sausage squash take tongs take spatula
take pan search container put move tray insert pan
take pan search container put pan put ladle take ladle
dry ladie take move tray tongs
Figure 3: Backward Analysis. The figure compares the top-5 original predicted actions with the counterfactual supportive actions optimized toward the target action take pan. Each column repre- sents timestep; the final column shows the anticipated output. Suppressed actions are highlighted in red and promoted actions appear in green. This example is drawn from the EK100 validation set.
actions and black segments representing background or non-action content. The adaptive gate bal- ances historical and current information: higher values emphasize past context, while lower values prioritize present input. Empirical observations reveal that fluctuations in the gating value often correlate with action transitions. For instance, the gate tends to rise during background segments to preserve past information, and then shifts attention to the current input when an action occurs. Importantly, the adaptive gate is not explicitly supervised by action boundaries; its context-aware behavior emerges naturally from training on the anticipation task.
CONCLUSION
We present AGA, Action-Guided Attention for video action anticipation, which leverages past pre- dictions to create semantic representations that guide the model's attention to parts of the history relevant for anticipating the immediate future. In addition, AGA dynamically mixes past and present visual information with its adaptive gating mechanism. Experiments show resistance to overfitting on top of overall competitive performance. Furthermore, AGA enables post-training forward and backward analysis, offering insight into the learned action dependencies and the reasoning process behind anticipation.
6 REPRODUCIBILITY STATEMENT
Implementation details are presented in Sections 3, 4.1, and Appendix B. The source code and processing scripts are publicly available at: prtps::/ggithnn htt.://ww....m//ooooooooo/n////.
Gating Value
Gating
0.5
Sequence
Action Sequence
The gating values, displayed alongside the tion sequence, demonstrate context-aware behavior. background regions (black), the gate retains historical context; in action regions (colored), it prioritizes current visual evidence.
Figure 4: Visualization of Adaptive Gating Ratio.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Yazan Abu Farha, Alexander Richard, and Juergen Gall. When Will You Do What? - Anticipat- ing Temporal Occurrences of Activities. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The EPIC-Kitchens dataset. In European Conference on Computer Vision, 2018.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC- FIC--itchenns1001 International Journal of Computer Vision, (30(1))33-55 2022.
Anxhelo Diko, Danilo Avola, Bardh Prenkaj, Federico Fontana, and Luigi Cinque. Semantically guided representation learning for action anticipation. In European Conference Computer Vision, 2024.
Antonino Furnari and Giovanni Maria Farinella. Rolling-unrolling LSTMs for action anticipation from first-person video. IEEE Transactions Pattern Analysis and Machine Intelligence, 43 (11):4021-4036 2020.
Harshayu Girase, Nakul Agarwal, Chiho Choi, and Karttikeya Mangalam. Latency matters: Real- time action forecasting transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In IEEE/CVF International Conference on Computer Vision, 2021.
Qiuhong Ke, Mario Fritz, and Bernt Schiele. Time-Conditioned Action Anticipation in One Shot. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Yin Li, Miao Liu, and James M Rehg. the eye of beholder: Joint learning of gaze and actions in first person video. In European Conference on Computer Vision, 2018.
Miao Liu, Siyu Tang, Yin Li, and James M Rehg. Forecasting human-object interaction: joint pre- diction of motor attention and actions first person video. In European Conference on Computer Vision, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF Interna- tional Conference on Computer Vision, 2021.
Himangi Mittal, Nakul Agarwal, Shao-Yuan Lo, and Kwonjoon Lee. Can't make an omelette with- out breaking some eggs: Plausible action anticipation using large video-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Zhaobo Qi, Shuhui Wang, Chi Su, Li Su, Qingming Huang, and Qi Tian. Self-regulated learning for egocentric video activity anticipation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(6))6715-6773 2021.
Yihui Qiu and Deepu Rajan. Action sequence augmentation for action anticipation. In International Conference on Learning Representations, 2025.
Debaditya Roy, Ramanathan Rajendiran, and Basura Fernando. Interaction region visual transformer for egocentric action anticipation. In IEFF/CVF Winter Conference on Applications of Computer Vision, 2024.
Fadime Sener, Dipika Singhania, and Angela Yao. Temporal aggregate representations for long-
range video understanding. In European Conference Computer Vision, 2020.
11

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, 2017.
Binglu Wang, Yao Tian, Shunzhou Wang, and Le Yang. Multimodal large models are effective action anticipators. IEEE Transactions on Multimedia, 2025.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. European Conference on Computer Vision, 2016.
Chao-Yual Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. MeMViT: Memory-augmenter multiscale vision transformer for effi- cient long-term video recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Yu Wu, Linchao Zhu, Xiaohan Wang, Yi Yang, and Fei Wu. Learning to anticipate egocentric actions by imagination. IEEE Transactions on Image Processing, 30: 143-1152, 2020.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, 2020.
Xinyu Xu, Yong-Lu Li, and Cewu Lu. Learning to anticipate future with dynamic context removal. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Olga Zatsarynna, Emad Bahrami, Yazan Abu Farha, Gianpiero Francesca, and Juergen Gall. Manta: Diffusion mamba for efficient and effective stochastic long-term dense action anticipation. In IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2025.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, 2019.
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMa: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0287 2023.
He Zhao and Richard P. Wilde. On Diverse Asynchronous Activity Anticipation. In European Conference on Computer Vision, 2020.
Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. AntGPT: Can large language models help long-term action anticipation from vidcos? In International Conference on Learning Representations, 2023.
Zeyun Zhong, David Schneider, Michael Voit, Rainer Stiefelhagen, and Jürgen Beyerer. Anticipative feature fusion transformer for multi-modal action anticipation. In /EEE/CVF Winter Conference on Applications of Computer Vision, 2023.
12

---

Published as a conference paper at ICLR 2026
Eq. Ex
Fy
FFN
MLP in Adapiive Gating
MLPP in classifier
LayerNorm
LayerNorm Rel.U Dropout
LayerNorm Rel.U
Linacr (2048, 1024)
Lincar (4096, 512)
LayerNorm
ReI.U
GELU
Rel.U
ReL.U
Dropout
Dropout
Drupout
Dropout Linear (512, 2048)
Dropout
Linear (Mbuckoonzs 2048)
Linear (2048, 512)
Linear (2048, 2048)
Linear (1024, 2048)
Linear(2048, Ng)
ScaleNorm
Table 7: Modeling components in AGA.
Queue Size
AGA (MTSR)
S=4
18.4
S=16
18.8
S=30
18.0
Table 8: Ablation study with different queue sizes on EK100.
A IMPLEMENTATION DETAILS
For the TSN baseline, utilized BN-Inception backbone with pre-extracted features provided by the official EPIC-Kitchens action anticipation repository For the Swin-B Transformer, we used open-source implementations with pre-trained weights from the timm library Input frames for both architectures were resized to 224 x 224 with pixel values rescaled to the range [-1,
The EK100 dataset was sampled at 1 fps over 30-frame sequences, the EK55 dataset fps over 10-frame sequences, and the EGTEA Gaze+ dataset at 0.5 fps over 10-frame sequences, These sampling configurations align with the anticipation intervals described in the main manuscript.
Figure 7 details the architectures of fa. Eq. EK. Ey, the FFN, and the MLPs. We applied a dropout rate 0.6 on EK100 and 0.4 on both EK55 and EGTEA Gaze+ datasets.
B ADDITIONAL EXPERIMENTS
We carried out experiments to determine optimal hyperparameters for AGA and identify its compu- tational cost. With another experiment, we investigated how errors in the predictions impact model accuracy. Finally, we examined how representing prediction uncertainty improves the accuracy of the AGA model.
B. SELECTION OF HYPERPARAMETERS
Two important hyperparameters for AGA are the length S of the queue from which the model can reference past information (Equation I and the smoothing factor the exponential moving average that determines the temporal window of the query (Equation
B.1.1 QUEUE LENGTH
The experiments with EK100 reported in the paper conducted with queue length of S 16. We also ran experiments for queue lengths and 30. The mean top-5 recall for the different queue lengths is shown in Table R
B.1.2 SMOOTHING FACTOR
Intuitively, EMA injects temporal context into the query and smooths out model prediction jitter. The optimal smoothing constant depends on factors such as the video sampling rate and the target action duration. As reported in Table 6 of the paper, we found in the 10 experiment that the MT5R was between 18.4 and 18.8 for 0.2 < < 0.8. This suggests that the accuracy is relatively stable for in that range.
v0.5.4
13

---

were forcibly Published as a conference paper at ICLR 2026
EMA
Top-1
Top-S
MT5R
EMA
Top-1
Recall
0.0
15.0
34,5
15.5
0.0
41.8
34.7
0.2
15.5
35.8
15.9
0.2
43.9
36.2
0.4
16.1
37.4
16,4
0.4
44.9
37.3
0.6
16.3
37.2
16.3
0.6
37.1
0.8
16.3
37.4
16.6
0.8
45,4
37.4
1.0
15.7
36.1
15.9
1.0
43.8
36.0
Table 9: Accuracy differences resulting from
Table 10: Accuracy differences resulting from
varying a on EK55.
varying a on EGTEA Gaze+.
Sequence Length
AVT (GFLLPS)
AGA (GFLOPS)
Sequence Length
AVT (ms)
AGA (ms)
137.28
123.91
14.97
11.4737
16
274.56
248.29
16
20.57
23.4944
32
549.12
497.57
32
33.93
47.3397
Table 11: Computational cost comparison.
Table 12: Inference time comparison.
We conducted the same study for the other two datasets used the experiments. Tables and 10 report the results of sweeping the smoothing factor over [0, 1]. We observe that the optimal scores result from a value of = 0.8, which is consistent with the findings on EK100, while even across datasets choice of within the range 0.2 S 0.8 leads relatively stable accuracy.
B.2 COMPUTATIONAL COST OF AGA
To evaluate the inference-time computational cost of AGA, we compared it against the AVT base- line. We created random tensors to mimic how sequences of frames are processed in real inference setting. each inference iteration we passed the random input to the model and let do its process- ing: AVT in go, due to its transformer architecture, AGA frame by frame, due to its sequential nature.
In Table 11 we show the total number of floating-poit point operations (FLOPs) computed for both models. AGA required slightly fewer FLOPs, despite using the heavier Swin-B backbone, versus AVT's ViT-B
In Table 12 we show the inference times for AVT and AGA (with backbone Swin-B) measured on single A100 GPU. The differences be explained the way in which the two models process the input. While AGA processes the sequence progressively, AVT handles the entire sequence in parallel.
B.3 PROPAGATION OF ERRORS
We wanted understand how robust our model is with respect errors in the predictions. We set up an experiment where AGA is subjected to inaccurate or noisy predictions that disrupt its agtion-guidance signal. As input, AGA received 30 frames from 30-second clip from the EK100 validation set, with second intervals in between. It had to predict the action in the last frame based on the preceding 29. For each sequence, we randomly created permutation of the 30 numbers i == 0, 29 and performed 30 runs. In run i, the predictions Úx()), where 0 < j assigned the probability N- In this manner made meaningless. In the table below, the first
uniform distribution, that is, each action an increasing random subset of the predictions column contains the number of frames for which the the classifier output (i.e., ÿ) has been reset and the second column shows the resulting mean Top-5 recall (MT5R).
The experiment was run over the entire validation set of EK 100. Figure shows that the accuracy steadily decreases from 18.8% to 16.3%
14

---

Published as a conference paper at ICLR 2026
19
AGA MTSR
18
17
10
15
20
25
# frames with randomized uniform predictions (out of 30)
Figure 5: Robustness analysis of AGA against the error occurred in frame prediction.
Overall Classes
Unseen Classes
Tail Classes Verb
Methods
Train On
Inference On
Action
Verb
Noun
Action
Verb
Noun
Action
Noun
GT
GT
GT
16.9
30.6
36.7
14.6
34.4
26.5
16.1
25.3
32,8
Self-Pred
17.1
31,9
36.0
14.1
32.9
26.8
32.2
Selt-Pred
ScIf-Pred
Self-Pred
(Top-1 One-Hot)
(Top-1 One-Hot)
(Top-1 One-Hot)
17.2
31.8
37.7
14.5
35.6
27.7
16.3
26.6
34.2
Self-
Self-Pred
(Top-1 One-Hot)
Self-Pred
(Top-1 One-Hot)
17.5
33.4
36.4
16.3
32.6
27.7
16.5
28.4
32.5
AGA
Self-Pred
Self-Pred
18.8
32.5
38.7
16.3
34.4
28.5
18.4
27.4
35.0
Table 13: Comparison between different action-guidane guidance signals for Q and K.
B.4 SELF-PREDICTION vs. GROUND TRUTH ACTIONS
AGA learns predictions ÿ that are probability distributions over the set of all action classes with loss function that measures the difference one-hot distributions Ya for specific action class as the target.
Intuitively, during its training, AGA strives generate predictions that ever closer to the one-hot ground truth. It is therefore natural to ask whether and how the performance of model changes if the predictions, which are generated resemble the ground truth, are replaced with actual ground truth distributions. Whether or not this is the case should give us more insight into how AGA works.
We created an experiment where we compared the use of the ground truth as opposed the self- predicted ÿ in query and keys. For frames with an action label we represented the ground truth with a one-hot Ya- For frames without label, occurring in background our transition segments, we used one-hot vectors representing generic background class.
For additional insight, we included experiments where we simplified the self prfe-preeeeted distributions to one-hot vectors ITI, carrying 1 for the action with the highest probability in ÿ. thus indicating the Top-1 predicted action, and this way aligning the representation to the ground truth format.
For both formats, ground truth and one-hot seff-prediction, ran two versions of the experiment: one where we used the revised format for both training and inference, and another one where we used it only for the inference.
We ran the experiments on the EK100 validation set. The results are shown in the Table 13 Here, "GT" stands for Ground Truth, that usage of the one-hot ya. "Self-Pred for Self-Predication. that is, usage of the S. and "Self-Pred (Top-1 One-Hot)" for the usage of the vectors ÿT1- The figures refer the accuracy 1K0 and the columns identical those in Tables Ind of the paper.
Tables[and222
The results show that AGA, which leverages the full self prediction distribution, achieves the highest accuracy. This can be intuitively explained by three factors:
15

---

Published as a conference paper at ICLR 2026
The full selff celffprediction carries richer intrinsic information than one-hot ground truth label (as further evidenced by the accuracy drop when binarizing the self- oeff-rrdiction into one- hot);
Self self-prediction still provides meaningful signals when no action is annotated during the observation window (for instance, background or transition segments);
Although GT provides the correct Top-1 future label, it lacks information about alterna- tive plausible futures, whereas the (imperfect but semantically informative) self self-prediction distribution better reflects the uncertainty and structure of the observations.
C MORE QUALITATIVE EXAMPLES
We provide additional qualitative examples from the EK100, EK55, and EGTEA Gaze+ datasets in Figures 11 where clips (a)-(d) illustrate successful predictions, with the ground-truth action revealed at the latest timestep. and clips (e)-(h) highlight failure cases in which the top-5 prediction of the model omit the target action. Figures are best viewed at full width and zoom in for full detail.
C.1 EPIC-KITCHENS- 100 (EK100)
Figure presents successful cases from the EK100 dataset. video (a), the scene evolves slowly and the washing activity is clear. The noun plate is also unambiguous, which allows the model to converge on wash plate, while alternative verbs such as hold, insert, and put are gradually ruled out. In video (b), the target action throw bin occurs between the third and seventh frames and is correctly identified by the model. Video (c) shows a case with little movement, where the verb prediction narrows to small set of consistent options over time, and the model ultimately selects the correct object. Video (d) is more challenging because the subject intent is unclear. The model nonetheless predicts coherent sequence: throw food followed by close bin, while assigning lower probability to open bin once the hand interaction with the bin becomes visible.
Figure iillustrates failure cases. In video (e). the verb predicted correctly but the noun mis- classified as pizza. In videos (f)- errors result either from insufficient visual detail ((f) or from ambiguity introduced by multiple plausible object candidates ((g) and (h)).
The 100 examples demonstrate that the model can handle everyday kitchen activities with gradual scene changes, but performance degrades when object categories are visually similar or when frames lack sufficient detail.
C.2 EPIC-KITCHENS-S (EK55)
Figure g highlights successful cases from the EK55 dataset, including challenging scenarios. In videos (a) and (b), the target objects (container and board:cutting, respectively) are not visible in the frames. Nevertheless, the model anticipates the actions take container and take board:cutting by leveraging contextual cues. Video (c) demonstrates the capacity to distinguish between similar actions, predicting put plate rather than take plate, based on the subject intention inferred from earlier frames. Accurate prediction requires retention of long-term evidence, which is supported by the adaptive gating mechanism and the AGA design. video (d), the ongoing activity of cooking pasta is anticipated, with the model predicting put-down spoon as the next action in the final frame.
Figure g presents failure cases where the model encounters ambiguity or insufficient context. In video (e), the object tofu is occluded, leading misprediction as container because the opaque surface prevents visibility of the contents. Video (f) illustrates noun misclassification caused by visual similarity, with olive confused for celery. In video (g), the verb is misclassified because the hand interaction with oil is only partially visible. In video (h), the model confuses chilli with visually similar objects such as tomato and pepper.
The EK55 examples highlight the advantage of AGA and adaptive gating for leveraging long-term context, while also revealing difficulties when objects are occluded or visually confusable.
16

---

Published as a conference paper at ICLR 2026
C.3 EGTEA GAZE+
Figure 10 provides qualitative examples from the EGTEA Gaze+ dataset, covering both successful and challenging scenarios. In video (a), the model captures subtle visual cues, such as the sponge disappearing from the bottom edge of the frame, which leads to the correct prediction of pul sponge. In video (b), the model predicts put container and links the container to the tomato by recalling formation from the initial frames, demonstrating the use of long-term memory to form object-action associations. Video (c) shows a case where observed movement suggests the verb take instead of the ground-truth verb put for the bread:container. Although direct evidence for returning the bread to the refrigerator is absent, the correct action still appears as the second-ranked prediction, reflecting the model strategy of coupling related verbs such as put and lake. In video (d), the model addresses a sequential scenario involving take contaiicr where one action visible and the other re- mains hidden. Accurate recognition requires identifying the container holding the cheese, which can only be inferred from the presence of its cover.
Figure I presents failure cases caused by insufficient evidence excessive ambiguity. In videos (e) and (f), the absence of salient details leads to incorrect predictions. Video (g) demonstrates noun ambiguity: the model predicts verbs confidently but assigns nearly equal probability to multiple visible objects. In video (h), the observation provides minimal context, leaving the model unable to determine the correct action.
The EGTEA Gaze+ examples highlight the model capacity to connect subtle visual cues with long- term memory, while also revealing the difficulty of disambiguating actions in situations with limited observations or an excessive number of potential object candidates.
17

---

Published as a conference paper at ICLR 2026
Time
open bin throx rubbish
squash
plate
check bas hang throw
put plate
fold cloth throw bin basit
scoup Sauce
#EX pasta
wazh tap
dry
pour Weildinessning
plate
drink beer throe Lura leaf check basil
Sc00p saucee
BIX pasta
wash hin
bin
vash
put
drink beer check Lon kitchten
pour 'd meshhin
scoop sauce
wash snsert takee plate
Food
fie sauce
wash spange
wash hend
Squeeie sponge
wash plate plate hold plate
drink beer a TH (Urt--n drawer
Ladle
put spatula
FiR
plate
take insertt clotites
pour sauce
Put tad
vosh sporge
uash pan
nove tao
wash
put
pour IQuidINse
wash plate put
plate
wesh throw bin pur. coMel
put spo0n
Nash spatu wash pan
put
seth tordrchwpping vazh pan Scoup food
with rork hold piate wesh tongs
cloth
cloth
put Tount hang elothes
open spocn
saatula
(b)
(c)
(d)
Figure 6: Four video clips from the EPIC-Kitchens-100 validation set, illustrating correct pre- dictions. Each clip demonstrates the last eight frames along with the top-5 action anticipations at Ta 1s, with the ground truth highlighted in bold green.
18

---

Published as a conference paper at ICLR 2026
Time
pizza LGT. take paper)
throw bin wach oven sear ch flour tiroe food drink beer ondon)
(GT put tongs)
Insert container
(GT: take plate)
put pizza take pizza
Take Tadtte
put fork
put spoon
dry tadle
take tongs
put
Insort cattery
take pizza insert pizza pizza
wish oven chrow Bin throw food take hab
move cont-iner lnerr
open container
put pizza
dry hand
put tadle
put kosfe
Insert
put pizza take pizza
wathh even wash cloth
throw food
squeeze cloth top
Take ladia
put spstula
close drawer
Insen containe container containet
close contairer
ripcaao
Insert container take container Rove open contaiine liieer
put pizza take pizza Intert renove
vosh oven throw food vosh top
take has
bin
Take ladle
put peeler
dry Taole
peripetto
Inwer conta lner insert contasiee close conteiine
dough
put pizza take pizza
wash even wesh top wash hop throw food put hob
close drawer
pul cutlery
take lodle
insert
put
Insert cuttery
put not fold cloth sordechopiin take
tale pizza put pizza insert Fenoo
wazh oven
wish top put hob
wesh heb
take tadle
close draner
Lass
Insert conta iner beard chopsin move contaiter insert toHell
cloth
close drawtr
put pizza Take pizza pizza
Mash oven vesht top
wazh hob
take pizza
take Laffie
put drawer
put
clase drowe tale Ladte put drower
opply Live put pizza
wish oven wesh top
throw Tood take pizza hob
nowe glass (h)
catlery
rat
(e)
(
(g)
Figure 7: Four video clips from the EPIC-Kitchens- validation set, showcasing incorrect pre- dictions. Fach clip demonstrates the last eight frames and the top-5 action anticipations at 1s, with the ground truth revealed the final frame. The correct verb and noun are highlighted bold green.
19

---

Published as a conference paper at ICLR 2026
put -dowi certel tale board seerdiicu cutting Lake kosid cereal
di--d spoons take sauce:: nove sauce-toowe toroto
Time
closs fridge close drawer take contatier tain mushroo
put plate take D plate clowe cloo open draver
stir pasts
close fridge Takos aushroos suchroos opent
Lake knu fe take nushroon con Lurneen Lep put russ
Laker plate put plate closs doer open
511r porto ta put do--doonn 5poon
soice:
put pasta
take ni milk:ooy take rustrroo close put slive squazh
Lake sushroon put Rushioson Lake iud le open bos
Lake plaue close door pul plate open door
But Prt-ddow spotn
$L.1 pasto
put pesta
pasta
Leke mush hor clese box bou:ceere ca open box San sppe sustrocs
open fridge Lake ni hissy down ritcchon
vot--inan spoon
doon cuttintnnnn take plate
openl Lap
open door
plate
stir pasts
put parta
cake sauce!
enterr estcceen
open boxt DoR sunuaaee close box cereel
Lake koile olive cutte
pitk- strairertien
opann Fridge
turn turn-ca down
put pasta
Kole pasta
stir pasta
put -o-d
openi Lep open bin
take
Lake kni fe tur--on too pour op Livvid
bourdrcuttre
take saices tooaato
take corriccutting
opan bin Lakee reg take
open Lop se5th plate
pul pla ae
stir pasts
put pasta
rove casta
fraicle:cres
open Ldge put-stown bog bin kitchhn
open door take snsfe turn on top dry drawer
Pet plate opcn t3o open 62
opcn dbor
stir pasta pus pais Enve pasta
Live
soue parta sauce:tonato
enter sutchen put dom bog hang 20.rrt fradge
drink water put lass put
LUT oniion Take kottle Ladee I pan
st1r pasta put pasta
Figure 8: Four video clips from the EPIC-Kitchens-55 validation set, illustrating correct predic- tions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth is highlighted in bold green in the latest frame, occurring at Ta 1s.
20

---

Published as a conference paper at ICLR 2026
put--oww celery)
spriakte adl)
Time
close fridge put aushroor
take container put contesner move conteiner
openi put sushroox pu tot-o alive take oiive
take oitzolive
put vinagar
put-dtae pour tonsto
stice pegper.
011
open o1l
(GT:
resoue band: band:rubber
sprinkle nozarella Str.sinnnn Lnkte Kozzarella
open bun put-com onion throw skin
tate ailsolive
close fridge close draw2r
put sushroonn
take mushrson
put vinegar
torato
pour e1l
putt-lon
sprinkle put-doon put-doun take conoto
put Lid
close fridge
cake container
open drawer
close drawer
take moo
skin
throw rubbissh
throw leftower
put--down oil pour c11 put vinegar
open bin
pour tossto
take glive put down Live put Kuchroom
throw ruboish
take citio Live put out-down pappoir
mozarette spr.ikle nozzarel sp #ozzare
Lake sushroos
clese tri8ge
throw open sin
throe bin
open sale
opread pesto
take milkssoy put down alive
put -dian squaeh
put container
open bin throw bin throw ruRbish
take silralive
open fridge
open T1l pour o11
get vinegar
sprinkie sun put Lid
throw open tin cleas
throw bin throw leftover
open fridge enter kittdhi
take lisssoy turn on Light put -own olive
take ailcolivve opin oil putt feert close doar
pour tometo
put 1id
starrchfor for paprika
aozaralla putt-cow bec Sprinkte sun
enter kitcienn
open fridge
opcn bin put- -o--don takee onionn
dry hand open drower
sprinkle onion
dry hand
open door open bin
close door
put berb
sear search--e rch-fo- paprale lk put neggu
enter kit thte put doun bin open door
take take nushrocn put cown onion close container put mushroon
sprinkte onion
sprinkle spriinte sprinkte sun
bin
dry hand
put here
clase
(h)
Figure 9: Four video clips from the SPICC-ittheen-55 validation set, showcasing incorrect pre- dictions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth highlighted in bold green in the latest frame, occurring at Ta 1s.
21

---

Published as a conference paper at ICLR 2026
Time
Reah eat.ngo otomaa
But tonate OUR tol
Frr c21 ceneiett CnEEentt contatte
Tele bread alne bradd
Tale 10rappi anstiaartaaie ratttaaeer opel creadrccm
fridp dtreeer
Take cheese OO
Tele Lestuce Take cicumber
Close Fridge
Nesh bows
Mealt
Take bread contelnce Take 30 contaler
Take chetr contatner
Serecce wathin
Hash eyzingiitess
05cn Tridge Closs fridge
bar Trigg
cuass:cmta
Take choese
sponge
Aut LER
Aul cieeee
Put bowe
Nash
Squeeree washing Ligui
Open restee draee taputo-con contttr
"ale broad rradiontte ontalee alner fut eveedd Doon Tridge fut chetso cessoionttaeee centaier
Take mmnracctntaa Close fr fridge Pat theneesonn
Open Tridge Upen drower
Take cheeze
Nach Iand
Nooli bowt
Siiecoe saoking layia Sqasie woohing Liuid
Nosh cutt:ngtboar Tur on raucet
Cloze fridge
Open fridge meatoicontaanne
Pot knerd: cont
2pmn tradgerrroo
Close fridge This breed: cotaiico
nut bread: irasttaaaain
Nut ondimeliontaal
cottaizer cheese liee receieneeee
fridge
N:: maeeerccaaiii eanterrrr
Take ciosse
Mosli bowe
Eripect/tod nee tne
Take breat medd-ratten rotta her
Take meeee-cottttee
lace invileaiion coniintiooaaie
Prt trass:cstaaiie
Open fridge
Close fridge
Put tosato
Close fridge
auttrnmsnn Open fridge
Take sporge
Hosli boot
Take
Moourt/te "ec0s tosato
Take breati kreadi cantaine Open fridge
Close fridge Tale te0300 to ontaaier
Tele cheracco concceettation aetteeee Take onion
ine
Take sponge
Put sponge
nash sowl
up9
hoe cppliantic
Came Tridee
Turn
Mesii
Turn faucet
Take tratercnntiee
Open fridge Ni arod aotaaa
Put ondimm woe.aatceeettt contatee
Tar 2n petacnes
PV conetccoonn www.tccconattten wowcoontattte
Take Traat cnsteirsnttioo
Put. soongee
Bash bowl Jasn hand
Take torato
Take egg Put tonato
Tura faucet ingrsteeesss
Put sponge Was catunguttmma
1ae teate: contimur Take egg
All cormttaan tester
broadtcortatee
Pur dissnioorttToo
Pe: Lopate: container
Take preidicontal containg
Take torato
Put tonato
tridge
tridge
(8)
(b)
(c)
(d)
Figure 10: Four video clips from the EGTEA Gaze+ validation set, highlighting correct predic- tions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth highlighted in bold green in the latest frame, occurring at Ta 0.5s.
22

---

Published as a conference paper at ICLR 2026
Time
hove Around pot.
pan)
Take taaat OO leer
(GT: Put tettuce)
1Ne eatinn on conaaatt lake cooking
[OT: Take bowl]
LGT www.eettonnateeee..ooo
Take grecorytigg
Open tradge
Take tosato
Pour cand rset
hit aLooot
Mach pot
Put pot
Take Ran
Clna or ciettitettteeee cocentatrr Soat
Turn faucet
Sove Around pct
Put tomettoinntaan
Talo asanning Inppet//ol recus
Clos aitrer aaleer Air 611:300tston foee
Dour comn Leant
Close "ridge
seasonung
Open triege
Kash pet
Put pot
Upen fridge aomatte TH dea tn aopponat contie for sesc0ning
Cles GoGIee conieeriionn conturr
Move Around pat
Open frisge cud Contoomun
Pst araaccaatauu
Clos ailcorrate
Pour condiment
Mash pot
Put poc
Citzt 1812 inrreittiaaainn taleer Mide COnEMMOt ooraure
taucet
Hove Around pot
Opan fridge eattacn fot brwndcontaimer concontttte thesee wwwecoontaiin close fridge
take 300 notirontaae Tuke sat:containes
Pour sessoning
Pour condiment
2ut atl: cottsine
kash pet
Put pat
Mix pasta
Weth tauces
Hove around pot
P.L br proddoo crsdzontauee aatu
Take toaate: (osatt) containdr
PUS tomits conticcantttee
Take stannng contatte
Mouting cater
Teis oslicintet Pusect/ooal colue
Pour condinent
weser
Open Tridge
Rit otlcomaaan
Put pot
pasta
tauces
Nove Atround pot
Doon fridge close fridge Put (horse conta Put breadicintaa Eeer
Cloce ailzaraaie cater
Pour soasningg
Thke stlcoonttor
Pour cand Irent
Wazh par
Put pot
Clee tngiam::
MDx sixture
Turn faacet
Nove Around pot
Open Fridge cos cnstentio cottoner
Sper Tradeeedn cruue Insject/Raad
hit sesserr
Pour condiment 1iz Ond ioleetttioooaa Aalicentaln
Fut altrooouurr
NiX nixture
Put pat
Tae 101
MAX pasta
Put crash
Wesh
Tur eff foucet
Meah Move Around pot Pot pot
opan rridge open drewer Open Put grocerysion
nu sestetts Corraser Pour seastnning
recicie conernseoooe
rlss unttentton comtane
Pour condirent
he altrentting
Pair ell
Put tresh
Mix pasta
Figure 11: Four video clips from the EGTEA Gaze+ validation set, showcasing incorrect predic-
tions. Each clip displays the last eight frames and the top-5 action predictions. The ground truth is highlighted in bold green in the latest frame, occurring at Ta 0.5s.
23

---

Published as a conference paper at ICLR 2026
D LLMS USAGE
The authors utilized LLM powered Al tools (GPT-5 and Grammarly) to proofread sentences and identify grammatical errors.
24


Reviews:
[{"review_id": "HtNnJH5qha", "reviewer": "Reviewer_3ZKi", "summary": "The paper proposes Action-guided attention (AGA) for addressing the task of video action anticipation. Instead of relying purely on visual features, AGA considers a transformer-based architecture that makes use of the estimated action labels' sequence. Specifically, queries are computed based on the exponential moving average of all previous actions, while keys are constructed based on the estimated action labels within a temporal window $ S $. Values are based on the frame embeddings computed based on a frozen frame-based backbone network. The paper also proposes an adaptive gating mechanism that adaptively fuses past and current evidence.", "questions": "- Can the authors provide clarifications regarding the timesteps/sequence notation and the way anticipation is achieved?\n- Have the authors considered the use of AGA for third-person and/or 3D (Mocap) action anticipation tasks?\n- How is the performance influenced if ground-truth action labels are used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of using a transformer-based architecture that considers the sequence of the observed action labels, in combination with the visual features, is interesting and reasonable. Surprisingly, except maybe from [R1], which follows a similar approach for long-term action anticipation, to my knowledge this direction has not received much attention yet.\n- The adaptive gating mechanism is also a reasonable addition, considering that the duration of the actions varies significantly and that the proposed method does not explicitly handle the estimation of the action duration.\n- The proposed method shows improved performance regarding several strong baseline methods on three widely used egocentric action anticipation datasets. It would be interesting to discuss also the suitability of the proposed method to third-person and/or 3D (Mocap) action anticipation tasks.\n\n[R1] Gong et al., \"Future transformer for long-term action anticipation\", CVPR 2022.", "weaknesses": "- There are some important clarity issues that make the comprehension of some crucial methodological aspects challenging. Specifically, there is some confusion between the use of the terms \"timesteps\" and \"frames\", as two separate notations are used ($ i $ and $ t $). The ambiguity is also because these two indices seem to refer to different quantities, i.e. $ i $ as frame sequence index and $ t $ as timestep (e.g., in seconds similarly to $\\tau_{\\alpha}$). \n- The text does not explicitly discuss how anticipation is performed. The ambiguity of how the anticipation is performed is closely related to the issue discussed in the previous point. For instance, in case $ t $ refers to seconds, it is not clear how the embeddings $ e_t $ are obtained from *multiple* frames that correspond to the interval between $ t-1 $ and $ t $. On the other hands, if $ t $ also refers to the frame sequence index (most probably), it is not clear how anticipation is performed as $\\tau_{\\alpha}$ corresponds to *multiple* steps ahead, which is not explicitly discussed in the text.\n- Some additional experimental analysis would be helpful. This includes the length of the queue size $S$. It would also be interesting to show how the proposed method's performance changes when the ground-truth action labels are used for anticipation.\n\nMinor comments\n--------------\n- It would be helpful to include a list of contributions in the introduction.\n- L.319-321: \"Nevertheless, empirical results show that AGA maintains strong performance, even under constrained annotation availability and sparse supervision.\". It would be interesting to provide more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GbtrhQvBnn", "reviewer": "Reviewer_f8ru", "summary": "This paper proposes Action-Guided Attention (AGA), a novel attention mechanism for video action anticipation that uses predicted action probabilities as queries and keys, rather than pixel-level features. The core insight is that action-level semantics provide more effective guidance for attention when dealing with the inherent uncertainty and visual clutter in anticipation tasks. The method includes an adaptive gating mechanism that balances historical context and current evidence, and enables post-training analysis via forward and backward analysis techniques. The paper demonstrates strong competitive performance on EPIC-Kitchens-100 and other benchmarks, with clear ablation studies and strong generalisation.", "questions": "1. What is the exact stopping criterion (e.g., L2 norm threshold, maximum iterations)? How sensitive are the results to the step size η and this stopping criterion?\n\n2. Concerning the EMA parameter: what does α=0.8  as an optimal parameter mean in terms of the effective temporal window or the balance between recent and historical information? Is this \"memory buffering\" optimal value dataset-dependent?\n\n3.  Computational cost and error propagation: How do the computational cost and inference time of AGA compare to baselines? \n\n4. Initialisation and relationship to avt: Two points of clarification: (a) At the start of a sequence, how are queries and keys initialised before the queue is filled? (b) A more detailed architectural comparison clarifying the exact relationship between AGA and the AVT baseline would be helpful.\n\n5. When does AGA perform poorly? Are there specific types of actions or scenarios where the approach struggles?\n\n6. What are the theoretical conditions under which action-guided attention should be more effective?\n\n7. How does this choice relate to information-theoretic principles or representational learning theory?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, this paper presents a novel, well-motivated approach to action anticipation, achieving competitive results and valuable interpretability. \n\n1. The core idea of using action-level semantics to guide attention is conceptually sound and addresses a real limitation of existing methods. The experimental validation is solid, with strong generalisation demonstrated across multiple benchmarks.\n\n2. The interpretability analysis offers valuable insights into model behaviour, with forward analysis revealing attention patterns and backward analysis providing counterfactual reasoning paths. \n\n3. The paper demonstrates competitive performance on EPIC-Kitchens-100 (17.5% MT5R with TSN backbone, 18.8% with Swin-B on validation), with test set performance of 16.9% MT5R (Swin-B), showing strong generalisation with a narrow validation-to-test gap (approximately 1.9 percentage points). \n\n4. The ablation studies provide clear evidence for the contribution of both action-guided attention and adaptive gating components: replacing standard causal attention with action-guided attention provides substantial gains (15.9% to 18.2% MT5R), and adaptive gating adds further improvements (18.2% to 18.8%). \n\n5. The method is also validated on additional benchmarks (EPIC-Kitchens-55 and EGTEA Gaze+), demonstrating robustness across different datasets and annotation regimes, including sparse supervision settings where the model must rely entirely on its own predictions.", "weaknesses": "1.  Theoretical grounding: The paper provides strong intuitive motivation for using action-level semantics over pixel-level features, but lacks a deeper theoretical analysis. A more formal justification would significantly strengthen the contribution.\n2.  Backward analysis convergence: In Section 3.4, the convergence criterion for the gradient descent in the backward analysis is stated imprecisely as \"until the sequence doesn't change any more.\" \n3.  Ema parameter insight: While Table 6 provides a thorough empirical evaluation of the EMA parameter α, the paper lacks insight into why α=0.8 is optimal. \n4.  Statistical significance and reproducibility: The paper doesn't report standard deviations, confidence intervals, or the number of runs. Providing statistical significance testing for the key comparisons (e.g., the improvements from ablations and the validation-test gap) is crucial.\n5.  Missing experimental details: The size of the FIFO queue 'S' is mentioned as a hyperparameter but not specified in the experimental setup. \n6. Furthermore, since AGA relies on self-predicted actions, a more explicit analysis of how prediction errors propagate and how the model handles error accumulation over time would be valuable.\n7.  Failure cases and limitations: The paper doesn't discuss limitations or failure cases.  \n\nMinor comments:\n\n*   Table organisation: Reorganising results tables to explicitly group RGB-only comparisons versus multi-modal/ensemble methods would improve clarity.\n*   Related work: The related work section could better position AGA relative to methods that also use semantic or high-level features (e.g., S-GEAR). How does AGA differ from these approaches?\n*   Notation and presentation: There are minor inconsistencies in notation (e.g., line 147) and formula placeholders that should be corrected for the final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jHvrJKKOUl", "reviewer": "Reviewer_XXT2", "summary": "The manuscript proposes Action-Guided Attention (AGA), a semantic attention mechanism for video action anticipation. Instead of relying on pixel-level similarity, AGA uses predicted action distributions as queries and keys to focus on temporally relevant past frames. An adaptive gating module further balances historical and current visual cues. The authors also introduce forward and backward analyses to interpret the model’s learned dependencies. Extensive experiments on EPIC-Kitchens-100, EPIC-Kitchens-55, and EGTEA Gaze+ demonstrate consistent improvements over prior methods.", "questions": "1. The paper states that past actions are treated as an unordered set. How does the model still maintain temporal consistency?\n2. Were any techniques applied to reduce frequent-class bias? How might uncalibrated probabilities affect the quality of semantic attention on long-tail classes? If none were used, could you discuss why AGA still improves tail classes and whether calibration might further help?\n3. AGA is tested with both TSN and Swin-B backbones. Did the authors observe any difference in how semantic attention behaves with different backbone architectures? For instance, does a stronger backbone reduce the relative performance gain of AGA, or does the improvement remain consistent?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript proposes a novel approach that leverages the model’s own predicted action distributions to guide attention, enabling it to focus on high-level contextual cues rather than low-level visual similarity. This design offers a fresh and effective perspective for modeling in action anticipation.\n2. The method shows consistent improvement on tail and unseen classes, indicating enhanced robustness across class distributions compared to prior work.\n3. The proposed forward and backward analyses offer interpretable insights into how the model captures causal dependencies in its high-level representations, illustrating meaningful internal reasoning.", "weaknesses": "1. The model uses its own predicted action distributions as inputs (queries and keys) in the attention mechanism. While this design enhances semantic reasoning, inaccurate early predictions could propagate errors through time due to the recursive dependency. The manuscript introduces an exponential moving average to smooth the predicted distributions and stabilize temporal dynamics. However, EMA may only partially mitigate the potential error accumulation caused by this recursive dependence rather than resolving it. It would be helpful for the authors to clarify whether any mechanism is explicitly designed to address this issue, or if they have other considerations regarding how such recursive errors are handled during training.\n2. The authors explicitly mention that AGA treats past actions as an “unordered set.” Without explicit temporal or positional encoding, the model might struggle to capture fine-grained causal order, which could limit performance in tasks where action sequences are strictly ordered.\n3. On datasets such as EPIC-Kitchens-100, the class distribution is highly imbalanced. Since the model uses predicted action distributions as semantic inputs, it would be helpful to clarify whether this design could potentially amplify frequent-class bias or, conversely, help mitigate it through semantic attention. But the reported tail-class gains remain unexplained, and there is no discussion about this aspect. Please discuss why AGA helps rare classes.\n4. The paper does not report quantitative efficiency statistics. Providing these measurements or comparisons with existing approaches would strengthen the empirical analysis and clarify the computational cost of AGA.\n5. The evaluation focuses exclusively on egocentric kitchen datasets (EPIC-Kitchens and EGTEA), which limits the demonstrated generality of the proposed approach. It would be valuable to verify whether the Action-Guided Attention mechanism generalizes to more open or diverse domains.\n6. The ablation study could be more fine-grained. While the paper evaluates the effect of the EMA coefficient and the presence of the gating module, it lacks comparisons between different gating architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YChobGKh2b", "reviewer": "Reviewer_TEVQ", "summary": "This paper focuses on the task of long-term activity anticipation in videos. The main technical contribution is the advocacy of using model frame-wise raw output probability as feature / input to another attention process, where the self-produced predictions serve both as query and key, meanwhile the frame embedding as the value. Moreover, standard attention for long-range (temporal) modelling is identified as sub-optimal in this paper, and authors instead propose to use standard temporal smoothing, i.e. EMA and adaptive filtering, i.e., gating, techniques in the model for better performance. Strong experimental results are demonstrated to support the method.", "questions": "A. How is the past predictions, \\hat{y}_{t-1: t-s}, produced? The reviewer guess that during training, the model is unrolled over S steps and computes each \\hat{y} auto-regressively, feeding those predictions into the next step’s queue. Please confirm the details.\n\nB. Add an equation for producing \\hat{y}.\n\nC. This reviewer believe the following work on acitivity anticipation are revalent for discussion.\n- When will you do what?- Anticipating Temporal Occurrences of Activities, CVPR 2018,\n- Time-Conditioned Action Anticipation in One Shot, CVPR 2019,\n- On diverse asynchronous activity anticipation, ECCV 2020,", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "A. The writing is smooth, math easy to read and overall clarity is good. The experimential results are strong across 3 different datasets for activity anticipation, which demonstrate the generalization ability of the proposed approach. The overall approach is quite simple and easy to implement.\n\nB. This reviewer has found the AGA design interesting, though less well-structured / supported throughout the documents. The way AGA works is: Action prediction (distribution over all predefined classes), \\hat{y}, is the self-produced outputs and it then works on the incoming frame-level embedding (pre-extraced) progressively along the time axis for next-step prediction. This core design bears a lot similarity to the autoregressive cross-attention~ish fashion, which, however, differs most critically in the information abstraction level - the AGA module works on the semantic level abstraction, e,g., re-use the raw probability prediction. A summary of Q/K/V design against the orignal applications (transformers) is as follows.\n\n| Model                                  | Query Source             | Key/Value Source                                       | Category                                       |\n| -------------------------------------- | ------------------------ | ------------------------------------------------------ | ---------------------------------------------- |\n| **Transformer Encoder**                | Same input sequence      | Same sequence                                          | Self-attention                                 |\n| **Transformer Decoder (e.g., in GPT)** | Generated tokens         | Encoder outputs                                        | Cross-attention                                |\n| **AGA**                                | EMA of predicted actions | Past actions (semantic) / past frame features (visual) | **Semantic-Visual Cross-Attention** |\n\nThe intution of transformer, by the best knowlege of this reviewer, is that re-weighting the “Value” vectors using relationships between “Query” and “Key” vectors. In this case, AGA is re-weighing the past frame visual representations using the relationship between semantic query and key vectors, which fits the demand of this paper. However, this leads to the 1st weakness point in the next section.", "weaknesses": "A.  It remains unclear what exactly make the AGA working for this task, and hinder readers summarize useful take-away message from this paper. Here are some questions to start the discussion. \n\n- According to the statement, quote \"only rely on dot-product attention over pixel representations...\", it seems the key finding here is that information in visual tokens being too preliminary, and the semantic feature is better. Yet, isn't \\hat{y} derived out of the pixel representation, e, in the first place through a output head, f_{\\theta}? So, what is the magic here empowering \\hat{y} with stronger information after some MLP projection?  Or say it in another way: if the pixel representations already encode the necessary semantics, why does the paper inject predicted logits (the model’s own action scores) into attention — instead of operating on the hidden-state space directly? A quick extra experiment using last-layer hidden states might shed lights on this matter.\n\n- The paper argues that visual cues might make the model prone to over-fitting. This reviewer would doubt if that's still true after the second encoder, f_{x}, especially after a few epochs of training. This argument is never supported with experimential study. A good one could be monitoring the validation set loss under two different settings - one with semantic feature and another one with visual features along training epochs. \n\n- The cross-attention alike mechansim in AGA seems unpredecent from previsouly published work. AVT (Anticipative Visual Transformer) can be seen as the canonical autoregressive transformer baseline, which only tested self-attention on visual tokens, not the cross-attention over past visual features and the next incoming frame-level feature. Maybe a study of a direct comparison where Q = current visual embedding and K/V = past visual embeddings (true autoregressive cross-attention)  would isolate the benefit of using semantic (action) representations from the benefit of directional cross-attention.\n\n- The EMA and gating techniques can be implicitly achieved using attention. Prior-driven technique are powerful but less generic. It seems the main motivation here is resolving the training difficulity. Replacing generic module with prior-driven one does not seem to be a scalable method. Maybe the authors can share the failure logs (training logs) of using learnable components (such as pure attention). The community might come up better optimization idea for this issue. \n\nB. The training recipe is missing quite a few important information for re-production purpose:\n- frame sampling rate\n- queue initiation - It’s unclear how the queue is filled at the start of a clip (zeros? repeats? skip first S frames?).\n- Any data augmentation? \n- EMA update scope - Is EMA applied to the raw logits or the softmax output (normalized prob.)?\n- Again on the past prediction, y_{t-1: t-s}, are they also being optimized with the cross-entropy loss? If so and they are produced auto-regressively, are the BTPP algorithm used to optimized a sequence of inter-dependent predictions or separately?\n- Prediction head architecture - the exact dim and layers of MLP here.\n\n\nC. The post-training analysis study is a bit pre-mature. The Forward and Backward analysis are interesting and visually compelling but lack methodological transparency and quantitative validation.\nKey implementation details (which layer, normalization, gradient target) are omitted, and no formal evaluation is conducted to confirm that these analyses genuinely capture causal or influential dependencies. This reviewer would not think of it as a well-rounded interpretability study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)