Action Sequence Augmentation for Action Anticipation
Authors: Yihui Qiu
Deepu Rajan
Venue: ICLR 2025 Poster (2025)
Keywords: Sequence Augmentation
Action Anticipation
Context-free Grammar
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Action anticipation models require an understanding of temporal action patterns and dependencies to predict future actions from previous events. The key challenges arise from the vast number of possible action sequences, given the flexibility in action ordering and the interleaving of multiple goals. Since only a subset of such action sequences are present in action anticipation datasets, there is an inherent ordering bias in them. Another challenge is the presence of noisy input to the models due to erroneous action recognition or other upstream tasks. This paper addresses these challenges by introducing a novel data augmentation strategy that separately augments observed action sequences and next actions. To address biased action ordering, we introduce a grammar induction algorithm that derives a powerful context-free grammar from action sequence data. We also develop an efficient parser to generate plausible next-action candidates beyond the ground truth. For noisy input, we enhance model robustness by randomly deleting or replacing actions in observed sequences. Our experiments on the 50Salads, EGTEA Gaze+, and Epic-Kitchens-100 datasets demonstrate significant performance improvements over existing state-of-the-art methods.

Published as a conference paper at ICLR 2025
ACTION SEQUENCE AUGMENTATION FOR ACTION AN- TICIPATION
Yihui Qiu & Deepu Rajan
College Computing and Data Science
Nanyang Technological University, Singapore {qiuy0007, ASDra jan @ntu. weatuu.edu.... edu.sg
ABSTRACT
Action anticipation models require an understanding of temporal action patterns and dependencies to predict future actions from previous events. The key chal- lenges arise from the vast number of possible action sequences, given the flexibil- ity in action ordering and the interleaving of multiple goals. Since only a subset of such action sequences herent ordering bias in them. Another challenge is the presence of noisy input to the models due to erroneous action recognition other upstream tasks. This pa- per addresses these challenges by introducing a novel data augmentation strategy that separately augments observed action sequences and actions. To address biased action ordering, we introduce a grammar induction algorithm that derives powerful context-free grammar from action sequence data. We also develop efficient parser to generate plausible next-action candidates beyond the ground truth. For noisy input, we enhance model robustness by randomly deleting or re- placing actions in observed sequences. Our experiments on the 50Salads, EGTEA Gaze+, and Epic-Kitchen BeiicKKichhens-ee 100 datasets demonstrate significant performance im- provements over existing state-of-the-ard methods.
present in action anticipation datasets, there is in-
INTRODUCTION
Action anticipation involves interpreting past events to predict future events in domains such as rohotics/Antonuca et al. (202T) and surveillance Duque al. (2007). This capability mirrors human foresight, enabling systems to predict future actions based on observed behaviors, thus allowing for proactive responses.
Predicting next action from observed action sequence is complex due to three main factors. Flex- ibility in the order of actions allows certain actions to occur any order, e.g., cut tomatoes, cut cucumber, and cut cheese in ingredient preparation. Interleaving goals, where multiple goals pursued simultancously, can result a sequence of potentially uncorrelated actions. For example, in the preparation of coffee and cereal, actions for make coffee, and make cereal in multiple valid orders, including in an interleaving manner. As illustrated in Figure [ the goal of make cereal can occur before the goal of make coffee (as sequence al), after (as sequence a2), or interleaved with the goal of make coffee (as in sequences a3 and a4). However, in practice, an action anticipation dataset typically captures only subset of these possibilities. which can lead to the learning of biased action orders, causing the model to favor specific action sequences and failing generalize to unseen or less frequent action patterns. Thirdly, action anticipation models often en- counter noisy inputs, which can arise from misclassified actions, or imprecise temporal annotations, which complicates the task by introducing uncertainty and variability action prediction.
be arranged
This paper tackles the above three challenges by generating new action sequences that, when inte- grated with the original training sequences, broaden the range of possible action orderings, thereby reducing bias and enhancing predictive accuracy. Sequence augmentation has been used in natural language processing Hou et al. (2018) and speech recognition Nguyen et al. (2020) to boost model generalization and mitigate over wver-fitting. Such techniques include synonym replacement, random insertion, deletion, or shuffling of elements. More recently, sequence augmentation has been used in video understanding Prananta al. (2022): Lee al. (2022): Falcon al. (2020), employing

---

Published as a conference paper at ICLR 2025
a1:
pour corcal bowl
pour inta
carcal
Take cup
Pour into cup
82:
Taka
Pour coffee into cup
Take bowf
pour cercal into bowl
pour milk into bowl
Stir corcal in bowl
83:
cup
into bowl
Into bow!
84:
Take
Take bowl
pour milk into bowl
pour cereal into bow!
Pour coffee into cup
Skir cereal in bowl
Figure A subset of valid action sequences for make coffee (red) and make cereal(
methods like temporal stretching, frame interpolation, and mirroring of video sequences. Despite its success in other domains, augmenting action sequences presents unique challenges that require maintaining logical and temporal coherence to ensure that the augmented sequences remain realistic and meaningful. An effective sequence augmentation strategy for action sequences has yet to be thoroughly investigated.
We introduce novel sequence augmentation strategy, called ActSeg, that separately augments the observed action sequence and the predicted next action. We first propose a novel grammar induction algorithm that extracts context-free grammar from action sequence labels. This grammar captures the order between actions, guided by the objects and their interactions within the sequence. We also develop modified version of the Earley Parser Earley 1970)) that efficiently generates diverse next action candidates across multiple goals, adhering to the grammar rules and the observed data. Additionally, enhance the model's robustness by borrowing proven effective augmentation technique from NLP of random modifications - deleting, or replacing actions - in the observed action sequences. The experiments on three benchmark datasets show that our approach effectively reduces action ordering biases and improves on the state-of-the-an action anticipation models' per- formances.
RELATED WORK
Action Anticipation. A variety of machine learning models have been explored for action antic- ipation, each leveraging different aspects of temporal and spatial data processing. Convolutional Neural Networks (CNNs) Abu Farha al. (2018). 9 long appreciated for their ability handle spatial features, have been adapted to address the temporal dimensions of video data. Among these adapta- tions, we focus on RULSTM Furnari & Farinella (2020). which combines recurrent neural network capabilities with CNNs to enhance temporal understanding through state retention over time. Re- cently, tranformer-bas
models such as AVT Girdhar & Grauman] (202I), RAFTformer [Grasse et al. (2023) and MotionFormer Patrick et al. (2021) incorporate attention mechanisms that weigh the relevance of different temporal segments. Zhao & Wildes (2020) uses Conditional Adversarial Generative Networks (CAGNs) jointly anticipate long-term activity labels and their corresponding times.
Sequence Augmentation is methodology that involves artificially expanding the diversity of train- ing datasets by altering existing sequences or synthetically generating new ones, thereby helping models to better generalize across unseen data and mitigate overfitting. Historically, sequence aug- mentation has been extensively applied in natural language processing (NLP) P) Li et al, (2022): Feng
al. 2021 Techniques such as synonym replacement Jungiewicz & Smywinski-Pohi (2019). ran-
dom insertion, deletion, or shuffling of words (or phonemes in speech) Wei & Zou 2019 have been standard practices to inflate text and speech datasets. These methods have proven effective in improving the performance of models on tasks ranging from language translation to speech-to-text recognition. Recently, sequence augmentation has seen innovative applications video processing Yun et al. (2020); Kwon et al. (2022), such temporal stretching [Pannanaa et al. (2022), frame in- terpolation Lee et al. (2022), mirroring Falcon Falcon et al. (2020), and generation with Monte-Carlo Tree Search (MTCS) Aziere & Todorovic (2023) of video sequences. To the best of our knowledge, a formal approach to sequence augmentation specifically for action anticipation has not yet been established. [Mittal et al. (2024) use lideo-language model that enhances the generalization of anticipation models by learning plausible future actions. While large language models (LLMs) ex- cel in scalability and adaptability, they are prone hallucinations, In contrast, grammar-based
2

---

Published as a conference paper at ICLR 2025
Video
Obeerved
Next action
Observed act
Next action
Observed action
actions
training sample
Augmented next action
Augmanted observed
3X training samples
The proposed action sequence augmentation strategy ActSeg initiates by deriving video grammar from an action sequence using the Action Grammar Induction (AGI) algorithm. Subse- quently, the Cross-Tree Earley Parser (CTEP) predicts alternative next actions based on the observed actions and the video grammar (the figure shows only one augmented sequence). These predicted actions combined with the original observed actions to compose random modifications (RM) of observed sequences are applied to generate additional training sam- ples. In this example, the training set expands to three times its original size.
Figure 2:
training samples. Finally,
method excels in interpretability with explicitly defined and traceable rules. Appendix A compares predictions from representative LLMs and our method.
Grammar for activity analysis. Grammar, as fundamental tool for representing the composi- tional structure of language, has been extensively explored in the realm of NLP Ginsburg (1755; ; Seki et al. (199). Beyond textual analysis, grammar's application in action anticipation was first explored by Pei et al. (01T) and et al. (201)) who utilized AND-OR grammars learned from strings of symbols, each representing an action according to the grammar's 's defined language. These early models, however, were limited to deterministic inputs and lacked flexibility. Vo Bobick (2014) extended this approach by introducing stochastic context-free grammars that could process probabilistic sequence inputs, enhancing the model's 's adaptability to more realistic, variable data sce- narios. More recently, Qi et (2020) employed the ADIOS grammar induction algorithm derive grammars directly from action corpora, marking a significant step towards automation in grammar generation for action sequences. |Piergiovanni et al. (2200) proposed differentiable grammar mod- els that learn sequential dependencies from the action sequence data. Gong et al. (2024) recently introduced a novel grammar induction algorithm that emphasizes key actions and temporal depen- dencies, incorporating recursive temporal structures to better mirror the repetitive and intertwined nature of human activities. Dessalene al. (2023) proposed mule-based, compositional, and hierar- chical approach to modeling actions from elemental motions, such as 'grasp" and 'release'. Despite these advancements, many existing grammatical frameworks either require hand-crafted rules, or depend on the limited training data that are biased in the order of actions.
ACTION SEQUENCE AUGMENTATION
Given a video an action sequence, training samples for action anticipation are typically generated using M + continuous actions from the sequence. Here, the first M actions form the observed sequence aobs = [a1, ......, aM) and the (M + 1)th action is considered as the next action to be predicted, apred a
In sequence augmentation for text classification, e.g., Wei Zou (011)), augmentation is applied to training sample (X, y), where X is the input sequence and is the output class. In our case, X rep-
3

---

Published as a conference paper at ICLR 2025
resents the observed actions, and is the next action. Our proposed action sequence augmentation strategy ActSeq applies distinct augmentation techniques to both the observed action sequences X and the next action y. The purpose of proposing alternative next actions is to enhance the diversity of potential future actions, aiming to reduce ordering bias. This ensures that the model can accu- rately predict variety of possible future actions without depending on single ground-truth action in the training data. The aim of augmenting observed action sequences is to simulate real-worle inaccuracies such as noisy labels in annotation or erroneous action recognition by an action segmen- tation model. It also forces the model to learn from a broader context of the action sequence under consideration.
The overall pipeline of ActSeq comprises three main steps, as illustrated in Figure 2 Action Gram- mar Induction (AGT): The AGI algorithm initially derives a video grammar from a given action sequence. This grammar serves as the framework for understanding and predicting action dynam- ies within the video. Prediction by Cross-Tree Earley Parser (CTEP): For each training sample extracted from the video sequence, CTEP predicts alternative next actions based on the observed actions and the video grammar. The predicted alternative next actions combine with the original observed actions to form new training samples. In Figure 2 for illustration, show the CTEP proposing a single next action and, hence, creating one new training sample. Random Modifi- cations (RM): For each new training sample generated by CTEP, additional training samples created through random modifications to the observed action sequences. In Figure for illustration, we show the RM is set to generate one modified observed sequence, hence, creating one additional training sample.
3.1 ACTION GRAMMAR INDUCTION (AGI)
We first define two types of ordering: dependent and independent. If action B is dependent on action A, then A needs to happen before B, denoted by (A, B). On the other hand, if action A and B are independent, they can happen in any order.
As stated in the introduction, learning the correct order of actions is challenging, primarily due to the numerous possibilities of combining actions to construct valid action sequences, particularly when limited by the availability of training samples. However, we observed that actions performed on a single object tend to follow more consistent sequences. For example, in all sequences shown in Fig. D actions performed object cup consistently follow the same order: "take" followed by "pour into." The same pattern holds for objects like the bowl and cereal.
The action grammar is defined tions and their orders pertaining to a single object (atomic action) or an interaction between multiple objects such as "pour cereal into bowl". Our grammar induction algorithm uses action depen- dency matrix, which is derived from the entire training set and maintained as a knowledge base, to construct a unique grammar for each action sequence.
context-free grammar (Section 3.1.2), designed to derive
3.1.1 ACTION DEPENDENCY MATRIX
Dependent actions linked to specific objects follow similar pattern across different goals. Con- versely, with independent actions, such as the addition of various ingredients to the bowl, a flexible order of execution is permitted. These aspects are captured in an action dependency matrix, which is a skew-symmetric matrix where M(.,) = 1 action is dependent on action j. and is 0 otherwise.
To construct the matrix, we first isolate the actions associated with each object from all action quences in the training data, creating object-specific action sequences. These actions could comprise of atomic actions and/or interactions. Interactions among objects are decomposed into respective atomic actions. For instance, for interaction 'pour cereal into bowl', the action for 'cereal" is de- noted as "pour', and for 'bowl' 'poured into'. Given the object-specife atomic action sequences, the dependencies between two actions are then computed by thresholding on
N(A=
P(A+B) N(A=B) N(B=A)
(I)
where P(A-- my denotes the likelihood that action B is dependent action A, NA my is the number of times action A precedes action B in sequences where both are present. Ideally, if they are inde- pendent, B) P(B=A) 50%, and if they are dependent (e.g. B depends A), P(A=B)
4

---

Published as a conference paper at ICLR 2025
Table 1: Decomposition of Interactions & Action Dependency Matrix. Interactions decom- posed into atomic actions and their dependencies. M(i.)) = 1: action is dependent on action j, M (i, j) reverse dependency, M (i, (shown -): dependency between actions.
Poured
Poured
Take
Pour
Pour
Stir cereal
Slir in
bowl
cereal
bowl
into
milk
bowl
bowl
Take bowl
-1
-1
Pour cereal
Pour cereal
into howl
Poured
Pour milk
Pour milk
into bowl
Poured into bowl
in bowl
Siir buwl
a Bakk bowl, pour caneal inoo bowl, pour milk bowl, coreal]
(0) Action sequences the training sot
Ist toke boal
Take bowl
Bow
Cereal
JND
Poured into Bowl
bowl
:2R
Take
poured, into
s0r_h
Pour
sttrr
Cereal
cenoal cowl
Action dependancy object Careal
Interaction Decomposition
3nd: pour milk into bowl 1"y!
Tako bowl y1 V/V 999)
pour
Pour Cereal
Info Bowl
Pour
Sür Geresl In Bowl
conoal bow!
boxd V 12Y)
Pour canaa into bowl
Strr Cereal
Pourod, Bowl
Pour Mik
Censal
Bowl
Bowl
Graph based of dependerc
Video Grammar corresponding
Figure 3: Example of Video Grammar construction. From an action sequence (a), action dependen- cies of atomic actions for each object (b), and interaction decomposition (c) of all atomic actions and interactions the sequence, are used to construct a dependency matrix for the sequence, visualized as graph (d). (e) demonstrates step by step construction of video grammar's production rules and corresponding grammar tree from the dependency matrix.
should be 100%. However in real datasets, this is usually not true due to the bias caused by limited samples and incorrect missing labels. Hence, introduce threshold, if P(A==)) B) is greater than this threshold, action is deemed dependent on action A and M(A,B) = 1. Otherwise the two actions are considered independent and M(A,F) 0. The action dependency matrix that captures the sequential dependencies of actions is shown in Tablc
Figure 3 (b) provides graph-based visualization of atomic action dependencies for objects. In (c). the graph shows the decomposition of interaction into atomic actions. Finally, (d) presents graphical representation of the action dependencies matrix in Table action sequence (a).
3.1.2 VIDEO GRAMMAR
Formally, a context-free grammar (CFG) Cremers & Ginsburg (1975) defined as 4-tuple (C,T,P, P,S), where C is a finite set of non-terminal symbols, representing higher-level constructs. is a finite set of terminal symbols. P is a finite set of production rules of the form - a, where A C'is non-terminal and (CUT) The represents the set of all strings that composed of non-terminals and terminals. The - denotes replacement the non-terminal A with the sequence S is the start symbol.
5

---

Published as a conference paper at ICLR 2025
Take bowl
Take cup
pour cereal into bowl
pour milk into bowl
Stir cereal in bowl
Pour coffee into cup
Pour Gereal into Bowl
Pour Milk Into Bowl
Stir Ceroal
in Bowl
Pour Coffee
Into Cup
Pour Cereal
Take
Poured, Into Bowl
Stir Dowl
Pour
Take Cup
Poured into
coffeee
Cereal
Bowl
Mikk
Cup
Figure 4: Given an action sequence with two entangled goals, AGI separates the two goals and represents them using two dependency matrices, visualised two graphs.
We types of production rules, 'NDD' and 'OR', defined (i) AND - aB3, and (ii) OR C + aB $a. The AND rule dictates that the non-terminal C' is replaced by the sequence aB, establishing that a occurs before 3. Conversely, the OR rule, containing the 'I' operator, allows for the non-terminal C' to be replaced by either a: or 3o, signifying that a and 3 can occur in any order. These rules enable the hierarchical generation of action sequences. The video grammar constructed using the dependency matrix. Actions that are dependent are linked with an AND operator, while independent actions are connected using an OR operator.
We demonstrate the construction of the video grammar with the example shown in Figure 3 starting with the initial action "Take bowl." As this action does not depend on any others, it directly added to the root node S as 0 in Figure e(.. The next action "Pour cereal into bowl" relies on "Take bowl", because its component action "poured into bowl" is dependent on "Take bowl". Consequently. it is connected to the right of O at the root node V using AND operator to indicate that 7 precedes V The third action, "Pour milk into bowl," also depends on "Take bowl" but independent of "Pour cereal into bowl." To manage this independence, dummy node V11 replaces the original V11 now termed V representing "Pour cereal into bowl." The action "Pour milk into bowl" then introduced as V/J and connected V with OR operator, indicating that Vi and V22 can occur in any order. Lastly, "Stir cereal into bowl" is added as V to the root node because it depends on both V1 and Vz2. Figure 3 () illustrates the fully constructed video grammar's production rules and corresponding grammar tree.
The example demonstrates that the proposed AGI algorithm both simple and effective capturing correct action ordering. Another advantage is that when there are multiple goals being pursued at same time, for instance, "make cereal" and "make coffee", AGI is able to separate the two goals and represent them using two dependency matrices, visualised as graph in Figure
3.2 CROSS-TREE EARLEY PARSER
The primary objective of the parser is to propose next action candidates based on observed actions, using the video grammar as a guide. When multiple grammar trees present, the original Earley Parser will not predict actions from a grammar tree if none of its actions has been observed. For instance, as illustrated in Figure 4 if the observed input is "Take bowl," belonging the "make ce- real" grammar tree, the Earley Parser will not predict "Take cup." which is part of the "make coffee" grammar tree, Instead, it will continue to process actions only from the "make cereal" grammar tree.
We propose modified version of the Earley parser called Cross-Tree Earley parser (CTEP) that can handle interleaving actions from multiple goals. It operates similarly to the Earley Parser but includes enhancements to handle goal switching and parallel goal execution. However, allowing parsing across multiple trees will exponentially expand the search space as the number trees in video increases. To enhance search efficiency, the CTEP Parser allows selective initialization of grammar trees. In our experiments, only the trees containing observed actions and the next N actions initialized.

---

Published as a conference paper at ICLR 2025
We first add an initialization process to allow initializing multiple grammar trees. A terminal symbol ">" is added as the left child of the root node of selected grammar trees; it also serves as the first input to initialize these grammars. After initialization, the CTEP parser maintains set of states at each position in the input sequence of observed actions, similar to the Farley Parser. As it moves through the input, it updates the set of states according to three primary operations Prediction, Scanning and Completion. In the following, the symbols /' and Q refer to the non-terminals. The symbols a and 3 represent arbitrary strings consisting of terminals and non-terminals. The dot (-) denotes the current position of the parser within the production rule.
(i Prediction: If the current state expects non-terminal (P + a Q3),, the parser outputs all possible productions of Q that can be executed. If the current state expects a terminal, the parser moves on to the Scanning phase. When all the actions in the input are processed, the parser outputs all possible next terminals in the current states as the candidates for next action. The candidates are then prioritized based on the order in which they appear in the video sequence. This prioritization reflects the natural progression of events in the video. (ii) Scanning: The parser checks the next input symbol in the current state. matches the expected terminal, the parser moves the dot to the right, indicating successful scanning of that part of the input. (iii) Completion: When the dot reaches the end of a production (P + a.3.), the parser "completes" " that production. The parser then revisits previous states that were anticipating this non-terminal P and advances the dot in those states.
3.3 RANDOM MODIFICATION ON OBSERVED ACTIONS
Inspired by the effeetive text classification technique of random insertion, deletion, and replacement of words, which robustly enhances training sample diversity and improves model generalization Weil Zou (2019). we have adopted similar augmentation strategy for action sequences. For each se- quence in the training set, randomly apply one of the following operations to n [T/k]] + actions, where L represents the total number of actions in a training sample and parameter k is an adjustable impact factor that controls the number of actions be modified. Action Replace- ment (AR): Randomly select actions from the sequence and replace each of these actions with one chosen random from the sequence. This simulates scenarios where actions may be misclassified by upstream models or by wrong labeling in the annotation. Action Deletion (AD): Remove ran- domly selected action(s) in the sequence. This simulates scenarios where actions may be missed or unobserved during recording or recognition.
4 EXPERIMENTS
4.1 DATASETS
We perform experiments on three action anticipation benchmarks. 50Salads Stein & McKenna 2013) consists of 901 action annotations, and 17 action classes. We report the average performance across the standard five splits. EGTEA Gaze+ Li et al. (2021) contains 10,325 action annotations, 19 verbs, 51 nouns and 106 action classes. Methods are evaluated on EGTEA Gaze+ reporting the average performance across the three splits provided by the authors of the dataset. EPIC-Kitchens- 100 100 Damen et al. 2022)) contains 3806 actions, with 97 verbs, and 300 nouns. Action is represented in (verb, noun) pair. For example, action "open fridge" consists of verb "open", and noun "fridge". work We evaluate our method on the validation dataset following previous work Guo et al. (2024).
4.2 METRICS
We use Mean Top-5 Recall as a class aware metric for Epic- Kitchenk and EGTEA Gaze+ following Furnari et al. (211)). Specifically, when assessing performance EGTEA Gaze+, the Top-5 Recall is calculated by averaging across actions. In contrast, for EPIC-Kitchens, the metric averages over verbs, nouns, and actions. For the 50Salads dataset, which contains significantly fewer actions, use T accuracy as the evaluation metric, following Guo et al. (2024).
We also propose a metric called Mean Action Order Bias (MAOB) measure the bias in action order within a dataset. Specifically, for each independent pair of actions, A and B, the action order bias for pair is defined as AO BAB (P(A= AiB= where P(A=B) B) is the probability that
7

---

Published as a conference paper at ICLR 2025
A occurs before B. These probabilities computed from training samples where both A and B are present. The Mean AOB is then computed by averaging over all action pairs.
4.3 STUDY OF ACTION ORDER BIAS IN SOSALADS DATASET
Table compares the MAOB for 50Salads dataset between the original sequence and with the aug- mented sequence generated from ActSeg. The MAOB of the original sequence is 0.32, indicating the presence of significant bias. The main source of bias arises from ingredients preparation (e.g.. cutting various ingredients and placing them in a bowl) and from sauce preparation (e.g., combining different sauces cup) processes. Both processes consist of 4 to independent actions, but some of these actions are significantly more likely to occur in particular order. For instance, in 67% of the videos, the action "cut tomato" occurs before "cut cucumber," and 71% of the videos, "add oil" precedes "put salt". After incorporating our augmented sequences into the training samples, the MAOB significantly decreased to 0.12, demonstrating the effectiveness our method in reducing ordering bias. MAOB studies on EGTEA Gaze+ and Epic- Sine--Ktchhnns1100 100 are in Appendix B
Table 2: Mean Action Order Bias on 50Salads.
Method
МАОB
Original sequence
0.32
Original augmented sequence by ActSeg (ours)
0.12
4.4 ABLATION STUDY OF ACTSEQ ON EGTEA GAZE+ DATASET
In this section, we study the impact of augmenting observed actions and next action individually, as well as in combination the performance of action anticipation the EGTEA Gaze+ dataset. We also compare our augmentation strategy with other common sequence augmentation methods using state-of-the-art model based on RAFTformer Girase al. (0033. Ablation study on 50Salads and Epic- Epic-Kitchens-IMe 100 can be found in Appendix
Augmentation to observed actions and next action. Table 3 and Table 4 show RAFTFormer's Mean Top Recall on EGTEA Gaze+ changes as function of two variables: (i) the size of aug- mented sequences by modifying the observed action sequences and (ii) the size of augmented quences by modifying the next action action-evt expressed as multiples of the original training sequences. For example, "3X" under Observed in Table[]indicatee that the number of new sequences generated by augmenting observed actions is three times the size of the original set.
Table 3: Ablation on number augmented observed actions EGTEA Gaze+.
Table 4: Ablation on number of augmented
next actions EGTEA Gaze+.
Observed Next
Top 5 Recall
Observed
Next
Top 5 Recall
original
62.1
original
62.
2X
62.8
XX
63.5
3X
original
63.1
original
2X
64.7
4X
63.0
4X
65.2
5X
62.5
5X
65.3
Table 5: Ablation on number of augmented observed and next actions on EGTEA Gazet.
Observed
Next original
Top 5 Recall
original
62.1
IX
XX
63.8
2X
2X
65.2
3X
4X
66.3
4X
5X
65.5
Table shows the augmentation effects observed action sequences. The recall rate initially im- proves from 62.1 1% with no augmentation to 63.1 % at a 3X increase, suggesting that moderate aug- mentation the observed action sequence positively impacts model training by forcing the model to
8

---

Published as a conference paper at ICLR 2025
learn from the context instead of relying on a few key actions. However, further increase to 4X and 5X augmentation leads to a decrease in performance, dropping to 62.5%. This indicates a potential werrmodiiiiationn of the observed action sequences, where learning next action from the augmented sequence becomes too difficult. The subtle fluctuation in recall rates across augmentation levels suggests that while augmentation helps certain extent, its benefits are non-linear and can lead to negative returns when excessive.
On the other hand, T shows slightly different trend when augmenting next actions. The recall rates steadily increase from 62. %% without augmentation to a peak of 65.3% at 5X augmentation, though with diminishing returns. Notably, 5X is the maximum number of candidates available for the ETGEA Gaze+ dataset. This indicates consistent improvement as augmenting next actions introduces valuable predictive complexity, helping the model better anticipate and generalize future actions.
providss an integrated view by comparing Top 5 Recall when both observed and next actions are augmented. For example, 3X under Observed and 4X under Next means that there are three times the augmented observed actions for each of the four augmented next actions. In this case, the recall peaks at 66.3%, which demonstrates that combining augmentation techniques next actions yields a synergistic effect, outperforming the application of either method in isolation. Additionally, the results suggest that there is an optimal scaling factor for the observed and next action sequences that maximizes performance.
both observed and
Comparison with other sequence augmentation methods. Since sequence augmentation methods have not been previously applied to action anticipation, we adopt two commonly used methods from other fields. Method is from text classification, which involves random modifications (deletion, insertion, addition, or replacement) of words in text[Wei& Wei & Zou (2019). Method 2 is from multi-label image classification, which utilizes prediction results from a downstream model Ke al. (2019). For method 1, we apply modifications to randomly selected actions as detailed in Section 3.3 to the video sequence, then generate training samples from the augmented sequence. For method 2, we take the predicted next actions from AVT & Grauman (202T). combine each prediction with the observed actions from the original training samples to create augmented samples. Additionally, propose new approach, Method 3, called "NEXT D." This method uses each of the next D actions following the observation as the "next action", combined with the original observed actions to generate D training sequences. For example, consider the notation (pred=apred]=[a.. b, c), d] where [a, b, c] represents the observed actions and d is the next action. If the action sequence in a video is (a, b, d,, e, Jn ..., and training sample is [[a, b, c). d). then the "NEXT D augmented sequences with = 2 are [[a, b, c), e) and [[a, b, f).
Table 6: Comparison of RAFTformer's performance trained other augmentation methods EGTEA Gaze+ dataset. aohs is the observed sequence, apred is the next action.
Method
Applied to
Recall (%)
Original training set
63.5
Original + Randomly delete, insert, add, replace actions (5X)
aobs, Opred
62.2
Original + Predicted results by AVT (5X)
apred
63.6
Original + Next (5X)
Apred
64.8
Original + ActSeq (pred only) (5X) (Ours)
Apred
65.3
Original + ActSeq (12X) (Ours)
aobes apred
66.3
T shows RAFTformer's performance with augmented sequences generated by different meth- ods on the EGTEA Gaze+ dataset. The application of randomly deleting, inserting, and replacing actions to the whole video sequence bith.,both observed and next actions) leads to decrease in per- formance. In contrast, as noted in Table augmentation applied solely to observed actions resulted in performance improvements. This suggests that random modifications of next actions introduce significant noise or irrelevant changes, leading to worse performance. Additionally, using predic- tions from AVT slightly enhances performance implying that while different models may learn dis- tinct insights from the dataset, they may yield less diversity and accuracy of possible next actions. Interestingly, the simpler "Next D" method significantly boosts performance to 64.8% when D = 5, indicating a strong correlation between observed actions and the next few actions. In comparison, Act,Seg, when applied next action only, achieves higher recall of 65.3%. Act.Seq is able to

---

Published as a conference paper at ICLR 2025
generate valid action sequences when the size of augmented sequences is 12 times the original (i.e. 12X) achieving the best recall of 66.3%.
4.5 IMPACT OF SEQUENCE AUGMENTATION STRATEGY ON SOTA
In this experiment, evaluate ffate-of-the-are models trained with and without ActSeq 50Sal- ads, EGTEA Gaze+, and EPIC-Kitchens- 100 datasets, each offering varying complexity in terms of action classes, goal diversity, and video length. This allows assess our method's generalizabil- ity across different models and datasets.
The 50Salads dataset (Table with fewer action classes, single goal and shorter sequences, re- sults in relatively fewer action ordering possibilities. Even so, reducing ordering bias through Act- Seq improves the performance of all models, albeit by a small amount. contrast, EGTEA Gaze+ (Table and EPIC- SHrCCKitchhensI 100 (Table present much more complex action space with diverse verbs and nouns across multiple goals, where ActSeq significantly improves recall by enabling mod- els to capture subtle action ordering variations.
Appendix D contains a step by step demonstration of how our ActSeq method generates augmented training samples using example from the Epic-Kitchenn dataset. Appendix demonstrates the model's s efficiency when trained with low-data.
In summary, these experiments highlight ActSeq's 's robustness across different models and bench- marks. Its consistent performance gains demonstrate its potential as a valuable strategy for improv- ing action anticipation for datasets with different complexities.
Table 7: Top
Accuracy on 50Salads.
Table 8: Top recall on EGTEA Gaze+.
Method
Action
Method
Action
CNNAbu Farha et al. 20188)
29.8
RULSTMFurnat & Farinella 2020)
58.6
+ ActSec
31.1
RULSTM ActSeq
60.4
AVrGGrd & Grauman (021)
48.0
AVTGirdhar & Grauman 2021
62.1
AVT'ActSSey
49.7
AVT ActSeq
64.8
Gireee et al. (2023)
53.2
RAFTformeGGrraee et al. 2023)
63.5
RAFTYormer+ ActSeq
53.9
RAFTformer ActSeq
66.3
Table Top recall metrics on the Epic-Kitchen- -00 dataset. Method
Verb
Noun
Action
RULSTM Furnari Farinella (020))
30.8
27.8
14
RULSTM + ActSeq
32.1
30.3
15.9
RAFTforme Girase et al. 2023
33.8
37.9
19.1
RAFTformer + ActSeq
35.1
40.4
20.8
MotionFormme.Parrick et (2021)
47.1
46.9
21.5
MotionFormer + ActSeq
48.4
49.4
22.4
5 CONCLUSION AND FUTURE WORK
This paper introduces comprehensive framework aimed at mitigating the challenges of action ordering bias datasets and noisy input action anticipation models through innovative sequence augmentation techniques. By tailoring distinct augmentation methods to the observed actions and the next action, we effectively reduce biases action order and improve the model's resilience to noisy input. This approach has demonstrated significant performance improvements across various datasets and state-of-the-art models.
However, our current framework assumes that when multiple goals are pursued simultaneously, the action sets for each goal are independent. This simplification may not hold in cases where a single action contributes to multiple goals. Future research could extend our model to address such complexities by exploring how intertwined action sequences and overlapping goals can be more effectively represented and anticipated.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what? what?-anticipating temporal occurrences of activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5343-3352 2018.
Alessandro Antonucci, Gastone Pietro Rosati Papini, Paolo Bevilacqua, Luigi Palopoli, and Daniele Fontanelli. Efficient prediction human motion for real-time robotics applications with physics- inspired neural networks. IEEE Access, 10:144-157, 2021.
Nicolas Aziere and Sinisa Todorovic. Markov game video augmentation for action segmentation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 13505-13514, 2023.
Armin Cremers and Seymour Ginsburg. Context-free grammar forms. Journal of Computer and System Sciences, 11(1):86-117, 1975.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocen- tric vision: Collection, pipeline and challenges for Booc-iichhess-000 100. International Journal of Computer Vision, pp. 1-23, 2022.
Eadom Dessalene, Michael Maynord, Cornelia Fermüller, and Yiannis Aloimonos. Therbligs in action: Video understanding through motion primitives. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10618-10626, 2023.
Duarte Duque Henrique Santos, and Paulo Cortez. Prediction of abnormal behaviors for intelligent video surveillance systems. In 2007 IEEE Symposium on Computational Intelligence and Data Mining, pp. 362-367. IEEE, 2007.
Jay Earley. An efficient context-free parsing algorithm. Communications of the ACM, 13(2):94-102, 1970.
Alex Falcon, Oswald Lanz, and Giuseppe Serra. Data augmentation techniques for the video ques- tion answering task. In European Conference on Computer Vision, pp. 511-525. Springer, 2020.
Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mita- mura, and Eduard Hovy. A survey of data augmentation approaches for nip. arXiv preprint arXiv:2105.0888 03375 2021.
Antonino Furnari and Giovanni Maria Farinella. Rolling-une unrolling Istms for action anticipation from first-person video. IEEE transactions pattern analysis and machine intelligence, 43(11): 4021-4036, 2020.
Antonino Furnari, Sebastiano Battiato, and Giovanni Maria Farinella. Leveraging uncertainty to rethink loss functions and evaluation measures for egocentric action anticipation. In Proceedings of the European conference computer vision (ECCV) workshops, pp. 389-405, 2019.
Harshayu Girase, Nakul Agarwal, Chiho Choi, and Karttikeya Mangalam. Latency matters: Real- time action forecasting transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18759-18769, 2023.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13505-13515, 2021.
Dayoung Gong, Joonseok Lee, Deunsol Jung, Suha Kwak, and Minsu Cho. Activity grammars for temporal action segmentation. Advances in Neural Information Processing Systems, 36, 2024.
Hongji Guo, Nakul Agarwal, Shao-Yusa Lo, Kwonjoon Lee, and Qiang Uncertainty-aware action decoupling transformer for action anticipation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18644-18654, 2024.
Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. Sequence-to-sequerae data augmentation for dialogue language understanding. arXiv preprint 1rxiv 2018.

---

Published as a conference paper at ICLR 2025
Michat Jungiewicz and Aleksander Smywińiski-Pole Towards textual data augmentation for neural networks: synonyms and maximum loss. Computer Science, 20, 2019.
Xiao Ke, Jiawei Zou, and Yuzhen Niu. End-to-end automatic image annotation based on deep cnn and multi-label data augmentation. IEEE Transactions on Multimedia, 21(8):2033-2100 2019. doi: 10.1009TTMM..11..SSS 28955111.
Taein Kwon, Bugra Tekin, Siyu Tang, and Mare Pollefeys. Context-aware sequence alignment using 4d skeletal augmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8172-8182, 2022.
Sungho Lee, Narae Choi, and Woong Il Choi. Enhanced correlation matching based video frame interpolation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2839-28477 2022.
Bohan Li, Yutai Hou, and Wanxiang Che. Data augmentation approaches in natural language pro- cessing: A survey, Ai Open, 3:71-900 2022.
Yin Li, Miao Liu, and James Rehg. In the eye of the beholder: Gaze and actions first person video. IEEE transactions pattern analysis and machine intelligence, 45(6):6731-6747 2021.
Himangi Mittal, Nakul Agarwal, Shao-Yuse Yuan Lo, and Kwonjoon Lee. Can't make an omelette without breaking some eggs: Plausible action anticipation using large video-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
18580-18590. 2024.
Thai-Son Nguyen. Sebastian Stueker, Jan Nichues, and Alex Waibel. Improving sequence-to- sequence speech recognition training with on-the-fly data augmentation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7689- 7693. IEEE, 2020.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichten- hofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye the ball: Trajectory attention in video transformers. Advances in neural information processing systems, 34: 12565, 2021.
Mingtao Pei, Yunde Jia, and Song-Chun Zhu. Parsing video events with goal inference and intent prediction. In 2011 International Conference Computer Vision, pp. 487-494 IEEE, 2011.
AJ Piergiovanni, Anelia Angelova, Alexander Toshev, and Michael S Ryoo. Adversarial genera- tive grammars for human activity prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part Il 16, pp. 507-523. Springer, 2020.
Luke Prananta, Bence Mark Halpern, Siyuan Feng, and Odette Scharenborg. The effectiveness of time stretching for enhancing dysarthric speech for improved dysarthric speech recognition. arXiv preprint arXiv:2201. 200000908, 2022.
Siyuan Qi, Baoxiong Jia, Siyuan Huang, Ping Wei, and Song- Chhu Zhu. A generalized earley parser for human activity parsing and prediction. IEEE Transactions on Puttern Analysis and Machine Intelligence, 43(8):2538-22 2554, 2020.
Hiroyuki Seki, Takashi Matsumura, Mamoru Fujii, and Tadao Kasami. On multiple context- grammars. Theoretical Computer Science, 8812):191-229, 1991.
Zhangzhang Si, Mingtao Pei, Benjamin Yao, and Song-Chun Chun Zhu. Unsupervised learning of event and-or grammar and semantics from video. In 2011 International Conference Computer Vi- sion, pp. 41 41-48. IEEE, 201
Sebastian Stein and Stephen J McKenna. Combining embedded accelerometers with computer vi- sion for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference Pervasive and ubiquitous computing, pp. 729-738, 2013.
12

---

Published as a conference paper at ICLR 2025
Nam N Vo and Aaron Bobick. From stochastic grammar to bayes network: Probabilistic parsing of complex activity. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2641-2648, 2014.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCLP)) pp- 6382 6388, Hong Kong, China, November 2019. Association for Computational Linguistics, doi: 10.18653/v1/D19- 1670. URL / htt.h//ooooo..oo///--11110
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020.
He Zhao and Richard Wildes. On diverse asynchronous activity anticipation. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 781-799. Springer, 2020.
A COMPARING GRAMMAR-BASED METHODS AND LLMs FOR ACTION SEQUENCE AUGMENTATION IN ACTION ANTICIPATION
The integration of grammar-basse based methods in action sequence augmentation has demonstrated promising results due to their interpretability and structured approach generating action quences. However, Large Language Models (LLMs) such as ChatGPT also offer compelling al- ternatives due their capability generate diverse and contextually rich action sequences from ex- tensive training data. Comparing these methodologies is essential for understanding their respective strengths and limitations in various application scenarios, particularly in dynamic and less-structured environments.
A.1 EXPERIMENTAL SETTING
Dataset: 50Salads
Task: Predict legitimate next actions given predefined sequence of cooking actions.
Models Evaluated:
Proposed Grammar-base Method (ActSeq): Utilizes rules derived from action narrations to generate sequences.
Gemini 1.5 Flash, Claude, ChatGPT-3.5 and ChatGPT-4.0: These models rep- resent LLMs, chosen for their varying capabilities in understanding and generating text-based sequences.
A.2 EXAMPLES AND COMPARISON OF RESULTS
Prompt given to LLM models:
Task: Given the complete action sequence, the observed actions and the goal, deter- mine what the next action could be (if there are multiple plausible actions, list them all). Provide a justification for your choice.
Complete Action Sequence:[cut tomato, place tomato into bowl, cut cheese, place cheese into bowl. cut lettuce, place lettuce into bowl, add salt, add vinegar, add oil, add pepper]
-Observed Actions:[cut tomato, place tomato into bowl]
GGoal: Prepare mixed salad with dressed vegetables.
Predictions and Analysis:
Gemini 1.5 Flash:
Predicted Actions: 'cut cheese'
13

---

Published as a conference paper at ICLR 2025
Justification: The goal is to prepare mixed salad with dressed vegetables. The observed actions indicate that the tomato has been prepared and added to the bowl. The next logical step in preparing mixed salad is to prepare another ingredient, such as the cheese. The action sequence shows that cutting and placing the cheese into the bowl follows the preparation of the tomato. Therefore, cutting cheese is the most plausible next action based on the goal and the observed actions.
* Analysis: The model correctly states that "The next logical step in preparing mixed salad is to prepare another ingredient." However missed "cut lettuce" in its prediction. Its prediction could be copied from the original action sequence.
Claude:
* Predicted Actions: Cut cheese and place into the bowl or Cut lettuce and place it into the bowl.
Justifications: goal is prepare mixed salad, and the complete action quence indicates that the next step after placing the tomato into the bowl is to cut the cheese and place it into the bowl, This aligns with the goal of creating a mixed salad with various ingredients.
the lettuce and place it into the bowl.
* Analysis: The model seems to interpret "Cut cheese and place it into the bowl" single action,
immediate sequence of actions, suggesting that once the cheese is cut, it should directly be placed in the bowl. However, there might be instances where multiple ingredients are cut, then put into bowl afterwards.
ChatGPT-3.5:
Predicted Actions:
'cut cheese', 'cut lettuce', 'add salt"
* Justifications:cut cheese and cut lettuce are independent of previous steps beyond what's observed. Add salt could technically happen after placing the tomato the bowl.
Analysis: The model suggests adding salt immediately after the initial salad preparation steps, showcasing lack of understanding of the structured sequence typically followed in salad preparation.
ChatGPT-4o:
* Predicted Actions: 'cut cheese', 'cut lettuce'
* Justifications: most plausible next action is "cut cheese." Alternatively, "cut lettuce" could also be valid action. Both align with the goal of preparing a mixed salad by incorporating the next ingredient into the preparation process.
* Analysis: The model correctly predicts the two candidates, however in its justifi- cation, prefers "cut cheese" over "cut lettuce. it could due to "cut cheese" the one that appears in the ground truth action sequence.
ActSeq (Grammar-based Method):
* Predicted Actions: 'cut cheese", 'cut lettuce'
* Analysis: Correctly identifies the need to continue with ingredient preparation before moving dressing components, demonstrating strong adherence the structured task rules.
The analysis of Large Language Models (LLMs) in the action sequence augmentation experiment reveals several weaknesses, including varied performance based on model capabilities, inherent bi- ases, and occasional misunderstandings of task. While more sophisticated prompt engineering can improve LLM responses by guiding them towards more accurate predictions, it remains challenging anticipate and cover all potential scenarios.
B EXTENDED MAOB STUDY ON EGTEA GAZE+ AND EPIC-KITCHENS- DATASETS
To evaluate the generalizability our ActSeq method, we extended the MAOB study to the EGTEA Gaze+ and Saic-Kitchens:: 100 datasets in addition to 50Salads. The purpose of this study is to assess how well ActSeq mitigates action order biases in datasets with varying levels of complexity. For reference, 50Salads contains fewer action classes, single goal, and shorter video sequences,
14

---

Published as a conference paper at ICLR 2025
while EGTEA Gaze+ and SppicKitchenss1000 100 feature more diverse set of verbs, nouns, and goals, presenting a significantly more complex action space.
As shown in Table I0 the augmented sequences generated by our ActSeq method are 12 times (3X for observed action and 4X for predicted action) of the original sequences. ActSeq consistently re- duces MAOB across all three datasets, significantly lowering the bias from 0.32 to 0.12 in 50Salads, from 0.28 to 0.10 EGTEA Gazet, and from 0.46 to 0. 9.19 in Spic-Kitchens- This demonstrates that ActSeq effectively mitigates action order biases, resulting in more balanced and representative datasets.
Notably, the reduction in MAOB correlates with improved performance across all datasets. By min- imizing biases in action ordering, ActSeq enables models to better generalize to unseen sequences, especially in datasets like EGTEA Gaze+ and Oppic-KichhensJO11 tion order can significantly impact performance. These findings further highlight the robustness of ActSeq and its potential for generalization across diverse datasets.
where subtle variations in ac-
Table 10: Mean Action Order Bias (MAOB) across datasets.
Method
50Salads
EGTEA Gaze+ Epic- Spic-Kitchens-100
Original sequence
0.32
0.28
0.46
Original Act.Seq (ours)
0.12
0.10
0.19
The extended MAOB study provides strong supporting evidence for the effectiveness of ActSeq in reducing action order biases across datasets with varying complexity. These results align with ob- served performance improvements, emphasizing the robustness and generalizability of our method. This analysis also underscores the importance addressing biases to enhance model performance in diverse and challenging action understanding tasks.
C ABLATION STUDY OF AUGMENTATION RATIO ON ALADS AND EPIC-KITCHENS- 100 DATASETS
This appendix conducts a similar ablation study as in Tables and 4 for the 50Salads and Epic- Kitchens- 100 datasets. These datasets differ from EGTEA Gaze+ in size, domain, and data dis- tribution. The additional studies aim to evaluate the generalizability of the ActSeq method across datasets with varying characteristics.
The results of the ablation studies are summarized below for both datasets. These studies assess the impact of varying augmentation ratios for observed and next action sequences on Top-5 Recall performance.
50Salads Dataset:
The 50Salads dataset, characterized by fewer action classes, shorter sequences, and a single goal, showed consistent improvements with ActSeq. As seen in Table 11 the best performance was observed with a configuration of 2X observed and 3X next actions.
Epic- Shie-Kitchens-100 Dataset: The Epic- SeiicKitchens-1000 100 dataset, with its complex action space and diverse goals, showed steady improvements in Top-5 Recall with increased augmentation ratios (Table [2]. The best performance of 20.8% achieved at 3X observed and 4X next configuration. These results further support the effectiveness of ActSeq in handling diverse and complex datasets.
The ablation studies on 50Salads and Epic-Kitchens 100 confirm the generalizability of ActSeq to datasets of varying size, domain, and complexity. Moderate augmentation levels (e.g.. 2X or 3X) strike a balance between computational efficiency and performance improvement, making ActSeq practical and adaptable solution for enhancing model performance in diverse applications.
D DEMONSTRATION EXAMPLE FROM THE EPIC-KITCHEN DATASET
This appendix demonstrates how our ActSeq method generates augmented training samples from the Spic-KKicher dataset. The example illustrates the derivation of action dependencies, their rep- resentation in a grammar tree, and the step-by st- generation of augmented sequences using our approach.
15

---

Published as a conference paper at ICLR 2025
Table 11: Ablation study the 50Salads dataset: Top-5 Recall (%).
Observed original
Next
Top-5 Recall (%)
original
53.2
1X
original
53.4
2X
original
53,5
3X
original
53.3
4X
original
53.1
original
IX
53.5
original
2X
53.6
original
3X
53.7
original
4X
53.7
IX
IX
53.5
2X
2X
53.7
2X
3X
53.9
2X
4X
53.5
Table 12: Ablation study the Epic-Kitchens-ID 100 dataset: Top-5 Recall (%).
Observed
Next
Top-5 Recall (%)
original
original
19.1
2X
original
19.4
3X
original
19.5
4X
original
19.6
5X
original
19.5
original
IX
19.8
original
2X
20.2
original
4X
20.5
original
5X
20.5
IX
IX
19.9
2X
2X
20.3
2X
4X
20.6
3X
4X
20.8
ORIGINAL SEQUENCE AND DEPENDENCIES
The original sequence of actions is follows:
Original Sequence: [open drawer, take knife from drawer, put knife on table, take pot from drawer, close drawer, put pot stove.]
Using bhject-based action transitions and interaction decomposition, the following dependencies are derived:
open drawer close drawer
open drawer take knife from drawer
open drawer + take pot from drawer
take knife from drawer > put knife on table
take pot from drawer - put pot stove
These dependencies can be represented as a grammar tree, as shown in Figure
TRAINING SAMPLE EXTRACTION
From the original sequence. training samples extracted by considering fixed-lengte observed ac- tion sequences. In this demonstration, we assume a length of two actions for the observed sequence. This results in the following training samples, each in the format: [[observed actions), next action]
16

---

Published as a conference paper at ICLR 2025
And Or
Open
Close Drawer
Take kntfe from
Takk pot from table drawer
Put
Put
knifa
drawer
Stove
Figure 5: Grammar tree representation of the action sequence: open drawer, take knife from drawer, put knife table, take pot from drawer, close drawer, put pot stove.
1. [[open drawer, take knife from drawer), put knife on table]
2. [[tkee knife from drawer, put knife table], take pot from drawer]
3. [[ut knife on table, take pot from drawer), close drawer]
4. [[take pot from drawer, close drawer), put pot on stove]
AUGMENTATION EXAMPLE
We demonstrate the augmentation process using the first training sample:
[[observed actions], GT next action] [[open drawer, take knife from drawer), put knife on table]
Predicting Next Action Candidates Using CTEP (our candidate generation method), the follow- ing next action candidates are predicted based on the observed actions:
{put knife on table, take pot from drawer, close drawer).
Generating Augmented Sequences To generate augmented sequences, we exclude the ground truth (GT) next action (put knife table) and combine the observed actions with other plausible candidates. The resulting augmented sequences are:
a. [[oen drawer, take knife from drawer), take pot from drawer]
b. [open drawer, take knife from drawer), close drawer]
Applying Random Modifications (RMs) To enhance diversity, random modifications (RMs) are applied to the observed actions. For example, switching the order of two actions produces:
c. [[take knife from drawer; open drawer). take pot from drawer)
d. |take knife from drawer, open drawer], close drawer)
In this example. a total of 4 augmented sequences (a.b,c, and d) are generated. This example demon- strates how ActSeq generates diverse augmented training samples. By combining plausible next tions with observed sequences and applying random modifications, our method effectively enriches the training data while adhering the dependencies derived from the grammar tree.
E DEMONSTRATION OF MODEL EFFECTIVENES IN LOW-DATA MODE
This appendix demonstrates how our approach improves data efficiency of a network, enabling it to require fewer original action sequences during training.
17

---

Published as a conference paper at ICLR 2025
EXPERIMENTAL SETUP
We simulated low-data conditions by selecting limited number of training videos per task from the EGTEA Gaze+ dataset. Specifically, we created subsets with 3 and 5 videos per task. These subsets were used to train the model both with and without ActSeq augmentations. The performance of these low-data models was compared that of full-data model trained with an average of 10 videos per task.
The RAFTFormer model was used as the base, and Top-5 Recall was selected the evaluation metric. Table [3] summarizes the results.
Table 13: Performance of the low-data model on EGTEA Gaze+ with ActSeq.
Videos per Task
With ActSeq (Top-5 Recall)
64.7
65.8
Full (on average 10 per task)
66.3
RESULTS AND DISCUSSION
Table demonstrates the consistent improvement in performance achieved with ActSeq across varying levels of training data availability. With only 3 videos per task, the model achieves Top-5 Recall 64.7%. This result highlights the robustness of ActSeq in extremely limited data set- tings, where it provides sufficient diversity to improve learning outcomes. It's worth noting that this performance is surpassing the RAFTFormer model trained with full original training set, which is 63.5%.
As the number of training videos per task increases to 5, the Top-5 Recall improves to 65.8%, showing that it enables models to achieve performance levels close to those of full-data models (66.3%). This result illustrates that while larger datasets naturally lead to better performance, ActSeq ensures that models trained on smaller datasets still perform competitively.
The experiments validate the data efficiency of ActSeq, showing that enables models to achieve performance levels close those of full-data models, samples. This makes ActSeq particularly valuable in scenarios where collecting large-scale datasets impractical costly. By enriching the training data with diverse and consistent augmentations,, ActSeq provides an effective solution for overcoming data scarcity while maintaining high model performance.
with fewer available original training
18


Reviews:
[{"review_id": "gBYfvhtGRa", "reviewer": "Reviewer_c22w", "summary": "The paper introduces a data augmentation strategy (ActSeq) for the task of action anticipation. The paper recognizes multiple challenges of action anticipation - 1). multiple possible action sequences, 2). action ordering bias from the datasets, 3). interleaved actions from interleaved goals and present an augmentation strategy to address these challenges.\n\nGiven the action sequence video, video grammar is derived using Action Grammar induction algorithm. Then the cross-tree early parser predicts alternative next actions based on the observed actions and random modifications of insertion, deletion and replacement is further applied. Overall, this augmentation gives 3x (3 times) the size of the original dataset.\n\nThe method is evaluated on three action anticipation datasets - epic kitchens, egtea gaze+, and 50salads. The data augmentation strategy is incorporated with multiple prior works and their performance with and without augmentation is compared. These methods show improvement with data augmentation.", "questions": "The datasets explored in the paper predict only a single action given the observed actions. Would the proposed data augmentation strategy also work in scenarios where multiple actions (e.g. 20 actions) have to be predicted given the observed actions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses concrete challenges that arise in the task of action anticipation and presents a simple approach of data augmentation to address the ordering bias and multiple scenarios possibility when anticipating actions. The paper also presents an exhaustive evaluation on three action anticipation datasets. The idea of interleaved goals and interleaved actions is interesting.", "weaknesses": "The contribution and the novelty of the paper seem to be limited. Previous works [1,2] have explored some components of the proposed approach such as exploring grammar for activity prediction and data augmentation for temporal bias which is similar to the random modification of the proposed idea. Can the authors emphasize the contributions and especially the novelty of the work?\n\n\n1. Piergiovanni, A. J., Anelia Angelova, Alexander Toshev, and Michael S. Ryoo. \"Adversarial generative grammars for human activity prediction.\" In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 507-523. Springer International Publishing, 2020.\n2. Mittal, Himangi, Nakul Agarwal, Shao-Yuan Lo, and Kwonjoon Lee. \"Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18580-18590. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUZUqE286I", "reviewer": "Reviewer_VMPA", "summary": "The authors introduce a data augmentation strategy, where they synthetically generate plausible, unobserved sequences of actions that obey the constraints of a grammar they induce over the action labels. They demonstrate this helps performance across a variety of models and datasets.", "questions": "1) Inducing a grammar over the observed actions is good, but I suspect an LLM could be used as an alternative source of action sequence augmentations, especially considering the Grammar operates only over narration-derived actions. Is there any reason to believe this isn't the case, or where one fails where the other would succeed? A qualitative example would be nice.\n2) There's a paper (Therbligs for Action Understanding) [1] below that imposes a grammar over actions based on commonsense rules. They perform action anticipation as well, with improved performance due to these rules enabling the modeling of plausible sequences that abide by the rules. This paper should be included in the Related Works.\n3) I wonder if this approach improves the data efficiency of a network, in that it needs less action sequences. This is claimed in Section 3 but not empirically explored (the way to do that would be to do an experiment where low-data model is compared with and without the augmentations).\n4) Is there an augmentation strategy that rather than increase the augmentations by a set factor, uses the set of all plausible sequences for data augmentation?\n\n\n[1] Dessalene, Eadom, et al. \"Therbligs in action: Video understanding through motion primitives.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The induction of a grammar over actions is vitally important due to the combinatorial nature of action sequence modeling. Actions can be arranged quite flexibly, and video datasets are not large enough to span all the variations over action sequences.\n2) The rules of the grammar capture well capture the hierarchical and compositional nature of action. This grammar they've induced can be applied in other domains (e.g. robotics, action recommenders, etc), which otherwise would normally be handwritten.\n3) The models and datasets were chosen well.", "weaknesses": "1) The results for Tables 8 and 9 are reporting Top 5. Top 1 results for action anticipation predictions are also helpful to see. Your data augmentation approach likely best improves the Top 5 results, but could possibly be hurting the Top 1 results by training over plausible but unlikely future actions.\n2) A qualitative figure would be nice to see. Or at least a visualization of the trees for one of the datasets. \n3) I don't fully understand Table 1. Why are there negative values? There is no caption and the Figure is confusing. This figure is vital for understanding how Figure 3e) is constructed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vUyyMZBxLh", "reviewer": "Reviewer_NV6R", "summary": "Thanks the authors for this ICLR submission.\n\nSummary:\n\nThis work identifies two major problem in action anticipation research: **1)** the default dataset split-up only allows for partial exploration over the entire action space, and **2)** noisey action labels induced by the imperfect preprocessing step (e.g., action recognition). To this end, this paper proposes a data augmentation method to enhance both the observable context and the prediction target (i.e., future steps). Moreover, a more noisy-robust version of data is generated by randomly deleting or replacing action elements.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Easy to follow.\n- Nice discussion on the three main issues in action anticiaption from line34-45!\n- Approach is technical sound. Although, as mentioned in the future work, the optimal solution is to perfectly de-compose interwined action sequences, the method in this paper serves as a practical temporary solution.  \n- Interesting study on the order bias. \n- Some Improvements over standard benchmarks across a variety of baseline methods.", "weaknesses": "- (Minor) The manuscript could be written better\n    - Inconsistent dataset name in Table 3 & 4 and line 407 - EGTEA Gaze+, EGTEA Gaze, EGTEA. Please consider make the dataset names consistent throughout the submission, which will make it easier to follow and make connections. \n    - Table 7, 8 & 9 captions are sentences, and should be ended with period. Please follow the basic academic writing protocol to better the  the wrtting.\n    - Given no appendix, it is better to explain the evaluated **verb, noun, action** items for epic-kitchen. Please include a section explaining the evaluated items and/or terminology. It is recommended that academic paper should be self-contained, so that readers can find all information within a single paper.  \n\n- (Minor) The MAOB study is interesting, yet it is incomplete - none has been done to other experimented datasets (e.g., EGTEA and Epic-Kitchen). I would like to suggest extending such study to the EGTEA and Epic-Kitchen-100 datasets. It will be insightful to check the correlation between the reduction of order bias and the performance improvements, which will be a strong supporting evidence for th ActSeq. method to be accepted. \n- (Minor) From Table 3 & 4, it is seen that the amount of useful information does not scale well the times (`X`) of augmentation, which is a concern. The solution to settle down the **X** hyperparameter seems critical. Please provide some discussion on the trade-offs here and initialize some considerations on how to decide this hyper-parameter.  \n- (Minor) This method requires really large number of augmentations, e.g., **12X** more sequences, to bring significant improvements (e.g., > 1 % in **Recall**). Arguably, it costs much longer training time and compute resources, specially on large datasets. The efficiency is questionable. Please provide information on the computational costs of their method compared to baseline approaches, or to discuss potential strategies for improving efficiency while maintaining performance gains.\n- (Minor) It would be remiss to ignore similar ablation study as in Table 3 & 4 on the other two datasets, i.e., 50 Salads and Epic-Kitchen-100, as they differ from the EGTEA Gaze+ in size, domain and data distribution. Showing more information will be essential to better evaluate the generalizability of the ActSeq. method. Please consider complete the ablation section with such experiments.\n\nMissing reference:\n- On diverse asynchronous activity anticipation, ECCV  2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "brKOhJDmBw", "reviewer": "Reviewer_s45F", "summary": "This paper introduces ActSeq, a data augmentation method for video action anticipation. To address the ordering bias in the training data, authors built a grammar base and alternate videos based on the grammar. Both observations and future action are alternated. The proposed method is evaluated on 50Salads, EGETA Gaze+, and EPIC-Kitchens-100. Results show ActSeq improves baseline methods consistently.", "questions": "1) How are the videos manimpulated? Do the augmented videos look realistic?\n2) From the reulsts in Tables 3-5, why 5x augmentation is worse than 3x?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) This paper tackels an important issue of action anticipation\n2) The action goal is specifically modeled and can be interpreted\n3) The proposed augmention method obtains promissing results on benchmark datasets.", "weaknesses": "1) For equation1, same atomic actions may appear multiple times in a video, the count may not be accurate.\n2) One action may have more than one prerequisite actions, only modeling the pairwise dependencies may not be optimal since the high-order dependencies are not considered.\n3) Some atomic actions have higher frequencies in training data, which leads may affect the AGI model\n4) There are few comparison with other augmentations methods. Only random processing is compared in Table 6. Simple augmentation methods, such as rotation and adding random noise, may also improve the performance. To demonstrate the superiority of ActSeq, a comparison with other augmentation methods is preferred. \n5) Details of generating augmented videos are missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)