AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?
Authors: Qi Zhao
Shijie Wang
Ce Zhang
Changcheng Fu
Minh Quan Do
Nakul Agarwal
Kwonjoon Lee
Chen Sun
Venue: ICLR 2024 poster (2024)
Keywords: long-term action anticipation
multimodal learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: LLMs provide temporal dynamics priors of human behaviors for long-term action anticipation

Abstract: Can we better anticipate an actor’s future actions (e.g. mix eggs) by knowing what commonly happens after the current action (e.g. crack eggs)? What if the actor also shares the goal (e.g. make fried rice) with us? The long-term action anticipation (LTA) task aims to predict an actor’s future behavior from video observations in the form of verb and noun sequences, and it is crucial for human-machine interaction.
We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and a top-down approach that infers the goal of the actor and plans the needed procedure to accomplish the goal. We hypothesize that large language models (LLMs), which have been pretrained on procedure text data (e.g. recipes, how-tos),
have the potential to help LTA from both perspectives. It can help provide the prior knowledge on the possible next actions, and infer the goal given the observed part of a procedure, respectively. We propose AntGPT, which represents video observations as sequences of human actions, and uses the action representation for an LLM to infer the goals and model temporal dynamics. AntGPT achieves state-
of-the-art performance on Ego4D LTA v1 and v2, EPIC-Kitchens-55, as well as EGTEA GAZE+, thanks to LLMs’ goal inference and temporal dynamics modeling capabilities. We further demonstrate that these capabilities can be effectively distilled into a compact neural network 1.3% of the original LLM model size. Code and model will be released upon acceptance.

Published as a conference paper at ICLR 2024
ANTGPT: CAN LARGE LANGUAGE MODELS HELP LONG-TERM ACTION ANTICIPATION FROM VIDEOS?
Qi "haoo
Shijie Wang" Brown University
Ce Zhang Brown University
Changeheng Fu Brown University
Brown University
Minh Quan Do Brown University
Nakul Agarwal Honda Research Institute
Kwonjoon Lee Honda Research Institute
Chen Sun Brown University
ABSTRACT
Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing what commonly happens after the current action (e.g. crack eggs)? What the actor also shares the goal (e.g. make fried rice) with us? The long-term action anticipation (LTAA) task aims to predict an actor's 's future behavior from video observations in the form of verb and noun sequences, and it is crucial for human-maching interaction. We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and top-down approach that infers the goal of the actor and plans the needed procedure accomplish the goal. We hypothesize that large language models (LLMs), which have been pretrained on procedure text data (c.g. recipes, how-tos), have the potential to help LTA from both perspectives. It can help provide the prior knowledge on the possible next actions, and infer the goal given the observed part of procedure, respectively. We propose AntGPT, which represents video observations as sequences of human actions, and uses the action representation for LLM infer the goals and model temporal dynamics. AntGPT achieves state- of-the-art performance on Ego4D LTA v1 and v2, EPIC-Kitchens, EGTEA GAZE+, thanks to LLMs' goal inference and temporal dynamics modeling capabilities. We further demonstrate that these capabilities can be effectively distilled into a compact neural network 1.3% of the original LLM model size. Code and model are available at www.wnpplm.ggine wtw.m..ithuu.ioAAnIGP
as well as
INTRODUCTION
Our work addresses the long-term action anticipation (LTA) task from video observations. Its desired outputs are sequences of verb and noun predictions actor of interest. LTA a crucial component for human-machine interaction. A machine agent could leverage LTA to assist humans in scenarios such as daily household tasks (Nagarajan et al.) 2020) and autonomous driving (Gao et al. (220)). The LTA task is challenging due to noisy perception (c.g. action recognition), and the inherent ambiguity and uncertainty that reside in human behaviors.
typically long-tern time horizon for
A common approach for LTA is bottom-up, which directly models the temporal dynamics of human behavior either in terms of the discrete action labels (Sun et al., 2019b), or the latent visual represen- tations (Vondrick et al. 2016). Meanwhile, human behaviors, especially in daily household scenarios, are often "purposive" (Kruglanski & Szumowskal 2020), and knowing an actor's longer-term goal can potentially help action anticipation (Tran et al. 2021). As such, we consider an alternative top-down framework: It first explicitly infers the longer-term goal of the human actor, and then plans the procedure needed to accomplish the goal. However, the goal information is often left unlabeled and thus latent in existing LTA benchmarks, making infeasible to directly apply goal-conditioned procedure planning for action anticipation.
Our paper seeks to address these challenges in modeling long-term temporal dynamics of human behaviors. Our research is inspired by prior work the mental representations of tasks as action grammars (Payne & Green 1986 Pastra & Aloimonos, 2012) in cognitive science, and by large language models' (LLMs) empirical success procedure planning (Ahn al. 2022 Driess et al.

---

Published as a conference paper at ICLR 2024
We hypothesize that the LLMs, which use procedure text data during pretraining, encode useful prior knowledge for the long-term action anticipation task. Ideally, the prior knowledge can help both bottom-up and top-down LTA approaches, they can not only answer questions such as "what are the most likely actions following this current action?", but also "what is the actor trying to achieve, and what are the remaining steps to achieve the goal?"
2023).
Concretely, our paper strives to answer four research questions on modeling human behaviors for long-term action anticipation: (I) Does top-down (i.e. goal-conditioned) LTA outperform the bottom- up approach? (2) Can LLMs infer the goals useful for top-down LTA, with minimal additional supervision? (3) Do LLMs capture prior knowledge useful for modeling the temporal dynamics of human actions? If so, what would be good interface between the videos and an LLM? And (4) Can we condense L.LMs prior knowledge into a compact neural network for efficient inference?
To perform quantitative and qualitative evaluations necessary to answer these questions, we pro- pose AntGPT, which constructs an action- based video representation, and leverages an LLM to perform goal inference and model the temporal dynamics. We conduct experiments on multiple LTA benchmarks, including Ego4D (Grauman et al. 2022). GPIC-Kitchens-55 (Damen et al. 2020), and EGTEA GAZE+ (Li et al. 2018). Our evaluations reveal the following observations: First, we find that our video representation, based on sequences of noisy action labels from action recogmition algorithms, serves as effective interface for an LLM to infer longer-term goals, both qualitatively from visualization, and quantitatively as the goals enable a top-down LITA pipeline to outperform its bottom-up counterpart. The goal inference is achieved via in-context learning (Brown et al.. 2020), which requires few numan-providee examples of action sequence and goal pairs. Second, we observe that the same video representation allows effective temporal dynamics modeling with an LLM, by formulating LTA as (action) sequence completion. Interestingly, we observe that the LLM-based tem- poral dynamics model appears to perform implicit goal- condicondi LTA, and achieves competitive performance without relying on explicitly inferred goals. These observations enable us to answer the final research question by distilling the bottom-up I.L. to compact student model 1.3% of the original model size, while achieving similar or even better LTA performance.
To summarize, our paper makes the following contributions:
1. We propose to investigate if large language models encode useful prior knowledge on modeling the temporal dynamics of human behaviors, in the context of bottom-up and top-down action anticipation.
2. We propose the AntGPT framework, which naturally bridges the L.MSs with computer vision algorithms for video understanding, and achieves state-of-the-art long-term action anticipation performance on the Ego4D LTA vI and v2 benchmarks, EPIC-Kitchens-55 and EGTEA GAZE+
3. We perform thorough experiments with two LLM variants and demonstrate that LLMs are indeed helpful for both goal inference and temporal dynamics modeling. We further demonstrate that the useful prior knowledge encoded by LLMs can be distilled into a very compact neural network (1.3% of the original LLM model size), which enables efficient inference.
2 RELATED WORK
Action anticipation can be mainly categorized into next action prediction (NAP) DDmen et al. 2020 Liet 2018) and long-term anticipation (LTA) (Grauman al.) (022)). Our work focuses on the LTA
task, where modeling the (latent) goals of the actors is intuitively helpful. Most prior works on action anticipation aim modeling the temporal dynamics directly from visual cues, such as by utilizing hierarchical representations (Lan et al. 2014), modeling the temporal dynamics of discrete action labels (Sun et al. 2019b). predicting future latent representations (Vondrick et al.) [2016] Gammulle et al. 2019b). or jointly predicting future labels and features (Girdhar & Grauman 2021] Girase et al.
)023). As the duration of each action is unknown, some prior work proposed to discover object state changes (Epstein et al. [021]] (Souček et al. 2022) as a proxy task for action anticipation. The temporal dynamics of labels or latent representations are modeled by neural networks, and are often jointly trained with the visual observation encoder in an end-to-end fashion. To predict longer sequences into the future for LTA, existing work either build autoregressive generative models (Abu Farha et al. 2018; Gammulle et al.) 2019a] Sener et al. 2020 Farha et al. 2020) use timestep conditional parameter and predict in one shot based on provided timestep (Ke et al. 2019). We consider these approaches as bottom-up as they model the shorter-term temporal transitions of human activities.
2

---

Published as a conference paper at ICLR 2024
Visual procedure planning is closely related to long-term action anticipation, but assumees that both source state and the goal state are explicitly specified. For example, [Chang et al. (2020) proposed to learn both forward and conjugate dynamics models in the latent space, and plans the actions to
take accordingly. Procedure planning algorithms can be trained and evaluated with video observa-
tions (Chang et al. 2020; Bi al. 2021 Sun et al. 2022 /Zhao et al., 2022 Narasimhan et al. 2023
Bansal ct al.) 2022). they can also be applied to visual navigation and object manipulation (Driess
al. 2023] Ahn et al. 2022 Lu al. 2023). Unlike procedure planning, our top-down LTA approach
does not assume access to the goal information. Our explicit inference of the high-level goals (with
LLMs) also differs from prior attempts to model the goal as a latent variable, which is optimized via
weakly-supervised learning (Roy & Fernando 2022:
Mascaro et al.).
Multimodal learning, such as joint vision and language modeling, have also been applied to the
action anticipation tasks. One approach is to treat the action labels as the language modality, and to distill the text-derived knowledge into
models. For example, Camporese et al. (202T) models label semantics with hand-engineered label prior based on statistics information from the training action labels. Ghosh et al. (2023) trains a teacher model with text input from the training set and distills the text-derived knowledge to uision-based student model. [Sener& & Yao (2019) transfers knowledge from a text-to-text encoder-decoder by projecting vision and language representations in shared space. Compared to these prior work, our focus is on investigating the benefits of large
language models for modeling the temporal dynamics of human activities.
3
METHOD
We introduce our proposed AntGPT framework for LTA. An overview is shown in Figure] I
3.1 LONG-TERM ACTION ANTICIPATION
The long-term action anticipation (LTA) task requires predicting a sequence of Z actions in a long future time horizon based on a video observation. In the LTA task, a long video V is split into an ordered set of N annotated short segments (5), a² J j- where S denotes the j-th segment in video V and a² denotes the corresponding action label in the form of noun-verb pair (n², v²). The video is also specified with stop time T, which is represented as the index of the last observed segment. In this way, a video is split into the observed segments Vo and the future segments of the video Vy whose labels (a(r+1), ,..; ((T+)) are to be predicted. A hyper-parameter Neeg controls how many segments the model can observe. Concretely,
take the observable video segments ( from Vo as input and output action sequence {ā(T+1) ..... a(T+))) as predictions. Alternatively, Ego-Topo (Nagarajan al. 2020) takes simplified approach, which only requires predicting the set of future actions, but not their ordering.
Bottom-up and Top-down LTA. We categorize action anticipation models into bottom-up and top- down. The bottom-up approach directly models the temporal dynamics from the history observations and predicts future actions autoregressively or in parallel. The top-down framework first explicitly infers the longer-term goal from the history actions, then plans the procedure according to both history and the goal. We define the prediction procedure of bottom-up model F as (a(T+1), ..., (((+2)) = (v.).. Here at denotes the j-th video segment's action label, and T is the index of the last observed segment. For the top-down model Fu, we formulate the prediction procedure into two steps: First, infer the goal by g = (V.)... then perform goal-conditioned planning as (ā(T+1),... ..., a(T+2)) = Fia (Vo, g), where g corresponds to the long-term goal inferred by the top-down model.
3.2 VIDEO REPRESENTATION
To understand the benefits of LLMs for video-based LTA, an important design choice is the inter- face (Zeng et al. 2022 Surís et al., 2023) between visual inputs and the language model. We are interested in investigating how to represent long-form videos in a compact, text-only bottleneck, while being helpful for goal inference and procedure planning with LLMs. The video data often contains complex and dynamic scenarios, with multiple characters, actions, and interactions occurring over an extended period. While such rich information can be potentially captured by (pretrained) visual embeddings even "video tokens" (Sun et al. 2019a Wang al. (2022). it remains unclear what visual representation would be sufficient to compress the long observed video context, while being friendly to the L.LMs.
3

---

Published as a conference paper at ICLR 2024
weah pooif.... pol polrts, by posao
[ntorpee inth, oir anneee, remoe ciite, poel prarn, potas, vemova paraos
potzar"
lustt.. was red, outrned
Temporal Model
Large Language Model
Action Recegn ton Model
pitts
Tamporal Modal
Gool Geneator
Di: baking caka
D2 nor parting tee
hos wood
Sihhepn nemore poina
Bakmup LTA
Visai Obverations V.
Gind.
j0 Crnviia Panadggna
fofeeettrr Gananator
rine, polmm.
Teacher LLM
bag, wich moce, mooe
Large Language Model
Case
Sharpen kells...
Studant st,
Di calr pince dagg equance 30x DS aed peod, hodd pood an held sandet veod
Ved uooiiii
NiTwwobt ixcoler-ieniigg
Figure I: Illustration of AntGPT. (a) Overview of LTA pradigms. The bottom-up approach predicts future actions directly based on observed human activities, while the top-down approach is guided by high-level goals inferred from observations (hence allows procedure planning). (b) Actions as video representations. A pre-trained action recognition model & takes visual observations V, as inputs and generates action labels, which can be noisy (shown in red). (c) Goal inferred by an LLM. We provide few human- provided examples of action sequences and the expected high-level goals, and leverage LLM infer the goal via in-context learning.
(d) Knowledge Distillation. We distill frozen LLM m, into a compact student model T. at sequence level. (e) Few-shot LTA by in-context learning (ICL), where the ICL prompts can be either bottom-up top-down.
We first consider the standard approach represent video frames as distributed embedding representa- tions, computed with pre-trained vision backbone models, such as the CLIP visual encoder (Radford et al. 2021). For each video segment S in Vo, the backbone extracts the representations of uniformly sampled frames from this segment to obtain E' = {er, e ). A neural network then take the embedding representation and predict action labels for the observed frames (action recognition), or the future timesteps (action anticipation).
Our action recognition network E is implemented as a Transformer encoder. It takes in the visual embeddings and one learnable query token as the input. We then apply two separate MLP heads to decode the verb and noun from the encoded query token. For each observed video segment S the recognition model E takes in randomly sampled image features E = fea. es.... )x E C E,, and outputs the corresponding action a for St. This process is repeated for every labeled segment in Vo, which results in Neeg actions (a(T-Na),... ,..., a(T)). in the format of noun-verb pairs. The recognition model & is trained on the training
Na)
minimize the Cross Entropy Loss between the
predictions and the ground-truth action labels.
How to Represent Videos for the LLMs? We consider simple approach to extract video representa- tions for large language model. We first compute the embedding representation of Vo, and then apply the action recognition model & to convert the distributed representation into discrete action labels, which can be directly consumed by off-the-shelf LLM. Despite its simplicity, we observe that this representation is strong enough for the LLM extract meaningful high-level goals for top-down LTA (see Section 3.3) and can even be applied directly to perform both bottom-up and top-down LTA with the LLMs. Alternative approaches, such as discretizing the videos via video captioning or object detection, or projecting the visual embedding via parameter-efficient fine-tuning (Hu et al. 202 Merullo et al. 2023). can also be applied under our framework.
3.3 ANTGPT: LONG-TERM ACTION ANTICIPATION WITH LLMS
We now describe AntGPT (Action Anticipation GPT), a framework that incorporates LLMs for the LTA task. An LLM serves both few-shot high-level goal predictor via in-context learning, and also as a temporal dynamics model which predicts the future actions conditioned on the observed actions. It hence benefits top-down and bottom-up LTA, in full-shot and fhw-shot scenarios.
4

---

Published as a conference paper at ICLR 2024
Few-shot Goal Inference. In order to perform top-down long-term action anticipation, conduct in-context learning on LLMs to infer the goals by taking the recognized action labels as inputs, as illustrated in Figure (b) (bb) and (c). The ICL prompts 9gol is formulated with examples in the format of <observed act i ons> => <goa 1> " and the final query in the format of "<bberrve rved actions> =>" The observed actions for the in-context examples are based on ground-truth annotations, and the observed actions in the final query are generated by recognition models, Since no ground truth goals are available, we either use the video metadata as pseudo goals when it is available, or design the goals manually. Figure 2 shows several examples for in-context goal inference with the LLM. We treat the raw output of the LLM T gea = x (9goal) ) as the high-level goal.
Bottom-up and Top-down LTA. We now describe unified framework to perform bottom- up and top-down LTT. The framework largely resembles the action recognition network which takes visual embeddings as inputs, but has a few important distinctions. Let's first consider the bottom-up model B. Its transformer encoder takes sub-sampled visual embeddings E from each segment S of Vo. The embeddings from different segments are concatenated together along the time axis to form the input tokens to the transformer encoder. To perform action anticipation, we append additional learnable query tokens to the input sequence of the Transformer encoder, each of which corresponds to a future step to predict. Each encoded query token is decoded into verb and noun predictions with two separate MLP heads. We minimize the Cross Entropy Loss for all future actions be predicted with equal weights. Note that one can choose to use either bidirectional or causal attention masks for the query tokens, resulting in parallel or autoregressive action prediction. We observe that this design choice has marginal impact on performance, and use parallel decoding unless otherwise mentioned.
Thanks to few-shot goal inference with in-context learning, implementing the top-down model Fia is straightforward: We first embed the inferred goals gea with a pre-trained CLIP text encoder. The goal token is then prepended the beginning of the visual embedding tokens to perform goal-conditin action anticipation. During training, we use ground -tutt action labels to infer the goals via in-context learning. During evaluation, we use the recognized action labels to infer the goals.
Modeling Temporal Dynamics with LLMs We further investigate if LLMs are able to model temporal dynamics via recognized action labels and perform action anticipation via autoregressive sequence completion. We first study the fully supervised scenario, where perform parameter- efficient (optionally) fine-tuning on LLMs on the training set of an LTA benchmark. Both the input prompt and the target sequence constructed by concatenating the action labels separated with commas. During training, the input sequences are formed either via teacher forcing (ground truth actions), or the (noisy) recognized actions. The LLM is optimized with the standard sequence completion objective. During inference, we use the action recognition model & to form input prompts from the recognized action labels. We perform postprocessing to convert the output sequence into action labels. Details of the postprocessing can be found in Section C.I To perform top-down LTA, we simply prepend an inferred goal at the beginning of each input prompt. The goals are again inferred from ground-truth truth actions during training, and recognized actions during evaluation.
Knowledge Distillation (Hinton et al. 2015) is applied understand if the knowledge encoded by LLMs about temporal dynamics can be condensed into a much more compact neural network for efficient inference. For sequence models such as I.LMs, the distillation loss is calculated as the sum of per-token losses between the encoded feature (e.g. logits) sequences by the teacher and the student. Formally, during distillation, given the input sequence x of length N, a well-trained LLM as the teacher model Te, the student model na is optimized to minimize the language modeling loss fim and distillation loss Lais Σn Dri (v) [!!"))) where Ot = (x) and f. = =x(x)) are the feature sequence encoded by म, and n. respectively, i is the token index of the target sequence, and DEL is the bullback-Leible divergence between the teacher and student distribution. The teacher model ne is frozen during training. An illustration is shown in Figure (d).
Few-shot Learning with LLMs. Beyond fine-tuning, we are also interested in understanding if LLM's in-context learning capability generalizes to the LTA task. Compared with fine-tuning model with the whole training set, in-context learning avoids updating the weights of pre-trained LLM. As illustrated in Figure [ (c), an ICL prompt consists of three parts: First, an instruction that specifies the anticipating action task, the output format, and the verb and noun vocabulary. Second, the in-context examples randomly sampled from the training set. They are in the format of "<observed actions> => <future act ions> with ground-truth actions. Finally, the query in the format "<observed actions> => " with recognized actions. An example of
5

---

Published as a conference paper at ICLR 2024
Resognized
Helpful
Goa. Paint doar and wood with paintbouth
put söck
paint docs
pannt paintbrush
put cortsiret
dip paint
paint docee
Resognized
(b)
Future
Coael Crean cet and med with valcaun doaams
vocJum
put pipe
vacuum
Resognized
(c)
Goal: Put on shocs
F.l pull push
Put
Figure 2: Examples of the goals inferred by LLMs. Goals are inferred from the recognized actions of the observed segments. The future actions are ground truth for illustration purposes.
the model's ss input and output is shown in Figure A1 (b). Alternatively, we also attempt to leverage bhain-of-thought prompts (Wei al. 2022) (Col) to ask the LLM first infer the goal, then perform LTA conditioned on the inferred goal. An example of CoT LTA is shown in Figure A1 (c).
EXPERIMENTS
We now present quantitative results and qualitative analysis on the Ego4D GGrauman al. 2022).
EPIC-Kitchen. (Damen et al. 2020), and EGTEA Gaze+ (Li et al. 2018) benchmarks.
EXPERIMENTAL. SETUP
Ego4D Grauman al. 2022) The dataset contains 3,670 hours of egocentric video of daily life activity spanning hundreds of scenarios. We focus on the videos in the Forecasting subset which contains 1723 clips with scenarios. The total duration is around 116 hours. There are 115 verbs and 478 nouns in total. The v2 dataset contains 3472 annotated clips with total duration of around 243 hours. There are 117 verbs and 521 nouns. We follow the datasets' standard splits.
EPIC-Kitchens-55 (Damen al. 2020) (EK-55) contains 55 hours egocentric videos of cooking activities of different video takers. Each video is densely annotated with action labels, spanning over 125 verbs and 352 nouns. We adopt the train and test splits from Nagarajan et al. (2020).
EGTEA Gaze+ (Li et al. 2018) (EGTEA) contains 86 densely labeled egocentric cooking videos over 26 hours. There are verbs and 53 nouns. We adopt the splits from Nagarajan et al. (2020).
Evaluation Metrics. For Ego4D, we use the edit distance (ED) metric. It is computed as the Damerau-Levenshan distance over sequences of predictions of verbs, nouns actions. We follow the standard practice in Grauman et al. (2022) and report the minimum edit distance between each of the top K 5 predicted sequences and the ground-truth. We report Edit Distance at 7 20 (ED@ 20) on the validation set and the test set. For EK-55 and EGTEA, we follow the evaluation metric described in Nagarajan et (202)). The first K% of each video given as input, and the goal is to predict the set of actions happening in the remaining (100-K)% of the video as multi-class classification. We sweep values of K [25%, 50%, 75%] representing different anticipation horizons and report mean average precision (mAP) on the validation sets. We report the performances on all target actions (All), the frequently appeared actions (Freq), and the rarely appeared actions (Rare) as in Nagarajan et al. (2020). A number of previous work reported performance on these two datasets. The order agnostic LTA setup these two datasets complements the Ego4D evaluation.
Implementation Details. We use the frozen CLIP (Radford et al. 2021) VIT-L/14 for image features. and a transformer encoder with 8 attention heads, and 2048 hidden size for the recognition model. To study the impact of vision backbones, we also include VgP, video backbone pre-trained on Ego4D datasets. For the large language models, we adopt open source Llama2-15 13B for in-context learning and 7B model for fine-tune fine-tuning. For comparison, also OpenAl's GPT-3.5 Turbo for in-context learning and GPT-3 curie for fine-tuning. More details and ablation study on recognition model, teacher forcing, L.LMs and other design choices descrihed in appendix.

---

Published as a conference paper at ICLR 2024
4.2 CAN LLMs INFER GOALS TO ASSIST TOP-DOWN LTA?
We compare two LLMs, GPT-3.5 Turbo and Llama2-chat-13B, on goal inference: To obtain the pscudo ground-truth goals for constructing the in-context examples, we use the video titles for EGTEA, and the video descriptions for EK-55. We manually annotate the goals for Ego4D. We use 12 in-context examples to infer the goals. For EK-55 and EGTEA, we always use the recognized actions in the first 25% of each video infer the goals. For Ego4D, set Naeg 8.
We first use the Transformer encoder model described in Section 3.33ion[3.3Jas the temporal model: allows us study the standalone impact of goal conditioning by comparing the bottom-up and the top-down LTA performances. The Transformer encoder takes in the same visual features as used for action recognition. The text embeddings of the inferred goals are provided for the top-down variant. Table [ shows results on Ego4D v1, EK-55, and EGTEA. We notice a clear trend that using the inferred goals leads to consistent improvements for the top-down approach, especially for the rare actions of EK-55 and EGTEA. We also noticed that both LLMs are able to infer helpful goals for top-down LTA and GPT-3.5 Turbo generates goals slightly better than the ones from Llama2-chat 13B. We also construct "oracle goals" using the video metadata provided by EK-55 and EGTEA datasets. We observe that using the oracle goals leads to slight improvements, indicating that the inferred goals already offer competitive performance improvements. Figure 2 provides some examples of the helpful and unhelpful goals inferred by Llama2.
Ego4d v1 (ED)
EK-55 (mAP)
EGTEA (mAP)
Method
Verb
Noun +
ALL
Freq f Rare
ALL f
Freq f Rare
image features
0.735
0.753
38.2
59.3
29.0
78.7
84.7
68.3
image features Llama2 inferred goals
0.728
0.747
40.1
58.1
32.1
80.0
84.6
70.0
image features + GPT-3.5 inferred goals
0.724
0.744
40.1
58.8
31.9
80.2
84.8
72.9
image features + oracle goals'
40.9
58.7
32.9
81.6
86.8
69.3
Table 1: Impact of goal conditioning on LTA performance. Goal-conditioned (top-down) models outperforms the bottom-up model in all three datasets. We report edit distance for Ego4D,, mAP for EK-55 and EGTEA. All results are reported on the validation set.
4.3 Do LLMs MODEL TEMPORAL DYNAMICS?
Model
Goal
Input
Verb
Noun
Model
Goal
Verb
Noun
Transformer
GPT-3.5
image features
0.724
0.744
GPT-3-curie
No
0.707
0.719
GPT-3-curie
GPT-3.5
recog actions
0.709
0.729
GPT-3-curie
Yes
0.709
0.729
Transformer
Llama2-13B
image features
0.728
0.747
Llama2-7B
No
0.704
0.705
Llama2-7B
Llama2-13B
recog actions
0.700
0.717
Llama2-7B
Yes
0.700
0.717
Table 2: Comparison of temporal models for top-down LTA. Results on Ego4D vl val set.
Table 3: Top-down vs Bottom-up for
LLM-based LTA. Results on v1 val set.
We further explore if LLMS can be directly applied model temporal dynamics. We focus on the Ego4D benchmark as it measures the ordering of the anticipated actions.
LLMs are able to model temporal dynamics. To utilize an LLM to predict future actions, we adopt the same video representation as used for in-context goal inference but fine-tune the LLM on the training set. For bottom-up LTA, we by default perform teacher forcing during training, and concatenate the Noeg ground-trutt action labels as the input sequence. Z ground-trutt action labels are concatenated as the target sequence. During evaluation, we concatenate Nogg recognized actions as input, and postprocess the output sequence into Z anticipated actions. For top-down LTA, we prepend the inferred goals to the input sequence.
We conduct top-down LTA with the open-sourced Llama2-7B LLM. During training, we adopt parameter-efficient fine-tuning (PEFT) with LoRA (Hu et al. 2021) and 8-bit quantization. We compare with the transformer baseline with image features, and report results on Ego4D vl validation set in Table 2 We observe that leveraging the LLM as the temporal dynamics model leads to significant improvement, especially for nouns. Additionally, layers (and hence increasing the model size) does not improve the performance of the image feature baseline (see Table AT in appendix), confirming that the improvement comes from the action
validate that simply adding more
7

---

Published as a conference paper at ICLR 2024
representation and better temporal dynamies modeling. The results demonstrate the effectiveness of action-based representation, when an LLM is used for temporal dynamics modeling.
LLMs can perform few-shot temporal modeling. We further tested LLMs' ability to model temporal dynamics when only shown a few examples. We consider both in-context learning (ICL) and chain-of- of-tthougtss (CoT) and compare them with transformer model trained from-scratch with the same examples. The results are illustrated in TabldA6] We observed that LLMs can model temporal dynamics competitively in a few-shot setting. As expected, chain-of-thoughee outperforms regular in-context learning, but both significantly outperform fine-tuning the Transformer model.
LLM-based temporal model performs implicit goal inference. We have shown that LLMs can assist LTA by providing the inferred goals, and serving as the temporal dynamics model, respectively, Does combining the two lead further improved performance? Table aims to answer this question. We report results with fine-tuned Llama2-7B 7B and GPTT--curie as the temporal model, which use Llama2-Chat-Ia 13B and GPT-3.5 Turbo for goal inference, respectively. We empirically observe that the bigger models lead to better inferred goals, while the smaller models are sufficient for temporal modeling. We observe that the bottom-up performance without explicitly inferred goals are on par (marginally better) with the top-down models for both LLMs. This indicates the LLM may implicitly inferred the goals when asked to predict the future actions, and performing explicit goal inference is not necessary. In the following experiments, we stick with this implicit goal inference setup.
Seq Type
Verb 4
Noun 4
Action +
Model
Setting
Verb 4
Noun 4
Action +
Action Labels
0.6794
0.6757
0,8912
7B
Pre-trained
0.6794
0.6757
0.8912
Shuffled Labels
0.6993
0.6972
0,9040
91M
From-scrated
0.7176
0.7191
0.9117
Label Indices 0.7249
0.6805
0.9070
91M
Distilled 0.6649
0.6752
0.8826
Table 4: Benefit of language prior. Results on
Table 5: LLM as temporal model. Results on Ego4D v2 test set. Llama2-7B model is fine-tuned on Ego4D v2 training set. 91M models are randomly initialized.
Ego4D v2 test set. We replace original action se- quences to semantically nonsensical sequences.
Language prior encoded by LLMs benefit LTA. We further investigate if the language (e.g. goals and action labels) used for video representation is actually helpful to utilize the language priors encoded by the LLMs. We first conduct experiments by replacing the action label representation with two representations that we assume the pretrained LLMs are unfamiliar with: (1) Shuffled Labels. We randomly generate a mapping of verbs and nouns so that the original verbs/nouns are 1-to-1 projected to randomly sampled words in the dictionary construct semantically nonsensical language sequence (e.g "open window" to "eat monitor"). (2) Label Indices. Instead of using words to represent actions in the format of verb-noun pairs, we can also use the index of the verb/noun in the dictionary to map the words to digits to form the input and output action sequence.
We fine-tune the Llama2-7B model on the three types of action representations on the Ego4D v2 dataset and report results on the test set. As shown in Table 4 the performance drops severely when shuffled action labels or label indices are used, especially for verb. The performance gap indicates that even LLMS have strong capability to model patterns beyond natural language (Mirchandani et al., 2023), the encoded language prior from large-scale pre-training still significantly benefits long- term video action anticipation.
LLM-encoded knowledge can be condensed into a compact model. We first introduce the baseline model I. Llama2-91M, which is 6-layer randomly initialized transformer decoder model with the similar structure as Llama2-7B. BB. The 91M model takes in the same input during training and evaluation and follows the same post-processing. We then conduct model distillation to use the Llama2-7B model tuned on Ego4D v2 training set as the teacher model and the same randomly initialized Llama2-91M as the student model. Results on test set are shown in Table [5] We observe that the distilled model achieves significant improvement comparing with model trained without distillation in the second row (7.3% and 6. 1%% % for verb and noun). It's also worth noting that the distilled 91M model even outperforms the 7B teacher model on all three metrics, while using 1.3% of the model size. The results confirm that LLM-encoded knowledge on implicit goal inference and explicit temporal modeling can be condensed into compact neural network.
8

---

Published as a conference paper at ICLR 2024
Method
Version
Verb
Noun
Action
HierVL (Ashutosh
2023
vI
0.7239
0.7350
0.9276
ICVAD(Mascaru el
vl
0.7410
0.7396
0,9304
VCLIP
Daa & Kyon
2022
VI
0.7389
0.7688
0.9412
Slowfast fast
Graamaan cl al
VI
0.7389
0.7800
0.9442
AntGPT (ours)
vI
0.6584
0.6546=338-3
0.8814#3.ee
Slowfast (Grauman et al.
2022
v2
0.7169
0.7359
0.9253
VideoLLM (Chen et al. 2025
V2
0.721
0.725
0.921
PaMsEgoA1 (Ishibashi Palm (Huang et 2025
0,6838
0.6785
0.8933
0.6956
0.6506
0,8856
AntGPT (ours)
v2
0.650
0.6498
0.8770
Table 6: Comparison with SOTA methods on the Ego4D v1 and v2 test sets in ED@ @02. Ego4d v1 and v2 share the same test set. V2 contains more training and validation examples than vl.
EK-55
EGTEA
Method
AL..
FREQ
RARE
ALI
FREQ
RARF
B3D (Carreira & Zisserman 2017
32.7
53.3
23.0
72.1
79.3
53.3
ActionVLAD (Girdna
29.8
53.5
18.6
73.3
58.6
Timeception Hussei
35.6
55.9
26.1
74.1
59.7
VidcoGraph Husscin ef al. 20196
22.5
49.4
14.0
67.7
77.1
47.2
EGO-TOPO Nagaraj cf al. 2020
38.0
56.9
29.2
73.5
80.7
54.7
Anticipatr (Nawhal et al. 2022
39.1
58.1
29.1
76.8
83.3
55.1
AntGPT (ours)
40.1
58.8=26-1
31.9+3-22
80.2+22-1
84.8=26-1
72.9
Table 7: Comparison with SOTA methods the EK-55 and EGTEA Dataset in mAP. ALL,
FREQ and RARE represent the performances on all. frequent, and rare target actions respectively.
4.4 COMPARISON WITH STATE-OF- TTA-
Finally, we compare AntGPT with the previous state-of-the-ar methods. We choose the model design settings such as recognition models and input segments number based on ablation study discussed in appendix [ection[A] For Ego4d v1 and v2, we train the action recognition and fine-tune the LLM temporal models with their corresponding training set. Table shows performance comparisons on Ego4D vl and v2 benchmarks. We observe that AntGPT achieves best performance and largely outperforms other SoTA baselines. Since Ego4d and v2 share the same test set, is also worth mentioning that our model trained solely v1 data is able outperform any other models trained on the v2 data, which indicates the data efficiency and the promise of our approach.
both datasets
For EK-55 and EGTEA, we compare the goal-conditioned AntGPT with the previous saate-of-the-ann results in Table 7 AntGPT achieves the overall best performance on both datasets. We observe that proposed model performs particularly well on rare actions.
CONCLUSION AND FUTURE WORK
In this paper, propose AntGPT to investigate if large language models encode useful prior knowledge on bottom-up and top-down long-term action anticipation. Thorough experiments with two L.M variants demonstrate that I.LMs are capable of inferring goals helpful for top-down LTA and also modeling the temporal dynamics of actions. Moreover, the useful encoded prior knowledge from LLMs can be distilled into very compact neural networks for efficient practical use. Our proposed method sets new state-of- performances on the Ego4D LTA, EPIC-Kitchens-55, and EGTEA GAZE+ benchmarks. We further study the advantages and limitations of applying LLM on video-based action anticipation, thereby laying the groundwork for future research in this field.
Limitations. Although our approach provides a promising new perspective tackling the LTA task, there are limitations that are worth pointing out. The choice of representing videos with fixed-length actions is both efficient and effective for LTA task. However, the lack of visual details may pose constraints other tasks. Another limitation is the prompt designs of ICL and CoT are still empirical, and varying the prompt strategy may cause significant performance differences. Finally, as studied in counterfactual experiments, the goal accuracy would have significant impact the action recognition outputs, and an important future direction is to improve the inferred goal accuracy, and also take multiple plausible goals into account.
Acknowledgements. We would like to thank Nate Gillman for valuable feedback. This work in part supported by Honda Research Institute, Meta AI, and Samsung Advanced Institute of Technology.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yazan Abu Farba, Alexander Richard, and Juergen Gall. When will you do what?-anticipating temporal occurrences of activities. CVPR, 2018.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166 2022.
Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video-language embeddings. CVPR, 2023.
Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In ECCV, 2022.
Jing Bi, Jiebo Luo, and Chenliang Xu. Procedure planning in instructional videos via contextual modeling and model-based policy learning. In ICCV, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Guglielmo Camporese, Pasquale Coscia, Antonino Furnari, Giovanni Maria Farinella, and Lamberto Ballan. Knowledge distillation for action anticipation via label smoothing. In ICPR, 2021.
J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In CVPR, 2017.
Chien-Yi Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. ECCV, 2020.
Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, al. Vidcollm: Modeling video sequence with large language models, arXiv preprint arXiv:2305. 13299 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and baselines. TPAMI, 2020.
Srijan Das and Michael S Ryoo. Video+ clip baseline for ego4d long-term action anticipation. arXiv preprint ar arXiv:2207.000 005000 2022.
Danny Dricss, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
Dave Epstein, Jiajun Wu, Cordelia Schmid, and Chen Sun. Learning temporal dynamics from cycles in narrated vidco. ICCV, 2021.
Yazan Abu Farha, Qiuhong Ke, Bernt Schiele, and Juergen Gall. Long-term anticipation of activities with cycle consistency. GCPR, 2020.
Harshala Gammulle. Simon Denman, Sridha Sridharan, and Clinton Fookes. Forecasting future action sequences with neural memory networks. BMVC, 2019a.
Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Predicting the future: A jointly learnt model for action anticipation. ICCV, 2019b.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In ICCV, 2020.
10

---

Published as a conference paper at ICLR 2024
Sayontan Ghosh, Tanvi Aggarwal, Minh Hoai, and Niranjan Balasubramanian. Text-derived knowl- edge helps vision: A simple cross- -oddl distillation for video-based action anticipation. In EACL, 2023.
Harshayu Girase, Nakul Agarwal, Chiho Choi, and Karttikeya Mangalam. Latency matters: Real-time action forecasting transformer. In CVPR, 2023.
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In ICCV, 2021.
Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell. Actionvlad: Learning spatio-temporal aggregation for action classification. CVPR, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXhv:2106.09685. 2021.
Daoji Huang, Otmar Hilliges, Luc Van Gool, and Xi Wang. Palm: Predicting actions through language models@ ego4d long-term action anticipation challenge 2023. arXiv preprint arXiv:2306.16555 2023.
Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders. Timeception for complex action recognition. CVPR, 2019a.
Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders. Videograph: Recognizing minutes-long human activities in videos. arXiv preprint arXiv:1905.05143 2019b.
Tatsuya Ishibashi, Kosuke Ono, Noriyuki Kugo, and Yuji Sato. Technical report for ego4d long term action anticipation challenge 2023. arXiv preprint arXiv:2307.01467, 2023.
Qiuhong Ke, Mario Fritz, and Bernt Schiele. Time-conditioned action anticipation one shot. In CVPR, 2019.
Arie W Kruglanski and Ewa Szumowska. Habitual behavior goal-driven. Perspectives on Psychological Science, (5(5):2256-12)) 2020.
Tian Lan, Tssug-Chuan Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In ECCV, 2014.
Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In ECCV, 2018.
Yujie Lu, Weixi Feng, Wanrong Zhu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Neuro-symbolic procedural planning with commonsense prompting. ICLR, 2023.
Esteve Valls Mascaro, Hyemin Ahn, and Dongheui Lee. Intention-conditionned long-term human egocentric action forecasting @ ego4d challenge 2022. URL 2207. 12080
Jack Merullo, Louis Castricato, Carsten Fickhoff, and Ellie Pavlick. Linearly mapping from image to text space. ICLR, 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models general pattern machines. arXiv preprint 20470 2023.
Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environ- ment affordances from egocentric video. In CVPR, 2020.

---

Published as a conference paper at ICLR 2024
Medhini Narasimhan, Licheng Yu, Sean Bell, Ning Zhang, and Trevor Darrell. Learning and verification of task structure in instructional videos. arXiv preprint arXiv:2303.13519, 2023.
Megha Nawhal, Akash Abdu Jyothi, and Greg Mori. Rethinking learning approaches for long-term action anticipation. In ECCV, 2022.
Katerina Pastra and Yiannis Aloimonos. The minimalist grammar of action. Philosophical Transac- tions of the Royal Society B: Biological Sciences, 367(1585):003-111 2012.
Stephen J Payne and Thomas RG Green. Task-action grammars: A model of the mental representation of task languages. Stuman-computer interaction, 2(2):93-333, 1986.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103..00020, 2021.
Debaditya Roy and Basura Fernando. Predicting the next action by modeling the abstract goal. arXiv preprint arXiv:2209.0515 05044, 2022.
Fadime Sener and Angela Yao. Zero-shot anticipation for instructional activities. In ICCV, 2019.
Fadime Sener, Dipika Singhania, and Angela Yao. Temporal aggregate representations for long-range video understanding. In ECCV, 2020.
Tomáš Souček, Jean-Baptiste Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. Look for the change: Learning object states and state-mdiifying actions from untrimmed web videos. In CVPR, 2022.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019a.
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Rahul Sukthankar, Kevin Murphy, and Cordelia Schmid. Relational action forecasting. In CVPR, 2019b.
Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, and Animesh Garg. Plate: Visually- grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters, 7(2):492444930, 2022.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python exccution for reasoning. ICCV, 2023.
Hung Tran, Vuong Le, and Truyen Tran. Goal-driven long-term trajectory prediction. In WACV, 2021.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11993. 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong. Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0000
2022.
He Zhao, Isma Hadji, Nikita Dvomik, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. P3iv: Probabilistic procedure planning from instructional videos with weak supervision. In CVPR, pp. 2938 2948, 2022.
12

---

Published as a conference paper at ICLR 2024
A ABLATION STUDY
In this section, we conduct ablation study on model designs for the Transformer model and LLM- based model Ego4D vl and v2 to show how choose the final settings. In the table, rows labeled as gray denote the final setting we use to report in Table
A.1 TRANSFORMERS: NUMBER OF LAYERS
We want to compare fine-tuned LLM with auto-regressive transformers trained from-scratch. However, our transformer model and the large language model we used do not have comparable parameter size. Here, we want to explore how parameter size affect the performance of the transformer and to investigate whether the performance gap we demonstrated in 4.4 is due to model size. We ran the transformer model with varying number of layers from to 6 and the corresponding results are in We observed that adding more layers to the auto-regressive transformer model does not
Table Al help performance. This further substantiate our conclusion that large language models are better in modeling action temporal dynamics.
# layers
Verb |
Noun
0.739
0.758
0.741
0.756
0.743
0.756
0.742
0.755
0.743
0.757
0.747
0.759
Table A1: Ablations on the number of layers to auto-regressive transformers. Transformers trained with actions labels from-scratch. Results on Ego4d v1 validation set. Layer number of Transformer model barely influence its performance.
A.2 TEACHER FORCING
We developed data augmentation technique to further boost the performance of our approach. When reporting on test set, we can augment the training set with recognized action tokens which comes from a recognition model that is trained on the validation set and makes inference on the training set. In this way, we double our training set with more noisy data, the distribution of which is closer the recognized action tokens that input into the fine-tuned model when making inference on the test set. Table A4 shows the comparison between three training paradigms.
A.3 RECOGNITION MODEL
To study the impact of vision backbones, altered the vision features from CLIP Ego and trained on the same architecture. The result shown Tabld/2 We observed that recognition models based ELP features outperformed CLIP features. This is expected since EgoVLP is pretrained Ego4D datasets. This finding suggests that our approach is robust to different recognition models and can be further improved with better recognition models.
A.4 LARGE LANGUAGE MODELS
We compared different LLMs for their performance on fine-tuning. Table shows the result on Ego4D v2 test set. We observe that Llama2-7B with PEFT performs better than fine-tuned GPT-3.
A.5 NUMBER OF INPUT SEGMENTS
Table A$ shows the influence of the number of input segments to LLM. A higher number of input segments allows the LLM to observe longer temporal context. We choose the input segments from (1, 3, 8). We did not go beyond because that was the maximum segments allowed by the Ego4d benchmark. general, AntGPT's performances on both verb and noun increase as we increase Nayy This is different from the vision baseline in Table 1, whose performance saturates at N = 3.
13

---

Published as a conference paper at ICLR 2024
Recog Model
Verb
Noun
Action +
Temporal Model
Verb 1
Noun
Action 1
CLIP
0.6794
0.6757
0.8912
GPT-3 curie
0.6969
0,6813
0.9060
EgoVLP
0.6677
0.6607
0.8854
Llama2-7B
0.6794
0.6757
0.8912
Table A2: Ablation on recognition model. Results on Ego4D) v2 test set. EgoVI.P as recognition model brings better performance.
Table A3: Ablation on LLM as temporal model. Results on Ego4D v2 test set. Llama2-7B perform better than GPT-3 curie as tempral model.
Input
Verb
Noun
Action +
seg
Verb
Noun +
gt 0.6794
0.6757
0.8912
0.734
0.748
recog
0.6618
0.6721
0.8839
0.717
0.726
gt+recog
0.6611
0.6506
0,8778
5
0.723
0.722
8
0.707
0.719
Table A4: Ablation on teacher forcing, Results on Ego4D v2 test set. Using ground-truth and recognized actions as training samples bring best performance.
Table A5: Ablations on input segments num-
ber to LLM. Results on Ego4d v1 validation set. Segment number of 8 brings best performance.
B FEW-SHOT ACTION ANTICIPATION
Model
Learning Method
With Goal
Verb J
Noun |
Transformer
SGD
No
0.770
0.968
Llama2-chat-133 13B
ICI.
No
0.761
0.803
GPT-3.5
ICL
No 0.758
0.728
GPT-3.5
ICL
Yes 0.775
0.728
GPT--3.5
CoT
Yes
0.756
0.725
Table A6: Few-shot results with LLM on Ego4D v1 validation set. The transformer baseline is trained with the same 12 examples from training set as the in-context examples for LLM.
We conduct quantitative few-shot learning experiments on the Ego4D vI validation set, and compare the transformer-bas baase model and LLM. The transformer baseline is trained on the same 12 examples sampled from the training set of Ego4D as the in-context examples used by LLM. The transformer is optimized by gradient descent. For top-down LTA with LLM, we compare the two-stage approach to infer goals and predict actions separately, and the one-stage approach based on chain-of offiin-ottthhhtts.. Results are shown in Table A6 We observe that all LLM-based methods perform much better than the transformer baseline for Tew-shot LTA. Among all the LLM-based methods, top-down prediction with CoT prompts achieves the best performance on both verb and noun. However, the gain of explicitly using goal conditioning marginal, similar to what we have observed when training the full set. Figure (b] (b) and (c), we illustrate the example input and output for ICL and CoT, respectively. More examples be found in Section D
B. COUNTERFACTUAL PREDICTION
To better understand the impact of goals for action anticipation, we design "counterfactual" pre- diction experiment: We first use GPT-3.5-Turbo to infer goals for examples as we did above and treat the outputs as the originally inferred goals, and then manually provide "altered goals" which are different than the original goals but are equally reasonable. We then perform in-context learning using both sets of the goals and with the same sequences of recognized actions as inputs to the LLM. Figure A2 illustrates our workflow and two examples. In the second example, although the recognized actions are the same, the output action predictions are all different. Despite the differences, using "fix machine" as the goal generates actions highly related to fixing such as "z "nighten nut"" and "turn screw" while switching to "examine machine' leads to actions like "test machi ne" and "record data". More examples can be found in Section D
can see that
14

---

Published as a conference paper at ICLR 2024
Observations
Actions
put pointbrush
paint wall
paint well
dip polmbrush
paint wall
dop brush
paint well
Predict
"rstruction: "Whst
tork, tork, put tork, wash intte take actions?
clear tabreware
Q stick pull string will,
point wall
(nnower ppanting room
Bb Buttomup Prodict Fulure Instruction: What will be the possible next actions according to given previous actions? You should follow the following rulos...
the lest comry leaf, put plant, adjust deth take plant, put plant, tave sickte take sidde no out plant, put sioce... out plant, cut plant, hold plant Q put stick put stick put string paint wall paint will point paint, paint paint paint wall
[Answer)
"okaan brush dp brush. point wal..... paint wall paint wall, dlean brush?
Ves, 2D 0.0
(6) Top-dowes
and Future
"inatructon: What
possibie
actions aczording givwn presious aethoned You shourd follow the fallmooing riles
D1: the leat carry leaf, put plant, adjss dloth tak plant put plant, tase sickle, take siide on: What' the goal according to previous actions? az: ahat the fuluww 20 actorsd wi The good gardaring Fulure 20 action see out plani put skodle cul plans, cut plant hold plant
Q put stck put slick pull stong, swin wall paint wal paint paint paett paint paint wall Q1: What's the goal according q provious action? 02 What are The future 20 actori??
[Arsow)
The goat is pairling Future 20 actiore are dip Eruals paint vol.
wall buahl,
No BD 8.50
Figure Al: Illustration of goal prediction and LTA with LLMs: (a) High-level goal pre- diction wth in-context learning (ICL), (b) Few-shot bottom-up action prediction with ICL. (c) Top-down prediction with thain-of- (CoT). The green word indicates correctly recognized actions (inputs to the LLM) and future predictions (outputs of the LLM), red indicates incorrectly recognized observations are [put paint ntbrush, adjust paintbrush, take container, dip container, paint wall, paint wall, dip wall, paint wall].
predicted actions. For this example, the ground-truth
Original Prediction
Counter Factual Predictions
[Inerred Goal]: gardening or trimming plants
[Allered Goof] harveeting crops
(Obeervations Inpul]: take plant, take plant, take trimmer. lake soil, take soil, take scil. soil, take
[Prediction.
[Prediction:
"put plant, adjust plant, adjoust soil take trimmer. trim plant, trimmer, ... put trimmor. take spmad adjust soil
"oul plant, out plant, nill basket, take basket, carry basket, baskett ..., pour soil, put soil, take soil, pour sail, soil'
[Inforrdd Goal]: machine
[Altered Goal]: examine machine
[Obeervations Imput]: screw screwdriver, hold pecial, hold pedal, hold lever. hold pipe, hold pipe. hald pipe, hold lever
[Prediction::
|Prediction):
Nighten balt, adjust lever.... .... turn screw, adjust pipe, tighten nut. hold lever, lighten nut, hold pedal, adjust pipe, hold lever, turn screwdriver"
usdust gauge, read gauge, ... lighten bolt, loosen screw, remove cover, examine circuit, adjus) setting, lest machine, record data, lool, close paner"
Figure A2: Illustaatoons of counterfactual prediction. We replace the originally inferred goal (gardening or trimming plants and fix machine) with altered goal (harvesting crops and examine machine), and observe that the anticipated actions change accordingly, even with the same set of recognized actions as the inputs to the LLM. Words marked in red highlight the different predictions.
Our qualitative analysis with the counterfactual prediction shows that the choice of goal can have large impacts on the action anticipation outputs. This indicates another future direction improve
15

---

Published as a conference paper at ICLR 2024
the top-down action anticipation performance is to improve the accurate and quality of the predicted goals, and to consider multiple plausible goals.
C ADDITIONAL EXPERIMENTAL DETAILS
In this section, we provide additional details for the fine-tuning language model experiments on Ego4D v1 and v2, well goal inference via in-context learning. We also provide additional details for all the transformer models involved in experiments on all the datasets. Finally, we describe the details of the different setups in Ego4d datasets versus EK-55 and Gaze. All experiments are accompolished on NVIDIA A6000 GPUs.
C.I PREPROCESSING AND POSTPROCESSING
During preprocessing for fine-tuning, we empirically observe that using a single token represent each verb or noun helps the performance of the fine-tuned model. Since the tokenization is handled by OpenAl's API, some verbs or nouns the Ego4D's word taxonomy may span over multiple tokens. For example, the verbs "turn-on" and "turn-off" are two-token long, and the noun "coconut" broken into three tokens. We attempt to minimize the token length of each Ego4D label without losing semantic information of the labels, which are important to leverage prior knowledge in the LLMs. As a result, we choose the first unique word to describe the verb and noun labels in the Ego4D label vocabulary. This preprocessing is performed for both the input action sequences, and the target action sequences for prediction. No additional task information is provided as input during fine-tuning,
observe that the fine-tuning training examples themselves clearly specify the task.
We observe that LLM models demonstrate strong capability on following the instructions as specified by fine-tuning in-context learning examples. However, the output sequences generated by the LLM are sometimes invalid, and post-processing is needed before the predictions can be evaluated by the edit distance metrics. Specifically, the standard Ego4D LTA evaluation protocol requires generating sequences of long-term action predictions, each in the form of 20 verb and noun pairs.
We consider the following scenarios to trigger postprocessing on the LLM's predictions: (1) the output sequence length is different than expected (i.e. 20); (2) an action in the output sequence cannot be parsed into pair of one verb and one noun; (3) a predicted verb or noun is outside the vocabulary. Table A7 shows the statistics on how often each type of incident would occur.
Instead of repeatedly sampling new responses from the LLM until it is valid, we choose use simple post-processing strategy: First, inspect each action in the predicted sequence, and remove the invalid ones. Second, truncate the sequence, or pad the sequence with the last predicted valid action until the sequence length is as desired. We then map the predicted words into the fixed noun and verb label space. This is achieved by retrieving the nearest neighbor of each word in our label vocabulary based on Levenshtein distance.
Incident
over all instances
Short Seq (1)
2.74%
Long Seq (1)
12.93%
Invalid Seq (2)
10.44%
Invalid Verb (3)
1.93%
Invalid Noun (3)
2.62%
Table A7: Statistics on scenarios which require post- post-processing, Invalid sequence refers to the scenario where the output sequence contains invalid strings that cannot be parsed into pairs of verb and a noun. Invalid verb/noun refers to the scenario where the word is outside of the vocabulary. Both scenarios often imply wrong sequence length.
C.2 BOTTOM-UP AND TOP-DOWN TRANSFORMER MODELS IMPLEMENTATION DETAILS.
For the bottom-up model that establishes our vision baseline, we use frozen CLIP Radford et al. (202T) image encoder (VIT-L/144@300 336px) as vision backbone. Each frame is encoded 768D)
16

---

Published as a conference paper at ICLR 2024
representation. For Ego4D, we use layers, 8 heads of a vanilla Transformer Encoder with hidden representation dimension of 2048. We use Nesterov Momentum SGD + CosineAnnealing Scheduler with learning rate 5e-4. We train the model for 30 epochs with the first 4 as warm-up epochs. All nyper-parameters are chosen by minimizing the loss on the validation set. In our top-down models, used the compatible text encoder of the ViT-L/14@336px CLIP model the text encoder. We used Adam optimizer with learning rate 2e-4.
For EK-55, we extracted features of all provided frames from the dataset. For the transformer, we use layers, 8 heads with a hidden representation dimension of 2048. We use Adam optimizer with learning rate 5e-5. All apperpparmeters are chosen by minimizing the loss on the validation set. In our top-down models, we used the compatible text encoder of the ViT-L/14 336px CLIP model as the text encoder. We used Adam optimizer with learning rate 5e-5.
For Gaze, we use 1 layers, 8 heads of a vanilla Transformer Encoder with hidden representation dimension of 2048. We use Nesterov Momentum SGD + CosineAAnealinng Scheduler with learning rate 2e-2. All hyper-parameters are chosen by minimizing the loss the validation set. In our top-down models, we used the compatible text encoder of the ViT-L/14 10LL/144366xx CLIP model as the text encoder. We used Nesterov Momentum SGD + CosineAnnealing with learning rate 15e-3.
C.3 RECOGNITION MODEL IMPLEMENTATION DETAILS.
For the recognition model, we use frozen CLIP Radford et al. (0021) image encoder (ViT- L/14@336ph as vision backbone. For all the datasets, we randomly sample frames from each segments. Each frame is encoded as a 768D representation. We use layers. heads of a vanilla Transformer Encoder with hidden representation dimension of 2048. We use Nesterov Momentum SGD Cosine Scheduler with learning rate 1e-3. We train the model for 40 epochs with 4 warm-up epochs. All hhper-paramenne parpmerrs are chosen by minimizing the loss on the validation set.
C.4 INFERRING GOAL DESCRIPTORS USING LLMS
In Section 4.2, describe ablation study where goal descriptor is used for in-context learning. More specifically, we use GPT-3.5 Turbo and Llama2-chat- 133 to infer the goal by giving recognized history actions and few samples as demonstrations with ICL as goal descriptors. An illustraion of the goals can be found in Figure AI For the experiments in Table [ the goal descriptors are added as LLM prompts in the form of "ooal <goal> Observed actions cobserved act ions>
=> " We note that this ablation study poses an interesting contrast to the CoT experiments: Instead of asking the LLM jointly infer the goal and predict the action sequences, the ablation experiment conducts goal inference and action prediction separately, and achieves slightly worse performance than COT.
C.5 ADDITIONAL DETAILS ON OPIC-KITCHENS- -5 AND EGTEA GAZE + EXPERIMENTS
In the experiment section, describe two setups of the LTA task different datasets. For the Ego4D datasets, we measure the edit distance of future action sequence predictions while for EK-55 and Gaze, measure the mean average precision for all actions to occur in the future. The distinction arises in how "action" is defined among these different datasets and setups. In Ego4D benchmarks, we follow the official definition of action, which a verb-prediction and a noun coen-ppreiitiin combining together to form an action. For EK-55 and Gaze, we adopt the task setup of Ego-Topo Nagarajan al. (2020) in order to compare with previous works. In their setup they defined "action" as only the verb-prediction, excluding the prediprrdiition in the final "action" prediction.
D ADDITIONAL VISUALIZATION AND ANALYSIS
D. EXAMPLES OF ANTGPT PREDICTION
We provide four examples of bottom-up fine-tuned AntGPT in Figure A3 two of which are positive, and the other two negative. Correctly recognized actions from the video observations (outputs of the vision module, and inputs to the LLM) and future predictions (outputs of the LLM) are marked in green while wrong observations or predictions are marked red. We observe that AT is able
17

---

Published as a conference paper at ICLR 2024
to predict reasonable and correct future actions when the recognized actions not perfect. However, when most of the recognized actions are incorrect (Figure(A3) (A3p), , the predictions would also fail as expected. Finally, the long-term goal based on the observed actions. In Figure A31, the actor keeps on mixing the ice while the LLM predicts curry-cooking related activities.
observe that the bottom-up approach sometimes fail to implicitly infer
D.2 ADDITIONAL EXAMPLES ICL AND CoT
Similarly with Figure 2, Figure A4 shows two additional examples of bottom-up prediction with ICL and top-down prediction with COT. We can further observe the influence of implicitly prediction future goal from the two examples. In the first example, due the low quality of recognized actions from the video observation (repeating "play card"), the bottom-up prediction follows the observation pattern to repeat "play card" while in the prediction of top-down prediction using CoT to consider the goal, the model succeed to predict "take card" and "put card" alternatively which both matches the ground-truth future actions and common sense in the scene of card playing better, with the verb edit distance significantly goes down from 0.95 to 0.45. The second example shows failure example of using COT. In this video of "playing tablet", the top-down model predicts the goal as "using electromic devices", thus influencing the following future action prediction to hallucinate actions like "operate comput er". This behavior is intuitive, as an incorrect goal sampled by the LLM is likely to mislead the predicted actions that conditioned on it. We believe adding additional cues for goal inference, and explicitly model the different "modes" of the future capture the goal diversity would mitigate this behavior.
D.3 ADDITIONAL EXAMPLES OF COUNTERFACTUAL PREDICTION
In Figure A5 we demonstrate four additional counterfactual prediction examples. In or- der to predicting with explicit goal using ICL, construct the prompts in the form of: "Goal: <inferrec//aleeee goal> Observed actions: a <observed act actions> <future actions>" " We observe that switching the goals from the "inferred goal" the "altered goal" indeed has huge impact on the predicted actions. This confirms that the goals are utilized by the LLM to predict (or plan) the future actions take. This behavior is consistent with our observations in Figure A4
18

---

Published as a conference paper at ICLR 2024
Past Observations
Future Predictions
[GT action labels]:
wash car
lift sponge
wash car
wash window
[Recogní it Lioned actions in obervatton];
wash car,
squeeze sponge,...
[LLM predicted sequence]:
wash wash wash wash hand
(a) Positive Sample 1
Future Predictions
[GT labe Is]
dough
flour
adjust
take stick
[Recognitioned actions in obervation]:
dough, put flour
[LLN predicted action sequence):
.... dough, adjust dough, ... dough, daugh
Positive Sample 2
Past Observations
Future Predictions
1 (ST action Tabels];
paintbrush
paintbrush
(Recogni rioned actions ohervatton] ... put paper, ... LL predicted action sequence]: remove wood, turn wood, smooth wood, adjust wood.
(c) Negative Sample
Past Observations
Future Predictions
[GT action Tabels]:
hold jug
put lce
mix Ice
take
[Recognitioned actions in obervation]: put fua, ... put ice,
scoop sequence]:
(d) Negative Sample 2
Figure A3; Four examples results from fine-tuned AntGPT. The green word indicates correctly recognized actions (inputs to the LLM) and future predictions (outputs of the LLM), red indicates incorrectly recognized or predicted actions.
19

---

Published as a conference paper at ICLR 2024
GT Observations
Future Actions
take
touch card
take card
divide
Predict Goal
"Instruction:
possible goal according alven previcus actions?
take fork, wash fork, put fors, take knife, wash knile, take spoon, wash speen, put spoon no deean tableware Qr play card. play card, card, card, play card, play card, play card, play card =>
[Answer]]
"llying
(b) Bottom-up: Predict Future
"Instruction: What
possible actions according
You follow the following
cirry put plant, adjust dltth, plani, plant, sickle, take out plant, sickle. plant, cut plant, hold plant Qr play card. play card, play card, play card, play card, play card, play card, play card
Vnswwrr::
Shuffio card, put card, pay cand. play card, play card, play carc?
Top-down: Prodict Goal and Futuukk
possiblc
according given provious
You
following nukes...
toe leat, sarry leat, put plant, adjust ploth, take plant, out plant, take sicale, take scdde Q1: What' the goal according to previbus actons? aZ: What are the futurn 20 actions? us The goal gardening Futur 20 actions are cut plant, put sickle. cul plant cut plant, holt plant
play card,
play play play card, play card, play play Q1:
gusl sccording previcus actione?
future 20 actons?
[Answor):
Web
"The goal playing card Futurn actions Are shuffle
pUt takon card, pull
(a) Example the ground-truth observations [take card, put card, pack card, take card, put card, take card, put card, touch card], the ground-truth predictions are [take card, put card, take card, put card, pack card, take card, put card, pack care, take card, put card, take card, put card, put card, pack card, arrange card, take card, shuffle card, put card, take card, divide card).
GT Observations
Future Actions
operate tablet
take pen
operate tablet
adjust pen
operate tablet
operate tablet
adjust tablet
"instruction: What the most possible according given previous actions?
take
take knita,
sake phone, operata phone, openate phona, operata tabiet, opanate tablar, operato sablet, operata opanate
[Answer]
usinn electonic devices
(b) Bottom-ux Prodict Future
"Instruction: What will the possible next 20 actions accanting to given previous actions? You should follow: the following nules... plant, hold
plant, adjust dioth, Q: take phono, operate phooe, operate phone, operate tablet, operate tablet, operate tablet, operate tablet, operate tablet
take
[Answer]:
Wet. 50 063 Noun 9.66
oou phooe, take wire operate tablet, put tablei, take phone"
(0)
"Inetaction: What will be the possible next 20 action accarding to given previous actions? You should follow the following ruleo...
D1: leaf, leaf, plant, adjust dioth, take plami, plani, take sicile, take scile What's goel sccording to previous action? Q2: What gansening, Future actions are: plant, put sicile. Qr take phone, operate phone, operate phone, operate tablet, operate tables, operate tablet, operate tablet, uperate tablet Q1: What's the goal according previcus aciione? Q2 What the fulure acions? =ox
20 actons?
plant, plant
(Answer):
Web. 80- R.89 Noun CD 2:35
The gool is using electronic devices Future 20 actioms are lake computer, open compuleer operaie tabiel, operals latile!"
(b) Example 2, the ground-truth observations are [hold table, operate tablet, hold pen, touch pen, take pen, operate tablet, operate tablet, operate tablet]. the ground-truth predictions
[adjust pen, hold pen, operate tablet, operate tablet, hold pen, operate tablet, operate tablet, operate tablet, operate tablet, opcrate tablet, move pen, operate tablet, lift tablet, put tablet, operate tablet, operate tablet, operate tablet, adjust tablet, operate tablet, adjust tablet].
Figure A4: Promts and results of TeL and CoT. Here we show three experiments: (1) Predict the final goal. (2) Botom-up predict future actions with ICL. (3) Top-down prediction by CoT prompting. Green word means correct observations or predictions and red means wrong.
20

---

Published as a conference paper at ICLR 2024
original Prediction
Counter Factual Predictions
[inferred Goal]: construction site
[elter Goal]: but 1d house
(observations Input]: neasure ladder, put rod, put rod, put rod, put rod, put rod, put red, put metal
[predictiin]:
"nove netal, adjust netal, .... adjust metal, lift metal, put netal. adjust metal, lift netal, netal, adjust netal, lift netal, put netal"
"take hanner, drive nail, put wood, neasure woad, cut wood, put waod, lift wood, put waod, woad,
wood"
[Inferred Goal]: building firewood stack
[A] tered Goal]; woodworking
[observations Input]: wood, wood, hold wood, put wood, put wood, put hold wood, hold wood
"put wood, [Prediction]:
[Prediction]:
put wood, wond, put wood, .... put wood, put wood, hold wood, put wcod, hold wood, hold wood"
"take saw, wond, wood, wood, put sandpaper, take paint, apply paint, take wood, put wood, adjust wood"
[tnferred Goal]: get dressed
[Alte coal]:
organize clothes
jobservations Input]: shirt, put shirt. shirt, adjust shirt, take shirt, take shirt, take shirt, adjust shirt
[Prediction):
[prediction):
"put stirt, take pants, put pants, adjust pants. take jacket, put jacket, adjust jacket, take bag, put bag, adjust bag"
"put shirt, take pants, put pants, take socks, take jacket, put jacket, take hanger, hang jacket, adjust hangers, tidy closet"
[inferred [oal]: clean kitchee sink
[A] tered Goal]: wash dishes
[observations Input]: wash faucet, wash faucet, wash faucet, wash plate, wash faucet, take faucet, put fauret, put faucet
[Prediction]:
[Predicttion::
sponge, put sponge, take soap, .... rinse sink. wipe sink, adjust faucet, turn faucet, rinse sponge, adjust towel, hang towel, adjust clase faucet"
mash cup, wash spoon, put spoon, .... wash bowl, put bowl, wash pot, put pot, wash pan, dry dish, put dish, dry bow1, put bowl"
Figure A5: Four examples of counterfactual prediction. LLM predict based on same observations but different goals. On the left shows the results with the "inferred goal" generated by G.T-3.5-Turbo according to observations generated by recognition model while the right part shows results with manually designed "alternative goal".
21


Reviews:
[{"review_id": "YOuXoamgnX", "reviewer": "Reviewer_Utid", "summary": "This paper proposes to utilize large language models for long-term action anticipation. CLIP visual features are extracted from each video segment and a sequence of action labels are predicted, which are input to the LLM. In-context learning is used to predict the final goal, which is embedded with the CLIP text encoder. Experiments on three datasets show the efficacy of the proposed method. The authors also show the distilled smaller models can be effective as well.", "questions": "1. The goals in Figure 2 are sometimes simply a repetition of the last action labels. Can the authors elaborate on how the goals are defined (manually annotated)? Should they be the last action of the videos?\n2. In Table 5, the distilled model is even better than the original model. How is that possible? Can the authors provide more explanation on that?\n\n------------Post rebuttal comments: the authors have addressed my questions and concerns. I have increased my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The idea of using LLMs for long-term action anticipation is interesting and the intuition makes sense.\n2. Experiments are thorough. The authors compare multiple usages of LLMs for the LTA task, including in-context learning and fine-tuning.\n3. The proposed method achieves SOTA on three benchmarks.", "weaknesses": "1. The paper writing/organization can be improved. The method description part of AntGPT is quite confusing. For example, an overview of how the separate parts of Figure 1 (b-e) are connected to each other will help. Also, for the experiment section, putting “comparison with state-of-the-art” to the front might be better IMO.\nOther minor presentation issues: Some text in Figure 1 is too small to read. \n2. There is no error analysis. When does the proposed method fail and why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V12hfK9bn0", "reviewer": "Reviewer_v3Zh", "summary": "This paper investigates using large language models to help action anticipation in bottom-up or top-down manners. For the bottom-up approach, future actions are predicted using observed videos or actions. For the top-down approach, the prediction of future actions is additionally conditioned on the goal. The LLMs are used by prompting or fine-tuning.", "questions": "I don't have too many questions about the technical part of this paper. I would appreciate comments on how should we position the task of action anticipation in the following research. Should this task actually be a compound task of action recognition (for observed video) and language understanding for anticipation? Or should we still study this task as a standalone task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The formulation of decomposing to bottom-up and top-down approaches is interesting and intuitive.\n2. Performance gains are observed on several datasets and ablation studies are good in general.\n3. The writing and presentation is good in general.", "weaknesses": "1. The interpretation of the results should be cautious. The performance gain is evident but not surprising since additional labels of goals and additional knowledge from LLMs are introduced. Although the boundary of using additional information has become vague since the popularity of LLMs, I still believe we should be cautious when comparing a new method with LLMs and previous methods without LLMs.\n\n2. Using LLMs for action anticipation makes me raises the question about the fundamental positioning of this task. In this sense, the action anticipation is mainly a language-understanding task rather than a vision task. If we look at Table 2, it is better to use actions as inputs than using visual features. If we still treat action anticipation as a vision task, what is the role and the importance of the visual modality? If we treat it as a language task, the contribution of this paper will be limited since it mainly utilizes prompting and fintuning of LLMs. If we treat it as a multi-modal task, we need to see more insights and experiments about the interaction between modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e6GJX1Dkka", "reviewer": "Reviewer_jEKR", "summary": "The paper studies whether and how large language models can assist long-term action anticipation (LTA) from videos. To do so, the paper introduces AntGPT, an approach to perform LTA based on a mixture of a bottom up (direct prediction from observations) and top down (goal prediction, then action sequence prediction conditioned on goals) approaches. The paper also shows that the large language models used in the approach can be effectively distilled to smaller models for convenience. Extensive evaluations assess the influence of each component on final performance. The proposed approach is shown to outperform state-of-the-art approaches on major datasets.", "questions": "The authors could discuss on few-shot learning as referenced above and comment on the most relevant minor comments.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The question, whether LLMs can assist action anticipation, is surely interesting.\n\nI found the paper informative and the study well designed to answer the research questions stated in the introduction.\n\nExperiments are extensive and designed to be informative for the reader, beyond showcasing the abilities of the proposed approach.\n\nThe final model is shown to outperform previous approaches on the main datasets.\n\nI appreciated the \"limitations\" section.", "weaknesses": "While I believe the paper is sound and an interesting contribution, one point is not clear to me:\n\nIn-context (few-shot) learning is used to anticipate future actions with LLMs. The idea of using few-shot seems to be motivated by constraints due to LLMs rather than due to the task. Indeed, while one may use the entire training set as a set of examples, this would lead to a large prompt, which could be both impractical and detrimental to performance. However, in principle, it would make perfect sense to avoid few-shot learning altogether and resort to some form of supervised learning. While I understand the constraints, I feel this part is not well discussed in the paper. Also, how are few-shot examples selected in practice?\n\nMinor comments:\n- the \"visual procedure planning\" paragraph in the related works section is not very easy to follow. It seems like the paragraph states what visual procedure planning is related to, but it does not give a clear definition of what it is intended by the term \"visual procedure planning\". I'd suggest revising.\n- in the same section, the reference (Mascaro et al.) is missing the publication year\n- \"an few-shot high-level goal predictor\" -> \"a few shot ...\"\n- at page 5, the paper states \"we either use the video metadata as pseudo goals when it is available, or design the goals manually\". It is not clear to me what \"design the goals manually\" means, and this does not seem to be well described in the main paper.\n- the paper relies on CLIP, but it would have been interesting to see how the use of other ego-specific video-language models such as EgoNLP, LaViLa or EgoNLPv2 would perform in this context\n- while experiments are performed on EPIC-KITCHENS-55, no experiments are performed on EPIC-KITCHENS-100. I believe experiments on EK55 are relevant enough, but EK100 is a larger dataset (of which EK55 is a subset) and a much denser and cleaner one, so it would seem a perfect test bed for LTA. I'd encourage the authors to consider it for future experiments.\n- In Table 1, there is a star (*) next to \"oracle goals\", but I could not find the explanation of this symbol anywhere\n- In table 5, \"from-scratch\" is reported for the second model as setting. Does this mean that the model was randomly initialized? From the text it seems that it was still pre-trained on some text corpus. Can the authors clarify (or fix) this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7zg5367lW", "reviewer": "Reviewer_R7fa", "summary": "This paper proposes a method named AntGPT to investigate if large language models encode useful prior knowledge on bottom-up and top-down long-term action anticipation. Experimental results on Ego4D LTA, EPIC-Kitchens-55, and EGTEA GAZE+ benchmarks indicate the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses for more details.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tThe motivation is clear, and the usage of large pre-trained models is interesting.\n-\tExperimental results on multiple benchmarks indicate the effectiveness of the proposed method.", "weaknesses": "-  The novelty is limited, AntGPT is a straightforward application of large language models to the action anticipation problem.\n- In Table 1, the improvement of the proposed method is marginal.\n- The proposed method relies on the classification results of previous fragments, and classification errors may lead to the accumulation of errors and affect the prediction results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)