LLaVAction: evaluating and training multi-modal large language models for action understanding
Authors: Haozhe Qi
Shaokai Ye
Alexander Mathis
Mackenzie W Mathis
Venue: ICLR 2026 Poster (2026)
Keywords: MLLM
action understanding
video understanding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Understanding human behavior requires measuring behavioral actions. Due to its complexity, behavior is best mapped onto a rich, semantic structure such as language. Emerging multimodal large language models (MLLMs) are promising candidates, but their fine-grained action understanding ability has not been fully examined. In this work, we reformulate EPIC-KITCHENS-100, one of the largest and most challenging egocentric action recognition datasets, into a MLLM benchmark (EPIC-KITCHENS-100-MQA). We demonstrate that when we sample difficult answers based on specialist models as distractors, leading MLLMs struggle to recognize the correct actions. How can we increase the performance of MLLMs? We curated a supervised finetuning dataset that includes `hard' action recognition, temporal detection, captioning, and free-form question answering to improve models' diverse action understanding capabilities. We introduce a new model called LLaVAction that adds an action token to boost models' attention on visual tokens and a two-stage pipeline to obtain structured actions. LLaVAction greatly improves the MLLMs' ability of action understanding, achieving strong improvements on both MLLM benchmarks (21 points in accuracy over GPT-4o on EPIC-KITCHENS-100-MQA) and established action recognition benchmarks, suggesting that our methods prepare MLLMs to be a promising path forward for complex action tasks. 
Code, data, the benchmark, and models are available at https://github.com/AdaptiveMotorControlLab/LLaVAction.

Published as a conference paper at ICLR 2026
LLAVACTION: EVALUATING AND TRAINING MULTI- MODAL LARGE LANGUAGE MODELS FOR ACTION UN- DERSTANDING
Haozhe Qi", Shaokai Ye", Alexander Mathis ** Mackenzie Weygandt Mathis"
École Polytechnique Fédérale de Lausanne (EPFL), Lausanne
mackenzie .mathis@eep..... alexancer www.mmsseethll.ommmmmm ch "Co-first, ""Co-seior
ABSTRACT
Understanding human behavior requires measuring behavioral actions. Due to its complexity. behavior is best mapped onto rich, semantic structure such language. Emerging multimodal large language models (MILMs) are promis- ing candidates, but their fine-grained action understanding ability has not been fully examined. In this work, we reformulate EPIC-KITCHENS- 100, one of the largest and most challenging egocentric action recognition datasets, into MLLM benchmark (EPIC-KITCHEN--OOG 100-MOA.. We demonstrate that when we sam- ple difficult answers based on specialist models as distractors, leading MLLMS struggle to recognize the correct actions. How can we increase the performance of MILMs? We curated supervised finetuning dataset that includes 'hard' action recognition, temporal detection, captioning, and free-form question answering improve models' diverse action understanding capahilities. We introduce a new model called LLa that adds action token to boost models' attention visual tokens and two-stage pipeline to obtain structured actions. VLaVtction greatly improves the MLLMs' ability of action understanding, achieving strong improvements on both MLLM benchmarks (21 points in accuracy over GPT-4o KITCC-KS PP05-100-MQAA and established action recognition benchmarks, suggesting that our methods prepare MLLMs to be a promising path forward for complex action tasks. Code, data, the benchmark, and models are available www.tluuubcom/AAdaaaaa
INTRODUCTION
Understanding human behavior is a complex challenge requiring multiple skills such as visual per-
ception, knowledge about the world and reasoning capabilities. Current State-of-the-AAe the-Art (SOTA)
methods in action understanding tasks (Chalk et al.] 2024;) Liu et al.] 2025 Shi et al. [023)] typi- cally rely visual foundation models imbue those kind of priors (Radford al." 2021 Wang jet al., 2022) 2023). However, they rely heavily on dataset-specific target heads and have limited language understanding ability, constraining their performance and especially generalizability. Re- cently, Multi-modal Large Language Models (MLLMs) (Zhang et al.) [2024c] Li et al.) [2024a] et al., 2024
Wang et al., 2024) have shown great potential for learning language priors to help understand visual content, making them promising alternatives.
MLLMs take visual and text information as inputs and can directly output text. For training and evaluating MLLMs on action understanding tasks, existing datasets (Kay et al. [0017] Caba Heilbron
et al. 2015) are converted into free text (either video caption or muestion-and-answer [QA] formats), thus creating new datasets (Liu et al.) 2024a] Li et al. 2024d) and benchmarks (Yu et al. 2019 Li ctt al., 2024c). Those free-text formats offer great flexibility and generalization across datasets, but also introduce limitations in model learning, evaluation, and application perspectives. For the model learning, directly predicting the action name or choosing from some randomly selected candidates (Figure prevents the model from learning the full action distributions and contrasting fine- -neigrri actions (Xiao et al. 202I) explicitly. For the model evaluation, free text output makes MLLMs un- able to directly compare with previous action task specialized models (Ramachandran al. 2025

---

Published as a conference paper at ICLR 2026
113.0
113.4
113.7
114.1
Kondin choices
ditraciors
80
Accuracy (86
60
LLavA-Video @ GPT-4o
LaVAction
ground trubh cick up dish
Random Choices
TIM Hard Distractors
40
Optiors:
Uprions:
ginger
A Epen drawa)
Similar Scene
pick up dish
20
phore fromt frencer
C. adjust hob
Temporal order
LLaVA-OV-78
LLaVA Vidoo-
GPT-4o-mini
LLaVA Action7BB
GPT-40
D. pick up dish
drop container
Similar objects
Figure I: LLaVAction-7B. Left: Qualitative inspection of distractors. We show an example clip with labels from random choices (which empirically is easy to solve), vs. our proposed harder benchmark with action labels generated by SOTA specialist (TIM (Chalk et al. 2024)). Our hard example mining strategy can automatically explore challenges such as temporal order and similar objects that are curated in other benchmarks. Right: While GPT-4o is strong when identifying correct answers among few random choices due to the large number of possible actions, it suffers in the harder benchmarking regime, and our method, VLatton outperforms GPT-4o.
For example, EPIC-KITCHENS 100 (Damen al. 2022), which one of the largest and most chal- lenging action datasets, has around four thousand actions. MLLMs may not always predict action that has an exact match in those action types, as we cannot put all the action types inside MLLMs' context prompt to let MLLMs select. This further limits the applications that require structured actions (e.g., behavior analysis with ethograms (Renner 2018)).
To address these issues, take inspiration from the hard example mining literature (Shrivastaval jtt al., 2016; Madry 2017) to improve the learning and evaluation of MLLMs. Specifically, for evaluation, reformulate EPIC-KITCHENS-ION (Damen et al. [2022) into a video multiple-choice question & answer (MQA) task with the correct ground truth action and four difficult incorrect actions, which we call recognition models (Chalk et al. 2024 Zhao & Krähenbühl (222) instead of humans or closed- sourced LLMs and MLLMs. This specialized model-based hard example mining reveals substantial drops in performance for existing MLLMs, including GPT-44o (Figure I and thus offers an efficient and challenging framework for evaluating MLLMs' action recognition abilities. We note that the hard example mining approach automatically picks distractors that pose challenges such as temporal order, which were purposefully curated in other benchmarks (Cai et al. 2024 Li et al. 2024c).
Incorrect choices are filtered by SOTA action
To improve MLLMs fine-grained action understanding, we proposed an action-related MILM data transformation regime and curated a training dataset that encompasses various aspects of action understanding, such hard action recognition, detailed captioning, free-form question answering, temporal detection and prior action association. With the training dataset, we propose LLa VAction models. We introduce action token designed improve the model'ss 's visual information utilization and a two-stage pipeline to output structured actions and fairly compare with other action recogni- tion models. These model designs could be naturally extended to different foundational MILMs (e.g.. Liu et al. (002a); /7hu et al. (2025).. ILLaVAcioon obtains SOTA performance on four action recognition datasets TEPIC-KITCHENS 100 (Damen al. 2022). EPFL-Smart WWFFL-Smar-Kitchhn--B3 (Bon- netto et al.) 2025). MECCANO (Ragusa et al., 2021) and Animal Kingdom (Ng et al. (022)) and shows strong generalization ability in comparison previous SOTA models. LLa VAction outper- forms GPT-4o on EPIC-KITCHENS, 100SS--00-MQAAA and achieves consistent improvements on ten video
MLLM benchmarks that require very different action understanding abilities and are either caption, open-ended, multi-choice format.
2 RELATED WORKS
Multi-modal large language models. Multi-modal large language models (MLLMs) are promis-
ing generalists (Li et al. 2024b). Early multi-modal models (Tsimpoukelli al. 2021) mostly
2

---

Published as a conference paper at ICLR 2026
LLaVAction
Action one-hot encoding
Response
Open-ended
MLP Head
Multiple choice GA
Structured action
Large Language Model
Projection Layer
Two-stage structured
Action-related MLLM data reformulation (training)
action (inference)
Vision Encoder
Temporal detection
Adversarial distractors
Top repurposing
Temporal order learning
GPT-4o reformulation
Specialized model
Direct prediction
Visual
Action
Video
token
token
token
Figure 2: LLa VActinn pipeline. Trained with our action-related MILM reformulated data,
LLaVicnoon outputs captions, action tokens and open-ended and multi-choice QAs. Our two-stage pipeline further enables LLa output structured action.
performed few tasks or relied on few-shot learning for task generalization. After the large success of Large Language Models (Achiam et al., 2023). multi-modal models appeared that supple- ment text with other modalities (Han et al., 2024). Among them, video MILMS (Li et al. 2024a Zhang et al. 2024c) promise robust and scalable solutions to understand and process video data. Our work falls into this direction, aiming at improving MLLMs' action understanding.
Action understanding is one of the fundamental abilities in video understanding and has been explored by several recent works with different specific focuses, such as InsTALL (Nguyen et al. 2025) mainly focusing on procedural action planning/prediction. HAIC (Wang al. (2025a) mainly focusing on detailed action captions and MotionLLM (Chen et al. (024b)) mainly focusing on human motion understanding. Instead, our work cares more about fine-grained action contrastiveness.
MLLM datasets and benchmarks. Significant efforts have been made to improve training (Lu al., 2024a| [i et al. 2024d) and benchmarking (Yue al.] 20044, Liu et al., (2024c) for MILMs. Based on the question type, they can be classified as video caption, open-ended question answering and multi-choice question answering (MQA) types. MQA format gains more popularity, especially for benchmarks (Xiao al. 2021 Fu et al. 2024). since it has need
another LLM/M- LLM to evaluate the model outputs and our SPIC-KITCHENS- SEEHHENS-IOO-MEEE 100-MQA benchmark falls into this direction. In comparison the existing MQA benchmarks whose choices either constructed by humans Yu et al. 2019 Fu et al., 2024) or by closed-source MLLMS (Maaz et al., 2023 Ye et al, 2024a), benchmark uses action recognition models efficiently find hard distractors, which is more efficient compared to human generation and is not be limted by closed-source MLLMs' performance.
Action recognition. Action recognition requires models to predict the action class for a trimmed segment (Shahroudy al.] 2016; Damen et 2022) and is fundamental task in video under- standing (Feichtenhofer et al. [019 Tong et al. 2022). Over the years, many methods have been proposed, yet suffer from fast camera movement, long-term temporal relations, and open vocabu- lary ability (Damen et al. [2222] Grauman et al. (022). We focus on MLLMs enhanced with video instruction-tuning (Zhang al. 2024c) to address those challenges.
METHODS
We introduce the EPIC-KITCHENS-IOOO.. benchmark (Section 3.1 and novel MLLM data reformulation paradigm (Section 3.2), followed by the LLaVAction model designs (Section 3.3)
3.1 HARD EXAMPLE MINING FOR MLLM EVALUATION
Existing MLLMs have shown the ability to understand video content including actions. However, whether MLLMs are good at contrasting fine-grained actions is not clear. Researchers have de- veloped benchmarks to focus on certain aspects of fine- ine-grained actions, such as temporal order (Liu al. 2024d), which are generated either by human effort or closed-souree MLLMs. In this work, propose to leverage SOTA action recognition models with hard example mining to construct new benchmark named EPIC-KITCHENS- COO-CHEEES----OOOAAAA which is efficient compared to human gen-
3

---

Published as a conference paper at ICLR 2026
eration and is not limited by closed-source MLLMs' performance. More importantly, the proposed hard example mining paradigm can also help MLLMs to enhance fine-graine fine-grained action understanding (Section 3.2) and enable the fair comparisons with specialized models (Section 3.3
We use SPIC-KITCHENS- 100 (Damen et al. 2022) as the data source for our benchmark for the following reasons. Firstly, EPIC-
100 boasts fine-grained action at scale (90K action segments comprising 100h, 4k action types in 100 hours). Secondly, despite numerous models being developed for this dataset, benchmark performance on tasks such as action recognition and segmentation remains far from saturated. Thirdly, the benchmark proves opportunities to compare against specialized models. Importantly, our hard example mining strategy generalizable and can be applied to any other action understanding dataset. (Section 4.3)
We constructed {uh, u2..... UN denote the set of video clips. Let = {nI, f?,,..., RN ), A= = fai, .2...., aN (a; E CJ be their corresponding clip narrations and action labels separately, where C represents the set of action classes. For each data sample i, we formulate the MQA task as:
CEC---ITCHENS-OO--WAE
with hard example mining follows: Let V
f (1i, 2,0,) ~ P2......... .PK] where Σ pk = Pk (0,
where 2 is the input sample (e.g., video clip). p; is the probability of picking the i-th option in the MQA the answer, Q is the space of possible questions, O (m., Dj represents the set of K answer options, n, is the correct narration, and D. represents K sampled distractors. These can be sampled randomly from narrations in other action classes:
DJ- = Uniform((ng EN (c(()))
However, random sampling D. likely contains trivially wrong answers (Figure D We utilize action recognition models g V - (0, 1) to find distractors. For video specific e; obtain the top K predicted classes: Ci Topx-1 \ {a:)). The distractor sampling becomes:
D Uniform( (n, E N 1 ECJ))
The complete of answers is formed as O [n:]UD] for random sampling O {nD for model-based sampling. We used K = 5 for our benchmark. Moreover, we use the action narrations A instead of action labels C build the choices to avoid implausible texts and confusions (more details are in Appendix F.I) We compared the two sampling strategies. We chose two leading action recognition methods on EPIC-KITCHENS- 100, namely. AVION (Zhao & Krähenbüh! 2023) and TIM (Chalk et al.) 2024). The results indicate that the TIM method consistently produced more challenging distractors for the evaluated MILMS (Table and Figure Consequently, we fixed with TIM for the EPIC- SPIC-KITCHENS- 100-MQA benchmark. We found that all tested models have huge performance drop in comparison to the easy setting, which illustrates that MLLMs struggle with fine-grained action recognition when tested with visually or semantically similar actions.
and qualitative examples Appendix
3.2 ACTION UNDERSTANDING FROM MULTIPLE PERSPECTIVES
When training MILMS on action understand- ing datasets (ayy et al., 2017). researchers usually re-formulate those datasets to MLLM- compatible tasks (video caption or question & answer (QA)), either by directly outputting the original action annotation or reformulating the annotations with closed-source MLLMs. Al- though this reformulation provides flexibility and generalization across video data even be- yond action understanding, fine-grained action differences are not fully explored and hence
Choice
Random 5
Avion-Top (Medium)
TIM-Top
Selection
(Easy)
(Mard)
GPT-4o-mini (07-18)
72,0
44,2
37.4
4P LIVVA-OV.0.5R
87.6
56.7
52.2
59.3
37.1
32.0
LLaVA-Ov-78
68.8
33.6
28.9
LLaVA-Video-7
65.0
40.0
35.7
Table 1: Comparison between different sources
of distractors. Models were evaluated on ei- ther random, AVION or TIM-generated distrac- tors. The values reported percent accuracy.
4

---

Published as a conference paper at ICLR 2026
10000
1354
352
whii our
dan aner
D. oighen
2007
2005
1/4
103
2074
2137
2341
211
Mi sen @ 200 oppeererr
Cie 2014
Li 9
Figure 3: Qualitative results. LLaVAction-7B consistently outperforms GPT-4o and LLaVA- Video-7B when tested on hard distractors. Bold option denotes ground truth, and the icons denote the selection of the models. See also Appendix
cause a performance drop on vious reformulation regime to encompasses various aspects of action understanding.
1000HCCESS-I0--OQ To address this, we expand the pre-
Adversarial distractors for fine-grained action contrasting. Thanks to the effectiveness of our hard example mining paradigm, we can also train MLLMs with 'adversarial' distractors that were generated by action recognition models. Since the PIIC-KITCHENS-I0O-MO) benchmark is based TIM's predictions, training MLLMs to pick the right answers using distractors gener- ated by TIM could lead to over-fitting to TIM's error distributions, resulting in an independent and identically distributed (IID) setting. To avoid methods from obtaining better performance by sim- ply overfitting on TIM's choice combinations, we used predictions from AVION to reformulate the OPIC-KITCCENN--OO training set. This provides us with an out-of-distribution (OOD) evaluation on EPIC-KITCHENSS-OOO 1000MMA (more OOD supports in Appendix (D). Moreover, we also present the results derived from training with distractors generated by TIM, which is the IID setting and (Table
yields the strongest results in EPICKKITCHENS--OO-MOO
Temporal detection for action boundary learning. Many actions (such as putting something) have clear initiation and conclusion and cannot be explicitly learned by the previous reformulation regime. Therefore, we instruct the model to predict the start and end timestamps of an action sample based on its randomly padded video clip. Specifically. given video segment V; with start timestamp 8; and end timestamp 6; we introduce a fixed temporal padding = seconds distributed between start and end. Let Uniform(O. 1) be the proportion of padding allocated the start:
$ = S; - aδ, é = e; + - a
(4)
We therefore use the new timestamps 8, and é to obtain the padded video segment 0, During training. our VActtin model takes 6; as input and predicts the start and end times as strings (e.g., "3.20", "1.20") corresponding to the true start and end times of the action in the padded video (prompts are detailed in the Appendix C.3)
Temporal order learning with prior actions. Actions exhibit certain natural continuity. This temporal aspect of actions can improve predictability. Therefore, we want to leverage prior actions T Di-na log (o a--1..... at-n,, where 0* is the optimal set of model parameters we are trying find, a, is the current action time t, at-1...., at-n represents the sequence of n previous actions. We set n = 2. We modulate this with additional visual in- structions to provide prior action information (prompts are detailed the Appendix C.5). During training, 30% of the MQAs are provided with prior action information ing evaluation (Table 3) we can either give the model no prior actions or give the model's own predictions of the previous actions to formulate sequential action prediction (SAP).
and learn: 0 = arg maxg
additional input. Dur-
Direct prediction. Following common practice, we also let the model to directly predict the action descriptions for a given video. The prompt is available in the Appendix C.4
General video understanding with closed-source MLLM-based reformulation. Although we mainly focus improving MLLMs' fine-grained action understanding, do not want weaken MLLMs' general video understanding. Therefore, we follow previous practice to let the model give general description of the video or answer video-related questions, where the annotations
5

---

Published as a conference paper at ICLR 2026
obtained from closed-source MILM GPT-4o. More details are in the Appendix B above, we argue that this reformulation alone would not help to gain better fine- one-grained action understandingg Since GPT-4o itself struggles with our empirically hard distractors, using annotations from GPT-4o alone harms the performance on EPIC-KITTCHENS-OOO-MG
As discussed
(Table
3.3 LLAVACTION MODELS FOR BETTER ACTION UNDERSTANDING
With the proposed new MLLM data reformulation paradigm, we propose LLa which is better at fine-grained action understanding. LLaVAction developed based on LLaVA series (Li et al. 2024a] ZZang et al.) 2024c). We further design a learnable action token to enhance the visual information utilization and two-stage pipeline to output structured action so that we can compare LLa fairly with other SOTA action recognition models (Figure 2).
Enhancing visual information utilization with action token. Most MLLMS rely only on the lan- guage prediction of the next token to train the model and extract information from visual tokens (Liu al. 2024a). However, recent findings suggest that this training strategy decreases the importance of vision tokens in late layers of MLLMs (Zhang et al., 2025a Liu et al. 2024b). Therefore, we de- signed an intermediate supervision of visual tokens. Specifically, ll,, we added learnable action token into the input tokens, which is analogically similar to the CLS token in the VIT model (Dosovitskiy 2020) that has been shown the ability in grasping image content (Wang et al., 2025b) The order of the input tokens was system text tokens, visual tokens, Iearnable action token and instruction text tokens; through causal attentions of the LLM backbone this enables the action token to integrate action information from visual tokens and then contribute to the subsequent language tasks. Let's denote the hidden states at the final layer of the MLLM as:
HG.
HY. ... HE, ho, H H E1
H
(5)
Hy E R2d are the hidden states corresponding to visual tokens, h E TRdd denotes the hidden state of the learnable action token, H E Re denotes the text tokens, d denotes the hidden dimension of the LLM. Ly and Lo denote the length of text tokens and length of visual tokens respectively. We apply three classification heads on top of the hidden state h" to predict nouns, verbs, and actions separately and use cross-entropy loss to train the elassifiers, in the belief that the classification training could guide the action token to learn better extract action-related visual information. Note that our action token only serves as an additional learning objective and is not used to give the final prediction. During inference, or when training with tasks that have no clear action labels (e.g., video captioning), we can simply compute the text generation loss.
Two-stage structured action prediction pipeline. MLLMs directly output free texts, which may hard to find an exact match with the action labels in the dataset. One could put all the possible actions in the question prompt and let MILMMs choose, but this will become in- feasible when the number of action types increases (c.g.. around 4k action types in EPIC-KITCHENS-IOO 100). This limitation prevents MLLMs from fairly comparing with action recognition models and constrains MILM from applying to applications that require structured actions. To this end, LLa designs two-stage pipeline when it needs to output structured actions. Similar to hard example mining (Section 3.I.. we use action recognition models' predictions to filter out easy actions. The difference is that we directly take the top K con- fident actions without adding the Ground Truth (GT) action. The two-stage pipeline will lower the upper-bound performance MILMM can obtain since the GT action may not be in the top K predic- tion. However, the value can control the trade-off between the upper-bound performance and the number of actions the MLLM needs to contrast. Note that our two-stage pipeline is only needed for datasets and applications that require structured actions, and is only applied at the inference stage. For the voca-locyy tasks, VAction can directly take the video and questions and give an open-endes answer. For training and benchmark construction, we always include the GT action into the choices to avoid misalignment.
Methods
8
16
zero-shot GPT-40
$2.2
N/A
zero-shor GPT-to-mini
37.4
N/A
zero-shot Video-
35.7
34.8
zero LLLVA, OV.7B
28.9
30.5
zero-shot LL
32.0
31.6
LLaVAction: LLaVA. Video 7B
71.7
73.4
LLaVAction LLaVA-OV-7B
71.3
72.3
LLaVAction: LLaVA-OV-0.58
64.8
65.4
Table 2:
Comparison on EPIC- KITCHENS-100-MOV Columns rep- resent the number of frames used for testing. Percent accuracy is shown.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
We conducted evaluations for the LLaVAction models on a wide range of benchmarks to show the models' strong action understanding abiliity. We used LMMs-E Evall (Zhang et al. 2024a) to evaluate LLaVAction on different MLLM benchmarks. We used eight frames for distractor experiments (Table D. 8 16 frames for our main results on CIC-KITCCHEN--IO-MOOG frames for ablation studies (Ta.le so that we can limit the cost of calling closed-sourced models during comparison. Meanwhile, we use 32 frames for EPIC-KITCHENS-IOO (Table 4 and 64
(Table 2 and 16
frames for the model reported in MLLM benchmarks (Table 7).
4.1 IMPLEMENTATION AND TRAINING DETAILS
We trained VLctton with three open-source baseline variants, LLLVV- LLaVA-Video-771, LLaVA-OV-7B and ALLLVA-OV-O.55 5B. With the proposed new MLLM data reformulation paradigm, re-annotated EPIC-KITCHENS- 100's training data, in total contributing to 530K annotated video-language pairs to train the VLaVAction models. The 7B and 0.5B models were trained for 12 and 11 hours 32 GH200 GPUs, respectively. Across all experiments and all baseline models, we set gradient accu- mulation to 2, batch size to 64 and total epochs to 2. Following LLa
the MLP connector, (Wang et al.] 2024)) and the visual encoder (SigLIP-384 (Zhai al. 2023)) were trained. The learning (see Zhang et al. (2024c) for additional details). Training on EPIC-KITCHENS- 100 data alone might result in over-fitting Therefore, we use data replay (Li et al., 2024a) to aid in generalization of the model. Thus, we mix with the training data of -LLVA-Video namely VA-VVVeo------ Video-178K.
LLM backbone (Qwen2-0.5B.77
was 2e-6 for the vision encoder and 1e-5 for the rest
4.2 RESULTS ON SPIC-KITCHENS 100-MMAA
OPIC-KITCHENS--O000 1O0-MAA contains hard distracting choices and is excellent to evaluate MI.LMs' fine grained action understanding. We report results for the MLLMs comparison (Table 2. LLaVAction additive ablations (Table 3) and leave- onu ablations (Appendix E..) on WITCHENS-100-MQA We start with MLLM comparisons (Table D. LLaVAction models perform much better than baselines and also obtain a 21-point improvement over GPT-4o (running GPT-4o beyond 8
LLaVA-Vikk-7B LLaVAction-7B
OOD Setting:
Zero-shot
34.8
GPT 4o hased reformulation
21.9
Random distractors
55.0
Adversarial distructers (AVIO)
Temporal Detection
65.2
Action token
69.1
GPT 4o hased reformulation
71.5
Direct Prediction
73.6
frames cost-prohibitive). Next, we verified VAc- tion's 's improvements do not just come from adding in- domain data by additive ablations (Table 3 Since both our MILM data reformulation paradigm (Section 3.2) and LLa VAction model design (Section (.3) contribute to the final performance, we ablate them together. We can see training with adversarial distractors (AVION) re- sults in the largest improvement (9.4 points). Addition of the action token gives the second most improvements (3.9 points improvement). Meanwhile, fine-tuning ALaVA-Video-T1 with the previous MI..M data reformulation paradigm (GPT-4o-based reformulation) results in a performance degradation (35.7 to 21.9). Based on the fact that it gives a meager 2.4 performance boost when we combine it with MQA task using AVION distractors, we believe this is a sign of catastrophic forgetting of MQA capability. Furthermore, directly predicting the action (direct prediction) and using contextual prior actions (temporal order learning w/ SAP) result in 2.1 and 0.5 points improvements, respectively. In summary, the combination of our MILM data reformulation paradigm and model design greatly improves the performance of the base LLa LA- -LaVA-Video-71 Video-7B model from 34.8 to 74.1 accuracy in the OOD setting and to 77.0 in the IID setting.
Temporal order Isarning
Temporal order learning w/ SAP
74.1
IID Setting:
Adversurial distractons (TIM)
76.3
Adversurial distractors (TIM) w/ SAP
77.0
Table LLaVAction additive ab- lations on EPIC-KITCHENS- MQA. Techniques are gradually added to achieve the final model. SAP denotes scquential action prediction during in- ference. Percent accuracy is shown.
note that simply
4.3 RESULTS ON ACTION RECOGNITION BENCHMARKS
With our two-stage structured action prediction pipeline (Section 3.3
LLaVAction can be fairly
compared with other action recognition models (Table 4 To assess the generalization of LLaVAc-
7

---

Published as a conference paper at ICLR 2026
tion, we tested on three datasets, which were carefully selected to exclude data used in pretraining while covering different domains.
ALLVActtoon achieves SOTA on EPIC-KITCHENS
100. Following common practice, we report the perfor- mance EPIC-KITCHENS- 100'ss validation set. We report performance in two settings ('w/ action label' and 'w/ action narration"), which differ in the candi- date choice generation. For the 'w/ action label' setting, we directly concatenated verb and noun action classes to obtain choices, which could produce implausible text. For example, the noun class for coffee maker represents "makerccoffee' in the noun definition. Meanwhile, "pour into' simplified as 'pour' in the verb definition, which could generate implausible text such as 'pour pot' that should be 'pour into pot'. Based on those observations, also report the 'w/ action narration' setting where we used the action narration of the corresponding video clip pirically observe that get better results scaling top-K from 5 to 20. Therefore, train and evaluate LLLVVccton with TIM's top 20 action predictions. LLaVAction achieves SOTA on EPIC- KITCHENS-100 under both settings.
Methods
Acc.
IPL Wang el. 2021
41.0
LaVila Zhap
2022
51,0
TAdaFormrr 7 Hiang et al 2023
51.8
LVMAE (Gundavarapu
52.1
M&M Kivng et al 2022)
53.6
AVION IZiaoo Kralienfch)
TIM (Chak of 2024]
56.4
Ours, LLaVAction-T8 action labe]
58.3
Ours, LLaVAction-BB w/ action narration
63.2
Table 4: Action recognition on EPIC- KITCHENS-IOG Top-1 accuracy on action classification. For specific verb- performance Figure
(more details in Appendix F.I). ...I) We em-
LLa VAction generalizes well to other datasets. We tested LLaVAc- tion on two recent action recognition benchmarks one testing generaliza- tion for a different cooking dataset (Bonnetto
Methods
Acc. Head Acc.
Tail Acc.
7em--soo generalization the second-stage mode!
AVION
Zhao & Krithenbühl 2023
19.3
LLaVA-Vieen
Vineo /B /hang at 20144
22,5
22.9
18.8
Ours, LLaVAction-78
36.2
38.1
24.6
REPHL-Smarr-Kiichen-3 et al.] 2025)) and for a differ- ent domain (tool assembly, Mce- cano, (Ragusa et al., (02II). EPFL- Kitce--Kt has 30 verbs and 46 nouns in common with EPIC- KITCHENS-100, which enables us also to compare with specialized models (such as AVION) for zero- shot generalization of second-stage models. We used the multi-modal VideoMAE's top predic- tions in APFL-Smart- Kitthe-ithhehntt to generate the MQAs for LLaVAction-7B and LLaVA-Video-7B To fairly compare with the specialized model AVION, we also use those top 5 predictions to filter AVION's predicted action logits. LLa obtained better zero-shot accuracy than AVION and LLaVA-Video (Table 5). Most importantly, the zero-shot VLaVAcnion even obtained similar over- all performance to the trained VideoMAE model and better tail action accuracy (Tail Acc.). When finetuning LLa VAction obtained SOTA performance (Table
Trained model
VideoMAB (ongg et al
2022
37,5
41.1
16.6
Multi-modal VidCOMAE Bonnetto et al.
2025
40.0
43.6
19.4
Ours, LLaVAction-BB 7B
46.6
49.7
27.0
Table 5: Action recognition on EPFL-SmarrtKitchen-B3 L.L outperforms prior methods the zero-shot and finetuned setting.
On Meccano we used FlowFast (Feichtenhofer et al. 2019) to generate hard distractors and finetuned LLa VAction. Even when trained for only one epoch, LLa VAction obtained 51.7 top-1 accuracy, beating SlowFast's 42.8.
Our training pipeline and model design is not limited to any particular domains or base MLLMs. To further support that, we tested our method on a very differ- ent domain, animal fine-grained behavior understand- ing. Specifically, we tested on the Animal Kingdom dataset (Ng et al. 2022). which has 140 fine-graing ine-grained actions and at most 12 actions can happen at the same time (i.e., multi--lasssffiaatie
Methods
Jaccard Acc.
VideoMAB: (Mamooler al. 2025
53.1
LLaVA-Vides (Zero-hoot
30.5
LLaVA- 7B (Random choice)
46.7
Ours, LLaVA Video-
IntmmLL3-88B7enn
28.
M.temVLL-8BBRRnddoo choice)
43.9
task). We adapted both LLaVA- VVVA-Viee--71 7B and InterVL33. (Zhu et al.) 2025) and trained with hard examples generated from Video- MAE MMamooler et al., 2025) to serve as our methods; for two base models (LLaVA. Video-7B and Intern VL3-8B) we compared to zero-shot and training with randomly generated options. Our methods, either using
Ours. InteraVL3-88
58.7
Table 6: Action recognition Animal Kingdom.
Intern VL3-8B
8

---

Published as a conference paper at ICLR 2026
Caption
Open-ended Q&A
Malti-choice Q&A
Mangalant ham et al. 2023
Patrioocnn CAAEE al. 2022
Khatak at 20255
Chial 3024
Chen et al 2024n
Video --l- Ma ml 2025
Yo Yo 2019)
VideoChnGPP
Masic et al 2023
Temp Compuss [Lut et al (2244)
[i (0225
Long LongVide
[Wa et al, 2024
Xio or al. 2002
Purce posspttonTTess
Acther QA
SenSehema
MVBench
MidcaMMM
wolo subs
VideoD
NesiA
ViC
CVRR
Li
Closed-soarce models
GPT-4V Achiam et
2023
70.8
43.5
59.9/63.3
61.3
GPT-4b
71.//7722
Gemini- F.5.Flsh Flash Team et
2021
35.1
65.7
70.3/75.0
61.6
Gemini- P.5-Pro Team
39.3
72.2
75.0/81.3
64.0
Open-source models
LongVA-7B Zhang et al
2024b
34.5
16.5
30.0
52.6/54.3
68.3
mPL.UG-Owa Owds
59 39..1
VideoCCar2-7B (L at
36.5
49.1
25.8
38.5
60.4
42.3/54.6
78.6
VideoLLaMA22113
C CRRb
2024
53.0
21.6
32.2
51.7
47.9/50.3
51.4
I.aVA-OV-7B [1i ci
38.8
56.6
60.1
56.7
58.2161.5
56.5
79.4
57.1
LLaVA-Vidoo-78 Zhang 0i 8l.
2024c
39.0
3.44
25.7
66.0
3.04
51.3
66.0
57.3
58.6
63.3/69.7
58.2
83.2
67.9
Ours, LLaVActiot=7
Relative improvement of ours the baseline LLaVA- VA
+1.2
-0.1
40.4
+0.9
-0.03
+4.3
+0.1
+1.7
+2.5
+0,6/+1.7
+0.4
-0.4
+2.3
Table Performance on other MLLM benchmarks that contain human actions. Please note, we noting that
not claiming SOTA, source model (LLaVA-Video-7B (Zhang et al.] 2024c)). We also show sub-task performances in Appendix G We show top-performance closed-source models for reference. Top open-source mod- els are shown in bold, and the second-best are underlined.
improve performance
baseline open-
the base MLLMs obtained much better performance (61.0/58.7) compared the original mod- els (30.5/28. I) and models finetuned with random choices (46.7/43.9) (Table 6). Furthermore, our method beats the baseline (VideoMAE) for both base models.
4.4 RESULTS OTHER MLLM BENCHMARKS
Apart from comparing with action recognition models, we want LLaVAction keep general video understanding abilities and also improve fine-grained action understanding on other zero- shot MILMM benchmarks. Therefore, we tested LLLVAActio-7BB on 13 MLLM benchmarks that test various MILM video understanding abilities. The evaluated benchmarks consist of two video caption benchmarks, five open-ended Q&A benchmarks and six multi-choice Q&A benchmarks. LLaVAction-7BB outperforms V-VVideo-77B on 10 benchmarks, indicating the enhanced video understanding ability of our model (Table
4.5 ATTENTION-BASER BASED ANALYSIS
We sought to analyze the impact of action-related training and LLa model design in an inter- pretable manner. Following the approach in (Zhang al. 20255), employed token attention anal- ysis to understand model behavior (Figure 4). We computed average text-visual token correlations for both LLaVA-Video-P7 7B and LLa LLaVAction-7B using the EPIC-KITCHENS. 100-MQA dataset. For -a-VV---ieeo-77 with randomly generated answer choices. Analy- sis Methodology: We first computed the text-visual attention tensors of size N x V, where N the number of data samples, I is the number of text tokens and V is the number of visual tokens. We then calculated the maximum across the text token dimension, followed by computing mean and 90th percentile values to estimate text-visual correlations.
fair comparison, we fine-tuned
Text- Visual Correlation Results: LLaVA- Video-7B 7B achieved mean and 90th percentile values 0.00476 and 0.0104, respectively, while LLaVAction-77 achieved mean and 90th percentile values 0.00769 and 0.0175, respectively. This indicates that LLaVAction attends more strongly to visual cues compared to L.VA-Vided likely due hard example mining strategy.

---

Published as a conference paper at ICLR 2026
Distracting options
wipe spatuls spoon
C, pick up tork put spatidi
spatula
LLWet-idee
Lovication
Figure 4: Qualitative attention for clip. Anecdotally, LLaVA- Video mainly attends to the wooden spatula that is placed in the drawer, LLaVAction also attends to the arms and, correctly, the plastic spatula that is being taken. We quantify visual-text token correlations in the main text.
Action Token Analysis: We computed the text-action token attention tensors of size N T 1, calculating the maximum over the text dimension followed by the mean to estimate text-action cor- relations. LLa VAction demonstrates higher than its text-visual correlation (0.00769). Notably, 99% of visual tokens exhibit lower text correlations compared to action tokens. Since action tokens exclusively attend visual tokens, this suggests they effectively aggregate visual information relevant to fine-grained actions, enhancing VActionn ' action understanding capabilities.
average text-action token correlation of 0.143, significantly
4.6 DIFFICULTY LEVELS OF FINE-GRAINED ACTION UNDERSTANDING ANALYSIS
features distinguishing fine-grained actions, individual samples may still vary in difficulty. To un- derstand how baseline models and LLaVAc- tion perform across different difficulty lev- els, we adopted the concept of psychometric curves (Boring. 1917). We utilized GPT-4o to rate the difficulty of distinguishing between the options (detailed prompt in Appendix (C6)
While EPIC-KITCHENS--OO-.
Difficulty levels
L. Video Acc.
0.405
0.360
0.342
0.339
LLaVAction
0.735
0.744
0.738
0.726
Samples
400
3529
2836
2903
Table 8: Model performance under different difficulty levels. LLaVAccion more robust with different semantic similarity between options.
from (very easy) to (hard) and then reported the performances of LLaVA-Vides and VLattction under different levels on EPIC-KITCHENS- 100-MQA (Table B. LLaVAction not only achieves higher overall accuracy but also maintains more robust performance difficulty increases, showing less performance degradation compared to baseline models.
5 CONCLUSION
Recent advances in MLLMs prompted our investigation of their fine-grained action understanding WATCCHEES--OO-MME
benchmark, which uses similar tions as distractors, we reveal that state-of-the-art MILMSs face significant challenges in action dis- crimination tasks. We address these limitations by introducing specialized data reformulation strate- gies and action-aware architectural components that substantially enhance MLLM action recogni- tion capabilities. The resulting LLaVAction model achieves robust performance and demonstrates strong generalization across our benchmark, three additional action recognition datasets, and ten comprehensive MILMM video understanding benchmarks.
abilities. Through our proposed
ACKNOWLEDGGENNTS
We thank the Swiss AI Initiative Project ID a03 and a144 from the Swiss National Supercomputing Centre (CSCS); H.Q. and A.M. thank the Boehringer Ingelheim Fonds PhD stipend; M. W.M.M. thanks the Vallee Foundation; M. W.M. and A.M. thank the SNSF by grant No. 320030-227871.
AUTHOR CONTRIBUTIONS
Conceptualization. S.Y.. H.Q., A.M.. M.W.M.: Methodology & Software: H.Q., S.Y., A.M.,
M. M.; Experiments: H.Q., S.Y.: Writing: H.Q., S.Y., M. M.W.M, A.M.; Visualization: H.Q., S.Y., M.W.M.: Funding acquisition: M. M.W.M., A.M.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work utilizes established human activity benchmarks and trains the resulting VLatiiio model exclusively on data that contain common daily activities. Given the benign nature of the training data and the focus on routine human behaviors, we anticipate minimal ethical concerns. All human activity data used come from publicly available datasets that have previously been used by the research community.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, released code. data, the benchmark, and models at http://giihbbooom///iiiieeooooot
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 1rXiv:2303.08772 2023.
Andy Bonnetto, Haozhe Qi, Franklin Leong, Matea Tashkovska, Mahdi Rad, Solaiman Shokur, Friedhelm Hummel, Silvestro Micera, Marc Pollefeys, and Alexander Mathis. Epfl-smart- kitchen-30: Densely annotated cooking dataset with 3d kinematics to challenge video and lan- guage models. arXiv preprint AXKv:25006006000 2025.
Edwin G Boring. chart of the psychometric function. The American journal of psychology, 28(4): 465-470, 1917.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine-gre grained temporal under- standing for multimodal video models. arXiv preprint arXiv:2410. 10818, 2024.
Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng. Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. arXiv preprint arXiv:2410.03051, 2024.
Jacob Chalk, Jaesung Huh, Evangelos Kazakos, Andrew Zisserman, and Dima Damen. Tim: time interval machine for audio-visual action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18153-18163, 2024.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang. Li Yuan, et al. Sharegpt4video: Improving video understanding and genera- tion with better captions. Advances in Neural Information Processing Systems, 37: 17:19472-19495, 2024a.
Ling-Hao Chen, Shunlin Lu, Ailing Zeng. Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.2034 2024b.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint NXiv22406.07476, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocen- trie vision: Collection, pipeline and challenges for epic-kitchens- 100. International Journal of Computer Vision pp. 1-23, 2022.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
OpenAl et al. Gpt-4o system card, 2024. UBLLtttpss//arrbr.rr//////111
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 6202-6211, 2019.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Iims in video analysis. arXiv preprint arXiv:2405.21075. 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Ilms video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 208 24818821111111111111111111111111 2025.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Nitesh Bharadwaj Gundavarapu, Luke Friedman, Raghav Goyal, Chaitra Hegde, Eirikur Agustsson, Sagar M. Waghmare, Mikhail Sirotenko, Ming-Hsuan Yang, Tobias Weyand, Boqing Gong, and Leonid Sigal. Extending video masked autoencoders 128 frames, 2024. URL https://
arxiv.org/abor 2411 13683
Jaaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang. Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 06584-26599 2024.
Ziyuan Huang, Shiwei Zhang, Liang Pan, Zhiwu Qing, Yingya Zhang, Ziwei Liu, and Marcelo H. Ang Jr. Temporally-adaptive models for efficient video understanding, 2023. URL https: //axxivv.rrg/ab//23008
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17X.v:1705.069500 2017.
Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Jameel Hassan, Muzammal Naseer, Fed- erico Tombari, Fahad Shahbaz Khan, and Salman Khan. How good is my video-Imm? complex video reasoning and robustness evaluation suite for video-Imms. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1642-3651, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li. Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint a4Xiv:2408. 03326, 2024a.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose Trends@ in Computer Graphics and Vision, 16(1-2):1-214 2024b.
purpose assistants. Foundations and
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp- 22195-22206, 2024c.
Xinhao Li, Yi Wang. Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flase Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.005: 2024d.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024a.
Shuming Liu, Chen Zhao, Fatimah Zohra, Mattia Soldan, Alejandro Pardo, Mengmeng Xu, Lama Alssum, Merey Ramazanova, Juan León Alcázar, Anthony Cioppa, et al. Opentad: A unified framework and comprehensive study of temporal action detection. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 2625-2635, 2025.
Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model, 2024b. URLhttps:
10803
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024c.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou, Tempcompass: Do video 1lms really understand videos? arXiv preprint arXiv:2403.00411 2024d.
Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. Videoeval-pro: Robust and realistic long video understanding evaluation. arXiv preprint arXiv:2505. 14640, 2025.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint MxXi::2306.055224. 2023.
Alcksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
Sepideh Mamooler, Haozhe Qi, Valentin Gabeff, Syrielle Montariol, Antoine Bosselut, and Alexan- der Mathis. Fine-tuning vision-language models for animal behavior analysis. In LLM for Scien- tific Discovery: Reasoning, Assistance, and Collaboration, 2025.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212 2023.
Xun Long Ng. Kian Eng Ong. Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19023-19034, 2022.
Pha Nguyen, Sailik Sengupta, Girik Malik, Arshit Gupta, and Bonan Min. Install: Context-
aware instructional task assistance with multi-modal large language models. arXiv preprint arXiv:2501. 12231, 2025.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36:42744-447777 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The mec- cano dataset: Understanding human-object interactions from egocentric videos in industrial- like domain. In Proceedings of the IEÉE/CVF Winter Conference on Applications of Computer Vision, pp. 1569-1578, 2021.
13

---

Published as a conference paper at ICLR 2026
Rahul Ramachandran, Ali Garjani, Roman Bachmann, Andrei Atanov. Oğuzhan Fatih Kar, and Amir Zamir. How well does gpt-4o understand vision? evaluating multimodal foundation models standard computer vision tasks. arXiv preprint arXiv:2507.01955, 2025.
Michael Renner, Ethogram. In Encyclopedia of Animal Cognition and Behavior, pp. 1-4. Springer, 2018.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010-1019, 2016.
Dingfeng Shi, Qiong Cao, Yujie Zhong, Shan An, Jian Cheng, Haogang Zhu, and Dacheng Tao. Temporal action localization with enhanced instant discriminability. arXiv preprint DrXiv:2309.05590. 2023.
Abhinay Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- based object detectors with online hard example mining. In Proceedings the the IEEE conference on computer vision and pattem recognition, pp. 761-769, 2016.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 12. 1805, 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 15:10078-10093, 2022.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14549-14560, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- model's perception of the world at any resolution. arXiv preprint 240i:2409.121991 2024.
Xiao Wang, Jingyun Hua, Weihong Lin, Yuanxing Zhang, Fuzheng Zhang, Jianlong Wu, Di Zhang, and Liqiang Nic. Haic: Improving human action understanding and generation with better cap- tions for mudti-modal large language models. arXiv preprint arXiv:2502. 2025a.
Xiaohan Wang. Linchao Zhu, Heng Wang, and Yi Yang. Interactive prototype learning for egocen- tric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8168 8177, 2021.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Interavideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.0319)) 2022.
Yuxuan Wang, Yiqi Song, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long streaming video understanding with recurrent memory bridges. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 241 70-2418, 2025b.
Haoning Wu, Dongxu Li. Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-conters interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828 28857, 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021.
14

---

Published as a conference paper at ICLR 2026
Xuchan Xiong, Anurag Arab, Arsha Nagrani, and Cordelia Schmid. M&m mix: A multimodal multiview transformer ensemble, 2022. URL wwtpt::t////ee http:://axxvv..oo orq/ ha////b///// 220222066..00 09852
Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal Ilms. arXiv preprint arXiv:2410.07177 2024a.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint
2024b.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. arXiv, 2019. URL aattps://arxe
org/ abs 906. 02467
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556 0166699577255 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training 2023. URL
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision.
arXiv preprint arXiv:2406. /6852, 2024b.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025a. URL httts::/aaxiisi www.//ooor//ooooo..oo abs/
2501 038895
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.027133, 2024c.
Yunzhu Zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang, and Linchao Zhu. Flexselect: Flexible token selection for efficient long video understanding. arXiv preprint wxXix22506.00993 2025b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models, 2024d. URL http:/::://rxii..rr/////// 2332.00922
Yue Zhao and Philipp Krähenbühl. Training large video model on a single machine in day. arXiv preprint arXiv:2309.1666) 2023.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models, 2022. httpp:// /tttppps///axi....// www.voog/aaao. hatrg//b///// abaabb//22. /2112.212222 04501
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
15

---

Published as a conference paper at ICLR 2026
A
METHODS: LICENSING INFORMATION
Code/Dataset
License
LLaVA-NeXT
Apache-2.0
AVION
MIT
T1M
CC-NC-SA-4.0
EPIC-KITCHENS-
Dataset 55 and extended
NC-Government
Annotations
CC-NC-SA-4.0
Imms-eval
MIT and Apache-2.0
Table 9: List of codes and datasets with their corresponding licenses.
B METHODS: GPT-40 DISTILLATION.
Due to the cost, we sample 4 frames per annotated video clip to go over the training set of EPIC- KITCHENS-I00.
We first get the caption corresponding to all video clips in the training set, and then we use the captions obtained create open-ended question-answert We show the corresponding prompts for the generations of captions and open-ended question-answen as follows.
B.1 GPT-40 AND GPT-40-MIN MIMII ANNOTATION PROMPT FOR THE CAPTION TASK.
You are viewing video frames from an egocentric perspective and you are the person. Describe the video frames in detail and reason about the actions you are performing. You will be provided with the human-annotated ground-truth for the action, but you should independently come to your own conclusion.
you disagree with the human annotation, indicate "true" wwt.:with.aa..amttttee.
field of your response, and provide your reasoning without mentioning the ground-truth answer. This will keep your reasoning clean. If you agree with the human annotation, indicate "false" in the "disagree with human annotation" field and provide your reasoning without referencing the ground-truth to maintain a clean description. The true ground-truth action is (gt.answer). Your reasoning steps should include supporting evidence for the action, such as the duration of the video, the sequence of actions the person performs, the objects they interact with, and the overall context of the video.
the
As a general guideline, for videos longer than 3 seconds, provide detailed reasoning steps, and for videos shorter than 3 seconds, generate less detailed reasoning. The video duration is (end_second start.second: 3ff] seconds. Make sure you use the -irst-person perspective in your reasoning.
first -peson
B.2 GPT-40 AND GPT-40-MINI ANNOTATION PROMPT FOR OPEN-ENDED QUESTION ANSWERING
Your job is to create 3 question-answer pairs based on the text below. The text contains first-person narrative of video frames from an egocentric perspective of a person interacting with objects in kitchen. caption text You ask questions such

---

Published as a conference paper at ICLR 2026
as: What object am I interacting with? What objects visible in the video? What is the sequence of the atomic actions am performing? Make sure your questions can be answered based on the information provided in the text. Do not ask questions ins that require additional context information beyond what is given.
C METHODS: LLAVACTION TASK PROMPTS
C.I LLAVACTION CAPTION PROMPT
Describe in details what you see from the video frames. Try to focus on what you are doing.
C.2 LLAVACTION PERSPECTIVE PROMPT
Egocentric. For the EgoSchema benchmark, given that our LLa is trained with egocen- tric perspective prompt on EPIC-KITCHENS-IO0 100, we use the same egocentric perspective prompt when we evaluate our model on EgoSchema benchmark.
You are seeing this video from egocentric view and you are the person. Your hands are sometimes interacting with objects. what action are you doing?
Allocentric.
The video is taken from egocentric view. The person's hands are sometimes interacting with objects. what action is the person doing?
C.3 LLAVACTION TEMPORAL DETECTION PROMPT
The provided video contains an action (ACTION NAME) that lasts 2696 seconds. What is the relative start and end time of the action in seconds? Format it as 'start.timestamp: end.timestampl and round to decimal places.
C.4 LLAVACTION DIRECT PREDICTION PROMPT
What action are you performing? Give a short sentence such as move knife'.
C.5 LLAVACTION PRIOR ACTION LEARNING PROMPT
(prev2.offset) seconds ago, you started an action marratioo. (prevloffset] seconds ago, you started an action (prevl narration]. What action are you currently performing? Here are the options of actions you can select:
C.6 DIFFICULTY LEVEL ASSESSMENT PROMPT
You are analyzing a multiple-choice question for fine-grained action recognition. Your task is to rate the difficulty of distinguishi between the options based on how similar they are to each other and the ground truth answer.
Ground Truth Answer: gt.answer
Options: options.text
Please analyze the semantic similarity between the options and rate the difficulty on scale of 1-4:
2

---

Published as a conference paper at ICLR 2026
-1 (Very Easy): Options very different from each other and the correct answer is obvious
2 (Easy) Options have clear differences, correct answer is fairly obvious
(Medium): Options have moderate similarity, requires some careful consideration
4 (Hard) Options are quite similar, subtle differences make it challenging
Consider factors like:
Semantic similarity between action descriptions
Specificity generality of actions
Whether options describe similar but distinct actions
How confusable the distractors are with the ground truth Respond with only a single number (1-4) representing the difficulty score.
D AVION AS OOD DISTRACTORS
2024 predictions build our IPIC-KITCHENS-IOOMOOAA bench- mark. When we use similar aar ideas to build the hard distractor training set, it results in IID setting if we still use TIM's (Chalk al. 2024) predictions. Methods could directly overfit on TIM's choice combination to obtain better performances instead of contrasting fine ome-grained actions. Therefore, we used AVION's (Zhao & Krähenbühl 2023) predictions during training to serve as an OOD setting.
We utilized TIM's (Chalk
Here, we provide three experiments to support using AVION can be OOD setting. First, we cal- culated the top-1 agreement percentage (65%) and top-5 overlap percentage (45%) between AVION and TIM in EPIC-KITCHENS-IO0 validation suggesting considerable difference in distri- bution, especially when K is larger.
Additionally. we computed the Jensen-Shannon Divergence (JSD) between the softmax outputs of Avion and Tim across the validation set (9668 samples). The mean JSD was 0.674 + 0.089, with a 95% confidence interval of [0.672, 0.765]. We obtained p- pavulee < 0.001 and Cohen's 's d of 7.57, indicating a large and statistically significant difference between Avion and Tim. As consequence, their generated distractors should be
coming from two different distributions.
An important signature of the IID vs. OOD argument that OOD is less vulnerable to overfitting when giving the model more chances to explore the training data. Following our OOD setting that uses Avion distractors for training and TIM distractors for testing, we performed experiments that vary K both training and testing, for K 10, 20, got 74.3, 69.5, 64.1, respectively. Since test-time distractors are generated by TIM and training-time distractors are generated by AVION, we believe increasing K training introduces overfitting, and thus it does not generalize well to TIM's distractors, which further supports our rationale for the OOD setting.
E ADDITIONAL ABLATIONS
E.
ACTION TOKEN DESIGN ABLATION
Our action token design (Section 3.3 effectively sees and encodes action-related video information, and hence benefits the guestion-answering task. To further support our action token design, we implement several other token-aggregation methods. Specifically, our action token is one learnable token added between the visual tokens and text tokens and supervised in the last layer with the action classification loss. We hence implement three variants: adding three action tokens and supervising them with verb, noun, and action separately in the last layer; 2) adding one action token and supervising in the first layer of the MILM;; adding action token and supervising it across
3

---

Published as a conference paper at ICLR 2026
all MILM layers. The results in Table 10 our benchmark. We can our action design performs the best while keeping simplicity.
Token designs
Acc.
3 tokens, last layer token, first layer
68.8
66.2
token, all layers token, last layer (Ours)
31.7
69.1
Table 10: Action token design ablation study. The top percent accuracy is shown.
E.2 LEAVE-ONE-OUT ABLATION
Remarkably, the 10-point gain over our baseline model cannot be attributed only a single factor. We took our full model, i.e.. the base plus all added methods, which we call VActionn.-7 and performed leave-on-out ablation (Table TT). Given our additions adds negligible overhead in the inference time (only one special vision token added to the baseline model), we then suggest using our full LLaVAction-777 7B and techniques in downstream tasks.
LLaVA-Video-7B
Acc.
Full (LLaVAction-7B)
74.1
Full w/o adversarial distractors (AVION)
69.7
Full w/o action token
73.6
Full w/o temporal detection
72.2
Full w/o GPT-4o-based reformulation
73.2
Full w/o direct prediction
73.2
Full w/o temporal order learning
72.3
Table 11: Leave-one-out ablation study. Full denotes having all the proposed methods. In each row we drop method from the full method and report the resulting performance. 16 frames were used for both training and testing, and the percent accuracy is shown.
E.3 STRUCTURED ACTION PREDICTION ABLATION
Our proposed two-stage pipeline enables MLLMs to fairly compare and outperform other SOTA action recognition methods. MILMS directly output free texts, which makes hard to find an exact match with the action labels in the dataset, especially when the action space is huge and fine- -raingrieed. With an external model applied in the first stage to filter out casy, irrelevant actions, MLLMs can mainly focus differentiating between the hard distracting actions. To support that, evalu- ated LLaVA-Video-78 with the same external model on the EPIC-KITCHEN action recognition benchmark (Table T2). We with an external model, showing that it struggles to solve the hard distractors. Meanwhile, we further implement another way (denoted 'Multi-round appending') achieve structured action output. Specifically, we first prefill and store the KV Cache for tideo+prompt-question to avoid repeated computation. After that, we append each action class compute the cross-entropy loss. The action class with the lowest loss is selected as the final action prediction. We test for both the zero-shot LL Video-78 7B and our LLaVAction models on the ÊPIC-KITCHEN action recognition benchmark (Table 12]. The results show our two-stage action prediction pipeline can obtain much better performance under both fine-tuned and zero-shot settings. Most importantly, the multi-round appending is extremely time-consuming. Although KV Cache storage avoids com- puting video-prompt+questtion repeatedly, the model still needs to infer 4K times to obtain the correct answer. Evaluating the model EPIC-KITCHENS-ION 100'ss validation set (9668 samples) takes around 820 GPU hours when using the multi-round appending approach. In comparison, our two-stage approach only takes 4.3 GPU hours, making two-stage method 190 times faster.
that the performance of
is much worse,
4

---

Published as a conference paper at ICLR 2026
Methods
Acc.
LLaVA Video-7B
Multi-round appending
10.3
Two-stage (Ours)
26.5
LLaVAction-78
Multi-round appending
40.0
Two-stage (Ours)
58.3
Table 12: Structured action prediction ablation study.
The top I percent accuracy is shown.
Methods
Action label
Action narration
zero-shot
26.5
35.7
zero-shot LLa LLaVA-OV-7B
19.6
28.9
zero-shot LLa VA-OV-0.5B
24.8
32.0
Table 13: Quantitative results for action label vs. action narration. Models are inferred with
eight frames inputs.
F QUALITATIVE EXAMPLES
F.1 COMPARING ACTION NARRATION AND ACTION LABEL IN EPIC-KITCHENS-IO0
The action labels in EPIC-KITCHENS- originate from the raw action narrations that are curated and compressed by combination of word clustering and iterative manual refinement (Damen al. 2022]. However, this compression might change the semantic meaning of both nouns, verbs and the way they combined. As result, large language models that sensitive to the meaning of words can be misled (see comparisons in Appendix Figure 5 and Table 13). While we show SOTA results using the action label, we note that we can achieve better performance if use the uncompressed, original narrations. We hope that our work could inspire future work to study the best text representation actions train and evaluate MILMS in action recognition.
Qualitatively: we illustrate some examples of choices represented in the action label manner (Ap- pendix F). We show the ground truth option in blue and the prediction of LLa LLLVAction-7BB in pink. We can see LLaVAAtion-7BB's predictions also make sense in those examples and hence cause ambiguity across choices. Instead, the corresponding action narration fits better to the language's nature and can better describe the video content with less ambiguity.
Quantitatively: furthermore, we also quantify MLLMs' zero-shot performance (LLaVA-OV-0.5B) LLaVA-OV-7B, LLaVA-Video-1e when using action labels action narrations as inputs (Table 13 The inferior zero-shot performance of 3 evaluated models when tested on the action labels action representation supports qualitative observations that action labels less ideal than narrations for MLLMs.
F.2 DIFFERENT CHOICES COMPARISON
Here, show examples of choices generated by random sampling. AVION top-5 predictions, and TIM top-5 predictions (Appendix Figure 6) We can see that the randomly selected choices have many trivial choices that can be easily distinguished with the correct answer. In comparison, choices generated based on AVION and TIM top-5 predictions become much more similar to the correct answer and exhibit features such as similar object/scene, temporal orders or object relationships that are emphasized by other benchmarks.
F.3 LLAVACTION CAPTION
Here, we show one video caption example of different models including GPT-4o, LLaVA- Video- 7B and our ILLVAccton-7B (Appendix Figure 7). We see the interacting object (pizza piece) is
5

---

Published as a conference paper at ICLR 2026
pretty small in the video and there are also many other distracting objects. Both GPT-4o and LLaVA- Video- cause hhallucinations' in their descriptions. For example, GPT-4o thinks the person holds the slice with both hands. Instead, LLa LLa still retains the video caption ability and can generate plausible descriptions of the video.
G SUB-CATEGORY PERFORMANCE COMPARISONS ON THE ADDITIONAL BENCHMARKS
Snice MVBench and LongVideo also have sub-category measurements with many of them re- lated action understanding, we also show the sub-category performances on these two benchmarks in this section.
G.1 PERFORMANCE COMPARISON ON SUB-CATEGORIES OF MVBENCII
Here show the performance comparison between sub-categories of MVBench. We can see LLaVAction- 7B boost the performance on many action- related categories such action count, action sequence and fine-grained action, etc.
and
Tasks
LLaVA-Video-7B
LLaVAction-7B (Ours)
Difference
Action antonym
76.0
75.0
-1.0
Action count
57.0
65.0
8.0
Action localization
61.0
63.5
2.5
Action prediction
62.0
59.0
-3,0
Action sequence
70.5
72.5
2.0
Character order
74.5
79.0
4.5
Counterfactual inference
50.0
52.0
2.0
Egocentric navigation
30.5
28.0
-2.5
Episodic reasoning
53.5
54.0
0.5
Fine-grained action
48.0
49.0
1.0
Fine Fine-grained pose
54.5
61.5
7.0
Moving attribute
71.0
72.5
1.5
Moving count
44.0
43.0
-1.0
Moving direction
35.5
31.0
4.5
Object existence
60.0
59,0
-10
Object interaction
84.5
83,5
-1.
Object shuffle
41.5
44.0
2.5
Scene transition
93.5
90.5
-3.0
State change
54.0
61.5
7.5
Unexpected action
81.5
79.0
-2.5
Table 14: Sub-category comparison with LLa VA-Vieoo7- on MVBench.
G.2 PERFORMANCE COMPARISON ON SUB-CATEGORIES LONGVIDEOBENCII
Here we show the performance comparison between LLaVA- V-Veeoo- and our LLa VAction- on sub-categories of Long VideoBench. We can see LLaVAction-7B also boosts the performance on many action-related categories such as event hefore/after, text-referred object attribute, and object- before/after object.
G.3 PERFORMANCE COMPARISON ON SUB-CATEGORIES OF VIDEOMME
Here show the performance comparison between LLaVA-Video. and LLa LLaViction-777 sub-categories VideoMME. Our model did not improve the action recognition performance on VideoMME possibly due the domain gap between VideoMME and EPIC-KITCHENS-100)

---

Published as a conference paper at ICLR 2026
Tasks
LLaVA- Video-7B
LLaVAction-7B (Ours)
Difference
Event-Referred object
72.31
69.23
-3,08
Beetee event
67.02
67.02
0.0
Rhferr-RRferred event
67.82
64.37
-3.45
object-Beforekl object
57.58
59.09
1.52
Scene-Referre object attribute
71.59
70.46
-1.14
event
72.04
66.67
-5.38
object
63,89
63.89
0.0
Refe-Reee object attribute change
55,56
52.78
-2.78
Referred object tracking
65,43
66.67
1.23
Sequence of scenes
41.24
41.24
0.0
object attribute
59,49
62.02
2.53
event
56,92
56.92
0.0
object
59.21
59.21
0.0
Event before/after text
50.68
58.90
8.22
Object before/after text
58.11
52.70
-5.41
Text-Referred object attribute change
47.56
50.00
2.44
Text-Refent object tracking
32.88
32.88
0.0
Table 15: Sub-category comparison with LLaVA-Video-7B on LongVideoBench.
H EXTENDED DISCUSSION
Egocentric vs. Allocentric perspective. MILMM can be assigned with different roles before seeing the video. Since the videos are taken from the first-person perspective, we believed the egocentric perspective aligns better with the LLM pretraining data. Therefore, we switch from the third-person (allocentric) perspective to the first-person (egocentric) perspective to better guide the model. We present the prompts in the Appendix C.2 When we fix the distractors from random sampling, using the egocentric prompt gives a 0.5 point improvement over using the allocentric prompt on EPIC-
Alternative approaches we tested. We also tested a few alternative approaches to improve MLLMS in our benchmark. We tried self-consistency predictions, which do not yield improvements, perhaps due to the task being vision-centric. Additionally, we explored multi-modal chain-of of-thoug-t (COT) reasoning by prompting the model generate caption prior addressing the multi-question an- swering task. However, we found that the model exhibited reluctance to perform this action, despite being capable of generating captions or answering multi-choice questions independently. of it is to inference the model twice, so we have the caption first and feed that into the instruction of answering multi-choice question task, similar to (Zhang al. 2024d). While minor improvement was observed, we think it is not worth the 2X compute. We believe that video action recognition is a good way to explore video reasoning for MILMSs. However, we leave COT improvements on this task for future work.
variant

---

Published as a conference paper at ICLR 2026
Tasks
LLaVA-Vidoo-7B
LLaVAction-7BB (Ours)
Difference
Categories: Artistic Performance
68.9
69.4
0.5
Categories: Film & Television
71.7
72.8
1.1
Categories: Knowledge
76.0
75.4
-0.6
Categories: Life Record
71.7
71.7
0.0
Categories: Multilingual
67.8
61.1
Categories: Sports Competition
65.6
66.4
0.8
Task Categories: Action Reasoning
69.5
69.1
-0.4
Task Categories: Action Recognition
69.0
68.4
-0.6
Task Categories: Attribute Perception
83.8
83.3
-05
Task Categories: Counting Problem
48.1
46.3
-18
Task Categories: Information Synopsis
87.0
86.7
-0.3
Task Categories: OCR Problems
73,4
74.1
0.7
Task Categories: Object Reasoning
73.6
73.3
-0.3
Task Categories: Object Recognition
75.7
77.1
1.4
Task Categories: Spatial Perception
68.5
72.2
3.7
Task Categories: Spatial Reasoning
82.1
82.1
0.0
Task Categories: Temporal Perception
76.4
78.2
1.8
Task Categories: Temporal Reasoning
51.4
52.0
0.6
Video Sub Categories: Acrobatics
65.6
64.4
-1.2
Video Sub Categories: Animation
58.9
60.0
1.1
Video Sub Categories: Astronomy
77.8
77.8
0.0
Video Sub Categories: Athletics
66.7
73.3
6.6
Video Sub Categories: Basketball
54.4
51.1
33,
Video Sub Categories: Biology Medicine
78.9
78,9
0.0
Video Sub Categories: Daily Life
78.9
75.6
-3.
Video Sub Categories: Documentary
74.4
76.7
2.3
Video Sub Categories: Esports
62.2
60.0
-2.
Video Sub Categories: Exercise
58.9
67.8
Video Sub Categories: Fashion
68.9
70.0
1.1
Video Sub Categories: Finance & Commerce
80.0
80.0
0,0
Video Sub Categories: Football
72.2
75,6
3.4
Video Sub Categories: Geography
76.7
75.6
Video Sub Categorics: Handicraft
77.8
76.7
-1.1
Vidco Sub Categories: Humanity & History
66.7
67.8
1.1
Vidco Sub Categories: Law
82.2
80.0
-22
Video Sub Categories: Life Tip
70.0
67.8
-22
Video Sub Categories: Literature
80.0
73.3
Video Sub Categories: Magic Show
62.2
65.6
3.4
Video Sub Categories: Movie & TV Show
68.9
70,0
1.1
Video Sub Categories: Multilingual
67.8
61.1
6.7
Video Sub Categories: News Report
84.4
84.4
Video Sub Categories: Other Sports
72.2
72.2
Video Sub Categories: Pet & Animal
78.9
78.9
0.0
Video Sub Categories: Stage Play
82.2
80.0
-22
Video Sub Categories: Technology
72.2
77.8
5.6
Video Sub Categories: Travel
75.6
77.8
2.2
Video Sub Categories: Variety Show
65.6
67.8
2.2
Table 16: Sub-category comparison with LLaVA-Video-11 on VideoMME.
8

---

Published as a conference paper at ICLR 2026
Narration
Nervation: food the pot with chapetioss
stritee into ceteeiern
spocn and ende
Action tabel optioss Insert
Acdion label options: mushroom E. put ran chopstics
Acdion lebel options:
spatuts
put corres maver putjeg
put acoon
out
Figure 5: Action labels vs. narrations. Blue option denotes ground truth and the pink option denotes LLa VActiionn--'' prediction. Action labels usually reduce multiple nouns into one noun, resulting in ambiguity that could mislead a MILM. Note that the narration also contains crucial particles with the phrasal verbs to clarify the meaning such as "put down", "put into".
Random
Avion
TIM
get jadio podto
pick podding equneas pagarr
Hurd cRE1
pas bove
caka pagoe picc dough
L.capappe
wesh
wan
with
poach
clive sconge
rins sporgs wethh
2omer idee lond
kanonn weter Hroo wwweihiing
open be open cen
bettt
Figure 6: TIM's choices are harder than AVION and random by introducing more visually similar objects and actions. Bold option denotes ground truth.
GPT-4o
L.aVActton 28
picking paza baking tray paper. pieas uniqus color with coppings like oliws mushrooms. I'm holding the sicee with both hands, canmining closely.
the video, interacting with
pizza that placed baking The hands
toppings with regetables. doing this over babing tray with parchment paper. the background, there arother place with more slices of pizza on
half
actions involves
the tray.
This
pomitty
manipulate titchen zutting presencx other pizza siics nearby mupports turning the pizza slice. liody to ensure even cooking or to reposition it for serving.
cornectly
Figure 7: The captioning capability can provide insights into models. A comparison among LL VLctionn,,-77, GPT-4o and VAV--Video- Video-7B.

---

Published as a conference paper at ICLR 2026
Verb Acturacyy
1.0
0.-
02-
Noun Accuracy with Counts (Part
1.0
0.2
0.0
Noun
Noun Accuracy with Counts (Part
0.0
Noun
Noun Nccuracy with
86
Accurary
Figure 8: Breakdown of the performance of LLaVAction-7B on Verbs and Nouns. We analyzed the accuracy per verb and noun in EPIC-KITCHENS. 100 for that our LLLVAccion-7B (32f), eval- uated on the validation set with action labels (i.e., the model reported in Table 4 that achieved 58.3 accuracy). There ization but otherwise are not separated in an intentional way. The number above each bar is the total per class.
nouns than verbs, thus nouns shown across four subplots for visual-
10


Reviews:
[{"review_id": "iSKQkZUCVO", "reviewer": "Reviewer_HUeB", "summary": "The paper introduces LLaVAction, a novel framework designed to evaluate and enhance the fine-grained action understanding capabilities of multi-modal large language models (MLLMs). The authors first reformulate the EPIC-KITCHENS-100 dataset into a video multiple-choice QA benchmark (EPIC-KITCHENS-100-MQA) using hard example mining with state-of-the-art specialist models such as TIM to generate challenging distractors. This benchmark exposes the current limitations of existing MLLMs (e.g., GPT-4o) in fine-grained action discrimination.\nTo improve MLLM performance, the authors curate a supervised fine-tuning dataset encompassing tasks like hard action recognition, temporal detection, captioning, and free-form QA. The proposed LLaVAction model introduces an action token to strengthen visual information utilization and a two-stage structured prediction pipeline for fair comparison with specialized models.\nEmpirical results show significant improvements—over 21-point accuracy gains on EPIC-KITCHENS-100-MQA compared to GPT-4o--and strong generalization across multiple benchmarks, suggesting that LLaVAction effectively enhances MLLMs’ action understanding.", "questions": "Could the authors discuss how LLaVAction’s two-stage pipeline scales to open-vocabulary or long-horizon tasks beyond EPIC-KITCHENS?\n\n\nAre there any failure cases where hard distractors cause misalignment between textual and visual cues?\n\n\nHow sensitive is LLaVAction to the choice of the base MLLM (e.g., Qwen2 backbone)?\n\n\nWould the authors consider releasing the adversarially generated distractor datasets for community benchmarking?\n\nIn related works, the authors should talk about how their method compares to other methods like InsTALL [Nguyen et al., 2025 arXiv]. \n\nHow do the results look with more traditional methods like ResNet3D, InceptionNet-3D (I3D), etc.?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Comprehensive evaluation across multiple datasets, benchmarks, and modalities.\n\n\nInnovative benchmark creation using model-based hard example mining—a principled and efficient alternative to human-generated distractors.\n\n\nClear empirical improvement over strong baselines (e.g., GPT-4o, LLaVA-Video).\n\n\nThoughtful model design, integrating both architectural and data-centric enhancements.\n\n\nStrong generalization across domains, including unseen datasets like EPFL-Smart-Kitchen-30 and Meccano.\n\n\nInterpretability analysis (Section 4.5) provides convincing evidence of improved visual-text alignment.", "weaknesses": "While comprehensive, the paper’s novelty leans more on system-level integration than on introducing fundamentally new model architectures.\n\n\nThe action token mechanism—though effective—could benefit from deeper theoretical or ablation-based exploration (e.g., why it captures action semantics more effectively than other token-aggregation methods).\n\n\nThe reformulation pipeline is heavily reliant on existing specialist models (e.g., TIM, AVION), which could limit generalizability to non-egocentric domains.\n\n\nThe benchmark’s dependence on EPIC-KITCHENS-100 might introduce dataset bias, and expanding evaluation to non-cooking activities would further strengthen claims of generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RBLh65vEWf", "reviewer": "Reviewer_quqa", "summary": "This paper tackles the challenge of action understanding in Mutli-modal Large Language Models (MLLMs). The paper address this challenge by introducing a new and challenging benchmark EPIC-KITCHENS-100-MQA, which a multiple-choice, fine-grained action-understanding benchmark by reformulating the EK100 datasets into a multiple-choice question-answering (MQA) task. More importantly, the paper makes its contribution by utilizing hard distractors, leveraging predictions from latest action recognition models to generate semantically similar and difficult distraction choices. The paper then proposes LLaVAction, which introduces an “action token” to enhance focus on visual information and a two-stage inference pipeline to produce structured action outputs. On EPIC-KITCHENS-100-MQA, LLaVAction outperforms baseline MLLMs and achieves SOTA results on several other action recognition and video understanding benchmarks.", "questions": "•\tFor the possible OOD problem, can you provide a clearer explanation or experiment to proof the current train/val setting remains OOD?\n\n•\tIn your claim, the GPT-4o reformulation alone downgrades the MQA performance but the two-stage pipeline itself relies on GPT-4o for caption during training process. Can you elaborate the balance between benefits and harms?\n\n•\tHow do results vary with K in both the miner and the two-stage pipeline? Is there a robust way to pick K beyond empirical tuning?\n\n•\tMinor notation issue: In Section 1.3, Formula 1 shows how the author formulate the MQA task. However, the author didn't definite p in the formula. Does it refer to the correct narration?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "•\tBenchmark Novelty: The method of using SOTA specialist models is not novel. However, the paper tackles the efficiency and effectiveness of MQA benchmarks by creating a challenging evaluation framework which successfully exposes the weaknesses of latest MLLMs model shown in Table.1.\n\n•\tStrong Performance: The proposed LLaVAction model demonstrates its significant performance gain over GPT-4o, LLAVA-OV, and LLAVA-Video on the new benchmark and also shows SOTA performance and strong generalization abilities on origin EK100 dataset and etc.\n\n•\tThe paper is well-written and easy to follow, the model itself is concrete and effective: 1. Using learnable action taken with independent heads and utilize 2. Two-stage structured prediction pipeline that narrow downs the candidates choice.", "weaknesses": "•\tUsing strong external models as support: The construction relies on specialized model and other action recognition model as strong support. It might create confusion that dependence on TIM and other action recognition model will effectively improve the result rather than improvement from the model itself.\n\n•\tPossible OOD problem:\nThe paper does partially address my concern that they use AVION as training support vs TIM as evaluation. However, the reported 65% top-1 agreement and 45% top-5 overlap between AVION and TIM’s results on EK-100 evaluation set. The out of distribution itself will not proofed by claiming of using two different mining models. The paper might need a more robust way of concluding the OOD Distractor by possibly including a domain test on EK-100.\n\n•\tNovelty for two-stage pipeline: The paper claims the two-stage pipeline predictor as one of their contributions.  While in Section 3.3, the paper demonstrates they use a specialist model to generate Top-K candidates, it acts like a classifier for post-processing the output.\n\n•\tSpecialized model confusion: In Section 4.3, the paper claims the task is zero-shot generalization. However, the specialized model is used to construct the candidate sets and also to ensure fair comparison against other model. It looks like a blur result rather than claiming as zero-shot since there is external help for inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CsfIsjNBwD", "reviewer": "Reviewer_V9cL", "summary": "This study focuses on improving the understanding of complex human actions by emerging multimodal large language models (MLLMs). The researchers reformulate the EPIC-KITCHENS-100 dataset into a new benchmark, EPIC-KITCHENS-100-MQA, to evaluate MLLMs' fine-grained action recognition. They find that leading MLLMs struggle with identifying correct actions, especially when presented with challenging distractors. To address this, they develop a supervised fine-tuning dataset encompassing tasks like hard action recognition, temporal detection, and free-form question answering. They also propose a new model, LLaVAction, which improves attention to visual tokens and utilizes a two-stage pipeline to better understand structured actions. The model significantly enhances performance, achieving notable accuracy gains on both MLLM and action recognition benchmarks, highlighting its potential for tackling complex human behavior tasks.", "questions": "Will the datasets, models, and related resources be made publicly available?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, presenting the methodology and experimental results in a clear and systematic manner.\n2. The research topic is highly practical and meaningful. Enhancing MLLMs' ability to better understand human actions could benefit a wide range of real-world applications.\n3. By approaching the problem from an action recognition perspective, the paper introduces a new, challenging benchmark. This dataset holds potential to contribute valuable insights to research within the relevant domain.\n4. The proposed LLaVAction model not only demonstrates strong performance in action recognition tasks but also achieves improvements across more generic benchmark datasets. This is a noteworthy phenomenon, showcasing the potential value and versatility of the approach to broader applications.", "weaknesses": "1. While the paper focuses on action understanding, its evaluation is limited to action recognition tasks. Comprehensive human action understanding involves other dimensions, such as temporal dynamics, the meaning or intent behind actions, and human interactions. Previous works like MotionLLM [1] and HAIC [2], which explore the use of MLLMs for human action understanding beyond just action recognition, have introduced broader benchmarks and methodologies covering these aspects. This type of works are highly relevant to this paper's domain and should be discussed as part of the related work. Furthermore, demonstrating LLaVAction's effectiveness on these more comprehensive action understanding benchmarks could strengthen the proposed method.\n2. Although the paper shows that LLaVAction improves performance on general video understanding benchmarks, a detailed analysis of the reasons behind these improvements is missing. Investigating which aspects of the proposed training methodology contribute to these gains would provide deeper insights into the strengths of the approach.\n\n[1] Chen, Ling-Hao, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. \"Motionllm: Understanding human behaviors from human motions and videos.\" arXiv preprint arXiv:2405.20340 (2024).\n\n[2] Wang, Xiao, Jingyun Hua, Weihong Lin, Yuanxing Zhang, Fuzheng Zhang, Jianlong Wu, Di Zhang, and Liqiang Nie. \"HAIC: Improving Human Action Understanding and Generation with Better Captions for Multi-modal Large Language Models.\" ACL 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0W8X8PsdpB", "reviewer": "Reviewer_BpZo", "summary": "The technical contributions, especially the comprehensive training data mixture, clearly demonstrate improved performance over baselines like GPT-4o. \n\nHowever, the effectiveness of the new benchmark is weakened by concerns regarding the **validity of its hard negative samples**, which undermines the strength of the evaluation methodology. \nFurthermore, the specialized Action Token unnecessarily introduce task-specific constraints that **limit the model’s generalization capability**. The two-stage structured action prediction pipeline is an **unnecessary design**.\n\nGiven that the benchmark validity is questionable and the proposed methods either lack necessity or generality, but recognizing the significant technical gains achieved, the paper lands at a borderline reject.", "questions": "The authors could discuss more about the Hard Example Mining strategy, since the other shortcomings are hopeless.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "LLaVAction integrates structured training tasks like temporal detection and temporal order learning to teach the model action boundaries and natural continuity. Such training data mixture strategy improves action understanding ability of MLLMs.", "weaknesses": "The Hard Example Mining strategy used for the new EPIC-KITCHENS-100-MQA benchmark is questionable due to the validity of hard negatives. \nThe generated \"hard distractors\"—which are chosen for being highly confusable errors—may be semantically correct actions due to potential annotation ambiguity/near synonyms, or missing labels. The EPIC-KITCHENS dataset has 4K action labels, which is hard to annotate exhaustively in the original dataset.\nAlthough the mechanism guarantees the exclusion of the exact ground truth class ID, it does not ensure the distractors are true semantic negatives.\n\n\n\nThe model’s Action Token design compromises MLLM generality by employing explicit classification heads to predict a finite set of nouns and verbs. While intended to enhance visual feature utilization, this hard-coded classification is specialized for the datasets with closed set labels, moving away from the flexible, open-vocabulary text generation paradigm characteristic of general MLLMs.\n\n\n\nThe proposed two-stage structured action prediction pipeline is unnecessary. \nThe authors contend that using this pipeline is essential because \"One could put all the possible actions in the question prompt and let MLLMs choose, but this will become impossible when the number of action types increases\" (e.g., 4k actions). However, this argument is questionable because simpler, direct classification methods exist that avoid context overflow, such as storing the video+prompt+question in the KV Cache and calculating the loss by appending individual class names to perform 4K classification natively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)