On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding
Authors: Zhanzhong Pang
Dibyadip Chatterjee
Fadime Sener
Angela Yao
Venue: ICLR 2026 Poster (2026)
Keywords: Temporal action understanding
multimodal large language models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal Large Language Models (MLLMs) have advanced open-world action understanding and can be adapted as generative classifiers for closed-set settings by autoregressively generating action labels as text. However, this approach is inefficient, and shared subwords across action labels introduce semantic overlap, leading to ambiguity in generation. In contrast, discriminative classifiers learn task-specific representations with clear decision boundaries, enabling efficient one-step classification without autoregressive decoding.
We first compare generative and discriminative classifiers with MLLMs for closed-set action understanding, revealing the superior accuracy and efficiency of the latter. To bridge the performance gap, we design strategies that elevate generative classifiers toward performance comparable with discriminative ones. Furthermore, we show that generative modeling can complement discriminative classifiers, leading to better performance while preserving efficiency.
To this end, we propose Generation-Assisted Discriminative (GAD) classifier for closed-set action understanding. GAD operates only during fine-tuning, preserving full compatibility with MLLM pretraining. Extensive experiments on temporal action understanding benchmarks demonstrate that GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-art results on four tasks across five datasets, including an average 2.5\% accuracy gain and 3$\times$ faster inference on our largest COIN benchmark.

Published as a conference paper at ICLR 2026
ON DISCRIMINATIVE VS. GENERATIVE CLASSIFIERS: RETHINKING MLLMS FOR ACTION UNDERSTANDING
Zhanzhong Pang', Dibyadip Chatterjee', Fadime Sener,
Angela Yao!
National University of Singapore (pang, dibyadip, ayao]0comp. nus. edu. sg,
famesener@gmail
ABSTRACT
Multimodal Large Language Models (MLLMs) have advanced open-world action understanding and can be adapted as generative classifiers for closed-set settings by autoregressively generating action labels as text. However, this approach is inefficient, and shared subwords across action labels introduce semantic overlap, leading to ambiguity in generation. In contrast, discriminative classifiers learn task- specific representations with clear decision boundaries, enabling efficient one-step classification without autoregressive decoding. We first compare generative and dis- criminative classifiers with MILMs for closed-set action understanding, revealing the superior accuracy and efficiency of the latter. To bridge the performance gap, we design strategies that elevate generative classifiers toward performance comparable with discriminative ones. Furthermore, we show that generative modeling can com- plement discriminative classifiers, leading to better performance while preserving efficiency. To this end, we propose inneerationAAsiislee Discriminative (GAD) clas- sifier I for closed-set action understanding. GAD operates only during fine-tuning, preserving full compatibility with MILM pretraining. Extensive experiments on temporal action understanding benchmarks demonstrate that GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-an results on four tasks across five datasets, including an average 2.5% accuracy gain and 3x faster inference on our largest COIN benchmark.
INTRODUCTION
Video understanding has traditionally focused on closed-set recognition and detection (Kay et al. 2017 Damen et al., 2022). Recent advances in Multimodal Large Language Models (MLLMs) (Li al. 2023 Maaz et al. 2023 Wang et al.) (024) have expanded the scope to open-world settings, enabling free-form language output via autoregressive (AR) token generation. This language-centric design enables MLLMs to solve diverse video tasks through text output, providing general and task-agnostic framework. This motivates examining their utility for conventional classification, where textual outputs enrich semantics while avoiding task-specific architectures.
To extend MLLMs to classification tasks in video understanding, prior works (Hu et al. 2023
He al., 2024 Chen et al.) 2024a Wu et al., 2024 Chatterree et al.) 2025) cast these tasks as generative problems, employing MLLMs as Generative Classifiers (Jaini et al. 2023). In this formulation, models are prompted with queries such "What is the action in the video?" and fine-tuned to autoregressively generate concise action labels (e.g.. "add onion") as free-form text. Yet the generative objective of MLLMs is not inherently tailored for classification, and the effectiveness of this reframing remains underexplored. In contrast, Discriminative Classifiers (Ng & Jordan 2001) align more naturally with classification, learning task-specifh representations and predicting actions directly, thus avoiding the additional step required to map free-form text back to predefined classes (Chen et al. 2024a). Exploring discriminative learning with MLLMs for video understanding therefore remains promising yet under-investigated direction.
In this paper, we demonstrate the advantages of discriminative classifiers over generative ones for temporal action understanding. We adapt pre-trained MLLMs into discriminative classifiers by appending a learnalle token to the visual input, enabling the model to
encode a global representation
CCode: httpp:s:/ggttubb.cm/////////////

---

Published as a conference paper at ICLR 2026
add
add coffee
add fish
add onion
add rice
add spices
add strawberrier cake
add sugar
add whipped cream
Generative classifier
(b) Discriminative classifier
Figure T-SNE plot comparing feature spaces of generative and discriminative classifiers on CrossTask dataset for actions sharing the verb 'add'. Generative feature mean of output token features; Discriminative feature the learnable token feature.
for classification. Our discriminative approach surpasses generative classifiers (Lin & Shou 2025 Chen et al. 2024a) in both accuracy and efficiency. By directly optimizing decision boundaries (Fig. D. the discriminative formulation reduces action confusion, leading to consistent performance gains. Furthermore, it predicts an action (e.g. "add strawberries cake") in a single forward pass, compared to the multiple passes required by generative classifiers to autoregressively output tokens (e.g. {'add', strawberries', to',"" cakee)), })
The superior performance of the discriminative classifier motivates an investigation into why it outperforms the generative one. Generative classifiers are more prone to confuse semantically similar actions, such as "add onion" and "add rice". This stems from the fact that action labels are intentionally annotated be concise and high-level rather than descriptive (Zhou et al. 2024 Kay et 2017)) which produces substantial semantic overlap across classes, e.g. the frequent use of verbs like "add" and "put". As a result, beyond the inherent visual ambiguities in video input, semantic overlap in the output space poses an additional challenge for generative classifiers. In contrast, the discriminative formulation ignores label semantics Ye & Guo 2017). eliminating semantic overlap and achieving clearer separation between actions. We further show that the generative and discriminative classifiers can become functionally equivalent when the action labels are introduced into the tokenizer's vocabulary singular tokens, and decoded in one autoregressive step. Adding single tokens prevent action labels being tokenized into subwords shared across actions, thus removing semantic overlap. This finding highlights the potential of extending MLLMs as generative classifiers to learn task-specific representations for discriminative learning.
However, the specifc discriminative classifiers lose the semantic richness conveyed by generated text. This motivates us to incorporate generative modeling to complement and thus enhance discrimi- native learning. To this end, we propose Generation-Assinn temporal action understanding, which integrates discriminative objectives with auxiliary generative objectives within a single end-to-end framework. This design maintains the strengths and efficiency of discriminative learning while incorporating additional semanties and contextual information through generative modeling, enabling context-ane aware and semantically enriched representations.
Discriminative (GAD) classifier for
We investigate broad spectrum of temporal action understanding tasks, spanning basic step and task recognition and step forecasting (Tang et al. 2019) to the more challenging setting of online action detection (Zhukov et al., 2019 Damen al. 2022 Song et al., (023). Our experiments demonstrate that the discriminative classifier achieves higher accuracy and lower inference latency than the generative one. Within the proposed GAD framework, generative modeling further strengthens classification by regularizing training with semantic encoding and contextual enrichment, while inference relies solely on classification, preserving the efficiency of discriminative learning. Notably GAD achieves tfte-of--the-rr across on four tasks across five datasets, including an average 6.8% F1 gain and 1.8x speedup on EPIC-Kitchens- 100, 1.5% Fl gain and 3x speedup on Ego4D GoalStep. and 2.5% Top-1 accuracy gain with 3x speedup on COIN.
Overall, our contributions are summarized as:
We demonstrate that generative classifiers underperform discriminative ones on classification tasks, primarily due to semantic overlap in the generative (textual) output space.
2

---

Published as a conference paper at ICLR 2026
We align generative and discriminative classifiers by interpreting classification single-step generation process, where predefined action labels are introduced as new tokens.
We propose a generation-assisted discriminative (GAD) framework, showing that auxiliary generative objectives enrich discriminative learning and improve temporal action understanding.
RELATED WORKS
Temporal action understanding involves recognition tasks such as temporal action detection and segmentation (Zhao et al., 2017 Ding et al.] 2023), step recognition and forecasting (Tang et al. 2019), and video recognition (Zhang et al. 2021D. Classical approaches model temporal dynamics using temporal convolutional networks ( (Farha & Gall 2019), recurrent neural networks (Xu et al..
2019), or transformers (Liu et al.] 2022). Recently, MLLMs have shown strong performance by formulating recognition as an autoregressive token generation task, where labels are expressed as free-form text and decomposed into tokens (Hu et al. 2023 He et al. 2024). Chen et al. (2024a); Wu al. 2024); Chatterjee al. (2025) extended MLLMs real-time interaction, temporal summariza- tion, and forecasting, while Ye et al. (2025) applied QA-style generation approach for video action recognition. However, autoregressive token generation inefficient, and its effectiveness for recogni- tion tasks remains underexplored. Our work demonstrates that generative learning underperforms its discriminative counterparts due to the added semantic complexity in text output.
Customized tokenization may use task-specific or optimized strategies (Liu et al. 2024 Zhang
ctt al., 2025) to reduce token length for faster decoding and improved semantic representation. Using Ego4D videos (Grauman et al. 2022). Lin & Shou (2025) constructed a hierarchical vocabulary of video narrations to enable faster inference. In image retrieval, Caron et al. (2024); Zhang et al. (2024) designed language- based discriminative entity codes, converting images into compact and semantically rich tokens serving as identifiers for efficient and accurate retrieval. For multiple-choice question answering, mapping each answer option to single symbol effectively tokenizes answers into one token, enabling faster inference (Joshua Robinson, 2023 Ranasinghe et al. (024). However, in action understanding, fine-grainet actions that share verbs and objects make distinctive tokenization difficult, and preserving full semantics is harmful. Similar observations have been made in closed-set image classification (Cooper et al., 2025; Conti et al., 2025), where distinguishing fine-grained categories remains challenging for LLMs. Specialized prompting has been proposed to improve differentiation, but the approach remains generative. In contrast, we adopt a classification-style approach, equivalent to encoding each action unique, unstructured atomic code.
Unified retrieval and generation within a single model represents a key step toward building general- purpose multi-task systems. Earlier vision-language foundation models, Yuu et al. 2022 Li et al. 2022 Chen et al. 2024b), constructed hybrid frameworks that combine a vision-text encoder and a text decoder, pretrained jointly with contrastive and language modeling objectives. Large language models (LLMs) (Koh et al.) 2023 Ma et al. 2024) have also shown potential as unified backbones when trained jointly with both losses, which can be further enhanced with task-specific instruction tuning (Muennighoff et al. 2024). Our work investigates MLLMs for classification tasks, aligning discriminative and generative learning within the same task, in contrast to existing approaches that address them separately for different tasks.
METHOD
We present the Generation-Assisted Discriminative (GAD) classifier, which leverages pre-trained LLMs for fine-tuning on downstream temporal action understanding tasks. Our focus is on the fine-tuning stage because general MLLMs typically struggle with task-specific requirements and must be further adapted for optimal performance We first formalize the problem of temporal action understanding, then introduce and analyze the generative classifier baseline and its challenges, before motivating our proposed solutions.
3.1 PRELIMINARIES
We focus on fine-tuning pre-trained L.LMs for temporal action understanding in a closed-set classifi- cation setting. Given a video sequence V, which may be short clip or an entire video sampled at a
2Qwen2.5- VL-7BB only achieves 16.1 16.1% and 8.9% zero-shot accuracy on COIN step and next-action prediction, respectively, compared to 67.3 % and 51. 51.6% after fine-tuning. Find more results in Appendix C
3

---

Published as a conference paper at ICLR 2026
Text token
[CLB) Leuraalle toket
presious action
*ageadi "rig''
75p Ineedd
brazd
aada"
Frand
eac-but
LM Head
Head
Cla Head
LM Head
Large Language Model
Large Language Model
Large Language Model
angrage
Vision Language
Viskond anguuge
Adaaacc
Vision Encoder
Vision
Encoder
+ Video
Task query
Task query
Video
[CLS]
Task query
[CLS]
(a) Generative classifier
(b) Discriminative classifier
(c) GAD
Figure Comparison between different architectures for downstream video-related recognition tasks: (1) Generative classifier: treating action labels as free text. (2) Discriminative classifier: learning an extra representation for downstream tasks. (3) Generation-assisted discriminative (GAD) classifier: learning an extra representation that is regularized through task-related generation.
predefined rate, and task query Q, the objective is to predict a label y from a predefined category set Y. This problem formulation is general, accommodating wide range of recogmition scenarios. Action labels. Temporal action understanding tasks typically involve actions that unfold over time and exhibit temporal correlations. These actions are generally expressed as concise and simplified verb- noun phrases (e.g. "add sugar"), and sometimes enriched with additional contextual elements (e.g. "add strawberries to cake"), to capture both the action and objects involved. The action vocabulary is typically closed, consisting of recurring motions ("add", "screw") applicd to a shared set of objects, which naturally leads to semantic overlap across labels.
3.2 ARCHITECTURE
We adopt LLaVA-style (Liu et al. 2023) architecture as the backbone, comprising visual encoder E.y. language decoder D., and a vision-lane langugge adaptor Avt. Video frames are first encoded by the vision encoder E,,, then mapped to text-aligned tokens via the vision-language adaptor Avt. yielding visual tokens F, = Avt (E, (V)). Meanwhile, the task query Q is tokenized into text tokens F., enabling joint processing with the visual tokens by the language decoder. During fine-tuning, only the Assion-lanuuaee adaptor and the language decoder are trained using LoRA, while all other components remain frozen.
Generative Classifier Baseline, a generative classifier, text and visual tokens are concatenated and fed into the causal language decoder to autoregressively generate subword tokens using language modeling head. During training, the generative classifier is provided with a target action label y. which is tokenized by the language model's tokenizer into sequence of subwords { (uo,"1;"" "1. un--i) Here, n represents the total number of subword tokens representing and these subwords are drawn from the tokenizer's vocabulary. For example, the action label "take pancake from pan" is tokenized as 'take'. 'panc', 'ake'. 'from'. 'pan") }} by the Llama-3 tokenizer (Grattafiori et al. 2024). The model is trained to maximize the conditional probability of each target subword u, over the vocabulary, conditioned on the visual and textual inputs and previously generated subwords uzi. The training done using the standard language modeling loss Lgens iee. the negative log-likelihood, applied to each subword token.
(Q, V, u.)) D. @ uui), Cym --m=- Pr (u 1 V, Uei, w)
(1)
where 9. is the generative classifier parameterized by w, G denotes concatenation. When applying the generative classifier for closed-set classification, the decoder is finetuned to generate subword tokens from a fixed vocabulary subset that covers the subwords required to represent all action labels.
Discriminative Classifier. Here, we extend the generative model to discriminative model for downstream classification tasks. Instead of directly classifying video features using specialized models, repurpose the existing MLLM task sssk-specific architectures. A learnable [CLS] token (Ye et al.) [025 Lin & Shou (025) is appended to the end of the language model input sequence, attending all preceding tokens and generating
task-agnostic classifier, eliminating the need for
4

---

Published as a conference paper at ICLR 2026
representations that integrates both the video input and the task query. We find that using learnable token enhances generalization, whereas relying on visual tokens directly can lead to overfitting (see Appendix [B.4). The resulting output representation is fed to customized classification heads for the downstream classification tasks by optimizing the traditional cross-entropy loss. Note that the language modeling head is disabled in this case, since generating subwords is unnecessary.
f2 V, [CLS]) Dr @ Fy @ [CLS]), Cele log Pr(y 0 d'),
(2)
where fa denotes the discriminative classifier parameterized by o, and ø' represents the parameters of the classification head.
This discriminative formulation supports faster inference in low-latency scenarios. It also prevents action labels from being tokenized into subwords, avoiding semantic overlap from shared subwords across labels. In Sec. we show that such overlap causes discriminative and generative classifiers to behave differently. with the generative one more prone to confusing verbally similar actions. Removing this overlap through specific designs, such as isolated tokenization, reduce these confusions among similar actions. In fact, when action labels are excluded from the text query and only used output target, the discriminative classifier becomes special case of generative classifier by 1) adding action labels as new vocabulary entries. 2) merging the classification head into the language modeling head. Since these added entries are not used input, their embeddings can be initialized randomly without affecting performance, and classification is equivalent to generating the entire action label in one single step. This equivalence demonstrates the potential of extending existing MILMMs to learn task-specific representations for discriminative learning.
Generation-Assiseed Discriminative Classifier. Generative modeling still remains valuable by providing deeper semantic cues through text generation, effectively encoding label semantics and context. To combine these benefits with the discriminative classifier, we propose the Generation- Assisted Discriminative (GAD) classifier, unified framework that augments the the discriminative model by adding the language modeling head, to produce auxiliary generative outputs.
We investigate three strategies for unification: (I) sequential learning with discriminative learning first, followed by generation conditioned on the learned representation. (2) sequential learning with generation first, followed by representation learning conditioned the generated output. (3) parallel learning, where both are learned simultaneously using shared text and visual input. We focus on the first strategy here it proves most effective, leaving details of other strategies to Appendix [B.2] Specifically, for (1), discriminative learning follows the standard discriminative classifier setup, after which generative modeling is applied conditioned the video V. the task query Q, and the learnable token [CLS]. The generation loss in Eq. iis then adapted as
Σ log (r uci Q, V, [CLS], 0),
gen
where 0 denotes the learnable parameters in this model, and u, represents the ith tokenized subword corresponding to the generation target. The overall training loss combines the adapted generation loss and the classification loss from Eq. 2 weighted by a balance factor X, as LEAD Lats XL
We use generative modeling as auxiliary task to regularize the representation learning in the discriminative classifier. For instance, it can implicitly capture intentions or reason about past and future steps, thereby supporting recognition of the current action in goal-oriented procedural videos. While this formulation could also treat discriminative and generative learning as separate, independent tasks, prior work shows that simply unifying their objectives in single model offers limited benefit (Ma et al., 2024; Muennighoff et al. (0244). In contrast, our auxiliary-task setting leverages generative modeling specifically to enhance discriminative learning.
3.3 MODEL TRAINING & INFERENCE
During training, we leverage pre-trained vision encoders and LLMs and perform instruction tuning with task-specific queries. The vision encoder is frozen; the vision-language adapter is fine-tuned, and the L.LM is updated via LoRA (Hu et al., 2022). We optimize both discriminative and generative objectives across all training instances, with the latter serving as auxiliary supervision to enrich representation learning. At inference, we disable the generative branch and use only the discriminative classifier to produce final predictions.
5

---

Published as a conference paper at ICLR 2026
Table Generative (Gen) vs. discriminative (Disc) classifier on OAD tasks. The runtime analysis performed on single NVIDIA RTX A5000 GPU.
THUMOS 14
CrossTask
EPIC-Kitchens-
100
Ego4DGoalStep
LLM
Model
S-F1 /P-FI
FPS
S-FI / P-F1
FPS
S-F1 / /P-F
FPS
S-F1 /P-F1
FPS
Llama3.2-
Gen
56.97 38.8
38.3
46.8 31.7
44.0
16.7 13.9
28.8
8,973.4
17.8
IB- Instruct
Disc
57.8 / 40.1
58.0
48.8 / 34.0
59.4
23.2 / 19.3
51.1
10.6/4.1
53.6
Qwen2.55
Gen
38.9
30.2
44.0 28.6
36.3
16.2
13.4
26.6
8.8/3.
12.3
0.5B Instruct
Disc
57.3/39.4
48.8
45.3 / 29.6
52.8
22.0/ 18.3
48.6
/
52.1
4 EXPERIMENTS
Our experiments aim to answer the following research questions for discriminative and generative classifiers: (I) How do they compare in performance? (2) What factors influence their performance? (3) When and how can a generative classifier enhance a discriminative one?
4.1 DATASETS, EVALUATION, AND IMPLEMENTATION DETAILS
Datasets & Tasks: We evaluate on four temporal action understanding tasks, including step recogni- tion, step forecasting, task (activity) recognition, and the more challenging setting of online action detection across five datasets. Step recognition identifies the occurred action in given video clip, while Step forecasting anticipates the upcoming action in the clip. The COIN dataset (Tang et al.) 2019) is adopted for these tasks, as it contains fine-grained, goal-orienter action steps. Task Recog- nition aims to detect the overall activity category of a given video with multiple steps. COIN also supports this task, providing high-level labels that reflect the hierarchical structure of such activities. Online Action Detection (OAD) focuses on recognizing actions in a streaming video using only past observations. Wideely used OAD datasets, including THUMOS 14 (Idrees et al., (017), EPIC- Kitchens- 100 (Damen et al.] 2022), and CrossTask (Zhukov et al., 2019), cover both sport videos with loosely related actions as well as procedural videos with more correlated and fine-grained actions. We also include Ego4D GoalStep (Song et al. 2023), which features more descriptive labels than the concise ones in the other datasets, allowing to assess the impact of label complexity on model performance.
Evaluation. To evaluate OAD tasks, following current SoTA (Pang al. 2025), F1 score with an IoU threshold of 0.1 (S-F1) and point-wise F1 score for action start detection with a Is threshold (P-F1). For step recognitionnfoorcaastnn and task recognition, report top-1 accuracy following existing literature (Chen et al. 2024a Wu et al. 2024).
segment- wise
Implementation. We use Llama3 (Dubey al. 2024) variants as the primary language decoder with
a 2-layer MLP adapter, and additionally evaluate Qwen2.5 (Qwen et al. 2025) variants. Task-specific visual encoders are employed following prior works (Wang et al. 2023
Pang et al. 2025 Chen et al., 2024a Wu et al. 2024). For OAD tasks, we use RGB features at or 4 FPS, with global token per frame. Since OAD requires untrimmed videos, video frames are sampled backwards from the current timestamp within fixed window to capture both short- and longer-term context (Xu et al. 2021). For step or task recognition, we follow Chen et al. (2024a) and use SigL.I--ViT: ViT-L-3844 (Zhai al., (023) with 2 FPS sampled videos, producing 10 tokens per frame (1 global and 9 patch tokens). Trimmed or full videos are used directly, with downsampling applied only when the sequence exceeds the predefined maximum length. The LLMs are finetuned using LoRA with balance factor is set to outputs by using Levenshtein edit distance to match the generated text to the closed-set action labels. Additional details are provided in Appendix B.1
128 and 256. The by default. In line with Chen et al. (2024a), we post-process generative
4.2 MAIN RESULTS
Experiments designed to systematically investigate the raised research questions.
Generative versus Discriminative Classifier: Tadles[[and and discriminative classifiers. The discriminative classifier (Disc) consistently outperforms the generative one (Gen) across datasets and LLM variants. It achieves notable improvements for OAD tasks, especially 3,600) increases semantic overlap, yielding gains of 6%, and 5% in segment-, and point-wise metrics. The discriminative classifier also surpasses the generative classifier by an average 6%, 3.5%, and 3%
the performance of generative
PPTC-Kitchens-100 100, where the large number of fine-grained actions (around

---

Published as a conference paper at ICLR 2026
Table 2: Generative (Gen) vs. discrim- inative (Disc) classifier on COIN for step/task and next step prediction.
Figure 3: False positives for "add sugar". Generative classi- fier incurs more diverse misclassifications.
Gen. Model
LLM
Model
COIN Benchmark
Falso Posiiives
Emb Model
Step
Next
Task
Llama3.2-
Gen
57.5
45.8
90.9
IB- IB-Instruct
Disc
64.1
50.1
92.8
Llama3-
Gen
61.3
48.3
92.3
whick pasturs pour lemon luece addl four pour viorgur pour alosked
pogr my It
your lemerpdd spreadd adl happeea cream
cafe
sti Turr
mnde lato glass
progon
caku
prur add arices
pourr jelni poorder
salt
slld must
tato beside
8B-Instruct
Disc
66.4
51.0
94.3
put mnixtore piz
lold straws
on step recognition, forecasting, and task recognition. Remarkably, the IB-version discriminative model even outperforms the 8B-version generative model.
We attribute these performance improvements to the discriminative classifier's ability to disregard output semantics. For example, on CrossTask, 3 shows that, due to the shared verb 'add', the generative classifier produces more false positives (after mapping the generative output to action categories), such as labeling "add sugar" as "add meat" or "add spices". These diverse errors not observed in the discriminative classifier's predictions. Appendix E for more analysis.) Following [Sikar et al.] (2024), we further introduce an entropy-based -barop diversity score to measure the spread of misclassification, with higher values indicating more diverse errors. The generative classifier scores 0.76, 1.3 and 1.8 on Cross compared to 0.66, 0.79 and 1.5 for the discriminative classifier, reflecting the diverse false predictions caused by the introduced semantics, see Appendix B.3
SPPIC-Kichenn-1000 and Ego4DGoalStep. respectively,
Discriminative classifiers also offer faster inference compared to generative ones, as it predicts outputs in single step rather than generating tokens autoregressively. As shown in Tab. the discriminative classifier achieves speedups proportional to the token count in action labels. On Ego4DGoalStep. the discriminative classifier is nearly 4x faster than generative, where the average action label length is 5.5 tokens, compared to 2.1 in THUMOS, 2.6 in CrossTask, and 2.4 in EK EK100, using the LLaMA3 tokenizer. Although the reported FPS excludes vision encoder computation by using pre-extracted features, the encoder itself runs at a comparable frame rate (Pang et al. 2025), making this speedup a meaningful improvement for real-time applications. Similarly, the discriminative classifier accelerates training by avoiding forward and backward passes through many tokens, achieving approximately 1.8x faster training on OAD tasks.
Bridging the Performance Gap. The observed performance gap between the discriminative and generative classifier motivates us to investigate its underlying cause. We observed that this gap should stem from the shared semantics in the generative output space. To verify this, we design three experimental settings in Fig. 4 that progressively disrupt these semantics, aiming to bridge the performance gap between the generative and discriminative classifier.
Randomized Consistent Mapping. Action labels are split into subwords by the tokenizer. Each subword is replaced with random token, but identical subwords are mapped to the same token across all labels.
Desynchromized Independent Mapping: Each subword is replaced independently with a random token, so the same subword will map to different tokens in different labels.
Extended Vocabulary: Add new action labels to the tokenizer vocabulary and initialize their embeddings by randomly selecting from the original vocabulary. Each action is thus represented by a single newly added token.
The corresponding results of these settings are presented in Tab. 3 highlighting several key observa- tions. First, randomizing tokens removes subword semantics and its relationships with neighbors, yet results remain unchanged, indicating that subword semantics and local word connections are not so critical. This is expected, as fine-tuning on predefined action labels encourages memoriza- tion. Additionally, action labels are concise and expressed in simplified style rather than natural language, making word transitions poorly captured and less important. By further desynchronizing removed, leading to significant
tokens to eliminate overlap among action labels, shared semantics performance gains that approach the discriminative classifier. This result supports our hypothesis that the shared semantics in the generative output space are the key cause of the performance gap.
7

---

Published as a conference paper at ICLR 2026
Table 3:
Bridging generative and discriminative classifiers using different tokenization with 1laam3322-1 1-1 I-Inttucct.
Figure 4: Tokenization strategies. ar and B denote random tokens.
Actions
tokenize
randomize
OAD Benchmark (segment-F11 /point-F))
Model
CrossTask EPIC-Kitchens-Ine 100 Ego4DGoalStep
add sugar
[add' 'uugur
[ai, cte]
add flour
[add', flour
[0r, Cts]
Gen
46.8 / 31.7
T6.7/ 13,9
8.9/3.4
[outt onion ]
[a4, as
Gen rand
46,9 / 31.8
16.8 14.0
8.5
Gen desync
48.7 / 34.1
23.0 19.0
1..4
extend
desynchronize
Gen_extend
48.9 / 33.9
23.3/ 19.2
10.5
['add sugar"] I'add flour']
[ex, ag)
Disc
48.8 34.0
23.2/ 19.3
[1s Sts]
['eut onion']
[04 (x))
Similarly, extending the tokenizer with action labels ensures non-overiapping tokens and achieves comparable performance. Unlike desynchronized mapping, each action is represented by a single token, allowing prediction in a single pass without token-by-token generation. This could help explain the discriminative classifier as a single-step generation process and align discriminative and generative approaches.
Inmeration-Assisterr Discriminative Classifier. While the discriminative classifier are more effective in classification settings, the generative modeling enables free-form generation, producing flexible outputs to support representation learning for classification. We investigate several scenarios to explore how generative capabilities enhance classifcation performance using the proposed Generation-Asssited Discriminative Classifier.
Labeling. The generative head produces the target action label as text, performing the same task as the discriminative classifier. is expected function as regularizer, incorporating the label lexical semantics that the discriminative classifier alone does not capture.
Context generation. The generative classifier generates additional outputs beyond the target labels, such as related past or future actions or the overall video goal. This process is expected to enhance representation learning by providing contextual and temporal cues.
To incorporate label semantics, we test two training strategies for labeling: two-stage and joint training. In two-stage training, the model is first trained using only the generative objective. All parameters then frozen to preserve the learned semantics, except the [CLS] token and classification head, which are trained in the subsequent stage. As shown in Tab, 4 the two-stage training results in significantly lower performance, suggesting that the generative classifier alone does not provide strong representations. On the other hand, joint training enables representation learning alongside generative responses, but with only slightly performance improvement, further suggesting that lexical semantics have limited impact in a closed vocabuulyyy setting.
Table 4: Comparison with GAD variants with Llama3.2 Instruct.
OAD Benchmark (segment-F1 point-F1)
COIN (Top-1 acc.)
Model
CrossTask EPiC-Kitchens-100 Ego4DGoalStep
Step
Next
Task
Disc
48.87 34.0
23.2 19.3
10.6/4.1
64.1
50.1
92.8
GAD (label 2stage)
41.8 / 26.7
9.3 / 6.7
5.9 / 1.7
35,9
28.0
85.5
GAD (label joint)
49.1 / 33.9
23.6/ 19.6
10.8 /4.1
64.4
50.4
93.2
GAD (context)
50.3 / 34.5
24.1 20.1
11.0/4.3
65.3
51.4
93.5
For context generation, the generation outputs the target label along with auxiliary information. In OAD, where only action labels are available, we explore action relationships as a form of context, specificallyy by generating the previous action the auxiliary output to support current predictions. This proves more effective than generating future actions, as the previous action is further supported by visual information from the input video, which contains only past observations in the online setting. More ablations can be found in Appendix [B.]] In COIN, where task information is available, we explore the role of task information as auxiliary outputs. Here, no extra information is used, since task label are already included during training. where task recognition and step recognition/forecating are jointly learned within single model. Results Tab. 4 show that both the task and previous action information help, with the task cues having stronger impact. Using the previous action as auxiliary output brings more benefit to CrossTask. where actions are closely aligned with a single goal, unlike other datasets where actions follow multiple, loosely related tasks. These results suggest that the generative modeling would more helpful when it generates information complementary to the discriminative classifier.
8

---

Published as a conference paper at ICLR 2026
For efficiency, while incorporating generative objectives slows down training similarly to the gen- erative classifier, inference speed matches that of the discriminative classifier by removing token generation, ensuring efficient inference.
4.3 ABLATION STUDIES
We conduct ablation studies on key design choices of our proposed GAD, evaluated OAD tasks with results reported in Tab More ablations are included in Appendix [.4
The learnable token. We adopt a learnable [CLS] token for discriminative prediction. An alternative is the last visual token, which also aggregates all preceding visual and text information. Results (w/o [CLS]) show that using [CLS] token performs better, likely due to its stronger generalization ability, while the last visual token tends to overfit the training data.
Auxiliary task. To confirm the benefits of capturing semanties via generation, we convert previous-action prediction into standalone classification objective, adding an extra classifier that uses either the existing learnable token (GAD_ _rpereee__eee _dicc) or an additional learnable token (GAD-prev_disee Results indicate that neither variant improves learning, and both can harm current-action accuracy, indicating that the advantage arises from generative semantic encoding rather than the auxiliary classification task itself.
Table 5: Ablation studies of our GAD classifier on OAD tasks.
OAD Benchmark. (frame acc//segment F1/point FI)
Model
CrossTask
EPIC-Kitchens-101
Ego4DGoalStep
Disc
11.7/48.8/344.
34.8/23.2/19.3
33.9/10.6/4.1
GAD
81.8/50.3/34.5
35.1/24.1/20.1
34.4/11.0/4
w/o [CLS]
81.5/49.2/33.8
34.6/20.7/17.7
33.7/8.6/3.6
GAD .pr_ppeevvue piic
80.8/48.2/31.6
34.8/23.6/19.3
33.2/10.1/3.8
GAD prev disc+
80.9/49.5/32.7
34.7/23.0/19.6
33.3/10.6/4.0
4.4 COMPARISON WITH SOTA.
We benchmark our approach against state-of- sate--ofttee of-the-art methods in Tables 6 and 7 Although not tailored any specific task, our approach still achieves state-of-the-an performance. Further gains can be expected through task-specific adaptations. To ensure fair comparison, we use the same visual features as state-of-the-ae methods, showing that our gains arise from the architectural design rather than from incorporating advanced features. For online action detection, present the first LLM-based method, significantly improving segment- and point-wise performance. For step/task recognition and step forecasting, approach also largely outperforms recent methods. Notably, our IB model even surpasses prior 8B models, highlighting the effectiveness of the discriminative classifier augmented with generative assistance.
4.5 QUALITATIVE RESULTS.
Fig- presents qualitative results the COIN benchmark. The generative classifier (Gen) likely generates semantically similar but incorrect outputs, while the discriminative classifier (Disc) demon- strate better discrimination. Our GAD model enhances Disc by generating contextual information that enables task-aware predictions. In the middle example, Disc outputs task-irrelevant action, while GAD accurately identifies the correct action. We also observe some edge cases where the
Table Comparison with SOTA on OAD tasks.
OAD Benchmark (segment-FF point-Fl)
Model
THUMOS'14
CrossTask HPIC-Kitchens- Ego4DGnalSttep
Testra (/hap & Krähenbühl 2022
43.0732.0
48.4733.8
16.5/14.7
8.773.5
MAT (Wang el al. 2023
49.4/34.2
49.7 / 34,2
17.5/
9,5/3.8
CMeRT (Pang el al. 2025
48.9 /4.66
50.1 / 34.3
17.7 / 15.8
9.7/3.9
Dise (Llama3.2-1m IB- Instrcct)
57.87 40.1
48.8 / 34.0
23.27 19.3
10.6/4.1
GAD (Llama3.2 2- Instruct)
58.1 40..
50.3 34.5
24.1 /0.1
11.0/4.

---

Published as a conference paper at ICLR 2026
Table Comparison with SOTA on COIN Benchmark.
COIN Benchmark (Top- acc.)
Model
Step
Next
Task
Video TaleolaskGraph (Ashutosh al. 2023
57.2
40.2
90.5
Vidcollm--nninn-88 (Chen al. 2024a
63.1
49.1
92.7
Videollm-Moo-SSB Wu et al.
2024
63.4
49.7
92.8
StreamMind-88 (Dng et al.
2025
63.7
49.9
93.2
Disc (Llama3.2-1 2-1. Instruct)
64.1
50.1
92.8
GAD (Llama3,2 3.111B 1at.:/..-------tteu--- IIstruct)
65.3
51.4
93.5
Dise Ba-mma-----nnstruee
66.4
51.0
94.3
GAD (Llama3-8B-Instruc))
67.3
51.6
94.5
model predicts correctly despite flawed generations. Commonly, the generation identifies the task but answers the query incorrectly, or is completely wrong misformatted. These edges cases highlight the benefit of performing generation after the learnable token. The learned representation remains robust to generative errors while still benefiting from generative regularization. More analysis is provided in Appendix and B.5
What is the action in the video?
What is the action in the video?
What the action the video?
put bread
pump up fael install
bisc
soap
vessel
"he the mutted roop Alvel toa fhr weurt Taok valll zwgs."
"ooual de lagle swwllooantnnipggperr Tol wyf.uce Bull."
Weyer Tist sonsfessk
Figure 5: Qualitative results the step recognition for Generative (Gen) discriminative (Disc) classifier. Red text denotes false predictions and green correct predictions. Callouts display GAD generation outputs.
4.6 GENERALIZATION TRADE-OFF
MLLMs are naturally strong in open-world scenarios due to their broad, data-driven pretraining, but perform poorly in closed-set action understanding (see zero-shot performance in Appendix [). Our classification setting involves task-specific fine-tuning, which boosts action-elassification performance but reduces general MLLM capabilities, reflecting fask-induced forgetting well-documented in the literature (Oh et al. 2024 Han et al.] (2024). We provide extra experiments to evaluate generalization and show this trade-off in Appendix D fact, the generative head in our GAD model enables lever- aging self-curated instruction-tuni -ttiing data to preserve generalization, e.g. augmenting action labels. In addition, since the backbone architecture remains unchanged, removing the introduced LoRA adapters can fully restore the model's open-world performance. We leave more comprehensive exploration of the trade-off between fine-tuning and generalization to future work.
CONCLUSION
This paper studies fine-tuning multimodal large language models for downstream temporal action understanding. We identify the fundamental limitations of generative modeling in classification tasks, showing how semantic label overlap constrains performance, and demonstrate the advantages of discrininative formulation. Building on these insights, we introduce a unified generative-assister discriminative (GAD) classifier that leverages generative modeling as an auxiliary objective to enhance discriminative learning. Crucially, our approach preserves full compatibility with pretrained models, requiring no changes to the pretraining process.
Limitations. While our generative-assistt discriminative classifier performs strongly in closed-set scenarios, discriminative models are still limited to the closed set and cannot directly handle novel or unseen actions. Additionally, spec-fc fine-tuning hurt general-purpott abilities like question answering. Future work could focus on improving generalization to new classes through generative components, while preserving the strengths of discriminative learning for the closed set.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2025).
ETHICS STATEMENT
The methods, data, and results presented in this paper raise no known ethical concerns. All ex- periments were conducted using publicly available datasets under their respective licenses. This paper also did not involve ethically sensitive activities, such as research with human subjects, dataset releases, potentially harmful experiments, issues related to discrimination & privacy.
REPRODUCIBILITY STATEMENT
All technical details, such as experiment settings, evaluation protocols and implement instructions detailed described in Sec.4. 1.1 of the paper and Sec.B. B.1 of the appendix to ensure reproducibility. All datasets and code used in this paper are publicly accessible.
REFERENCES
Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras, and Kristen Grauman. Video-mined task graphs for keystep recognition in instructional videos. Advances in Neural Information Processing Systems, 36:67733-67844 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint 1392 2025.
Mathilde Caron, Ahmet Iscen, Alireza Fathi, and Cordelia Schmid. A generative approach for wikipedia-scale visual entity recognition. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 17313-17322 2024.
Dibyadip Chatterjee, Edoardo Remelli, Yale Song, Bugra Tekin, Abhay Mittal, Bharat Bhatnagar, Necati Cihan CamgÅkz, Shreyas Hampali, Eric Sauser, Shugao Ma, et al. Memory-efficien stream- ing videollms for real-time procedural video understanding. arXiv preprint arXiv:2504.1399 13915, 2025.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18407-18418, 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguillig tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 24185-24198, 2024b.
Alessandro Conti, Massimiliano Mancini, Enrico Fini, Yiming Wang, Paolo Rota, and Elisa Ricci. On large multimodal models as open-world image classifiers. arXiv preprint arXiv:2503.21851, 2025.
Avi Cooper, Keizo Kato, Chia-Hsien Shih, Hiroaki Yamane, Kasper Vinken, Kentaro Takemoto, Taro Sunagawa, Hao-Wei Yeh, Jin Yamanaka, Ian Mason, et al. Rethinking vlms and llms for image classification. Scientific Reports, 15(1):19692, 2025.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro. Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for spic-kitchens-ll1 International Journal of Computer Vision, (30(1))33-55 2022.
11

---

Published as a conference paper at ICLR 2026
Guodong Ding, Fadime Sener, and Angela Yao. Temporal action segmentation: An analysis of modern techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2): 1011-1030, 2023.
Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Donglin Bai, Zhibo Chen, and Ting Cao. Streammind: Unlocking full frame rate streaming video dialogue through event weent-gated cogmition. arXiv preprint arXiv:2503.06220, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yazan Abu Farha and Jurgen Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3575-3584, 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 18995-19012, 2022.
Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao, Ke Yan, Shouhong Ding, Yuan Gao, and Gui-Song Xia. Anchor-based robust finetuning of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26919-26928, 2024.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13514 2024.
large multimodal model for long-term video
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wes Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12065 -2075, 2023.
Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos "in the wild". Computer Vision and Image Understanding, 155:1-23, 2017.
Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. arXiv preprint arXiv:2309. 16779, 2023.
David Wingate Joshua Robinson, Christopher Michael Rytting. Leveraging large language models for multiple choice question answering. arXiv preprint arXiv:2210. 12353, 2023.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasinhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1010/7055969999 06950,, 2017.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning, pp. 17283- 17300. PMLR, 2023.
Hilde Kuchne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 780-787, 2014.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified vision-language understanding and generation. In International conference on PMLR, 2022.
machine learning, pp.
KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXivv2305.0633 2023.
Kevin Qinghong Lin and Mike Zheng Shou. Vlog: Video-language models by generative retrieval of narration vocabulary. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1-228, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34999 2023.
Yanting Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Xiaoling Wang. Generation with dynamic vocabulary. arXiv preprint arXiv:2410.08488,, 2024.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei. Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3202-3211, 2022.
Feipeng Ma. Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Multi-modal generative embedding model. arXiv preprint arXiv:2405. /9333, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.080 054244 2023.
Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Conference on Learning Representations, 2024.
Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi- -ii Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision- language models. Advances in Neural Information Processing Systems, 17:12677-1 2024.
Zhanzhong Pang, Fadime Sener, and Angela Yao. Context-enhanced memory-refined transformer for online action detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8700-8710, 2025.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL nttts:://erxiv.. org/ abs 221..212211..22 15115
Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, and Michael S Ryoo. Understanding long videos with multimodal language models. arXiv preprint arXiv:2403.16998 2024.
Daniel Sikar, Artur Garcez, Robin Bloomfield, Tillman Weyde, Kaleem Peeroo, Naman Singh, Maeve Hutchinson, Dany Laksono, and Mirela Seljan-Delang The misclassification likelihood matrix: Some classes are more likely to be misclassified than others. arXiv preprint arXiv:2407.078.1. 2024.
Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang. Miguel Martin, and Lorenzo Torresani. Ego4d gtal-step: Toward hierarchical understanding of procedural activities. Advances Neural Information Processing Systems, 36:38863-3888 2023.
13

---

Published as a conference paper at ICLR 2026
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Jiahao Wang, Guo Chen, Yifei Huang. Limin Wang, and Tong Lu. Memory-and hemory-and-anticipaan -ndaniiiiiottion transformer for online action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13824-13835, 2023.
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pp. 38-76. Springer, 2024.
Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang, Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videollm-mon Efficient video-language streaming with mixture-of-depths vision computation. Advances in Neural Information Processing Systems, 37: 109922-109947, 2024.
Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5532-5541, 2019.
Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. Advances Neural Information Processing Systems, 34:1086-1099, 2021.
Meng Ye and Yuhong Guo. Zero-shot classification with discriminative semantic representation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7140-7148, 2017.
Shaokai Ye, Haozhe Qi, Alexander Mathis, and Mackenzie Mathis. Llavaction: evaluat- ing and training multi-modal large language models for action recognition. arXiv preprint 28X 2025.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0100 01917. 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Hanling Zhang, Yayu Zhou, Tongcheng Fang. Zhihang Yuan, Guohao Dai, and Yu Wang. Vocabtailor: Dynamic vocabulary selection for downstream tasks in small language models. arXiv preprint arXiv:2508.1 15229, 2025.
Xing Zhang, Zuxuan Wu, Zejia Weng, Huazhu Fu, Jingjing Chen, Yu-Gang Jiang, and Larry S Davis. Videolt: Large-scale long-tailed video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7960-7969, 2021.
Yidan Zhang, Ting Zhang, Dong Chen, Yujing Wang, Qi Chen, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, Fan Yang, et al. Irgen: Generative modeling for image retrieval. In European Conference on Computer Vision, pp. 21-41. Springer, 2024.
Yue Zhao and Philipp Krähenbühl. Real-time online video detection with temporal smoothing transformers. In European Conference on Computer Vision, pp. 485-502. Springer, 2022.
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. Proceedings of the IEEE international conference on computer vision, pp.
2017.
Jamming Zhou, Junwei Liang, Kun- Yu Lin, Jinrui Yang, and Wei-Shi Zheng. Actionhub: large-ccale action video description dataset for zero-shot action recognition. arXiv preprint arXiv:2401. 11654, 2024.
14

---

Published as a conference paper at ICLR 2026
Dimitri Zhukov, Jean- Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 3537-3545, 2019.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
THE USE OF LARGE LANGUAGE MODELS
In this paper, Large Language Models (LLMs) were used exclusively for polishing the manuseript, including improving writing style, enhancing readability, and correcting grammatical emrors. LLMs were not employed for research purposes such as literature retrieval, idea generation, or discovery. All methodological proposals, experimental designs, analyses, and conclusions developed without the involvement of L.LMs.
A FEATURE ANALYSIS OF GENERATIVE VS. DISCRIMINATIVE CLASSIFIER
We examine the feature quality of the generative and discriminative classifiers. For generative outputs, action labels are tokenized into multiple tokens, and representations are derived using one of four strategies: mean, max, first, last token. Full t-SNE visualizations reveal the clear decision boundaries of the discriminative classifier, further highlighting the comparatively inferior performance of generative decoding.
aldel
add
ald ooson
(b.1) Generative-- meon
(a.2) Geusrative - max
odd lice
add
odd stunbeciis to vake
add
whippel
(b) Diseriminative elaesificr
(a.3)
4)
Figure 6: T-SNE plot comparing feature spaces of the generative and discriminative classifier CrossTask dataset for actions sharing the verb 'add'. Generative feature is derived from token representations using four strategies: mean, max, first, or last token. Discriminative feature the representation of the learnable token.
B EXPERIMENTS
B.1 IMPLEMENTATION DETAILS
We conduct evaluations on two benchmarks: OAD, for online action detection, and COIN, for step/task recognition and step forecasting, The used task-specific queries are summarized in Tab
For the OAD benchmark, training involves sampling a "current" timestamp f, and long- and short-term memories constructed by padding cropping past observations up to f. The timestamp is sampled using a sliding window with a random start and a stride equal to the short-term length, while frames in short- and long-tern streaming setup is simulated with a sliding window of stride and fixed start time at 0, predicting one frame time. We train for 50 epochs THUMOS' 14, CrossTask, and SPIC-Kitchens-I
selected at fixed predefined sample rate. During inference,
online
16

---

Published as a conference paper at ICLR 2026
Table 8: Task-specific queries.
Task
Query
Online action detection
What is the action in the last frume?
Step recognition
What is the action in the video?
Step forecasting
What is the next action in the video?
Task recognition
What the overall activity in the video?
and 20 epochs Ego4DGoalStep,, where the latter contains fewer distracting background frames, making it easier to train. As training samples are generated with a stride equal to the short-term length, only a fraction of frames is used per epoch. For example, 50 epochs on CrossTask with stride 20 is equivalent to 2.5 full passes over the dataset. Optimization is performed with AdamW using learning rate of 0.0001, a warmup ratio of 0.1, and batch size of 32 across all datasets. Unless otherwise specified, we employ 1 1--1--InsttuC as the language model. Details of the hyper-parameters for OAD can be found Tab.
Table Hyperparameters for OAD setting.
Datasets
long-term (s)
short-term (8)
sample rate
visual feature FPS
THUMOS'I4 14
128
32
ResNet50@4
CrossTask
128
20
DINOV2@1
PPIC-Kitchens-100
128
20
TSN
Ego4DGoalStep
128
16
DINOV2@1
For the COIN benchmark, we adopt multi-task learning framework to jointly train step recognition, next-step forecasting, and task recognition using task-specific queries, with separate MLP heads for step and task predictions. Training is conducted for 5 epochs using the Adam optimizer with learning rate of 0.0001, warmup ratio of 0.05, and batch size of 8.
For OAD tasks, we report only segment-wise wise and point-wise F1 scores, omitting frame- wise performance, since frame-level accuracy may not provide sufficiently informative reliable evaluation, as shown in [aaag et al. (2025). For completeness, frame-wise accuracy results are included in the supplementary material.
Metric:
B.2 GAD UNIFICATION STRATEGY
The generative and discriminative classifiers can be unified either sequentially or in parallel. We investigate three unification strategies, illustrated with representative examples.
Sequential learning with discriminative learning first. The learnable [CLS] token first pro- cessed, serving as conditioning context for the generative output.
<(begin of text|>
User: What is the action in last frame?
Assistant: flip bread <cos>
Sequential learning with generation first. The learnable [CLS] token positioned after genera- tion, enabling the discriminative learning to be conditioned on the generative outputs.
<(begin of textl>
User: What the action the last frame?
Assistant: flip bread <eoss [cls]
Parallel learning. Discriminative learning and generative decoding are performed concurrently in branches, sharing the same and visual tokens, but without conditioning another. To fairly assess the impact unification strategies, we align the generative task with discriminative learning in GAD by generating only the target label, generating extra previous action could
17

---

Published as a conference paper at ICLR 2026
<lbegin of texis-
User: What is the action in the last frame?
[cls)
<lbegin of texip>
User: What is the action in the last frame?
Assistant: flip bread <eos?
Table 10: Discriminative and generative performance in the proposed GAD classifier, where genera- tive modeling is for labeling task same as the discriminative learning.
OAD Benchmark. (frame acc./segment F1/point FI)
Model
CrossTask
EPIC-Kitchens-100 100 Ego4DGoalStep
Gen
41.3/46.8/31.7
32.9/16.7/13.9
32.9/8.9/3.4
Disc
41.7/48.8/340
34.8/23.2/19.3
33.9/10.6/4.1
GAD gen
81.5/46.8/31.8
32.1/16.0/13.2
31.4/7.8/3.0
GAD emb
81.6/49.1/3..9
34.8/23.6/19.6
34.0/10.8/4.1
GAD seq-g gen
81.2/46.3/31.3
33.3/17.1/14.2
32.5/8.9/3.3
GAD_seq-g.co
81.2/46.3/31.3
33.3/17.1/14.2
31.5/8.7/3.4
GAD GAD.parallel_gen
81.5/46.//11.8
32.4/16.1/1.44
32.2/8.4/3.1
GAD GAD parallel.emb
81.7/49.4/34.1
34.9/23.6/19.6
34.1/10.7/4.1
interfere with current action prediction and bias performance. From the results in Tab. 10 we observe the conflicts between generation and discriminative learning in sequential learning setting. In the discriminative learning first strategy (GAD), discriminative predictions are largely unaffected and can even benefit slightly from semantic regularization through generation, whereas generation performance suffers when conditioned on the learned token. In contrast, in the generation first strategy (GAD_seq-2 generation performance remains less affected, but discriminative performance degrades, sometimes exactly matching the generative one. This happens because conditioning generative outputs enables the discriminative classifier to learn a shortcut to aligns its representation with the generative outputs during training, causing it effectively replicate the generative output inference. Since generative outputs are more prone to confusion due to semantic overlap, which can turn degrade discriminative classifier performance. Meanwhile, placing the learnable token the end slows inference, as autoregressive generation must be completed first.
In the parallel learning strategy ..ADllarallll), discriminative and generative performance less affected by interference. Training them simultancously is analogous to multi-task learning with shared backbone, allowing the model capture cross-task knowledge while benefiting from regularization. However, this strategy needs two forward passes for each output, which reduces training efficiency.
In conclusion, given our focus on discriminative outputs, we stick with the discriminative learning first strategy, which yields strong representations, exploits generative regularization, and ensures efficient training and inference.
B.3 ENTROPY-BASED DIVERSITY SCORE FOR MISCLASSIFFCATTINS
Since the generative classifier treats action labels as subword tokens, shared subwords across similar actions can lead to greater confusion. To quantify this effect, we introduce an entropy- based Diversity Score (DScore) measure the variability of misclassified predictions. We first compute the confusion matrix C E N of a given classifier, where N denotes the number of classes, with rows representing ground truth labels and columns representing predictions. Then, we set the diagonal entries (true positives) of the confusion matrix to zero focus misclassifications. For OAD datasts, we additionally exclude misclassifications assigned to the background class to avoid diluting the results, as background is semantically unrelated to other classes and overwhelmingly represented in the datasets. Finally, we normalize each row of the modified confusion matrix C' to obtain an error distribution for each target class, and quantify the diversity of misclassification for each class
18

---

Published as a conference paper at ICLR 2026
using the Shannon entropy H. The final DScore is computed as the average entropy over actions, excluding the background and classes absent at test time.
C..y Ek=D
N-1
Pij
=4=-1
Σ
Pr. log(pe +
Vi -1],
(4)
C...
k=0
where is a small constant ensuring valid input to the logarithm.
Table presents diversity scores on the OAD datasets, showing that the generative classifier produces more diverse misclassifications due to semantic overlap. In contrast, our GAD model can encode semantics through generation while mitigating the error diversity, leading to more consistent predictions and thus potentially reducing action over-segmentation.
Table 11: Comparison of misclassification diversity for generative (Gen), discriminative (Disc), and generation-assiste discriminative (GAD) classifiers.
OAD Benchmak. DScore
Model
CrossTask
SPIC-Kitchens-100 Ego4DGoalStep
Gen
0.76
1.3
1.8
Disc
0.66
0.79
1.5
GAD
0.67
0.80
1.49
B.4 ABLATION STUDIES
We conduct ablation studies on key design choices of our proposed GAD, evaluated on OAD tasks with results reported in Tab 12
The generative output can be any text, but focus generating the previous action. Beyond the flexibility of free-form outputs, we aim to show that the benefits stem from generation itself rather than the auxiliary task. To this end, we reformulate previous action prediction as separate classification task, introducing an additional classifier that uses either the representation of the existing learnable token (GAD-prev -rev_discc or a new learnable token appended after (GAD_pre -pev disc+). Results indicate that neither approach improves learning and can even disrupt current action classification, causing a performance drop. This further highlights the advantage of using generation to encode semantics an auxiliary task.
We use the generative head for previous step generation. Alternatives include generating the next action or past actions within a fixed short-term window. Results show that these alternatives still outperform the discriminative baseline by capturing action relationships, but remains less effective than previous step generation. Next-step generation lacks corresponding visual inputs, while generating multiple past actions reduces performance due increased complexity that diverts focus from discriminative learning.
For discriminative outputs, we use the representation of the learnable [CLS] token for prediction. An alternative use the last visual token, which also aggregates all preceding visual information. This design fits the setting of online action detection, which requires recognizing the action in the last frame. However, results (w/o (CLS]) show that using [CLS] ] token performs better, likely due to its stronger generalization ability, while the last visual token tends to overfit the training data.
The generative response is designed to include the target label along with the context information describing the previous action. Therefore, generative and discriminative learning somehow aligned toward predicting the same target action. To evaluate the effect separating the tasks, we remove the target label from the generative output, leaving only the context generation (GAD (GAD_sep). While GAD_sep still performs well, it underperforms the original version, suggesting that sharing the same target better aligns the discriminative and generative modules, while also providing valuable context for the action relationship learning.
B.5 ANALYSIS ON GAD GENERATION OUTPUT
We use generation as an auxiliary task and rely solely on the discriminative output during inference. Nevertheless, it is important to examine the quality of the generative outputs and their role in
19

---

Published as a conference paper at ICLR 2026
Table 12: Ablation studies of GAD classifier on OAD tasks.
OAD Benchmark. (frame acc./segment F1/point FI)
Model
CrossTask
Ego4DGoalStep
Disc
11.7/48./344
34.8/23.2/19.3
33.9/10.6/4.1
GAD
81.8/50.3/34.5
35.1/24.1/20.1
34.4/1 1.0/43
GAD BaDD_prev_diss.
80.8/48.2/31.6
34.8/23.6/19.3
33.2/10.1/3.8
GAD GAD-prev pre.prrevise
80.9/49.5/32.7
34.7/23.0/19.6
33.3/10.6/4.0
GnD_
81.6/49.8/34.
34.6/23.7/19.8
34.1/10.8/4.2
past
81.7/49.5/34.1
34.7/23.9/19.9
33.9/10.4/4.1
w/o [CLS]
81.5/49.2/33.
34.6/20.7/17.7
33.7/8.6/3.6
GAD_sep
81.3/50.0/33.9
34.7/24.0/20.11
34.3/10.9/4.3
enhancing the discriminative learning. To this end, we specifically analyze the generation results. We conduct this analysis on the COIN benchmark, as its accuracy metric provides clearer basis for evaluation.
analyze the correctness of three types GAD output: the discriminative output (Disc), generation output for the target label (Gen), and the generation output for the task label (Gen_extra). These yield eight possible output combinations for step recognition and forecasting, as illustrated in Fig. 7 In the of task recognition, the target label and task label are identical, Gen_extra is not applicable, leading to four possible output combinations.
By examining the distribution of output correctness combinations, make several observations. First, the three types of outputs are generally consistent, with most cases fully correct. Second, step generation is more challenging than task, tasks exhibit clearer separation with lower semantic overlap. As such, cases where the task generation is incorrect while the step generation is correct are uncommon. Third, there instances where both generative outputs flawed, yet the discriminative output remains accurate, highlighting the benefit of placing the discriminative learning ahead of generation to avoid the impact of erroneous generated content. Although the discriminative learning is not conditioned on generation, it is still influenced by generative outputs, which regularize it be generation aware.. Finally, some cases show the generative outputs are accurate while the discrininative output does not, highlighting complementarity between the two and suggesting potential gains through ensembling.
recognition
USc
Qen
Ge
Sentr
Talk recogmitio
Dieca
Genta
can eic
Gen x
Gen #
Gari
Gamii
Figure 7: Quantitative analysis of discriminative ouputs (Disc) and generative outputs (Gen & Gen ex). Larger areas indicate higher occurrence. Checks and crosses denote correct and incorrect outputs, respectively.
B.6 QUALITATIVE RESULTS
We provide additional qualitative results on the OAD task and COIN benchmark in Fig. We observe that even when the previous step generation is occasionally incorrect, the relationship between the generated previous step and the current step remains meaningful, e.g. "close cupboard' following 'open cupboard'. This demonstrates that the model can captures action relationships through generation. Similarly the COIN benchmark, we also observe that the generative and discriminative
20

---

Published as a conference paper at ICLR 2026
Table 13: Comparison with SOTA on OAD tasks.
OAD Benchmark, (frame acc acc./segment F1/point F1)
Model
THUMOS'14 14
CrossTask
EPIC-Kitchens- 100
Ego4DGoalStep
Testra (Zao & Krähenbühl
2022
76.9/43.0/3200
81.2/48.4/33.
34.7/16.5/14.7
31.7/8.7/3.5
MAT Wang et al,
2023
78.4/49.4/34.2
$1.4/49.7/44.2
35.0/17.5/15.5
32.9/9.5/3.8
CMeRT (Pagg
2025)
78.2/48.9/34.6
81.3/50.1/34.33
35.2/17.7/15.8
32.8/9.7/3.9
Gen (Llama3.2- BB B-Instruct)
78.6/56.9/38.8
81.3/46.8/31.7
32.9/16.7/13.9
32.8/8.9/3.4
Dise (Llama3.22 1amma33.2-BB- B- (B-srructt)
78.5/57.8/40.1
81.7/48.8/34.0
34.8/23.2/19.3
33.9/10.6/4.1
GAD (Llama3.22 I-IIntrucct
78.8/58.1/40.2
81.8/50.3/34.5
35.1/24.1120.11
34.4/11.0/4.3
Table 14: Comparison with SOTA on OAD tasks.
OAD Benchmark (segment-Fl point-F1)
Model
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Gen
46.8/314
16.7 / 13.9
8
Testra [1]
48.4 33.88
16.5 / 14.7
8.7 3.5
MAT [2]
49.7 34.2
17.5 / 15.5
9.5 / 3.8
CMeRT [3]
50.1 / 34.3
17.7 155.
9.7
Disc
48.8 34.0
23.2/ 19.3
10.6/4.1
Gen
46.9 31.8
16.8 / 14.0
8.8
Gen desyne
48.7 34.1
23.0 19.0
3.9
Gen_extend
48.9 33.9
23.3 19.2
10.5 / 4.0
GAD
50.3 / 34.5
24.1 20.1
11.0 4.3
outputs are not always consistent, highlighting their complementary nature and motivating further analysis of their ensemble performance.
B.7 SOTA COMPARISON ON OAD TASKS
We include additional frame-wise accuracy results for a more comprehensive assessment. The results show significant improvements our GAD classifer in segment- and point-wise performance while maintaining competitive frame-level accuracy.
C ZERO-SHOT ACTION UNDERSTANDING
We evaluate two MLLMs zero-shot, Qwen2.5-VL-7B (Bai al. 2025) and VideoL.LM-online (Chen ct al. 2024a), on the COIN dataset ( 750 actions). For Videollm- model, a procedural video understanding MLLM, we use for open-ended generation. For Qwen2.5-VL, action prediction is treated either as an open-ended generation task, or by providing action labels as candidate options in the prompt. Posc-procssiing always applied match the generation action categories using the CLIP text encoder and cosine similarity.
As shown in Table 15 Qwen2.5 performs best but still significantly trails our fine-tuned method. These results suggest that action understanding with large, semantically similar actions is challenging, making fine-tuning essential. In addition, providing action categories in the prompt worsens the performance, suggesting that including a large candidate set in the prompt can hinder MLLMs ability to correctly interpret instructions.
TRADE-OFF
We prove the necessity of tine-tuning in Sec. C. However, extending fine-tuning boosts performance, but increases memorization and reduces generalization. To quantify memorization, define the
21

---

Published as a conference paper at ICLR 2026
Streaming video
time
time 100s
time
time 107s
backgrond
aeemmco cil
Das backgrmuul
badiegrond
Diszi bockgrwand GAD ndd kinela
Biso misoue 6AD add kimehi
GAD Inckgrourd
SAD your sesimn of
$AD tackground
6Ab add kinehi
Vluur
pow sesane all
time 133s
170g
Gan
Gan backgrenn
Gon bockgroend
Disc: cdd rice
Dise: mistare
Disd sii mistarc GAD sie misture
Disc cal usion
Disz edd esiens
GAD add nice
SAD sir mosure
G4D add oniin
GAD sdd soaree
Jate sockire
uhl roc
Ground Trush
Disc
timels)
(a) On CrossTask.
Streaming
time
time 5.5s
time 5.75s
time 9
time
11.25
time 11.5%
time 11.75%
open oaphoard
bockgroard
Disc backgroodd
close cepboard
Disci badogmond
Dic: backgmand
GAO eaploard
GAD take bowl
GAD cluse cuphoard
SAD opes Eklge
GAD tackgnounal
6AD close fridge
BAD open squaii
Sidg
open oxphoast
oyen inphosee"
dane expeasa"
Padge"
Wooy Pugge"
time 20ss
time
time
28
-29.56
Gen pou lamon
Cen backgnand
Gani finle: Diec open thiger 54b open filgee
Gen backgnard
Gan backgmunl
Gen take kanon background
Diec bockggoonnd
Diec backgmond GAD beckgrondd "Rolgnows!
Diec badkgmond
Ditc backgronnd
6Ab
pour Jakc
6AD beckgroudd
SAD clne filge
badigmuul
Thur solos Prevosur
(Yjux nugge Peewsor your
"Cios wonr
yook
huml
balk lkk
dins
dew hoter
juwe
dorr nighort
coe Intor
yowr pome
Gin
Disc
GAD
Emneet
(b) On MIIC-Kithhens-110
What is the next action in the video?
Whot the overall activiry the video?
What is the overall activity in the video?
pour maw
Gen
Gen
make
pour maw
change
Disc
quift
6AD
pour ingredicnts
GAD
change hike tires
6AD
quilt
Your the rngredores cato Avadt Fisk make iosassasl craase"
Vilange hibe taw.
"okle feal
(c) On COIN benchmark
Figure 8: Qualitative results on the OAD and step recognition tasks. Gen generative classifier, Disc- discriminative classifier. Text in red denotes false predictions, while in green represents correct predictions. Callouts display the generation outputs produced by our GAD. Bar charts in the bottom show predictions for streaming videos.
22

---

Published as a conference paper at ICLR 2026
Table 15: Zero-shot performance of existing MLLMs on COIN dataset.
Model
Step
Next
GAD (fine-tuned)
67.3
51.6
Videollm-online-SB (open-ended)
4.8
3.5
Qwen2.5-VL-7B (open-ended)
16.1
8.9
Qwen2.5-VI.-7B (categories in prompt)
11.9
6.5
memorization rate as the proportion of generated outputs that match training action categories during inference. As shown in Table [6] achieving optimal performance on COIN task recognition (Tang) al. 2019) requires epochs of fine-tuning, yet the memorization rate evaluated on the unseen Breakfast dataset (Kuehne et al. 2014) exceeds 98% after just one epoch, and the zero-shot perfor- mance on unseen dataset Breakfast decreases over the course of training. This shows the inherent trade-off between optimal performance and generalization ability.
In fact, the generative head our GAD model enables leveraging self-curated instruction-tume -unnnng data to preserve generalization for example, by augmenting action labels. However, this comes the cost of some closed-set performance. We leave more thorough exploration of these generalization capabilities to future work.
Table 16: Perfommance-enncaalzzzioon trade-off our generative bascline.
Epoch
Epoch
Epoch
Epoch
Epoch
Epoch
Memorization_ratee ratio (on Breakfast)
0.0
98.1
99.6
99.8
99.9
99.9
Accuracy COIN (test set)
8.3
77.0
85.3
91.3
92.8
92.7
Accuracy Breakfast (unseen)
6.3
4.5
3.5
3.6
3.6
3.6
E
ERROR ANALYSIS: GENERATIVE VS. DISCRIMINATIVE
We provide extra analysis why the generative baseline produce more diverse misclassifications than the discriminative one. In Figure ? we compare the confusion matrix comparison CrossTask for actions shared verb 'add'. The generative baseline exhibits greater difficulty handling semantic overlap among these actions.
Figure 9: Confusion matrix comparison on CrossTask for actions shared verb 'add'. Generative classifier incurs more diverse misclassifications
Generative Classifier
Discriminative Classifier
add oesion
4
3 2
add letuuee
add meal
tomato
adi choes:
13
tomill
add rice
ham
add kimchi
2
Ground
wid sugar
add spices
add chili powder
add mustard seeds
carry
add fish
add flour
add strwheeries to cake
add salan
lateee
add torate
ald chease
ald korilla
ald hare
kinchi
ssgar
add chill powder steds
old syiere
salt
ald curry to fish add flour
add stmwhantis vakee
Prediction
23


Reviews:
[{"review_id": "FJBzeWzS3W", "reviewer": "Reviewer_CrzN", "summary": "This paper investigates the use of Multimodal Large Language Models (MLLMs) for closed-set temporal action understanding. The authors identify a performance gap between generative and discriminative classifiers, attributing it to semantic overlap in label tokenization. To address this, they propose a Generation-Assisted Discriminative (GAD) classifier that incorporates an auxiliary generative objective during training to provide semantic regularization, while maintaining the discriminative model's efficient inference. Evaluations on multiple action understanding benchmarks show GAD achieves state-of-the-art results.", "questions": "- Given that the architecture of the discriminative classifier (appending a [CLS] token) and the idea of multi-task learning with an auxiliary objective are well-established, what is the specific conceptual novelty of the GAD framework beyond its application to the video modality?\n- Can you design experiments to isolate the benefit of the auxiliary task? For instance, compare against a model trained with an equivalent amount of additional discriminative data or a different, non-generative auxiliary task to prove that the generative nature of the objective is key?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper's greatest strength is its thorough, apples-to-apples comparison between generative and discriminative approaches across a wide range of tasks and datasets. \n-  The implementation details in the main paper and appendix are extensive.", "weaknesses": "- Using a discriminative head on top of a generative model and training it with an auxiliary task is a standard practice in machine learning. The conceptual contribution is limited.\n- The paper lacks a deep analysis of how the auxiliary generative task helps. \n- The GAD framework is only evaluated in closed-set settings. Its applicability to open-world or few-shot scenarios is not discussed, limiting the scope of its claimed generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T28BuVh660", "reviewer": "Reviewer_fcN1", "summary": "The paper presents a study on generative vs discriminative methods for action classification, illustrating how the latter outperform the former in most cases. Several potential causes are mentioned in the paper, including the partitioning of actions into different tokens leading to overlapping and lack of distinctiveness in the representation of actions. The authors then propose a generative assisted classification approach where an LLM is endowed with a class token tasked with predicting actions in a closed-set environment, followed by a language decoder to decode past and future actions around the one classified with the CLS token. The paper is accompanied by supporting experiments illustrating the drawbacks of existing LLMs for generative action classification. Additional results show that their proposed approach can enhance existing LLMs for action classification.", "questions": "The runtimes in Table 1 illustrate that the discriminative counterparts of the corresponding LLMs are much faster than the generative ones. I understand that this might be due to the fact that when the actions are split into several tokens, it requires several forward calls to the model, incurring in slower inference. However, if the model is provided with action-specific tokenization as mentioned in l. 363, then inference should be achieved in a single forward pass. In such case the speed should be similar for both generative and discriminative approaches. In any case, what is the difference between a generative approach with extended vocabulary, whereby each full action corresponds to a token, and a discriminative approach with as many [CLS] learnable tokens as class labels? In the latter approach a one-vs-all discriminative approach could be learned (please refer to works on meta queries). I find that referring to the extended-vocabulary approach to generative is a bit overstretched. Can the authors elaborate on this?\n\n\nIn a closed-set setting with N classes, comparing discriminative and generative methods with the latter having a classifier that has L logits with L >> N is obviously biased. Have the authors tried to compare both methods by capping the classifier of the generative approach to N logits only, or to N' logits with N' being only the logits corresponding to the tokens involved in tokenizing the target classes? \n\n\nHow sensitive are the generative results to the tokenizer? I believe a proper study with further text encoders is necessary to study the influence of the tokenizers in the generative results. \n\nHow is the method trained in the case that there's a single action in a video? e.g. Kinetics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and presented. The motivation is clear and the paper is properly threaded. The authors identify specific challenges of generative models for action classification, and enumerate and study the possible reasons behind this subpar performance w.r.t. discriminative methods. Then, the authors devise a combined approach that is shown to produce better results. The paper is accompanied by proper ablation studies and an efficiency analysis, illustrating how discriminative (or single-token decoding) is not only more accurate in terms of closed-set prediction, but also faster at inference.", "weaknesses": "While the paper is well threaded and motivated, part of the story is driven towards obvious conclusions that leave aside some potential alternatives (please see the questions below). In particular, \n\n1. The fact that discriminative approaches, or closed-set approaches, outperform generative ones, is not a finding or contribution of this paper, it is general knowledge. It is not expected that open-vocabulary models will outperform discriminative approaches in closed-set environments. The narrative is driven in a way that benefits discriminative models, without proper reference to the fact that this is constrained to the specific environments where the set of actions to be recognized is closed. In the questions below I suggest a potential study for a fairer comparison between generative and discriminative approaches. I also believe that bridging the gap between both methods is indeed making them the same (i.e. if a single token is used per action by extending the vocabulary, and the logits are capped to those of the target actions, then a generative approach boils down to a discriminative one with learnable queries and a one-hot binary classification approach). \n\n2. It is a bit misleading that the narrative is built and tested in an ego-centric scenario where discriminative approaches are prone to thrive. How would the whole story sustain itself in a more generic scenario where several actors are performing potentially overlapping actions? E.g. the AVA setting where actions are not exclusive (i.e. in a multi-class scenario). In such case it looks to me that a generative approach can outperform discriminative approaches. If it is out of the scope of this paper to study such scenario, then it should be clearly stated in the paper to avoid leading to misinterpretations. \n\n3. There's a missing text-to-text action retrieval from the predictions of generative approaches to closed-set predictions. In particular, it is not clear (or at least I missed that in the paper) how the generative predictions are mapped to actions. \n\n4. There's no breakdown (unless I missed that, in which case I hope the authors can address this in the rebuttal) of where the confusing is coming from. In Fig. 3 the authors plot the false positives for the \"add sugar\" action, but it is not clear to me if this refers to action within the target set, or free-form actions.\n\n\nIn summary, I am borderline with the paper, as I believe it holds some valuable contribution, it is well written and presented, but lacks some proper contribution that is properly sustained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9h7fg3X6k", "reviewer": "Reviewer_4Sdb", "summary": "Authors explore how Multimodal LLMs (MLLMs) can be used for closed-set action classification, comparing generative and discriminative approaches. They claim that generative classifiers (autoregressively producing text) are less efficient and accurate compared to discriminative classifiers.\nThe authors proposed a Generation-Assisted Discriminative (GAD) classifier that combined strengths of both aspects. This model uses a discriminative framework for efficient inference but incorporates generative modeling as an auxiliary task during fine-tuning to enhance the model's semantic understanding. Evaluation across five datasets establish clear performance gains in terms of both accuracy and inference speed.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Strong results across multiple datasets with multiple baselines with focus on both accuracy and speed. \n2. Interesting dual training strategy. \n3. Clear presentation of key idea (e.g. great use of diagrams in Figure 2).", "weaknesses": "1. \"approximately 1.8× faster training\": Discuss this more - how is the training faster than next-token prediction? This is unclear.\n\n2. Lost generality: does this classification training remove the general QnA abilities of these VLMs? This is unclear. \n\n3. How do these VLMs perform zero-shot (generative classifier) on these tasks? This will provide an interesting point of comparison. \n\n4. Inference compute scaling: if you generate one or two tokens (with the generative part of GAD), and then run the classifier on all tokens, does this improve performance? This will be an interesting analysis.\n\n5. How does cls head scale with more output categories? For example, say 20,000 categories? What is the impact on inference speed? \n\n6. Considering discussing prior work exploring similar ideas for video QnA and NLP. Two examples below. \n  - https://arxiv.org/abs/2403.16998 (using MLLMs for closed-set video classification)\n  - https://arxiv.org/abs/2210.12353 (using LLMs for closed-set classification)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)