Measure Twice, Cut Once: A Semantic-Oriented Approach to Video Temporal Localization with Video LLMs
Authors: Zongshang Pang
Mayu Otani
Yuta Nakashima
Venue: ICLR 2026 Poster (2026)
Keywords: Video LLMs
Video Temporal Localization
Contrastive Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Temporally localizing user-queried events through natural language is a crucial capability for video models. Recent methods predominantly adapt video LLMs to generate event boundary timestamps for temporal localization tasks, which struggle to leverage LLMs' pre-trained semantic understanding capabilities due to the uninformative nature of timestamp outputs. In this work, we explore a timestamp-free, semantic-oriented framework that fine-tunes video LLMs using two generative learning tasks and one discriminative learning task. We first introduce a structural token generation task that enables the video LLM to recognize the temporal structure of input videos based on the input query. Through this task, the video LLM generates a sequence of special tokens, called structural tokens, which partition the video into consecutive segments and categorize them as either target events or background transitions. To enhance precise recognition of event segments, we further propose a query-focused captioning task that enables the video LLM to extract fine-grained event semantics that can be effectively utilized by the structural tokens. Finally, we introduce a structural token grounding module driven by contrastive learning to associate each structural token with its corresponding video segment, achieving holistic temporal segmentation of the input video and readily yielding the target event segments for localization. Extensive experiments across diverse temporal localization tasks demonstrate that our proposed framework, MeCo, consistently outperforms methods relying on boundary timestamp generation, highlighting the potential of a semantic-driven approach for temporal localization with video LLMs.

Published as a conference paper at ICLR 2026
MEASURE TWICE, CUT ONCE: A SEMANTIC- ORIENTED APPROACH TO VIDEO TEMPORAL LOCAL- IZATION WITH VIDEO LLMS
Zongshang Pang
Mayu Otani
Yuta Nakashima
The University of Osaka 2CyberAgent, Inc. {pangzs, n-yutaaeimm sanken. saneen osaka-u.ac.
otanimayuleybeeagent.cco co.
ABSTRACT
Temporally localizing user-queried events through natural language is a crucial capability for video models. Recent methods predominantly adapt video LLMs to generate event boundary timestamps for temporal localization tasks, which strug- gle to leverage LLMs pre-trained semantic understanding capabilities due to the uninformative nature of timestamp outputs. In this work, we explore timestamp- free, semantic-oriented framework that fine-tunes video LLMs using two genera- tive learning tasks and one discriminative learning task. We first introduce struc- tural token generation task that enables the video LLM to recognize the temporal structure of input videos based on the input query. Through this task, the video LLM generates sequence of special tokens, called structural tokens, which par- tition the video into consecutive segments and categorize them as either target events or background transitions. To enhance precise recognition of event seg- ments, we further propose query-focused captioning task that enables the video LLM to extract fine-grained event semantics that can be effectively utilized by the structural tokens. Finally, we introduce a structural token grounding module driven by contrastive learning to associate each structural token with its corre- sponding video segment, achieving holistic temporal segmentation of the input video and readily yielding the target event segments for localization. Extensive experiments across diverse temporal localization tasks demonstrate that our pro- posed framework, MeCo, consistently outperforms methods relying boundary timestamp generation, highlighting the potential of a semantie-driven approach for temporal localization with video LLMs
INTRODUCTION
Localizing temporal events based on user interests is an essential capability for video recognition systems to handle practical video tasks such as moment retrieval (Lei et al. (202T), action localiza-
tion (Chao et al. 2018 Cheng & Bertasius] 2022), video summarization (Song et al. 2015 Gygli
et al. 2014), and dense video captioning (Krishna et al., 2017; Wang et al. 2021a Yang et al.
2023 While such temporal localization tasks were traditionally handled by specialist models,
cent research has begun leveraging video LLMs to unify them within a single framework by adapting
I.LMs for boundary timestamp generation (Ren al. 2024 Liu et al. 2024c Zeng et al. 2025).
Accurately localizing the boundary timestamps of a target event requires understanding the semantic content of the target event by both examining its relevance to the localization query (e.g., event description or action label) and differentiating it from adjacent events to identify event boundaries. However, current methods expect video LLMs internally handle the semantic understanding and directly provide the final boundary timestamps. Consequently, a major focus of this line of research has been to develop various video LLM- LLMM-finndly timestamp representations to boost performance. Nonetheless, direct timestamp generation may limit the potential of video I.MMs, as they are backed by L.MMs that are built to process semantic information (Brown et al., (2220) and have primarily been pre-trained on video captioning and question answering tasks that require mapping visual inputs to
Code available httpttttpss.. htttt:pp//gihhhbe com/ http:hhtwwcppppppg.//ooo//o

---

Published as a conference paper at ICLR 2026
Localize the event clip showing: A person preparing plate of food. Divide the video into Transition and Event segments and provide Query Focused Caption for the segment.
Localize event clip showing: person preparing plate of food. Provide the start and end timestamps of the clip.
8
<start time> to <and time> seconds
<tat>
fOrc
<ent>
<tst>
varions dugrofients plate..
KBA
Transition
Event
Transition
(a) Boundary Timestamp Generation
(b) MeCo (Ours)
Figure 1:
In contrast to previous boundary timestamp generation approaches (Ren et al. 2024
Guo et al. 2025a Liu et al. 2024c Huang al. 2024; 2025; Guo al. 2025b), MeCo leverages the semantic understanding capability of video LLMs to capture the video temporal structure and categorize video segments into transition and event structural tokens <tst> and <ent>. It can also generate query-focused captions (QFC) facilitate more accurate localization of the event segment via the event token. The semantic-oriented pipeline completely frees video LLMs from timestamps generation.
scrutinize the detailed event semantics, which can
outputs with concrete semantic meanings (Maaz et al., 2023 Lin al. 2023a). Additionally, it has been shown that LLMs struggle with highly uninformative outputs in both language (Wei et al. Kojima al.,
2022
2022) and multimodal scenarios (Lai et al. 2024 Liu et al. (024c). In this
work, we initiate the exploration of semantic-oriented strategies for adapting video LLMs to video temporal localization tasks. We adapt video LLMs' semantic understanding ability for such tasks through carefully designed supervised fine-tuning tasks involving both generative and discriminative learning objectives.
To start, we propose a structural token generation task to enable the video LLM to distinguish semantic differences between queried events and background transitions by capturing the overall temporal video structure. The output consists of consecutive segments in the video categorized as a sequence of event tokens <ent> and transition tokens <tst>>, arranged in temporal order. To facilitate more precise categorization of event segments via event structural tokens, we propose query-focused captioning task that enables the video LLM to scrutinize the details in each queried event before localization by generating detailed captions for it, akin to the role of Chain-of-Thoupe before LLMs' final answers (Wei et al., 2022 Kojima et al., (022). These two tasks exploit the innate generative power of video LLMs adapt their semantic understanding abilities for temporal localization tasks.
Building the temporal categorization from structural tokens and the semantie refinement from query-focused captions, we propose a sfructural token grounding module based on contrastive learn- ing (He et al. 2020; Radford al. 2021] Wang et 2021b; (quab et al. 2023 Pang et al. 2024) to map the rich semantics encoded in each structural token to the corresponding video segments. This enables holistic video segmentation, where queried events can be precisely localized through their corresponding structural tokens. The contrastive learning objective effectively taps into the
hidden discriminative power LLMs (BehnamGhader et 2024 localization tasks.
Liu al. 2024c) for temporal
The proposed framework, named MeCo, enables video LLMs to Measure twice by prioritizing the semantic understanding of holistic video structure and fine-grained event content before Cutting once to extract all queried event segments. This design fundamentally differs from timestamp generation-centrie approaches, making MeCo fully semantic-centric approach for video LLM- based temporal localization. An illustration of the three proposed objectives is shown in Fig. I Extensive experiments show that MeCo consistently outperforms timestamp-centee approaches, of- ten by significant margins, across tasks spanning grounding, dense video captioning, and complex reasoning domains (Liu et al. 2024c).
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Video Temporal Localization Tasks. Video temporal localization tasks such as moment retrieval
and highlight detection (Lei et al. (021). extractive video summarization (Gygli et al. 20144] [Sog
ct al. 2015 Pang et al. 2023), and action localization (Chao et al. 2018 Cheng & Bertasius 2022)
require localizing salient event segments in response to user query in the form of event boundary timestamps. Furthermore, tasks such as dense video captioning (Krishna et al. 2017 Yang et al,)
2023) and grounded video question answering (Bärmann & Waibel [022] Di & Xie 2024) involve generating captions and performing complex reasoning about these localized events. Traditionally, these tasks have been addressed by specialist models with task-speeific designs and domain-specifie training data. Although unified models for localization-ons only tasks have been proposed (Lin et al.)
2023c
Yan et al. 2023 Liu et al. 2024b), they cannot handle generative tasks like captioning.
Video LLMs. Early efforts to enable LLMS to perform video-level tasks used LLMs as agents built on chain-of-thought reasoning and tool-use mechanisms (Zeng al. 2022 Surís et al. 2023
Lin ct al., 2023b). Advances in end-to-end multimodal pretraining (Radford et al. 2021 Li et al.
2023) and instruction tuning (Ouyang et 2022 Liu al. 2024a) have led the development of
powerful video LLMs (Zhang et al. 2023 Song et al. 2024 Li et al. 2024 Wang et al. 2024b
al. 2025 Yuan et al. 2025), which have have shown that these models excel at temporal reasoning
over very long videos. beneñiting from LLMs' long-context semantic retrieval abilities. However, while effective for general video understanding tasks such as captioning and question answering.
they do not address tasks that require event temporal localization.
Temporal Localization Video LLMs. Recent developments in temporal localization video LLMs have enabled unified approaches for both localization and generation tasks. Models such as TimeChat (Ren et al., 2024) fine-tune pre-trained video LLMs to output numeric tokens that rep- resent event boundary timestamps. Subsequent works (Huang et al. 2025: Qian al. 2024| Guo ct al. 2025a b) augment the LLM's vocabulary with learnable timestamp tokens. For example, VTG-LLM represents timestamps with set of learnable digit tokens and pads the timestamp token sequence to a fixed length (Guo et al.
2025a). Meanwhile, TRACE introduces specialized times- tamp encoder and decoder (Guo al.) 2025b) and VideoChat-T proposes teporal adaptive position encoding module (Zeng et 2025) There are also works that directly interleave the textual timet- samps with frame tokens (Meinardus et al. 2024 Lu et al. 2024). Observing that LLMs struggle with numeric tokens and many newly introduced tokens, E. Chat (Liu et al. 2024ch instead fine- tunes LLMs on a boundary embedding matching task using single boundary matching token. As departure from such works relying on timestamp generation, we propose MeCo to explore how the innate semantic understanding capabilities of video LLMs can be leveraged to build a unified temporal localization framework.
3 METHOD
3.1 OVERVIEW
Video temporal localization involves understanding user-queried events and determining their tem- poral boundaries {(t}, M with M 2 1. Depending on the query, the localized boundaries should sometimes be accompanied by event captions (Krishna et al., (017) or answers to event- related questions (Bärmann & Waibel 2022). which are treated as textual tokens { where N is the total number of tokens. Recently such temporal localization tasks have been unified within a single video LLsed framework (Ren et al. 2024 Guo al., 2025a).
Current video LLM- based methods focus on boundary timestamp generation, which fails to leverage video LLMs' core strength: their pre-trained semantic understanding capabilities. In contrast, we leverage video LLMs' semantic understanding and retrieval capabilities to partition the input video into segments, categorize them as target events or background transitions, and summarize their se- mantics in the hidden states of the proposed structural tokens. To ensure precise retrieval of event semantics, we augment the structural tokens with query-focusee captions. Finally, the structural token grounding module maps the structural tokens to their corresponding video segments via the retrieved and summarized semantics in their hidden states. We now describe in detail how we equip the video LLM with these capabilities. An illustration of the proposed components is in Fig.
3

---

Published as a conference paper at ICLR 2026
Seuctural Token Crounding
Projector
Projector
Honnes black glaves applr veludt (0r, since foans olean the fruse
Houts Maci plawe poisty uije deass atmons pabli houdlyih....
LoRA
Askregrooiv Gencraiion
(b) Query-Focused Captions
Learaabee
Frezzn
Tokzae
Resumpker
Tokenirer
Treaition
Tranaition
Tranitiinn
Toraition Tokens ctetns
Localzo the cips showing hand
Strudiural Tokeons
Query Fecesed Caption Tokees
Stuctaral Tokena
(a) Generating Structural Tokens and Query-Focsued Captions
Structural Token Grounding
Overview of the proposed MeCo framework. Given an input video and localization- related user query, MeCo generates structural tokens (Sec. 3.2). including the event token <ent> and the transition token <tst>,, enhanced by query-focused captions (Sec. (.3) to encode more precise semantic information and can facilitate holistic temporal segmentation via structural token grounding (Sec. 3.4b.
Figure 2:
3.2 STRUCTURAL TOKEN GENERATION
Although video LLMs have demonstrated excellent temporal structure understanding for video ques- tion answering through causal reasoning and for video captioning by describing narrative flow (Song al. 2024 Wang al. 2024c Zhang al., 2024 Xue al. 2024). this capability has not yet been explicitly leveraged for temporal localization. To hill this gap, we propose novel structural token generation task for training the video LLM materialize its temporal structure understanding into temporally ordered sequence of video segments to facilitate temporal localization.
Given a T -framme video as input visual en- coder (Fang et al. 2023b) and resampler (Liu jet al. 2024c) from the video I.LM extract from the video a set of frame feature maps {F, )T=) where F; E PxC has P token embedding vectors, each of dimension C. An LLM de- coder then takes {Fi)T-) and the tokenized user query {q:)=1 as inputs. The structural to- ken generation task requires the video LLM to distinguish between event segments and back- ground transition segments in the input video based on the user query. The segments are then represented in the LLM's autoregressively generated output as either event tokens <ent> or transition tokens <tst>>, collectively called structural tokens, which are newly added to the LLM vocabulary before fine-tuning.
Video Span:
SFT Label: <tst> [QPC] <ent><tet>
Single segment: GT the beginning
Video Span:
Label: [QFC] <an <ttt>
Multiple segments
Video Span: 6t
Label: [QFC] <ent> [QFC] <ont><tat>> [QFC] <ent>
Figure 3: Examples of supervised fine-tuning (SFT) labels created from temporal localiza- tion data, where GT stands for Ground Truth windows and QFC for Query-Focused Caption (Sec. 3.3). For illustration clarify, we only show non-overlapping events for the multi-segment sce- nario, overlapping events can be represented in way similar to consecutive events, e.g., [OFC] <ent> [QFC] <ent>.
The preparation of supervised fine-tuning data for the structural token generation task builds on ground-truth event boundary timestamps from general temporal localization data. Given a T-frame video with a set of M ground-truth event boundary timestamps {(4g, ( augment them with neighboring transition segments to form an augmented set of segments {(t), M+X where is the total number of transition segments. Let Tem be the set of indices of
By default, we sample frames from videos 1 fps unless specified otherwise.
4

---

Published as a conference paper at ICLR 2026
the queried event segments; the sequence of structural tokens can be defined {ST()))== M+ with
<ent> if E Ments
ST(1)
<tst> otherwise.
Importamily, event segments may appear at video boundaries or occur consecutively without inter- vening transitions. These cases provide crucial learning signals: the <tst> token does always trivially precede and follow the <ent> token, and should only appear based on the actual presence of transition segments in the video. Several illustrative examples are provided in Fig. 3 Over- all, structural tokens transform the video's temporal flow into sequence of events and transitions. Through auto-regressive generation, each structural token must attend to its corresponding segment, thereby summarizing the segment's ' semantic information in its hidden state. This creates the founda- tion for grounding structural tokens to their corresponding video segments for localization purposes.
3.3 QUERY-FOCUSED CAPTIONING
Just humans rewatch clips identify specific details of interest, we hypothesize that the summa- rized segment semantics encoded in structural tokens can be refined by having the LLM examine each event segment more closely. To this end, we introduce query-focuse captioning task that requires the model to generate detailed captions for queried segments. By generating these captions immediately before each corresponding event token <ent>, we provide rich semantic information that the token can attend to via causal attention. This process mirrors -hain-of-thought reasoning (Wei al. 2022), where intermediate reasoning steps enhance the quality of final outputs, but resorts to more obtainable captions.
As shown in Fig. and Fig. 3 the overall tokens that the LLM needs to generate now be- come an interleaved sequence of structural tokens and query-focused caption tokens X (T())
M-K with
[OFC] if Ient
CAP(1)
2
otherwise,
where [QFC], encloses all query-focused caption tokens for the i-th event segment, and Ø indicates no such token is placed (we omit the end-of-sequence token for notational clarity). is worth noting that textual response tokens for tasks involving captioning and question answering are treated as part of the query-focused caption tokens to unify the output format across all temporal localization tasks. The training objectives of structural token generation and query- qucuu-focused captioning can be unified as single language modeling loss:
LIM
Σ
logní X, X,
( 1a Xzn),
(3)
N
where X,, is the n-th token in X, X <n (X,,) W= and N is the total number of tokens in X.
As query-focused captioning is a novel task with no existing dataset, we leverage the ground-truth event timestamps in I.T..nstr (Liu et al. 2024c) to extract event clips, which are then sent to video captioning model to generate detailed clip captions. Further details regarding the generation pipeline presented in the Appendix
3.4 STRUCTURAL. TOKEN GROUNDING
To ground the generated structural tokens to their corresponding video segments, we leverage LLM hidden states to maximize the log-likelihood of the structural tokens with respect their correspond- ing segment frames. Formally, given the projected LLM hidden states (H,}²= and { M (from two learnable MLP projectors following Liu et al. (2024ch) of the video frames and the structural tokens, respectively, where H, E PxC and S; E R formulated as:
the structural token grounding loss can be
M+K &
log (h.
CST +
t
t=
5

---

Published as a conference paper at ICLR 2026
where h RC is spatially average-pooled from H,, is learnable temperature parameter (Rad- ford et al. 2021). and both h, and 8, are normalized to the unit sphere following (Radford et al. 2021 He et al. 2020). The conditional probability p(hr|s:) of frame t given structural token is
computed as:
exp(s.-he/T)
p(h((s;) = E exp(s,
(5)
which essentially makes Eq. (4) contrastvve learning objective (He et al. 2020 Radford et al. 2021
that pulls together structural tokens and their corresponding segment frames. We also attempted the symmetric version of Eq. (4) by including p(s; hy), similar to Radford et al. (0021) but observed
performance drops, for which provide analysis in Sec. 4.4 and thus choose Eq. (4) as the final loss function.
3.5 TRAINING AND INFERENCE
The overall training objective of the video LLM is a combination of the language modeling loss and the structural token grounding loss:
L LLM + LST,
(6)
where LiM and LsT are defined in Eq. (3) and Eq. (4), respectively. We use the E.T. Instruct dataset (Liu et al. 2024c) containing 164K samples augmented with query-focused captions and updated instructions as the supervised fine-tuning dataset. More details regarding the dataset and instructions
provided in Appendix Cnd Appendix G
During inference, the video LLM recognizes the vidco's temporal structure to determine the pres- ence of event and transition segments, and then auto-regressively generates corresponding structural tokens (<ent> and <tst>). Before generating each <ent> token, the model first produces a query-foeusee caption for that event segment. Upon completion of structural token generation indi- cated by the end-of-sentence token <50S>, we compute (( as in Eq. (5) for all frames with respect to each structural token. We then obtain holistic temporal segmentation by assigning each frame to the structural token that yields the highest conditional probability. This directly yields the queried event segments via the event tokens <ent>, The pseudocode for MeCo inference provided in Appendix E.
EXPERIMENTS
4.1 BENCHMARKS
We evaluate MeCo's zero-shot temporal localization performance on three benchmarks: E.T. Bench (Liu et al. 2024c), Charades-STT (Gao et al. 2017). and HVHighlights (Lei et al. 2021).
E.T. Bench is comprehensive benchmark composed of curated data from various public bench- marks for event-level and time-sensitive video tasks. We utilize the grounding. dense captioning, and complex temporal reasoning domains in E.T. Bench for evaluating our method. The grounding domain, with F1 score as the evaluation metric, includes five tasks: Temporal Video Grounding
(TVG) LLei et al., 2021 Gao et al. (017). Episodic Memory (EPM) (Grauman et al. (2022). Tem- poral Action Localization (TAL) Pattraucean et al.]
2023 Gorban et al. 2015; Jiang et al. 2014), Extractive Video Summarization (EVS) (Song et al. 2015 Gygli et al. 2014), and Video Highlight
Detection (VHD) (Song et al.) 2015 Gygli et al. (0144). 4. The dense captioning domain, with F1 score and sentence similarity as evaluation metrics, consists of two tasks: Dense Video Captioning (DVC) (Zala et al., 20233 Zhou et al. 2018) and Step Localization and Captioning (SLC) (Zhukov al. 2019] Afouras et al. [023). The complex temporal reasoning domain, with recall as the eval-
uation metric, includes two tasks: Temporal Event Matching (TEM) (Patraucean al. 2023 Lei
ct al. 2021) and Grounded Vidco Question Answering (GVQ) (Bärmann & Waibel 2022)
Charades-STA and QVHighlights are widely adopted benchmarks for temporal grounding and video highlight detection. The TVG task of E.T. Bench contains samples from both these two bench- marks, but does not evaluate the highlight detection performance for QVHighlights. We thus report additional results directly on the original benchmarks to facilitate comparisons with existing meth- recall at temporal Intersection over Union thresh-
ods. For Charades-STA (Gao et al. 2017).

---

Published as a conference paper at ICLR 2026
Table 1: Zero-shot performance comparisons on E.T. Bench. "SFT. Data" refers the supervised fine-tuning data used in the temporal localization tuning stage, which may include both localization- specific and other video data. Grayed out metrics are not zero-shot results as the model accessed the training data of the corresponding evaluation data in E.T. Bench. Bold and italic only are used for comparisons in the self-implemented Instrut fine-tuning-basse setting.
Madel
SFT Dada
Egeche
ToRA
Gronding
Dasse Captiniing
Comples
TY4p
BPM
TAlay
EVSr:
VHDly
DVC,;
DNC
SLEy,
SLC
TEMa
GVQ
prospred
eneresapps Laretal
MLLAW
Valoo LLaVA
3.0
13.0
113
28.9
25.0
15.0
0.9
9.3
7.5
Vilco Moao-LLaMM2 (2) Car S
0.1
0.0
06
10.0
1.5
0.5
14.5
0.0
15.2
040
0.1
VThred] M (R)L Btere 1044
143K
7.6
1.5
15.2
15MM
76.5
13.4
13.0
3.7
6-4
6.8
1.4
VIT-EE (10
117K
15.9
14.0
70.8
14.4
(158)
22.3
1800
33.8
13.3
31.0
13.3
16.0
9008
68.5
11.1
19.1
65.0
23:2
350
18.1
19.2
0.0
DNqquker (Th) QUR
6198
20.0
17.8
16.1
13:2
Fitersardos
KTNesran
Tummet 5175
31.
QWea VL (7)) wwt.eeppp000
36.7
26.9
17.1
663
41.3
25.3
25.7
15.6
Antiaiool/ fro per-rrissi chodlpoist withost lncalinatinn daaed E.T.ummai Seleoopleo
VITALLM (o0- line J00
20.11
F2.6
19.5
41.6
389
19.6
19.5
11.6
18.2
limoChst (18). Aaree -io0
20.5
2.3
12.7
23.6
43.0
39.4
16.5
18.0
11.8
19.1
0.8
22.3
17.3
23.4
13:2
12.
MoCa (trCCist S307
11.2
32.6
31.2
65.9
41.4
25.3
21.
15.2
23.6
MeCa (ETChai 18)
623
13.4
35.1
35.1
66.3
43.4
2007
16.3
19.1
5/9
MeCi (QWeesVV
34.2
13.8
18.3
18.4
olds of 0.3 (R@ (o.3). 0.5 (R@ lo.s) and 0.7 (R@ (R@lo.7). For QVHighlights, we use mean Average Precision (mAP) and HIT @1 for highlight detection.
4.2 IMPLEMENTATION DETAILS
We develop MeCo using the E.T.Chat (Liu et al. 2024c) WWen2VL-7B (Wang al. 2024b). For ETChat, we use its original Phi-3-Mini-3.88 (Abdin et al. 2024) version and implement a 7B ver- sion based on QWen2-7B (Yao et al., 2024) by following ETChat's three-stage training protocol, where the final stage involves temporal localization fine-tuning with LoRA (Hu et al., (02T) for one epoch. For QWen2VL-7B, we directly fine-tune the pre-trained checkpoint plus the newly initial- ized projectors from ETChat with the temporal localization data. More implementation details are provided in Appendix
4.3
MAIN RESULTS
Comprehensive comparisons. As shown in Tab. I although previous temporal local- ization video LLMs demonstrate promising zero-shot results compared to general video LLMs, they underperform on most tasks compared to MeCo. Specifically, MeCo (3.8B) achieves substantial gains across all domains. Notably, many models use larger base LLMs and train for consider- ably more steps. The results demonstrate that MeCo leverages video LLMs' semantic understand- ing more effectively for temporal localization than boundary-centric methods. Furthermore, when MeCo uses more powerful base LLM (7B), its performance consistently improves on most tasks, reinforeing its scalability and effectiveness.
E.T. Bench:
E.T. Bench: Comparisons with E. Inst.ustt fine-tuning, When fine-tuned on E. E.T.Instruct, VTG- LLM (Guo et al. (025a) and TimeChat (Ren et al.) (024) retain performance levels comparable to their original settings. TRACE (Guo et al. 2025b) experiences the greatest performance drop, likely because its specialized timestamp encoder/decoder and newly introduced timestamp tokens require extensive tuning for effective adaptation the LLM. In contrast, MeCo emphasizes leveraging the inherent semantic understanding video LLMs, making more amenable to efficient fine-tuning.
Temporal Grounding, As shown in Tab. 2 MeCo achieves consistently better zero-shot perfor- mance on Charades-STA compared to either previous methods' official checkpoints or E.T.Instruct- fine-tuned checkpoints. After fine-tuning on the training set of Charades-STI MeCo achieves sig- nificantly better results and retain the best performance for R@ 1o.3 and R 1o.5- However, MeCo prioritizes capturing general semantic differences between query-relevant and background frames

---

Published as a conference paper at ICLR 2026
Table 2: Zero-shot and dataset-wise fine-tuning performance on Charades- and QVHighlights.
Charades-STA
QVHiighlihhs
Model
@ lo.3
R@
R@ lo.7
mAP
HIT@1
Zemt performance (official checkpoints).
VLmeLLLM (7B) (Huang
2024
51.0
27.5
11.4
ViimLLLM (13B) (Huang ct al. 2024
55.3
34.3
14.7
Momentor (7B) Qian
42.6
26.6
11.6
HawkEye (7B) Wang
20244
50.6
31.4
4.5
TimeChat (7B) [Ren et al.
32.2
13.4
23.9
VTG-LLM (7B) Guo ct 2025a
33.8
15.7
33.5
TRACE (7B) (Guo et | 2025b
40.3
19.4
42.7
E.T.Chat (3 (3.8B) (L1U et 20244
64.4
19.4
58.9
Seq2Time (7B) (Deng et al.
12025
31.2
13.7
NumPro-FT (7B)
al.
2025
63.8
42.0
20.6
37.2
VideoChat- T (7B) Zeng al.
2025
69.9
48.7
24.0
54.1
Zero-shot performance (Fine-tumed on E. E.T.nstructt.
TimeChat (7B) (Ren et al.
2024
43.4
24.9
9.2
16.4
30.6
VTGLLM (7B) (Guo
2025a
24.8
9.8
3.5
15.9
27.1
TRACE (7B) Guo et 20256
39,4
23.7
11.5
16.4
30.6
E.T.Chat (3.8B) [Liu
2024c
64.4
43.2
19.4
23.2
58.9
MeCo (FTChat 3.8B)
66.7
44.4
17.5
39.2
61.8
MeCo (ETChat 7B)
69.6
46.4
19.1
39.5
64.3
MeCo VQWen2VL 7B)
71.1
50.1
23.3
37.2
57.9
Dataset- wise fine-tuning performance.
M-DETR [Lii et al. 2021
65.8
52.1
30.6
35.7
55.6
UMT (Liu ct al. 2022
39.9
64.2
QD-DETR Moon cf al.
2023b
57.3
32.6
39.1
63.0
CG-DETR Moon of al. 2023a
70.4
58.4
36.3
40.8
66.7
UniVTG (Lin et 2023c
72.6
60.2
38.6
38.8
61.8
HawkEyye Wang ci al
2024d
72.5
58.3
28.8
TimeChat (7B) [Ren ct al. 2024
46.7
23.7
21.7
37.9
VTG-LLM (7B) Guo et al. 2025a
57.2
33.4
TRACE (7B) (Guo al. 20255
61.7
41.4
VideoChat- (7B) Zeeg et al. 2025
79.4
67.1
43.0
27.0
55.3
MeCo (ETChat 3,8B)
75.3
61.6
38.5
44.6
71.8
MeCo (ETChat 7B)
77.2
63.9
40.1
44.7
74.3
MeCo (Q 7B)
82.3
68.5
41.6
45.3
75.1
rather than modeling dataset- speaase--ppeificc phase-in and phase-out boundary patterns, it may achieve less impressive gains in terms of R @ 10.7.
Highlight detection. As the continuous semantic similarities derived from Eq. (5) can directly be utlized as highlight scores, MeCo achieves much higher performance in mAP and HIT@ for highlight detection than previous methods, most of which generate numeric tokens to approximate highlight scores which struggle to capture the underlying semantic information. Impressively, fine- tuning with the training set of QVHighlights significantly improves MeCo's performance, which even prominently surpasses that of the specialist models. Importantly, MeCo is the only method that achieves a decent balance between temporal grounding and highlight detection.
4.4 ABLATION STUDY
In this section, all experiments are conducted with MeCo (3.8B) unless otherwise specified, and all metrics reported the average across all tasks within the corresponding domain in E.T. Bench.
Semantic-based methods excel; video LLMS amplify. Tab, 3 we compare contrastive vision language models with temporal localization video LLMs. For each contrastive model, compute the cosine similarities between the localization query feature and the frame features (sampled at fps). We then apply threshold to these similarity scores and merge contiguous points above the threshold as localized segments. Contrastive models built on semantic similarities perform impres- sively grounding tasks without additional training. This provides solid proof for the strength of
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparisons with beundcry-centric methods using the same base video LLMs and frame sampling strategies.
Table 3: Comparisons between contrastive vi- sion-language models and video LLMs.
TYori
TAL
Modh
Flu.
CLUP5-11-254 Rattonee bir
15.1
166
19a
313
62.2
127
TineCaal
9.9
VTGLLM
25.7
9.3
McCo
34,3
30.2
17.8
15.7
MeCa
111
66.9
ET 2130 E.Caaaa Li stal. 20246
33.5
29.1
163
8.6
McCo
40.6
35.4
18,
16.6
Table 5: The necessity of <tst> token and query-focused captioning (QFC).
Table 7: Investigation on the compatibility of QFC and different boundary-centric localization strategies, where Positional Embedding is from Ren et al. (2024), Interleaving from Meinardus et al. 2024). and Boundary Matching from Liu
Method
Flye
Flaap
Simas
ReCera
<ent>
26.7
15.0
14.2
<ent> <Let>>
38.1
33.8
20,5
14.5
et al. 2024c
QFC
40.4
19.9
<ent> + Query Copying
26.6
15.2
14.3
9.5
40.6
35.4
20.3
Loc. Strategy
Flyzd
Flop
Simee
Receon
Positional Embedddin
22.8
16.2
14.0
8.7
Table 6: The effect of using different variants of the structural token grounding loss LSt-
+ QFC
14.8
15.1
11.8
8.0
Interleaving
27.1
31.8
15.9
12.4
QFC
23.2
Fl.-
Fluge
Simou
Boundary Matching
33.5
29.1
16.3
Garfeth (ent))ll)))))) (m.))
41.6
QFC
30.5
26.9
18.9
7,9
Calptha Carp(h [x]]] (1)+)+ (a
Structural Tokens
38.1
20.5
14.5
243
13.6
QFC
40.6
35.4
20.3
16.6
setaantccbbsse approaches in temporal localization. By harnessing video LLMs' powerful semantic understanding capabilities, MeCo further amplifies this strength.
Replacing boundary-centric methods with MeCo yields consistent benefits. To isolate the ben- efits of MeCo, we compare it against boundary-ceme centricc methods under their respective settings. As shown in Tab. [ MeCo consistently outperforms the original methods under the same setting across all tasks. Details of the experiments in Tab. are provided in the Appendix
The necessity of both holistic and localized understanding. As shown in Tab. 5 optimizing the structural token grounding loss without transition tokens (<tst>), with segments derived via thresholding, yields significantly poorer performance than when <tst> is used. Notably, <ent> tokens begin to take effect once query- query-focused focused captioning (QFC) is introduced. However, replacing QFC with an uninformative query-cope query-copying task (Guo et al. (025b) reduces performance to the level achieved using only <ent> tokens. By combining holistic structural information via <tst> tokens with localized details from QFC, MeCo achieves the best performance.
Sufficient negative samples matter in LST. We now discuss the observation that adding a "sym- metric" loss term with p(s,(h,) for Lst (Eq. (4) led to performance drops, as mentioned in Sec. 3.4 In p(h, (s:) (Eq. (5). he is a frame-level feature and S; intended to capture a segment. Taking the softmax over frames involves significantly more terms in the summation of Eq. (5) than taking it over the structural tokens, e.g., 100 frames might correspond to only 3 structural tokens. Thus, the losses over h, (s.) and (h,) are not "symmetric" in terms of the softmax. Fewer terms in the denominator of Eq. (5) imply fewer negative samples in Eq. (4), which could negatively affect contrastive learning (He al. 2020), shown in Tab. 6 (row 2). To highlight the influence of neg- ative samples, we replace the frame-level features in p p(h, (s;) with segment-level features by using p( (h, (s:)"))))) where h s8g 1 drastically reduces performance (row 3-4).
पー ΣE h4- This leads significantly fewer negative samples and
Boundary-centtre methods fail to leverage query-focused captioning. As shown in Tab. 7 var- ious methods that focus solely on boundary timestamp generation fail to exploit the rich semantic cues provided by QFC. While the former focuses on phase-in and phase-out changes, the latter em-

---

Published as a conference paper at ICLR 2026
lring Slew fiom of
Fid clee heleagig in de aciee conpary: Shoving sawsiing sirire Parklc coutoter
Provicc Lataled Aocrptice cheep.
<tacc bekkgg Biadd luckical
vitk
the aritan lunbrage and othe vehidkrii methe (eac) <tach
Cat A betiic highighhnn to iostiien fnum
eat A basd lesses. drovng shstien nlea vtber contrxtats
10
(a) Temporal Grounding
(b) Temporal Action Localization
Figure 4: Qualitative examples from temporal grounding and temporal action localization tasks, which involve both mngleeseement and multi-segment scenarios.
phasizes the most relevant semantic cues. In contrast, MeCo's structural tokens effectively leverage this detailed semantic information to enhance performance.
Qualitative analysis. As shown in Fig. MeCo can generate detailed query-focused captions and accurately localize event segments in both single-event and multi-event scenarios. However, there remains considerable room for improvement as there still exist predicted windows that are prominently off from the ground-truth. We provide the visualization of failure cases in Appendix Sec.
CONCLUSION
In this work, we provide a novel perspective of utilizing a semantic-based approach, contrast to previous boundary-centric methods, to enable video LLMs handle temporal localization tasks. Instead of directly fine-tuning the video LLMs to generate boundary timestamps, we propose a semantic-oriented framework, MeCo, better leverage LLM's pre-trained semantic retrieval cap- bility for temporal localization tasks. MeCo is equipped with structural token generation to capture holistic video structures and query-focusee focused captioning tp extract fine- grained event semantics. Facil- itated by the structural token grounding module. MeCo can perform a holistic temporal segmentation of the video, readily yielding the timestamps of the queried event segments. Extensive experiments have proven the effectiveness of MeCo on a suite of temporal localization tasks in zero-shot set- ting. MeCo also achieves impressive performance dataset-wise fine-tuning setting for temporal grounding and highlight detection tasks.
LIMITATION AND FUTURE WORK
Despite the impressive effectiveness of MeCo, we observed that it has relatively lower boost in fine-grained grounding metrics, egg., R la.7- Intrinsically, MeCo prioritizes capturing semantic differences between query-relevant and background frames rather than modeling fine-graine phase- in and phase-out boundary patterns. This reflects an inherent trade-off between semantic-oriented strategies that enable strong zero-shot generalization and boundary-focused modeling that excels at fine-grat ine-grained localization. Thus, we do not claim to totally replace boundary-centrr approaches, which inherently enjoy decent compatibility with the generative modeling of LLMs and can directly model the patterns segment boundaries. Exploring ways to integrate the strengths of both worlds promising avenue for future work. Moreover, believe that the proposed components by no means the only options for semantic-oriented approach.
ACKNOWLEDDGEMENTS
This work was supported by JST ASPIRE Grant No. JPMIAP2502 and JST FOREST Grant No. JP- MJFR2160.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To enhance the reproducibility of our work, we made the following efforts. Firstly, we provided detailed steps of the training data synthesis process in Sec. 3.2 with visual aid Fig. and in the appendix Sec. [ and Sec. G Secondly, we provided detailed pseudo code to facilitate the under- standing of the inference process in Sec. t Thirdly,
provided concrete implementation details in the appendix Sec. Bnnd the annotated code base in the supplementary material, which also contains the generated supervised fine-tuning data in the "data/" folder. Finally, we provided all the evalua- tion instructions in the appendix Sec. as well the evaluation code in the supplementary material.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404. 142199 2024.
Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to vidcos. Advances in Neural Information Processing Systems, 36:50310 2023.
Leonard Bärmann and Alex Waibel. Where did leave my keys?-episodic--memor--bee question answering on egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1560-1568, 2022.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
2020.
Yu- YuiWei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. Rethinking the faster r-cnn architecture for temporal action localization. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 1130-1139, 2018.
Feng Cheng and Gedas Bertasius. Tallformer: Temporal action localization with a long-memory transformer. In European Conference on Computer Vision, pp. 503-521. Springer, 2022.
Andong Deng, Zhongpai Gao, Anwesa Choudhuri, Benjamin Planche, Meng Zheng, Bin Wang, Ter- rence Chen, Chen Chen, and Ziyan Wu. Seq2time: Sequential knowledge transfer for video Ilm temporal grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13766-13775, 2025.
Shangzhe Di and Weidi Xie. Grounded guestion-answerigg in long egocentric videos. In Pro- ceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 12934 12943, 2024.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19358-19369, June 2023a.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp- 19358-19369 2023b.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp- 5267-5275, 2017.
11

---

Published as a conference paper at ICLR 2026
Alex Gorban, Haroon Idrees, Yu-Gang Jiang, A. Roshan Zamir, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar. Thumos challenge: Action recognition with large number of classes, 2015.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18995-19012, 2022.
Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-Ilm: Integrating timestamp knowledge into video Ilms for en- hanced video temporal grounding. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 39, pp. 3302-3310, 2025a.
Yongxin Guo, Jingyu Liu, Mingda Li, Qingbin Liu, Xi Chen, and Xiaoying Tang. TRACE: Tem- poral grounding video L.M via causal event modeling. In The Thirteenth International Confer- on Learning Representations, 2025b. httpss/:: /wop/rpeeeeeeeeeeee net. forum?ide
14fFVOchUS
Michael Gygli, Helmut Grabner, Hayko Riemenschneiderr and Luc Van Gool. Creating Summaries from User Videos. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), ECCV, pp. 505-520, 2014. doi: 10.1007997--3311--0066//
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
/arxiv.org/abset
09685
Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower Ilm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14271-14280, 2024.
De-An Huang. Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed camporra-localization assistant. In European Confer- ence on Computer Vision, pp. 202-218. Springer, 2025.
Yu-Gang Jiang, Jingen Liu, A. Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar. Thumos challenge: Action recognition with large number of classes, 2014.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213. 2022.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp- 706-715, 2017.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9579-9589, June 2024.
Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 10111846-18855 2021.
Bo Li, Yuanhan Zhang, Dong Guo, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03322 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-trate pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19742. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2025.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilavas Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012,, 2023a.
Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Chiin Ching Lin, Ehsan Azarnasa,, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, al. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310. 19773, 2023b.
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univig: Towards unified video-language temporal ground- ing. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pp- 2794-2804, October 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Ye Liu, Siyuan Li, Yang Wu, Chang- Wen Chen, Ying Shan, and Xiaohu Qie Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3042-3051, 2022.
Ye Liu, Jixuan He, Wanhua Li, Junsik Kim, Donglai Wei, Hanspeter Pfister, and Chang Wen Chen. y - Efficient image-to-video transfer learning for video temporal grounding, 2024b, URL
attps://arxins org/ abs/ /404, 00801
Ye Liu, Zongyang Ma. Zhongang Qi, Yang Wu, Ying Shan, and Chang W Chen. Et bench: To- wards open-ended event-level video-language understanding. Advances in Neural Information Processing Systems, 37:32076-324 -200770--210... 2024c.
Weiheng Lu, Jian Li, An Yu, Ming-Ching Chang, Shengpeng Ji, and Min Xia. Llava-mr: Large language-and-vision assistant for video moment retrieval. arXiv preprint M.Nv::2411.15000 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint (rXiv:2306.05424, 2023.
Boris Meinardus, Hector Rodriguez, Anil Batra, Anna Rohrbach, and Marcus Rohrbach. Chrono: A simple blueprint for representing time milmss arXiv preprint arXiv:2406. 18113, 2024.
WonJun Moon, Sangeek Hyun, SuBeen Lee, and Pil Heo. Correlation-guidedd query-dependency calibration for video temporal grounding. arXiv preprint arXiv:2311.085: 2023a.
WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. Proceedings of the 2023b.
IEEE/CVF conference computer vision and pattern recognition, pp.
OpenAl. Gpt-4o system card, 2024. URL http:pp:://avvvv.orr/aba////////
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:3344007193 2023.
Mayu Otani, Yuta Nakashima, Esa Rahtu, and Janne Heikkilä. Uncovering hidden challenges in query-based video moment retrieval. arXiv preprint arXiv:2009.0000 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
13

---

Published as a conference paper at ICLR 2026
Zongshang Pang, Yuta Nakashima, Mayu Otani, and Hajime Nagahara. Contrastive Losses Are Natural Criteria for Unsupervised Video Summarization. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2010-2019, 2023.
Zongshang Pang, Yuta Nakashima, Mayu Otani, and Hajime Nagahara, Revisiting pixel-level con- trastive pre-training scene images. In Proceedings of the IEEE/CVF winter conference applicatioon of computer vision, pp. 1784-1793, 2024.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 16:42748-42761, 2023.
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning.
arXiv preprint arXivv2402.11445, 2024.
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video iims with active real-time interaction via disentangled per- ception, decision, and reaction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24045-24055, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313-14323, 2024.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18221-18232. 2024.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recog- nilion, pp. 5179-5187, 2015.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 11888-11898, 2023.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.0066 240.0000740044 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv: 12191, 2024b.
Teng Wang, Ruimao Zhang. Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6847-6857,
2021a. 2021a.
Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Ilms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02889 2024c.
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for sulf-supe visual pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3024-3033, 2021b.
14

---

Published as a conference paper at ICLR 2026
Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang. Qun Liu, and Dongyan Zhao. Hawk- eye: Training video-text Ilms for grounding text in videos. arXiv preprint arXiv:2403.1022: 2024d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. ohain-of-thouuht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13754-13765, 2025.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava Parameter- free llava extension from images to videos for video dense captioning, 2024. URL https:
16994
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13633, October 2023.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10714-10726, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, et al. Minicpm-v: A gpt-4v level milm on your phone, 2024. URL pettps://arxiz:
01800
Liping Yuan, Jiawei Wang. Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding. arXiv preprint arXiv:2501.07888, 2025.
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mo- hit Bansal Hierarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23056-23066 2023.
Andy Zeng. Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint wrxXvv2204.00599 2022.
Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, Yali Wang, Yu Qiao, and Limin Wang. Timesuite: Improving MLLMs for long video understanding via grounded tuning. In The Thirteenth Intera- tional Conference on Learning Representations, 2025. URL
.et/
forum? d=nAVVe JJURZZ)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986. 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuna audio-visual language model for video understanding. arXiv preprint arXiv:2306.022 028558 2023.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv preprint arXiv:2406. /6852, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, 2018.
15

---

Published as a conference paper at ICLR 2026
Dimitri Zhukov, Jean-Baptist Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 3537-3545, 2019.
16

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the drafting and revision of this manuscript, we used GPT5 and Claude Opus 4.1 for checking grammatical errors, formatting the LaTex code for tables and figures, and getting suggestions on academic writing.
B IMPLEMENTATION DETAILS
Following E.T.Chat, we perform three stages of training. Throughout all three stages, the visual encoder remains frozen while the frame compressor and projection layer are trainable. In stage 1, we additionally freeze the Q-Former and LLM. In stage 2, we unfreeze the Q-Former and train the LLM with LoRA. In stage 3, we conduct fine-tuning by freezing part of the Q-Former, initializing new LoRA module for the LLM, and initializing two MLP projectors for the frame and structural token hidden states, respectively, for the structural token grounding module. Following (Liu et al.) 2024c), the structural token hidden states extracted from the second-to-last LLM layer and the frame hidden states from the last layer. Table X provides the detailed hyperparameter setup used for performed 4 NVIDIA A100-806 800 GPUs.
stage-3 training. All training
We develop MeCo using the E.T.Chat architecture (Liu et al. 2024c), which employs a pre-trained ViT-G/14 from EVA-CLIP (Fang al. 2023b) the visual encoder and resampler consisting of pre-trained instruction-conditinee Q-Former (Li et al. 2023) followed by frame compressor (Liu 2024c) that produces one token per video frame. A projection layer projects visual tokens
al.
into LLM inputs. The QWen2 base LLM (Wang et al. (024b) is from MiniCPM-V-2.6 (Yao et al.) 2024 We also implemented MeCo with QWen2VL-7H Wang et al. (2024b) by mounting two newly initialized projects from ETChat. For pre-training, we format the structural token sequence based on the temporal order of event and transition segments. For dataset-wise fine-tuning, we observed that the temporal order of structural tokens may inherit the dataset-specifie biases in the temporal positions of ground-truth windows (Otani et al., 2020). Therefore, we relax format of the structural token squence by simply appending single <tst> token to the end which attends to all the transition frames during training and inference.
Table 8: Hyperparameters for stage-3 training.
MLP Projectors
Number of Layers
2
Hidden Size
1536
Output Size
3072
Large Range LoRA
LoRA r
128
LoRA
256
LoRA Dropout
0.05
LoRA Modules
QVO Layers
Model Training
Max Number of Tokens
2048
Number of Epochs
1
Batch Size
2
Learning Rate for LoRA
5e-5
LR Decay Type
Cosine
Warmup Ratio
0.03
Optimizer
AdamW
AdamW 31, 32
0.9, 0.997
17

---

Published as a conference paper at ICLR 2026
C QUERY-FOCUSED CAPTIONING
Based on the temporal localization data in E.T.Instruct (Liu et al.) 2024c). we extract event segments and input them to video captioning model, MiniCPM-V-2.66 Yao et al. 2024). to generate detailed captions. As these captions often contain redundant information, we summarize them using GPT-4o- mini (OpenAl 2024) under the condition that the final captions are more detailed than the original localization queries. The annotation pipeline and representative QFC examples are shown in Fig.
D ADAPTING TIMECHAT AND VTGLLM TO WORK WITH MECO
In Tab. we show the results of integrating MeCo into the TimeChat (Ren et al. 2024) and VT-
GLLM (Guo al. 2025b) models. TimeChat and VTGLLM share the same architecture, with ViT-G/14 from EVA-CLIP (Fang et al., 2023b) as the visual encoder, pre-trained Q-Former (Li et al. 2023) as the visual resampler, and Lade (Zhang et al., 2023) as the base video LLM. The key difference is that TimeChat applies a sliding video Q-Former to compress the visual tokens to 96, while VTGLLM applies slot-based visual compressor obtain 256 tokens. Both use 96 as the maximum number of sampled frames.
To integrate MeCo into this architecture, we modify the video Q-Former in TimeChat to a standard image Q-Former, which resamples 32 tokens to token per frame. Additionally, apply bidirec- tional self- attentinn to the visual token components, following (Liu et al. 2024c). Other components remain unchanged. We then train TimeChat, VTGLLM, and McCo (adapted) on E.T. using the same hyperparameters as (Ren et al. 2024).
E MECO INFERENCE
We provide the pseudo code for MeCo's inference process in Algorithm | Specifically, after the LLM finishes its generation, we first extract all the structural tokens from the LLM's generated token indices and extract the indices of the event tokens (<ent>) ent_idees in the extracted structural token list. We then obtain the frame embeddings fr emb and the structural token em- beddings st emb by extracting the I.LM hidden states and feeding them into the MLP projectors. The pair-wise cosine similarities between the two sets embeddings are then calculated, which can then be normalized by softmax () operation along the temporal axis obtain the conditional probability p_hs in Eq. (5). Each frame is then assigned to the structural token that leads to be highest conditional probability using the argmax () operation. For each <ent> token, find all the frames that have been assigned to it, merge temporally consecutive frames into segments by split at gaps, and add the obtained segments into ent segs (we observed that sometimes the LLM tends to represent multiple semantically-releaan segments by a single <ent>>.. The start and end timestamps for each segment are simply the timesatmps of the first and last frames in the seg- ment. We do not conduct post-proessing for cases where the segment timestamps do not follow the appearance order of their corresponding <ent> tokens. The above pseudo code assumes the most common case where all the event segments non-overlgp for clean demonstration of the in- ference process. In the actual implementation, we process each <ent> > token one by one and assign frames to each <ent> token after suppressing its adacent <ent> tokens' (if any) frame scores to take care of overlapping cases.
F FAILURE CASES
In Fig. 6 we visualize several typical falure cases: (a) the <ent> token attends to the totally wrong segment; (b) the <ent> token attends to the correct semantic information but the boundaries are not well estimated by the <tst> tokens: (c) the model only generates actually three; (d) the video starts directly with the event segment while the model considers the first few frames transition frames and thus generates <tst> token at first.
events while there
18

---

Published as a conference paper at ICLR 2026
Algorithm Pseudocode of MeCo Inference.
Ez_ eeme frame embeddings (IxC) at omb: structurai token emboddings (1848) xC)
tau: temperatere
ent_idces: indices of event tokens <ent>
calculaee conditional probablittien (Rq. (51)
21: -22 www.emmmleeeeee (C
12_nermalize (st_eem))
mataul (zfz, zat.1) (X-H)
2_h5 = softmax (nim tas, dimm0) $ Eg. (5)
Assign frames structural
assign p__hsarrmaxe (iimmml (dim=l)
ent sega []
(ede for i, idx in emmeratee (ent, idces):
Get indices where assign == 1dx
indices wherelassign
idx)
Split at discantinuitics to segments
segs spilt_at__apa
Get start and end timestamps
timestampa [[seg[0], seg[1;] for in segs]
Add to the segment 1ist [tt__egs.extendlssee [ite_a, te_ll, [te_2, te_21, ...]
TaTEorE lourarg
Query-focusee captisns
Quagy (swry) Segenee
J0NY de WN noander
groaney
Terpora Gosloater
[S021]
regowve soneento cions
Plynng Andrattinn
[3602]: compoctiv
cosciee foourey de sour enportent pay reltted Sueool
LLM
awwiloood movements preces
match..
Vikoo bdlt Dnfectee
[8611]: sker jocket descesds a trewy skpp,, sulll di ravegating
segmanieii cantions
Zelg
lutta as pondery arow so aia with eacr alsap cave down the recartalss
Decrde vidaze fecossiion
[500]] sue propent load by plaoing a plats on the countaiiss and thx
Caption Mord
Wher pore?
ad5ing an 922 to
[S8G]]: Affectorats interactinss betweer persen and Highight
P Bdo
eratdd Sonwectc setting
[deG2): A light usieral out lamt nu bock to the camare wii atrped out fexs riht. daker eat fres contert recing apandt ther
[38G1]: acers snow person cooking stove varioas Uheralaa und ingendienta around hand. perscn using chopaticts
Hivi dkd sex wih?
pots, indicated by the novemee choperibee
comient
Figure Query- captioning pipeline and examples.
G EVALUATION AND TRAINING PROMPT TEMPLATES
For evaluation, we modify E.T.Bench templates to work with MeCo. Example templates are pro- vided in Figure 7 For training, we manually craft a query-focused captioning-aware instruction template for each task domain in E.T.Instruct and diversify it with GPT-4o (OpenAll 2024) to gen- crate four additional templates. The instruction templates for all domains are provided Fig.
19

---

Published as a conference paper at ICLR 2026
Fisd dlip descrieed by the sereacer wiss persos oat food out of logs doscrption the
Fird cip deserbed hod by the seneece the persoe eatr food ofinffo fogi Provide a detilld cescittion the dipp
Left -1BEN
Right AEBE
Viad suprert
aa) Wrong attention of the <ent> token.
(b) Wrong enclosing of the <tst> token.
bekonging xctin extpgyy:
deaathee semmene peron rven donce a Axstrr descigtion of dllp
201p
(c) Wrong estimate of the number of events.
(d) Wrong prediction of the <tst> token.
Figure 6: Visualization of figure cases.
Given 'Vvidoo video, find described the 'Iqueyy" Provide detslled description of
Epoodic Wemory
Given a '[video domain] video, fin a dip, that can answer the question: "Iquery"" Provide detailed description the cllp and give the answer.
Highiggt
'(vidco domain)" highlgght moment, according sentence: '(uurry".
summary the moment.
Extractive Video Summarizztin
synopals
Temporal Action Localestion
Eiven i Vivideo demaan?" video, find all the dips bwiongnng to the aution categary: 'iqurry". Provids corciss description for nech clp.
Dense Video Captioning
Given '(vddo domain]' video, densely summaras the video by localzzng that depict major activity events. Describe event focusing
Locallaation
Given a 'Vvddo domain]" video, localize scries citss that contain the major stops in the viseo and briefly descrite be each clpp
Temporal Event Matching
Gieen al 'videa domain! video, kientify the setion in ovids ovids and lucolice smother muxt simbe actiorn Provide detalied description the action.
Figure 7: Evaluation prompt templates.
20

---

Published as a conference paper at ICLR 2026
Toriporal Video Brunding
Vicco Highlghi Dcteciion
Locatze the dip deaotited by the goven reatual query' 'stweyl' in video and provide a more dotalled description of the dip.
representatvv dip video that contain hignlght mement depicted by the given query 'Iqueeyl" and provie a discrppius to help the use understand
givan quary 'Quaayl'
what hoppening in HiMlight
in the vicco and provide a desoriston with more relevant tomsoral and spatial deta le.
Identity reprewntaa Highlight dip in the sideo commezoncc the query
"oueyy" and provide a detalled description rich in importane visual activiies for the
by quary in the vadeo and describe it with more important debels"
querry 'ppaeyf'
Guenn video and cuery 'sowry'', a representstive highlghh do video that corroponds the and descr be it with recessory detal's relited to the undwerarding
Locakze the dip related the testual "Iquery" the video
onhonce the descriptio by adding more tomporal spenial detalls relevant
qvery
to the query.
Find representative Highight dip in the video that comresonns the query "sawry!! and wirch the query
Find vides comesponding toxtual query "Iqueryl" and
relevart temporal and debill
the suery with aobiboool but relevuntt temporal and sputial
faciitotn bestor understondng.
Locoliie regrosetative highight dip in the video that comragonds to the quary "owwryy' and deccrbee focusing higllgh--owiitt debile enfonce the
Extrocevv Vicco Summa lcation
user's expanerce.
Summarize the video into brief synocci focusing on the closs that Hentily cipp the video thet comben the most importeen vical caes relatted to the video doman 'iquryl' and provile conc syropes of the
Terppraal action Lncetoetien
"Iquary"
Localize of the clss rolazed to the human action "quaryl" and provids concie deecrpp for each them.
identify the dipa that depict the human action 'Juuryy' the video prov for wach of them a condoe descepption.
Givon video and a vidco domain 'lourrs!' summariz the dips that contain the most pap tant voual cass generate concse syropas of the video. Summariz ze the video into concse sytops's focusing on cips the most impor tant voual case neated to the video dumann 'iqaany":
For the given action 'loueyfi localize
conresonningg cips the video
decrmbe each af them by fonusing on actioneeiated viuol actitttes.
Extract all the dips in the vidzo that depice the action "lauery"" and provide
dip.
Summorze the video into concie syropes lncalosing the most important cips with weaal cane releted to the video domman' (qpuryy:
Gven a video an action 'iqueryl', all the cipp the video that clep the action and cancusly descmbee each ckp.
(a) Temporal grounding.
Dence Video Carticning
Stop Locol and Copcooiing
Denealy summariee the video by loceleng claa that copect a scrica major setiity everts For each dip, firs provide debaled description and then gave triet
Locakiee a serie of cipw the contann the major stape the video. First provide desailed explonation each summary.
then provide
Denscly summarize the videe's storyine by describing the dips contaiingg major everts in tomporal onder. For each cioo provide a desalted description followed by cornesponding brisf aummary.
Summarow the video by localcnng ches that desect saries mayer state. For each provide detaile oxglanation followed by lorel summary.
Dipture condowy
idwmiiy
aocuring
a decaltd description for each event and then summarize it briefl.
provide desiled explonation and then summoriz it briefy.
Veinity the dipp thet comsin the major events in the video and triefly deccribe sech tomporal
Locate the dlos involvnng soquentill el series of stops the vidoo, For cuch step, provide
Provice detalles
dli followed
summary.
bnst summary.
Summarue major stape she video. Provide detaied axplanation wxch step foloaed by leriet summany..
Prouide deese cantions the wide viea by provitigg oequenos of (dersaed dip
descrlttor,
pois the
(b) Dense video captioning.
Grounded Waice Opeation Arsncciig
Gronnedd Vilco Spestion Aronctiny
Watch video caretully with the qumntinn 'iquers' mind and the cip thet
contains the arowar Provide your anubyia of the die that supports your anowe. and finally provid the answer itsef".
Densely capture all the clss thet construst the storylne of the video For each din provide detaled decription then summorin briety.
Identity
"jauers"
video. for each event, provide detailed desoription and thas summarize it briefy.
Donodl the evers
the arower.
Givon video and question "iquerl"' find the dlo the vidco that heps answer find arswer the and yiualy provide the
Derocty identlly the events the vicco. For each ovent, provide deteilod and than
tall
briefty
Lookzze the dip the video that containe the contect needed amswer the questin "suary"' mavigate through the dip and provede the snswer to the
major evenss. provide dctallss descrittion and then summaatce briefly.
Dennely summarie the
focusing
Denocly identlly the cipo that depict the mejor events the vidoo, For each die detaledd desorption
qussion.
summorien briefly"
question "jusrys'
andyya the dip and provide the answer
(c) Complex reasoning.
Figure& 8: Instruction templates for different task domains: (a) temporal grounding, (b) dense video captioning, and (c) complex reasoning.
21


Reviews:
[{"review_id": "X3pTiOjKxo", "reviewer": "Reviewer_c33n", "summary": "This paper presents MeCo, a semantic-oriented framework that adapts video LLMs for temporal localization by replacing direct timestamp prediction with structured generation. MeCo first emits structural tokens—<ent> (events) and <tst> (transitions)—to outline video structure, augments each event with query-focused captions to enrich semantics, and then grounds tokens to segments using a contrastive objective. Evaluated on E.T. Bench and standard temporal localization benchmarks, MeCo delivers strong and consistent performance.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) **Motivation & insight.**\n   The paper makes a compelling case that asking LLMs to emit *uninformative numeric timestamps* underutilizes their semantic reasoning. This observation is clearly articulated and convincingly motivates a *semantic-oriented* alternative.\n\n2) **Strong, broad improvements.**\n   MeCo delivers substantial gains across diverse tasks—notably **59.1% vs. 38.6% F1 on TVG (E.T. Bench)**—with **consistent improvements across nine tasks**. It is especially effective for **multi-segment** scenarios (e.g., **26.2 mAP vs. 1.5** for E.T.Chat on QVHighlights).\n\n3) **Comprehensive evaluation & fair comparisons.**\n   The experiments span multiple domains (grounding, dense captioning, complex reasoning) and benchmark suites, with thorough comparisons against numerous baselines under matched, transparent settings.", "weaknesses": "1) **Incremental novelty via integration.**\n   The triad—**structural tokens**, **query-focused captions (QFC)**, and **contrastive grounding**—is well motivated but each component echoes established ideas (tokenized structure, CoT-style captioning, CLIP-like contrast). The contribution lies primarily in a **reformulation and clean integration for localization**, rather than a single, fundamentally new algorithmic primitive.\n\n2) **Temporal granularity concerns.**\n   Real videos exhibit **richer, nested structures** (actions ↔ sub-actions, scene changes, shot boundaries). A binary **event/transition** scheme may be too coarse: how are **gradual transitions**, **overlapping events**, or **multi-threaded activities** represented?\n\n3) **Caption overhead vs. payoff.**\n   Generating and then attending to QFC introduces **non-trivial compute/latency**. How much of the gain is attributable to **QFC** versus **structural tokens** alone? While Table 5 indicates both matter, the **interaction and marginal utility** (A → A+QFC, A → A+QFC+contrast) aren’t dissected in depth.\n\n4) **Failure-mode analysis is thin.**\n   Beyond noting “prominently off” cases (Fig. 4), there’s no **systematic breakdown** of when/why the method fails (e.g., heavy occlusion, rapid camera motion, fine-grained action boundaries, dialogue-driven cues). A short taxonomy with **representative examples** would be valuable.\n\n5) **Limited sensitivity studies.**\n   Only **β** and **λ** receive analysis (Fig. 4). Important **design knobs**—**NMS threshold** (0.7), **temperature τ**, **projector architecture/width**, **negative sampling strategy**, **frame sampling rate**—are not explored. A small **hyper-sweep and robustness table** would strengthen claims of generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "86xjNqGb1V", "reviewer": "Reviewer_HVDc", "summary": "The paper introduces MeCo (Measure Twice, Cut Once), a semantic-oriented framework for video temporal localization using Large Language Models (LLMs). It proposes a conceptual shift from previous methods that rely on generating uninformative boundary timestamps, which often fail to fully leverage the LLM's pre-trained semantic understanding capabilities.MeCo fine-tunes a video LLM using a combination of two generative and one discriminative learning task:Structural Token Generation: A generative task where the LLM partitions the input video based on the user query by outputting a sequence of new, special structural tokens.Query-Focused Captioning (QFC): A second generative task that requires the LLM to produce a detailed caption for each queried event segment immediately. Structural Token Grounding: A discriminative task, implemented using a contrastive learning objective, that maps the semantic information encoded in the structural tokens to the corresponding video segments. Extensive experiments demonstrate that MeCo consistently outperforms timestamp-centric approaches across diverse temporal localization tasks.", "questions": "na", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The paper is written with good clarity.\n2.The motivation is clear\n3. The framework is well-presented", "weaknesses": "1.The structural token generation relies on Supervised Fine-Tuning (SFT) labels that require a contiguous, non-overlapping segmentation of the entire video, achieved by augmenting Ground Truth (GT) event boundaries with neighboring transition segments ($\\langle tst \\rangle$ tokens) to cover the full duration. This presupposes the existence of exhaustive, high-quality, segment-level annotations for the whole video.\n2.  The current asymmetric Structural Token Grounding loss introduces a significant imbalance: the number of negative frame samples for a structural token $s_i$ is $T$ (total frames), whereas the number of negative structural token samples for a frame $h_t$ is only $M+K$ (total segments)28. This massive disparity in negative sample cardinality could lead to the loss function being overly dominated by the frame-level discrimination task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wpYcQlzGWp", "reviewer": "Reviewer_Sykj", "summary": "The paper introduces MeCo, a framework for temporal localization that replaces timestamp generation with structural tokens: event <ent> and transition <tst>. It augments these tokens with query-focused captioning (QFC), generating detailed captions before each <ent> to refine event semantics. A contrastive structural token grounding module aligns tokens with corresponding video frames to achieve holistic temporal segmentation. Trained on E.T.Instruct along with QFC data, MeCo has strong zero-shot performance across grounding, dense captioning, and complex reasoning tasks.", "questions": "Please mainly see the weaknesses section above. In general I will give a borderline reject rating.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I think the paper has proposed a new shift from timestamp prediction to semantic segmentation with the help of structural tokens (<ent>, <tst>) plus query-focused captioning (QFC). \n\nAlso, the author present some results under self conducted experiments with fair comparisons to demonstrate the effectiveness of the proposed method.", "weaknesses": "However, I have some severe concern about the paper:\n\n1. Unfair comparison between the proposed method and previous methods. Since the author utilized extra model (MiniCPM-V-2.6) to build the training set, dense captioning will bring more labels to the training set. Thus compared to those timestamp prediction models, it is naturally that the proposed method will have better results. Thus, I think the author should compare previous models with their best results, rather than some complex settings. \n\n2. Critical comparison are missing including TimeChat and TimeChat-T series model. Also, for Charades-STA and QVHighlights, What if the proposed method also fine-tuned on target dataset, rather than the E.T. Instruct dataset? I think the performance gap will be minor or none. \n\n3. the method shows smaller gains for action-focused queries, and the paper acknowledges this limitation without a concrete explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oBEozo4ZJ9", "reviewer": "Reviewer_NFTY", "summary": "The paper proposes MeCo, a semantic-oriented approach to temporal localization with Video LLMs. Instead of directly predicting boundary timestamps, MeCo first produces structural tokens, which contain an event token (<ent>) and a transition token (<tst>), to segment a video. Before each <ent>, the model inserts an event caption to enrich semantics. During training, each structural token is grounded to its corresponding temporal segment via a contrastive objective. Experiments across grounding and dense captioning tasks show consistent gains over timestamp-centric baselines.", "questions": "Please see the weakness part. Particularly, what were the reasons for omitting recent baselines (e.g., DisTime, LLaVA-MR, UniVTG, Mr.BLIP)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Effective grounding objective.** The contrastive grounding loss (Eq. 4) is well-aligned with temporal localization and empirically appears to improve boundary precision.\n2. **Strong zero-shot generalization.** Trained on E.T.-Instruct data, MeCo reports competitive or superior zero-shot performance across grounding and dense captioning, indicating good transfer without task-specific fine-tuning.\n3. **Ablations support design choices.** The ablation study isolates the contributions of structural tokens and event captions, showing both components are necessary for the full effect.", "weaknesses": "1. **Missing implementation details.** The paper presents an inference approach but does not fully specify how timestamps are derived from selected frames or how invalid sequences are handled (e.g., if the index chosen for the first <ent> exceeds that of the second <ent>, whether re-ordering or smoothing is applied).\n2. **Limited comparisons to recent methods.** The paper primarily compares with earlier methods (e.g., TimeChat, E.T.Chat). ** More recent temporal grounding approaches, such as DisTime, LLaVA-MR, UniVTG, or Mr.BLIP, are not included. On Charades-STA in particular, several of these report strong results. The paper should either compare these approaches or discuss why MeCo lags behind them.\n3. **Title–method mismatch.** The metaphorical title “Measure Twice, Cut Once” does not clearly convey the core technical contribution (semantic structural tokens + contrastive grounding), which could be confusing for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)