Aha! - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
Authors: Aiden Chang
Celso M de Melo
Stephanie M. Lukin
Venue: NeurIPS 2025 poster ()
Keywords: Online Highlight Detection
Real-time Video Analysis
Vision Language Models
Primary area: deep_learning
TL;DR: Aha! enables real-time, autoregressive highlight detection in continuous video using language tasks. Using a VLM and a Dynamic SinkCache for constant memory, it achieves SOTA on benchmarks & shows robotics potential.

Abstract: Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet, most existing video understanding and highlight detection methods assume access to the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-step reasoning needed for real-time decision-making.
We introduce Aha, an autoregressive highlight detection framework that predicts the relevance of each video frame against a task described in natural language. Without accessing future video frames, Aha utilizes a multimodal vision-language model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To enable scalability, we introduce the Dynamic SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation to capture high-level task objectives, enabling effective frame-level rankings for informativeness, relevance, and uncertainty with respect to the natural language task. Aha achieves state-of-the-art (SOTA) performance on highlight detection benchmarks, surpassing even prior offline, full-context approaches and video-language models by +5.9\% on TVSum and +8.3\% on Mr.Hisum in mAP (mean Average Precision). We explore Aha’s potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video. Both experiments demonstrate Aha's potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.

AHA Predicting What Matters Next: Online
Highlight Detection Without Looking Ahead
Aiden Chang
University of Southern California
Los Angeles, CA 90089
aidenchang@gmail. com com
Celso De Melo
Stephanie M. Lukin
DEVCOM Army Research Laboratory Adelphi, MD 20783
DEVCOM Army Research Laboratory Adelphi, MD 20783
Abstract
Real-time understanding of continuous video streams essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet. most existing video un- derstanding and highlight detection methods assume access the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-sta step reasoning needed for real-time decision -maiing.. We introduce AHA, an autoregressive highlight detection framework that predicts the relevance of each video frame against task described in natural language. Without accessing future video frames, AIIA utilizes multimodal vision-language model and lightweight, decoupled heads trained on large, curated dataset of human-centric video la- bels. To enable scalability, we introduce the Dynamic SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation capture high-level task objectives, enabling effective frame-level rankings for in- formativeness, relevance, and uncertainty with respect to the natural language task. АнА achieves oftte-of-the-ar (SOTA) performance highlight detection bench- marks, surpassing even prior offline, full-context approaches and ladgu-la models by +5.9% on TVSum and +8.3% Mr.Hisum in mAP (mean Average Precision). We explore AHA's 's potential for real-world robotics applications given task-oriented natural language input and continuous, robot-centric video. Both experiments demonstrate AHA's 's potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.
Introduction
Real-time understanding of continuous video streams is crucial for intelligent agents operating in high- stakes environments, from autonomous vehicles and surveillance drones to field-deployed robotics in disaster relief scenarios (1-4]. Despite this need, while earlier explorations into Online Highlight Detection (OHD) existed (c.g.. using LSTMs (5)). the trajectory contemporary HD research, particularly leveraging powerful modern transformer based architectures, has overwhelmingly cen- tered on offline, full-context processing (6-8) Even approaches incorporating task-conditioning via
"Conducted research fellow the DEVCOM Army Research Laboratory. gillub.com/aidee220//Al
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

natural language queries operate offline assume the entire video is available during inference [9]10] This fundamental reliance on full-context renders existing HD methods unsuitable for streaming applications requiring step-by-step reasoning and immediate action based on unfolding events.
Concurrently, a separate area of research has explored streaming video analysis, often leveraging Large Language Models (Video-LLMs) for tasks like dense video captioning or generating dialogue responses about ongoing events [ [2]. While some of these models have explored HD as an auxiliary capability, their application to OHD faces significant limitations. These Video-LLMs often necessitate modifications to standard HD benchmarks, employ post-hoe smoothing techniques that violate strict online constraints by implicitly using future information, and ultimately yield suboptimal HD performance [13. This leaves a critical gap: robust method designed specifically for accurate, online, dusk-conditiones highlight detection on standard benchmarks.
We address this gap by introducing novel framework built for OHD. We define OHD as the method of analyzing a streaming video by observing frames strictly one at a time and, for each current frame, predicting its highlight score using only past and present information, without accessing any future frames. This sequential, causal processing is fundamental for enabling real-time decision-making in dynamic environments. Given natural language task description, our model, AIIA, performs OHD by employing a lightweight, autoregressive scoring mechanism focused directly on highlight detection. This allows AHA to operate effectively on traditional HD benchmarks in truly online fashion, without requiring benchmark modifications or non non-causal smoothing, and achieving SOTA performance even zero-shot settings. Our main contributions are:
We propose AHA, an autoregressive framework featuring lightweight prediction heads (scoring relevance, informativeness, uncertainty) and our novel Dynamic SinkCache memory for efficient, constant-cost, OHD under natural language conditioning, and a video quality dropout mechanism enhance robustness against real-world noise.
AHA Framework for Efficient OHD:
A Large-Scale Dataset for OHD f We construct and release the Human Intuition Highlight Dataset (HIHD)) novel dataset of -23k videos incorporating user engagement signals and task-driven captions, specifically designed to train and benchmark task-conditioned OHD models.
SOTA OHD: AHA surpasses prior methods, including offline approaches, on the HD benchmarks TVSum 14 (+5.9% mAP) and Mr.Hisum [15] (+8.3% mAP). We validate AHA'S robustness and
real-world applicability through comprehensive experiments, ablations, and on challenging long- horizon, noisy robotics video from SCOUT [66, demonstrating task-relevant understanding where offline processing is infeasible.
2 Related Works
Offline and OHD, HD research, especially with modern architectures, has predominantly focused on offline, full-context processing. Techniques evolved from early handerafted features deep attention models [17119 10]. but fundamentally require offline access. These methods, while achieving strong offline results, require bidirectional temporal access, making them unsuited for streaming. To the best of our knowledge, one of the few recent attempts at dedicated OHD using sequential models was Lal et al. [5] with LSTMs; yet, the challenge of frame-wise highlight prediction under strict online causality remains mostly open.
A central and persistent challenge in HD is the difficulty of obtaining labels that accurately reflect what constitutes a highlight, and, critically, generalizing this understanding across diverse video domains and content types 20 15). While early benchmarks like TVSum [4] offered rich but small-scale human annotations (50 videos), later datasets like Hisum
[5] leveraged large-scale user engagement signals (e.g., replay spikes) for scalability and capturing broader interest. Their underlying hypothesis, which we also explore, is that these large-scale engagement patterns effectively capture moments of high viewer interest that align with worthy content, which correspond to human intuition. While existing datasets capture this intuition, robust OHD for diverse, raw streaming often hindered by limitations such as the small-scale of benchmarks (c.g.. TVSum) or, even in larger collections, by 'clearer' signals and content less representative of the variable quality and uncurated nature of many live streams. Building on prior theory to address these gaps, we introduce a new large-scale dataset that utilizes engagement-styee signals akin to these datasets about
"The instructions how download the dataset will be included in the github.

---

human intuition, but is distinctively curated for broader visual quality variance and explicit support for condiiondt learning.
Streaming Video-language Models. Recent Streaming Video-Languuae Models (Video-LI.Ms) TO
12 11 21] have significantly advanced multimodal reasoning for streaming video, offering archi- processing (c.g., StreamingLLM [21]. Token Turing Machines (221), which inform our work. However, their direct application to robust OHD reveals limitations. These models often prioritize interactive tasks (e.g., dialogue, VQA) over the continuous, fine- mme-grained scoring essential for OHD.
tectural inspirations, particularly in
Consequently, when highlight detection is addressed, is often as an auxiliary function, with modified evaluation criteria that is not aligned with traditional HD benchmarks strict online constraints, leading to performance that can be suboptimal for specialized HD 13 Moreover, their evaluations often use short clips, rarely demonstrating sustained OHD performance long videos (>10 minutes).
ÁHA diverges from this trend by being specifically architected for efficient, high-performance OHD, It adapts memory-efficient streaming concepts for the video-language domain by employing task- aware scoring heads that process video as a continuous stream against a persistent task. Crucially, AIIA demonstrates its effectiveness not only by outperforming traditional highlight detection bench- marks under strict online settings but also by maintaining robust performance on long-form videos, addressing key gap in current streaming literature.
Uncertainty-Aware Online Modeling Unlike HD models offering deterministic scores, AHA incorporates an uncertainty head drawing from probabilistic sequence learning (33 to model predictive uncertainty, crucial for online settings with limited context. our knowledge, AHA the first application of explicit probabilistic uncertainty modeling for task-conditioned, frame-level OHD,
Vision Encoder (SigLip) MultiModal Projector
Frame
Tokens
LM
Video Stream
Loss
44
NLL Loss
(Decoder)
Uncertalntty
Vid start
LoRA
SmoothL1
token
Loss
Color Key
fi
Relevance
System Prompt
Trainable Layers
BCE Loss
Objective
Layers
Informativeness
Textual Encoder
Figure 1: AНA architecture, showing the flow from video stream and text prompts through the visual encoder, multimodal projector, decoder, to the multi-objective prediction heads.
3 Methodology
We treat task-conditioned OHD in streaming video as follows: given continuous stream of video Jt).
frames {Jo, Ji,..,,,e the goal at each timestep / is to predict sealar highlight score ÿe for the current frame ft, indicating its relevance and an optional system prompt S {*1.-.., 8m are provided once at the beginning of the video and remain fixed throughout inference. At each inference timestep, the model observes (1) the current frame ft. (2) the fixed task and system prompt embeddings Q and S, and (3)x a memory mechanism (e.g., KV cache (26)) that stores previously computed tokens for efficient autoregressive decoding. The model must emit a scalar highlight score ÿ, without access to future frames, full- sellseeqeece context, or bidirectional attention, enabling real-time low- latency operation. During training, instead of a KV cache, the model uses fixed-length window of preceding frame and text tokens. This constrained context ensures the model learns operate under streaming conditions while still allowing for full gradient flow. Auxiliary objectives (e.g.. language modeling or captioning) may be incorporated to enhance semantic representations, the training regime mirrors the causal constraint: the model sees only the current and past tokens within a limited window.
user-specified natural language task objective. This
objective, Q = (91....
3

---

To solve this OHD problem, we propose AHA, lightweight autoregressive framework built on recent advances in streaming multimodal LLMs. Its architecture (Fig. I consists of four key components: (I) A Frozen Visual Encoder (pretrained SigI.I (27 extracts frame features, facilitating generalization without visual fine-tuning [13 12 (2) Minimal Multimodal Projection, a single linear layer that maps visual embeddings to the LLM token space for fast per-frame tokenization. (3) A Token-Level Autoregressive Decoder, a decoder-only transformer [28] 29), processes interleaved text (including S and Q initially) and visual tokens in unified sequence, enabling continuous, single-pass decoding for streaming inference. (4) Multi-Ohjective Prediction Heads (relevance, informativeness 13], uncertainty) added top of the decoder's final hidden layer h to capture frame-level semanties for HD, which are then combined to produce ÿt. An auxiliary language modeling head (LM head) [2] also enriches representations during training. The selection of SigLIP and the Qwen2 based decoder is grounded in their SOTA performance and widespread adoption in recent vision-language literature (see Appendix
3.1 Training Objectives
We supervise AlIA by jointly training four lightweight prediction heads, each targeting a distinet objective: task-conditioned relevance, informativeness, uncertainty, and auxiliary captioning. The total loss is a fixed, weighted sum of these objectives, a simple yet theoretically grounded multi-task learning strategy (see Appendix C.5 for detailed justification of this approach and our model backbone selection).
Relevance Head.
The relevance head estimates task-conditioned highlight relevance for frame ft via a scalar prediction é.- = Wr where h4 E TR. PRDddd is the decoder's final hidden state and W. E R Pridden x are learned linear projection weights. This prediction is supervised against human engagement scores r. (Sec. (.3) using a Smooth LI loss [30], Crelovance (Eqq Tap. To encourage smooth temporal predictions reflecting common user engagement patterns often observed in video data (e.g.. gradual build-up and fall-off interest around key moments, as noted in datasets like Mr.Hisum), we incorporate an additional total variation (TV) regularizer [31] LTV (Eq. Tb). These are given by:
Twin
CTV
ve 1=-12
(Ib)
Lrelevance = Sf (1a)
Twwnn
where E [0, in Eq. 1b indicates if both in and f--1 are valid within a temporal window of size T Twin The total relevance objective, Lretevance-total: combines these terms:
Erelevance-tniil Lrelevance ATV
Informativeness Head. Informativeness measures whether a frame introduces new information relative to recent context. Following prior work in dialog- based VideoLLMs [13), we incorporate a binary classification head to estimate if frame ft introduces new information, outputting a score i, = softmax SaatmaxxXWW,,, hW,,,), where W; E R Pradden x2 are learned weights projecting ht to a 2D output for binary classification. It is trained to recognize temporally novel or redundant frames using Binary Cross-Entropy (BCE) with ground truth i, E {0. 1} (Sec. 3.3
Cinto = BCE((,, 44)
(3)
Uncertainty Head. Uncertainty captures the model's confidence in its frame-level predictions under partial observability. This head predicts the logarithm of Gaussian variance. From the hidden state he, its linear projection W. outputs raw log-variance i, = W..h. This is clamped to a predefined range [Triiny [mmas] (yielding 14,e) to obtain the predicted variance ou exp((e.). The primary loss component the Gaussian negative log-likelihood (NLL) 25 For this, the mean f4 is taken as the linear output of the relevance head (i.c., #4 = fz), used alongside the ground truth relevance and the predicted variance of:
(r-- 11)
LNL
+ log((266)) +
20? +
where is a small stability constant (e.g., 10-6), However, relying solely on the NLL loss can lead to
mode collapse: known pitfall where the model learns degenerate solution by predicting a single, uninformatively high variance for all frames to trivially minimize the loss (3] To counteract this, we introduce variance diversity penalty, Laiv (Eq. 5aj. based on the batch standard deviation of the

---

clamped log-var varianesss This regularizer forces the model to produce a dynamic and meaningful range of uncertainty values. The final uncertainty loss, Luncertainty (Eq. 55b, combines the expected NLL with this penalty. A detailed derivation and justification for this uncertainty formulation is provided in Appendix
Cdiv -ddiv std((4,e)eetaee)) (5a)
Cuncortainty max ([CLL + Ldiv) (5b)
LM Head. Following prior streaming LLMs [3 12 an auxiliary LM head encourages semantically rich hidden representations. At randomly sampled training timesteps, the model generates short captions (Sec. (.3) for the current frame conditioned on prior context using standard cross-entropy loss for next token prediction:
LIM CrossEntroppy (ontEntrooyyLMMine (LMHead Me)
(6)
Generated text is not injected back into the context, focusing on unidirectional frame-wise scoring.
Total Loss. The final training objective, Lusat, a weighted combination of the Lrelevance total (Eq. 2 Linto (Eq. 3), Lancertuinty (Eq. 5b), and L.M (Eq. 6):
Liaal XyCrelevancce total XpCinio + AuCuncertaint ALMCLM
Fixed weights À are used during training (see Appendix C.2 for values). We use fixed weights to ensure training stability and interpretability. avoiding the complexity of joint optimization or dynamic reweighting across heterogeneous objectives. Additional implementation details and motivation for each loss component are provided in Appendix C.T
3.2 Inference and Memory Management
Transformer self-attention scales quadratically with sequence length [3). making streaming inference costly. To mitigate this, we adopt a KV Cache [26). storing previously computed attention keys and values at each layer, avoiding redundant computation. Each layer's cache grows with sequence length L, storing tensors of shape [B H, L, Dj, where B is batch size, H is number of heads, and is head dimension. However, for long videos (Lmux 127k in evaluations). this unbounded growth leads to GPU out-of-memory (OOM) errors, highlighting the need for a memory eeffore-efficientt alternative.
Dynamic SinkCache. To solve this, our framework introduces the Dynamic SinkCache, novel modification of the hybrid memory approach from SinkCache (2I). Unlike the standard method that uses the first few generic tokens its sink, our mechanism creates more targeted long long-term memory. dynamically constructs the sink to contain exclusively the natural language task objective tokens (Q), while the sliding window is dedicated to recent visual context. This design carries a constant memory footprint, supporting inference over arbitrarily long videos. In our implementation, the task objective sink averages -45 tokens, which pair with sliding window of 2048 recent tokens. This configuration, requires only 17% of the standard cache (Lavg = 12,421 and conserves memory while achieving improved performance (see Section 4.2)
Formally, the highlight score at timestep is computed as ÿt = (Je, K... The term K., = (Ω, kt-nct) ) represents the memory accessible at timestep t. Here, the sink is precisely the set of task objective tokens Q, and k is the sliding window recent visual tokens.
A comprehensive explanation of the Dynamic SinkCache mechanism, including its operational details, comparisons against other caching mechanisms, the role of sink tokens in maintaining long-term context, and an illustrative diagram, is provided in Appendix F
Scoring. The final highlight score dL is computed by fusing the relevance (ft). informativeness (G)), and uncertainty (û,) heads using an uncertainty-aware, piecewise scoring function. Specifically, let P. be the predicted relevance (W,h & the predicted informativeness Nooftax(WW,h,, and â the predicted uncertainty score (taken as the clamped log-variance 4.cc output by the uncertainty head, where i4.c = Mlamp(Whhg Lminy Lmax). We then apply an uncertainty-awarr linear weighting function:
(xi + 3ir, 0t ade + 3ic E(ū,- TT),
û S Tu (low uncertainty)
(8)
Tw). if û > Tu (high uncertainty)
The parameters (e,,,,,T,)) are set using a static approach, which our analysis shows is more robust than unstable dynamic alternatives (see Appendix C.4). This framework offers flexible trade-off:
5

---

for a truly zero-shot configuration, we use a fixed heuristic (based on a 10:7 ratio for a 9). For optimal domain-adapted performance, all four parameters are tuned via a lightweight grid search. Our results in Section 4.1 are presented for both settings to demonstrate the model's capabilities.
3.3 Video Datasets for Prediction Head Supervision
We train AllA using a combination of existing video-language datasets and novel dataset tailored for highlight relevance in videos. These datasets supervise different model heads and are critical for enabling frame-level semantic understanding.
To effectively supervise the multi-objectivve prediction heads of AHA, particularly the core relevance head, we construct novel, large-scale dataset, named the Human Intuition Highlight Dataset (HIHD). The construction of HIHD begins with the Mr. benchmark [15]: for each video entry therein, we retrieve its original full version from YouTube 34 via webscraping. Videos with fewer than 70,000 original views are subsequently discarded to ensure data quality. From the retained videos, systematically sample frames at 1 fps to align with our model's visual processing rate. The corresponding YouTube replay counts (engagement scores [15) are then normalized to [o, 1] range, serving as our primary relevance signal rr, the ground truth scores for the relevance head's Smooth LI loss (Eq. Taa. While this engagement based signal enables scalability far beyond manually labeled datasets, we acknowledge it is an imperfect proxy for true importance. It may introduce biases by amplifying content designed for high engagement (c.g.. "clickbait") and misaligning with expert judgment in safety-critical domains. We provide detailed discussion of these limitations in Appendix I and Section 5 respectively. For our condiiiiod setting, relevant task objectives Q are generated by programmatically transforming each video's original YouTube title into diverse natural language queries using predefined templates (e.g., title "Exploring the Riemann Hypothesis" might become "What segment of the video addresses 'Exploring the Riemann Hypothesis" """); see Appendix H Finally, to simulate real-world video stream degradation and enhance model robustness, we introduce "quality dropouts": 5-20% of each video's i duration is randomly selected, and frames within these segments undergo perturbations such as resolution reduction, block noise, color banding, or blackouts, with corresponding dropout masks generated (dctailed in Appendix [E.I]. Crucially, HIHD adopts the exact caann//aliattion/eest splits from HiSHiSum to ensure fair comparability, and its training set explicitly excludes vidcos present in common highlight detection evaluation datasets.
The resulting HIHD comprises 22,463 videos, each with frame-level normalized engagement scores (re), a synthetic task objective Q, and quality dropout masks. This dataset provides rich, frame-level supervision specifically designed for training and evaluating task- cossicondiiooned OHD models like AHA, By combining large-scale implicit human engagement signals with synthetically generated task conditioning and targeted robustness augmentations, HIHD aims to foster the development of models that can model human intuition in dynamic, task-driven, and imperfect streaming environments.
To supervise the informativeness head, which predicts whether frame fi introduces new information, we adapt strategies from MMDuet [13] a streaming framework. Ground truth labels is {0, 1} are derived from segment-level captions in the human-annotated subset of Shot2Story [35] and procedural videos from COIN (36.. For each segment, a "point of sufficient understanding" is randomly sampled between 50% and 75% of its duration. Frames from the 50% mark up to this point are labeled informative (i, = 1): others before or after are labeled non-informative (ie = O). This reflects the intuition that early frames lack context and later ones become redundant once understanding is achieved. The informativeness head is trained using BCE loss (Eq. 3). with the hypothesis that this signal correlates with highlight moments in OHD. Crucially, our framework is designed to explicitly decouple this signal of informational novelty from task-relevance, using separate heads to learn these distinct concepts. We provide detailed justification and qualitative analysis demonstrating its effectiveness on a real-world robotics video in Appendix D.I
To enhance semantic representations, we train an auxiliary LM head using the same Shot2Story and COIN annotations. At random timesteps, AHA generates a short caption for the current frame conditioned on prior context and the task prompt, supervised via next-token cross-entropy (Eq, ( Unlike interactive systems ge.g., MMDuet), AНA does not re-inject generated text into the context or use it during inference. This preserves its unidirectional, non-dialogue streaming setup. The LM task solely improves the quality of hidden representations h4 used by the highlight prediction heads (see Appendix D for details).

---

Experiments
This section details the comprehensive experimental evaluation of AHA. We first assess its core perfor- mance as an OHD model under strict streaming constraints on two standard HD benchmarks, TVSum and Mr.HiSum (Section 4.1). We then evaluate its robustness common video degradations and con- duct ablation studies to analyze the contributions of its key components (Section 4.2). To demonstrate its practical applicability in challenging real-world conditions, we further test AHA'S 's capabilities on a long-form form robotics video (Section 4.3), and generalization potential to other unoptimized video understanding tasks (Section 4.4). Our results are averaged over runs.
4.1 Highlight Detection
The widely-used TVSum HD benchmark [14 provides multi-rater frame-level importance scores for 50 diverse videos. However, its small size can cause topic bias in standard splits, hindering reliable generalization assessment [15]. Therefore, to rigorously assess generalization from its pre-training (Sec. we evaluate AIIA on TVSum zero-shot (i.e., without TVSum-specific fine-tuning) and with lightweight grid search. Following [10), report Kendall's 's (ordinal association) and Spearman's (monotonic relationship) rank correlations. We also report top-5 mAP (mean Average Precision for top 5 summary segments) per established TVSum protocols.
TVSum (Table D. AHA establishes a new SOTA demonstrating remarkable performance even in truly zero-shot setting. Using fixed heuristic without any domain specific tuning, our model achieves 91.6 top-5 mAP, significantly outperforming the previous best tuned model, TR-DETR [T9] (87.1 mAP). This zero-shot configuration also produces the most faithful overall frame ranking, setting a new SOTA on both Kendall's 7 (0.304) and Spearman's 's (0.433).
Furthermore, performance summary retrieval can be pushed even higher. By adapting the scoring parameters a lightweight grid search on the TVSum validation set, the top-5 mAP is boosted to 93.0. This domain-adapted adapted configuration slightly alters the global ranking but excels at the primary goal of identifying the most critical highlight segments.
Table 1: TVSum Performance. We report top-5 mAP, T, and p. 'Tuned?' indicates if fine-tuned on TVSum (Y) or not (N). Modalities: V (visual), (text), (audio). Bold is SOTA. (Per-category
details: Appendix B.2
Model
Tuned?
Modality
mAP
Kendall -
Spearman
Human 20
N
0.177
0.204
PGL-SUM 18
N
V
57.1
0.206
0.157
LLMVS 10
N
V+T
0.211
0.275
UniVTG 11
N
84.6
QD-DETR
V+A
86.6
TR-DETR
Y
V+A
87.1
AHA (Zero-Shae
N
V+T
91.6
0.304
0.433
AHA (Domain-Adaaped)
N
V+T
93.0
0.285
0.406
The large-scale Mr.Hisum HD benchmark 15 uses YouTube replay statistics ("most replayed" data reflecting broad viewer engagement) as scalable ground truth, forming a key component our HIHD (Sec. 3.3 Since AHA'S 's training (via HIHD) uses only Mr.Hisum's 's training split data, evaluation on its test set is strictly on held-out data. Per protocol [15], we report mAP@ 50 and mAP@ 15 (top 50/15 ranked segments) to assess relevance assignment to frequently rewatched frames.
To specifically evaluate our relevance head on the task it was trained for, we use scoring configuration that isolates its output (3 = 1, with all other weights set to zero). On the Mr.Hisum test set (Table this focused approach achieves new SOTA of 64.19 19 mAP@ 50 and 32.66 mAP@ 15, a significant improvement (e.g., +8.3 mAP@50 50 over PGL-SUM (8).. These results validate that our relevance head, trained on large-scale engagement, successfully identifies salient moments correlated with user engagement under strict future access constraints.

---

Table 2: Overall HiSum performance on the full test set. Bold highlights our SOTA results.
SL-module [8]
iPTNet [39]
DSNet [40]
AHA PGL-SUM [18]
Metric
(Ours)
mAP@55 mAP@@ 15
55.31 24.95
50.53 22.74
50.78 24.35
55.89 27.45
64.19
32.66
4.2 Ablations
We conduct ablations TVSum for its evaluation of streaming scoring and ranking; Mr.HiSum is omitted as mainly tests the relevance head. AHA is tested with the optimal sliding window (n 2048) unless otherwise specified. Core component and SinkCache configuration results shown in Table 3 We also demonstrate the efficacy of AHA'S video quality dropout training in Tyy evaluating its performance under various visual degradations.
Head Importance. The decoupled prediction heads are crucial (Table left). Removing the relevance (3 = 0) or informativeness (a 0) heads severely degrades Top-5 mAP by 15.7 and 9.8 points, respectively, from our 93.0 mAP baseline. Omitting uncertainty (e 0) results a smaller drop (3.2 mAP points), suggesting it aids calibration but is less critical here than the other heads.
Language Conditioning, Eliminating language conditioning (empty task string Q) significantly reduces Top-5 mAP by 11.8 points (93.0 + 81.2) and drastically lowers rank correlations (c.g.. S-p: 0.406 -> 0.342). This highlights the critical role of persistent language grounding for task- conditioned HD in streaming video, where retaining the task objective enables AHA to maintain long-ran rong-range semantic alignment. Furthermore, the model exhibits graceful degradation under imperfect conditioning. When tested with ambiguous (i.e., overly general) or entirely irrelevant prompts, performance declines proportionally rather than catastrophically, confirming that the language prompt strongly guides but does not dominate the underlying visual saliency detection (see Appendix (B.5).
Memory Mechanism Analysis. Our most significant architectural finding comes from ablating the memory mechanism itself (Table 3] right). We found that simpler strategies relying on only recent context ('Sliding Window Only") or only initial context ('Static Window Only') performed poorly. Our proposed Dynamic SinkCache outperforms not only these simpler methods but also an "Unbounded KV Cache" and the "Standard SinkCache", proving that task-focused sink the optimal memory strategy for this problem. The choice of a 2048 token window for recent context provides an excellent balance of performance and efficiency, as detailed in our window size analysis in Appendix F2 The details of each memory mechanism can also be found in Appendix F.4
Table 3: Ablation study on TVSum. Left: Core component ablations. Right: Memory mechanism ablations. Our default model (top row) uses the Dynamic SinkCache.
Variant
mAP
S-p
K-r
Memory Mechanism
mAP
S-p
K-7
Аha (Ours)
93.0
0.406
0.285
Dynamic SinkCache (Ours)
93.0
0.406
0.285
a =
83.2
0.341
0.237
Standard SinkCache
92.6
0.401
0.280
8 0
77.3
(1.221
0.221
Unbounded KV Cache
91.7
0.400
0.277
= 0
89.8
0.401
0.278
Sliding Window Only
69.5
0.063
0,043
w/o Q
81.2
0.342 0.238
Static Window Only
63.2 0
-0.01
Impact of Video Quality Dropout Training. A key design goal for AHA is reliable performance despite visual degradations common in real-world streaming. Its training incorporates video quality dropout mechanisms (Appendix [E.I) specifically to build resilience against such artifacts. To demonstrate the efficacy of this training approach, we evaluated AНА on the TVSum dataset under both clean conditions and with several simulated visual degradations [41]: color banding, block noise, quality degradation, and blackout, each applied 20% of frames within each video. As detailed in Table 4 AHA exhibits notable resilience. For instance, its Top-5 mAP drops by only (Ap. 4) and 1..) percentage points when subjected to color banding and block noise, respectively. Even when faced with more severe artifacts like quality degradation and complete blackout, AHA maintains strong absolute performance with mAP scores of 88.9 1A4.1) and 88.2 (A4.s). These findings confirm that the video quality dropout strategy employed during AHA's 's training is effective in preparing it for
8

---

imperfect visual inputs. This enables graceful degradation and underscores its potential for reliable deployment real-world streaming environments where video quality can be unpredictable.
Table 4: Robustness video corruptions on TVSum (Top-5 5 mAP). A indicates drop from clean.
Clean
+ColorBanding
+BlockNoise
+Quality
+Blackout
Aha (Ours) 93.0
92.6 (Au.a)
91.2 (A1s)
88.9 (Aa.1)
88.2 (A..s)
4.3 Real-World Evaluation on Long-Form Robotics Video
We evaluate AHA on video from the SCOUT dataset [I6], a long-horizon (20+ min), egocentric video captured during indoor robot navigation trials from human-robor collaborative exploration exercises Unlike web videos, SCOUT features continuous footage with no cuts, degraded quality (e.g., static, warping), and sparse, mission-relevan events, providing challenging, real-world testbed for OHD.
АНА generates highlight scores (ÿe) in real-time. To facilitate qualitative analysis and enable creation of structured highlight reel from these continuous online scores, we apply Savitzky-Golay smoothing 42 followed by peak detection as a post-procees proepppccsssing step to isolate segments of high salience. Ground truth video annotations were established by domain experts by aligning these predicted peaks with human-issued navigation commands (obtained from experiment transcripts) and key visual transitions observed in the video footage. In 8-minute analysis (Fig. 2). 16 of 18 predicted peaks aligned with human-issued commands or meaningful actions (e.g., "robot take better picture of the shoes", "enter the room"). Despite the aforementioned noise and visual corruption, AHA remained stable, This stability is consistent with its designed resilience to such artifacts (as demonstrated in Sec. 4.2 and Table 4 and AНА also showed strong alignment with semantic shifts. These results, while preliminary in application to this dataset, suggest that AHA can detect high-salience moments in real-time, supporting both operator alerting and automated highlight generation in field robotics deployments (see Appendix G for more details).
dal 1007
door
Bright roor
dwr
Approuch
ioeee
Portall
paates
Turn aany
fellway
Faee Move to foor
light una
Taak Objective: objects
Score
are in this roor??"
V1
(trame
Figure 2: SCOUT results. Colored lines mark annotated events from video (e.g., room entry. turns). Highlighted regions indicate degraded video. Black line AHA'S predicted highlight scores.
4.4 Generalization to Broader Streaming Video Understanding
Beyond its primary application in highlight detection, AHA'S 's architectural design and learned representations demonstrate strong potential for broader video understanding. When evaluated on streaming moment retrieval (MR) protocol
[13] using the Charades-STA dataset (4) AHA achieves SOTA performance among streaming methods, yielding 50.7% @ at an IoU of 0.5, and 27.9 7.9% @ at IoU of 0.7. This result highlights the robustness of our approach for fine-grained temporal understanding in streaming video. Full details of this streaming MR evaluation, including comparative results, are provided in Appendix B.T Additionally, AIlA's capabilities on other video-language tasks such as dense captioning and multi mulli--nswer grounding are discussed in Appendix
A subset of video frames available in SCOUT repository. Full videos planned for near-term public release.

---

5 Conclusion
We introduced AHA, real-time, task-conditioned OHD framework. Its lightweight prediction heads and novel Dynamic SinkCache-baseer memory achieve SOTA performance on standard HD benchmarks, remarkably outperforming even traditional offline methods while maintaining constant computational cost across arbitrarily long video streams. Trained on our HIHD data, derived from user engagement signals and task-conditioned prompts, AlIA aligns with human-like intuition for relevance. We validated its robustness on standard benchmarks and challenging real-world settings, including streaming MR and the long-horizon SCOUT roboties dataset, demonstrating AHA's "s capability for consistent, task-relevant understanding in noisy, real-world conditions where offline processing is often infeasible.
Looking ahead, AHA offers a scalable solution for intelligent agents requiring real-time, context- aware video understanding, such as for surveillance drones, satellite analysis, embedded systems, and disaster response. Our ongoing work is developing further analysis of the SCOUT videos, and extending AHA to publicly available drone footage from disaster response efforts (e.g.. wildfire monitoring), where its online, task-conditioned highlight detection can be tailored to aid responders and investigators in identifying mission-critical information from continuous video streams.
I Limitations and Future Work, Although AHA achieves strong results, we identify opportunities for future improvement.
Uncertainty Modeling: Our uncertainty head is trained without ground uncertainty labels due to the subjective nature of highlights and the difficulty of capturing annotator confidence at scale, limiting interpretabillity in high-stakes settings. Future work could explore supervised, contrastive, or calibrated approaches. For instance, datasets with human confidence scores, such as Multi VENT-
G 44, offer a promising path towards direct supervision (see Appendix [I]. Despite this, our model demonstrates improved performance when incorporating uncertainty into its scoring.
Training Efficiency and Backbone Generalization: High compute costs restricted our architectural ablations, including the validation of the AHA framework on wider range of vision-language backbones (see Appendix for our selection rationale). Future work could explore distilled variants of ÁНА to improve runtime efficiency, which would facilitate this broader testing and confirm the framework's s adaptability across different underlying models.
Static Inference Weighting: Our framework relies on a static weighting scheme for inference, design choice empirically validated in our ablations (Appendix C.4), where proved more effective than the dynamic alternatives we tested. While this modular approach yields SOTA performance, the exploration of more sophisticated adaptive weighting mechanisms remains a compelling direction for future research. Expert annotated datasets like Multi VENT-G 44 could provide an important testhed for validating any weighting strategy against human defined importance (see Appendix D.
Memory Constraints in Training: Training uses fixed-length windows without persistent memory across segments for efficient batching. While the Dynamic SinkCache inference provides stable, bounded memory, future work could explore augmenting training with recurrent or retrieval- -bsed memory mechanisms to potentially enhance global reasoning capabilities learned by the model.
Ethical Considerations and Broader Impact. Despite its benefits for applications like disaster response, AHA could be misused in surveillance contexts amplify societal biases if trained on biased data. We recommend its deployment with pivacy-preserving measures (e.g.. blur filters for faces), robust access controls, and domain-specifie ethical audits. To guide responsible research and application, a code of conduct will accompany our public repository. As this technology develops, we encourage continued open discussion regarding its ethical deployment, particularly in sensitive domains such as public safety, surveillance, or defense.
Acknowledgments and Disclosure of Funding
This research was conducted while the first author was a research fellow at the DEVCOM Army Re- search Laboratory (ARL), supported by an Army Educational Outreach Program (AEOP) fellowship administered through the Rochester Institute of Technology. We thank ARL for additional compute support and mentorship, and the anonymous reviewers for their valuable feedback, which improved the paper. including the development of the Dynamic SinkCache.
10

---

References
[1] Peng Liu, Bozhao Qi. and Suman Banerjee. EdgeEye: An Edge Service Framework for Real-time Intelligent Video Analytics. In Proceedings of the 1st International Workshop Edge Systems, Analytics and Networking, EdgeSys' 18, pages 1-6, New York, NY, USA, June 2018. Association for Computing Machinery. ISBN 978- 14503-55377-- doi: 10. 00.1145/52113444.2154444
URL.ttpps:/// /ttp:::://a.acmmmm. acm. org/
doi/100. /145/3213344 3213345
[2] Koffka Khan. Swarm Intelligence- Based Decision Support Systems for Adaptive Video Streaming: Navigating Real-Time Challenges in Dynamic Fnvironments. Swarm Intelligence, 6(8).
[3] Hartmut Surmann, Kevin Daun, Marius Schnaubelt, Oskar von Stryk, Manuel Patchou, Stefan Böcker, Christian Wietfeld, Jan Quenzel, Daniel Schleich, Sven Behnke, Robert Grafc, Nils Heidemann, Dominik Slomma, and Ivana Kruijff-Korbayonn
Lessons from robot-assisted disaster response deployments by the German Rescue Robotics Center task force. Journal of Field Robotics, 7163) 2024. ISSN 1556-4967. doi: 10.1002/....2255 http://wwwwaaaaacooooooooooo///a rob. 22276
htttttpp:://onmmmeeleeeeee http:///w...ee..cc//o/////aa///a http:////oaaaaaaaaoaaa 1002/
eprint:
[4] M Furqan Ayub, Faiq Ghawash, M Aumns Shabbir, M Kamran, and Farhan A Butt. Next Generation Security And Surveillance System Using Autonomous Vehicles. In 2018 Ubiquitous Positioning, Indoor Navigation and Location Based Services (UPINLBS), pages 1- 5, March 2018. doi: 10.1 109 855597444
URL httpss://eeeexpplrrrr leee. www..o.ggabstrrttteeeee////oe/ee
[5] Shamit Lal. Shivam Duggal, and Indu Sreedevi. Online Video Summarization: Predicting Future Better Summarize Present. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 471-480, January 2019. doi: 10. N0.1109/WACV.2199 19.000................ URL
leee. org/
6ocument/8669035 ISSN: 1550-5790.
[6] Apostoliiis, Eleni Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, and loannis Patras. Video Summarization Using Deep Neural Networks: A Survey, Proceedings of the IEEE. 109(11): 1838-1863, November 2021. ISSN 1558-2256. doi: (100///PRCC.222.31777777 URL https
//ieeexplore.. iewe. ora.eee.rr//ooumeentte 9594911
[7] Jie Lei, Tamara L. Berg. and Mohit Bansal. QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries, November 2021. URL httt:://aaxxiyi org/ aba/2107 2020009999999 #xXiv:2107.066/ [es].
[8] Pulkit Narwal, Neelam Duhan, and Komal Kumar Bhatia. A comprehensive survey and mathematical insights towards video summarization. Journal of Visual Communication and Image Representation, 89: 10367, November 2022. ISSN 1047-3203. doi: 10. ivcii.02 103670. URL (ttts//t:://// :/wp:s::/www
[9] Haopeng Li, Qiuhong Kc, Mingming Gong, and Tom Drummond. Progressive Vidco Summarization via Multimodal Self-supervised Learning. In 2023 /EEF/CVF Winter Conference Applications of Computer Vision (WACV), pages 5573-5582, January 2023. doi: 10.1 ISSN: 2642-9381.
URL
https://ieeexplore ieee. htt.///weeeeeeeeeeeeeeoooo////ee docmmee/110000000
[10] Min Jung Lee, Daynung Gong, and Minsu Cho. Video Summarization with Large Language Models, April
2025. http:/ttt////xiiii httpa/aawwwaarrraa///aaa///// 11199 arXiv:2504.1 199 [cs].
[11] Rui Qian, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming Long Video Understanding with Large Language Models. Advances in Neural Information Processing Systems, 177311933-111966 19360, December 2024. URL https: /ttetp:::/ppooo httpps://proceeddnnggg nwwripa. cc/ pappr/pp..... http://///0..................000
Conference htm
[12] Joya Chen, Zhaoyang Lv. Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. VideonL Online Video Large Language Model for Streaming Video, June 2024. URL hRtp::////aaii otg//ba////aeeeeeeee 1186. arXiv:2406. 11816 [es].
[13] Yueqian Wang, Xiaojun Meng. Yuxuan Wang, Jianxin I Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Text Duet Interaction Format, November 2024. URL arXiv:2411.1999 [es].
[14] Yale Song. Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. TVSum: Summarizing web videos using titles. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5179- 5187, Boston, MA, USA. June 2015. IEEE ISBN 978- 1778-1-2 1-4673-6964444 URL http://eeeexxpoore 1eee. org//bccummn//229115//
doi:
11

---

[15] Jinhwan Sul. Jihoon Han, and Joonseok Lee. Mr. HiSum: A Large-scale Dataset for Video High- light Detection and Summarization. Advances in Neural Information Processing Systems. 36:40542- 40555, December 2023. URL https: https://proceeeinng wwwpeiinnnnnn.uureeeeeeeeeeee.ee wwtppppperrlllleepppapeeeeeeeeea 7aal/718006332550666601.. 1aeaaaa344448488 Abstract- Datasets, and and
Bww.hearncchh..ee.e aad.eessssaddeeechrarareee html
[16] Stephanie M. Lukin, Claire Bonial, Matthew Marge, Taylor A. Hudson, Cory J. Hayes, Kimberly Pollard, Anthony Baker, Ashley N. Foots, Ron Artstein, Felix Gervits. Mitchell Abrams. Cassidy Henry, Lucia Donatelli, Anton Leuski, Susan G. Hill, David Traum, and Clare Voss. SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpus. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Compulational Linguistics, Language Resources and Evalu- ation (LREC- 2024), pages 14445-14458, Torino, Italia, May 2024. ELRA and ICCL. URL
https://aallantholooy wwwpllloggggoogg.geooooo wtooo/////// 2024. 1rec4mairmmmmmmm 1259/
[17] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jinpeng Wang. Rui Yan, and Mike Zheng Shou. UniVTG: Towards Unilied Video-Language Temporal Grounding, August 2023. URL URL http::tttt/:iiiie wrgpeiiiiiorraaaaae hate//bb/22/0 /200 16715 arXiv:2307.1677 [cs].
[18] Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Joannis Patras. Combining Global and Local Attention with Positional Encoding for Video Summarization. In 2021 IEEE International Symposium Multimedia (ISM), pages 226 234, November 2021. doi: 10.1 N0.10991MM52913.22110000 URL BtKLLattt///eeeepp..oo ieee org (tw.occmet/9660600
[19] Hao Sun, Mingyao Zhou, Wenjing Chen, and Wei Xie. TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection, January 2024. URL arXiv:2401 ,02309 [es].
TransiirrrcaaTaTaaoaaoooaema 02309
[20] Mayu Otani, Yuta Nakashima, Esa Rahtu, and Janne Heikkila. Rethinking the Evaluation of Video Summaries, April 2019. http: www.aos/////////o..e/...o... httg//bbb/999....../ 1903. 11328 arXiv:1903.1 11328 [cs].
[21] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Streaming Language Models with Attention Sinks, April 2024. URL http://arxiv. wtt.:oo//////0////oo 17453 arXiv:2309. 17453 [es].
[22] Purvish Jajal, Nick John Eliopoulos, Benjamin Shiue-Hal Chou, George K. Thiruvathukal, James Davis, and Yung- Maing- Lu. Token Turing Machines are Efficient Vision Models, January 2025. URL http://arxiv org/ abte///ss240.....// 2209/ 07613 arXiv:2409.0760 [es].
[23] Pavia Bera and Sanjukta Bhanja. Quantification of Uncertainties in Probabilistic Deep Neural Network by Implementing Boosting of Variational Inference, March 2025. URL http://arxiv org/ abs /203
13909 arXiv:2503. 13909 [es].
[24] Unyamanee Kummaraka and Patchanok Srisuradetchai. Time-Series Interval Forecasting with Dual Outputt Monte Carlo Dropout: A Case Study on Durian Exports. Forecasting, 6(3):166 636, September 2024. ISSN 2571-9394. doi: (0.:390fforecast66300333
URL.https:://// htttppa://www.... http::////wwmw.m......./ com wwwm//7719299///////11 Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.
[25] Erik Englesson, Amir Mchrpanah, and Hossein Azizpour. Logistic-Normal Likelihoods for Heteroscedastic Label Noise.
[26] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently Scaling Transformer Inference, November 2022. URL http://tt http::::///ariioor///bbbbaa//
wwwp:bs////a//////ea http:///2......1....00001000 .5102 arXiv:221 110222111001100 1.05102 [cs].
[27] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer Sigmoid Loss for Language Image Pre- Pre-Training. September 2023. URL http://///aa org/ /2003. 15343 #rXiv:2303.15523 [es].
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, December 2023.
URL htt/http:// http://arxiv. org/ wwtg//a///eee abs//////////../////////////// 08485 arXiv:2304.08485 [cs].
[29] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VIL VL: A Versatile LisiounLanguage Model Understanding, Localization, Text Reading, and Beyond, October 2023, URL htt/:tt/t//xiiii wwwp://wwwrr//bb//////// 12966 arXiv:2308. [cs].
[30] Ross Girshick. Fast R-CNN, September 2015. arXiv:1504.0808: [es].
URL http: /ttp://axxiv..m http://axxxvvv.ooaabba Sarg/abs//ie 15obs//114... 18088 08083
12

---

[31] Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal al- gorithms. Physica D: Nonlinear Phenomena. 60(1):259-266, November 1992. ISSN 0167-2789. doi: 016727899290242F
10.10166016--779000))000---1111
[32] Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the Pitfalls Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks, April 2022. URL huttpp////arxx org/ /bb// 09068 ar arXiv:2203.09 0966 [es].
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates. Inc.. 2017. URL httpp::/ppaprss. nips. cc/paper_filaeeee files// -ba.eccaaaaaaa-aaaaat.t.... html
[34] YouTube-8M: A Large and Diverse Labeled Video Dataset for Video Understanding Research, URL
https://reseachh. htt.://eseerrrr...ggl.l..o www.gooo.ooooooooooooooooooooooe wwt.///ooooooooeeeeeeoooc doooleeeem///ooooadaaaaaa htmll
[35] Mingfei Han, Linjie Yang. Xiaojun Chang. Lina Yao. and Heng Wang. Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Vidcos, February 2025. URL httth:t////xiiii org/ abs/
2312. 10300 arXiv:23 10300 [cs].
[36] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis, March 2019. URL actp://arxivg org/ aba/ 1903. .28744 arXiv:1903.022. [cs].
[37] WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jac- Pil Heo. Meery-Dependent Vidco Representation for Moment Retrieval and Highlight Detection, March 2023, URL/attp://arxiv.or arXiv:2303. [es].
abs/2303. 13874
[38] Minghao Xu, Hang Wang. Bingbing Ni, Riheng Zhu, Zhenbang Sun, and Changhu Wang. Cross-category Video Highlight Detection via Set-based Learning, August 2021. URL http: org/ 2208
11770 arXiv:2108. 11770 [es].
[39] Hao Jiang and Yadong Mu. Joint Video Summarization and Moment Localization by Cross Cross-Task Sample Transfer. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16367-16377, New Orleans, LA, USA, June 2022. IEEE ISBN 978-1-6655469466-3
doi: 10.1109/
CVPR55268..2222.011
URI bttt:ttt//eeeeeppp.l..
leee. wtg//accumnt//88788888
[40] Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. DSNet: A Flexible Detect-to-Sumenn Summarize Network for Video Summarization. IEEE Transactions on Image Processing, 30:948-962, 2021. ISSN 1057-7149, 1941- 0042. doi: 10.1 19910T11.20200303998886
BRLL:tttps://ieeexpplor ieee org wttg/boocument/92551444
[41] Dan Hendrycks and Thomas Dictterich. Benchmarking Neural Network Robustness to Common Corrup- tions and Perturbations, March 2019. URU/http:///axxvv.rr//// http://bss////..... 12261 1r03v 10261
[es].
[42] Abraham. Savitzky and M. J. F. Golay. Smoothing and Differentiation of Data by Simplified Least Squares Procedures. Analytical Chemistry, 1679)166 July 1964. ISSN 0003-2700. doi: 10. 1021/ac60114a047
Publisher: American Chemical
Society.
[43] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: Temporal Activity Localization via Language Query. pages 5267-5275, 2017. URL htt.aaaaaa//ppaacccaattaa thecvf www.ww..co/ccoaaaaaoaoe..ec.eecc com/ http://onteeccccmmm
2017/html/Gao Temppoorrr Activity ICCV 201P22217___222 www.rrappppp..... html
[44] Kate Sanders, Reno Kriz, David Etter. Hannah Recknor, Alexander Martin, Cameron Carpenter, Jingyang Lin, and Benjamin Van Durme. Grounding Partially-eeiined Events in Multimodal Data. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15905 15927, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1inn/////n////....inggggg-/ org/2024.findinge findings-iinniii e.m.p. 9
URL http:httss//accttttttt aclthooooyy
[45] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. URL http: // /arxiv.org/abs//177 05101 arXiv:1711.05110 [es].
[46] Samyam Rajbhandari. Jeff Rasley, Olatunji Ruwase, and Yuxiong He, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, May 2020. URL http:://axxxs //rr:iv////ii/////a///// org/ httg/abs/1S0 4:Xiv:1910.20044 [es].
13

---

[47] Tri Dao. FlashAttenttio- Faster Attention with Better Parallelism and Work Partitioning, July 2023.
/RRLptt:///axii http://.rrgabbba//aaaa 08691 arXiv:2307.0869) [cs].
[48] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen, LoRA: LawkRank Adaptation of Large Language Models, October 2021. URL
http:// attt://arxiv, org/ wwwg//////210...// 2206 09685 arXiv:2106.0966 [cs].
[49] Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li. Ziwei Liu, and Chunyuan Li. VaVA-OneVision: Easy Visual Task Transfer, October 2024.
URL arxiv. org/ hbs///ba//// htte///b//4408/// 2408 03326 urXiv:2408.03326 [es].
[50] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding. March 2024. URL htt.tt//////xi..... wwt.a/s/rrr//////////o////oo 2312.02055 arXiv:2312.0200 [es].
[51] Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu, VTi Empower LLM to Grasp Video Moments, November 2023. http:////ars
18445 arXivv2311. 18445 [es].
[52] Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang. Dianho Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding, February 2025. http:://aaaii..rgg//////// httg://bb//44055/ 13382 [cs].
arXiv:2405. 13382
[53] YouCook2: Large-scale Cooking Vidco Datasct for Procedure Understanding and Description Generation, URL http:// eecs. umich... edu/
[54] Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7482-7491, Salt Lake City, UT, USA, June 2018. IEEE. ISBN 978- --3-6-9986-64220-. doi: 10.1109/CVPR.20E 000781. URL /ttL:tttps//eeeeepllrr ieee org/ /tt////cmeettt//88799/
[55] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollar. Focal Loss for Dense Object Detection.
[56] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-L Language Understanding and Generation, February 2022. URL (RR.phtt:://arzit org/ aba/2201. 12086 arXiv:2201. 12006 [cs].
[57] David Nix and Andreas Weigend. Learning Local Error Bars for Nonlincar Regression. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994. URL httpp::/
nwwrips. heurips.cc/paper: wttpc//ppp..leeleeeppppeeeeeeeee wttpss//pppeeee99/////////////// fflle/
pdf
[58] Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, October 2017. URL, http:/aap///axxivorr///aat
04977 arXiv: 703.04977 [cs].
[59] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncer- tainty in Deep Learning, October 2016. URL Uttp:tt/://arxi org. httpg/abs/15000 02142 arXiv:1506.021 0212 [stat].
[60] Charles Blundell, Julien Cornehise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Networks, May 2015. URI htt.hhttp///arxiworgggg aba/1505. 05424 arXiv:1505.05424 [stat].
[61] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017, URL httpt:: nips. cc/paper files/llaaape / 2017 httpa///eeeeeeeeeeeeeeeeeeeeeeea
html
[62] Erik Englesson, Amir Mehrpanah, and Hossein Azizpour. Label Noise, August 2023. URI. http//tttt//arziv.ooo www.:////////......a./ 03884444440000999 arXiv:2304.02849 [es].
haggr:iniiiiiipoooooooooiiiiiiii Logistic-Normal Likelihoods for Heteroscedastic
[63] Yonatan Geifman and Ran El-Yaniv. SelectiveNet: Deep Neural Network with an Integrated Reject Ntt.19901.09999 [cs].
Option, June 2019, URL http::///aarxx...og/////////// 09192
[64] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL http://arxiv. wwwpexwwwwwwoooaaeeeooeee org/abs/ 1810. 04805 1IX.v1810.008005 [cs].
14

---

[65] Kaisa Miettinen. Nonlinear Multiobjecttive Optimization, volume 12 of International Series in Opera- tions Rexearch Management Science. Springer US, Boston, MA, 1998. ISBN 97849-4444444 978-1-4615-5563-6
10.1007/978 1-4615-5563-6
URL BtRLtttp:///ink httpp://iinnk..piiigr.ccc pppriinger.co/1..1100.
978 1- 4615 5563-6
[66] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent Component Alignment for Multi-Task Learning. In 2023 JEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pages 20083-20093, June 2023. doi: 10.1 1199/C/PR52299.022..11399 URL
https://Aeeexploor leee. www.:rr///ooee//o/aaaooo 102033221
ISSN: 2575-7075.
[67] Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting Scalarization in Multi Learning: A Theoretical Perspective.
[68] Claire Bonial, Stephanic M. Lukin, Mitchell Abrams, Anthony Baker, Lucia Donatelli, Ashley Foots, Cory J. Hayes, Cassidy Henry, Taylor Hudson, Matthew Marge, Kimberly A. Pollard, Ron Artstein, David Traum, and Clare R. Voss. Human- Robot Dialogue Annotation for Multi-Modal Common Ground. Language Resources and Evaluation, November 2024. ISSN 1574-020X, 1574-0218, doi: 10.1007/ arXiv:2411. [es].
210579-024-09788-2. URL httt::ttp///xii org/ aba/ aba/2411, 2411. 12829
15

---

A
Training Hyperparameters
Table gives gives the full set of hyperparameters used fine-tune Aha on the Qwen-7B backbone.
Table 5: Key hyperparameters for training AHA.
Category
Hyperparameter (Value)
Optimization
Optimizer
AdamW 45
Betas (optimizer)
(0.9, 0.999)
Epsilon (optimizer)
10-
Weight decay
Learning rate
2 x 10
LR scheduler
Cosine decay with linear warmup
Warmup ratio
0.05 warmup steps)
Gradient norm clipping
1.0
Gradient checkpointing
Enabled
Batching
Per-device train batch size
Gradient accumulation steps
2 (effective batch size 2)
Num epochs
Precision & Acceleration
BF16 training
Enabled
DeepSpeed
zero2 46 + CPU affload
Attn implementation
Flash Attention2 47
Data loading
Dataloader workers
4 Truc False
Pin memory
Drop last batch
Video preprocessing
Frame rate
fps 384 x 384
Frame resolution
Pooling stride
Frame tokens (#)
49 [7.
Token pooling dims
Model bockhones
LLM backbone
Imms comss-aab/laan amm-s--aa/laaaaaaneiiiii---o-oo http://llll-larr-pattltieleie
Vision backbone
Multimodal projector
3x3 conv + linear layers
Losses regularization
Stream loss weight
1.0
TV loss window
49
Saving & logging
Save strategy
steps (every 25 steps) 5 checkpoints
Save total limit
Logging strategy
steps (every step)
A.1 Implementation Details
on a frozen Qwen-2.7B backbone for one epoch. Training was performed on 3 compute nodes, each with 2xNVIDIA A6000 GPUs (48GB VRAM), totaling 6 GPUs. The full training run took approximately 28 hours. Videos were sampled fps for both training and inference. AНA was trained using Py PoToh 2.5.1, Transformers 4.49.0. 00 and CUDA 12.4 on Ubuntu 22.04. Training runs were executed on Paperspace, with all checkpoints and video data stored in Amazon S3 bucket.
We fine-tune AILA using Low-Rank Adaptation (LoRA) 48
LLM backbone: Frozen Qwen-2.7B 29 (1mms- (tt.::/:ha///aaaaacceeeooaooooaa
Vision encoder: Frozen SigLIP Large 27 spogle/ssigiip--age--aaaoo
16

---

MM projector: Single linear layer nn. Linear (em__mideeee hidd_hddidde,, hidden_size) map- ping each 1152-d patch to Qwen's 3584-d hidden space.
A.2 Inference Performance
We conducted a detailed performance analysis of our framework on a 1062 second video (-17 minutes) using two NVIDIA A6000 GPUs. The system achieved sustained throughput of frame per second (FPS), demonstrating high efficiency with 100% peak GPU utilization and 90% peak memory controller utilization. During this process, the framework consumed peak of 30.49 GB of VRAM across both GPUs and operated well within safe thermal limits at peak temperature of 659C, all while maintaining a minimal system RAM footprint of 3.66 GB. While this establishes strong performance baseline, the FPS rate means that in a live scenario, the system would "drift" and fall behind the incoming video feed. Therefore, for real-time deployment, implementing logic to strategically skip frames would be necessary to keep the analysis on track. Given that our framework already leverages the available compute effectively, it a strong candidate for such optimizations to achieve the higher throughput needed for live applications.
B Additional Results
B.1 Streaming Moment Retrieval
We follow the streaming moment retrieval (MR) protocol introduced in MMDuet 13,, applying AHA to Charades-STA 43 and treating frame-level relevance scores as soft temporal indicators. Using smoothing window w as in prior work, we compute R@ at loU thresholds 0.5 and 0.7.
As shown in Table AHA with w 8 achieves the highest temporal grounding performance on Charades-STXX attaining 50.7% R@ @.5 and 27.5 27.9% @@0.7. This constitutes an absolute improvement of 8.3 and 9.9 points, respectively, over the strongest baseline (MMDuet with w = 8), highlighting the benefits our direct frame-level scoring and creaming-orreette design, even in the absence of span-level supervision.
To further contextualize these results, we train two additional streaming MR baselines following MMDuet's 's framework [13., using the same initialization (LLaVA-One' Vision (49]), training data (MMDuetl (13). and learning schedules. We reformat the data into the interaction and segment representation formats used by TimeChat [50] and VTmmeLLM [51], yielding LLaVA-OV-TC and LLaVA-OV-VT. Comparisons to these variants further validate the advantages of our frame-level design.
We note, however, that this formulation is still approximation of full moment retrieval. Accurate span localization under strict streaming constraints remains an open challenge. Extending Alla with autoregressive span prediction memory aware temporal boundary modeling is promising direction for future work.
Table 6: Performance on Charades-ST:: for temporal grounding.
LLaVA (OV-TC)
LLaVA (OV-VT)
MMDuet
AHA (Ours)
ÁhA
Metric VTG-LLM 52
MMDuet B
(w 8)
(w
8)
R@0. R@0.7
33.8
33.1
36.5
27.3
42.4
42.8
50.7
15.7
12,4
12.3
2.1
18.0
18.1
27.9
B.2 TVSum's Categorical Evaluation
In addition our overall Top-5 mAP results, we analyze performance across TVSum's ten activity categories [14]: Changing Vehicle Tire (VT), Getting Vehicle Unstuck (VU), Grooming an Animal (GA), Making Sandwich (MS), Parkour (PK), Parade (PR), Flash Mob Gathering (FM), Bee Keeping (BK), Attempting Bike Tricks (BT), and Dog Show (DS). As shown in Table our full multimodal model (V+T) achieves new state-of-the-art in nearly every category, with particularly large gains in visually complex tasks like Changing Vehicle Tire.
17

---

Table 7: Top-5 mAP (56 on TVSum categories. Bold indicates state-of-the-an art per category.
Method
Modality
VT
VU
GA
MS
PK
PR
FM
BK
BT
DS
Avg
QD-DETR 37
(V)
V
88.2
87.4
85.6
85.0
85,8
86.9
76.4
91.3
89.2
73.7
85.0
UniVTG 17
V
83.9
85.1
89.0
80.1
84.6
87.0
70.9
91.7
73.5
69.3
81.0
TR-DETR 19
(V)
V
89.3
93.0
94.3
85.1
88.0
88.6
80.4
91.3
89.5
81.6
88.1
QD-DETR (V+A)
V+A
87.6
91.7
90.2
88.3
84.1
88.3
78.7
91.2
87.8
77.7
86.6
TR-DETR (V+A)
V+A
90.6
92.4
91.7
81.3
86.9
85.5
79.8
93.4
88.3
81.0
87.1
AhA (Ours)
V+T
98.3
99.2
99.4
84.8
81.2
94.8
97.4
94.2
93.1
87.6
93.0
B.3
Multi-Answer Grounded Video Question Answering
The Multi-Answer Grounded Video Question Answering (MAGQA) benchmark 13 extends con- ventional Video QA by requiring models to generate multiple answers at semantically relevant time points within a single video, rather than single response per question. In MAGQA, each question corresponds to Teurns ground-truth answer turns, each defined by a start time starty, an end time Rarms and an answer text goldy. Models must decide, at each frame, whether to respond based on the sum of informative and relevance scores exceeding a threshold t, and then produce the answer in real-time. Performance measured using the in-span score, which combines textual relevance (scored 1-5 via an LLM) with temporal accuracy by averaging the scores of all predicted answers falling within each ground truth interval and then averaging across intervals. This setup simulates realistic streaming video comprehension, emphasizing both promptness and answer correctness without access to future frames.
Our model attains an in-span score of 2.42 (GPT-scored) at / = 0.5 and 2.37 at / = 0.3, compared to MMDuet's peak of 2.93 (Table(8), indicating that AHA still produce timely, relevant multi-answer responses even without task-specific training. After deduplication, we average only about 2.02-2.09 unique turns per video, despite generating over 30 raw turns, showing that our streaming design reliably spots answerable moments but tends to repeat predictions when it isn't explicitly optimized for MAGQA. This highlights both the versatility of our framework in auxiliary QA tasks and the opportunity to further improve answer diversity and precision through dedicated fine-tuning.
Table 8: MAGQA evaluation results: In-span score and response turns
Model
In-Span Score (LLaMA / GPT) # Turns (w/o. / w/. dedup)
LLaVA-OV-TC -LVVA-VVVTT
2.92 / 2.79 2.94 / 2.78
3.4/1.9
5,4/2.2
MMDuet [I3
w/ t = 0.6
2,46 / 2.33
13.7/4.0
w/ & 0.5
2.77 / 2.61
18,4/5.3
w/ t- = 0.4
3.00 / 2.81
23.0/6.6
w/ t 0.3
3.13 2.93
27.0/7.6
AHA (Ours)
w/ t 0.5
2.68 / 2.42
30.55 2.02
w/ t = 0.3
2.63 / 2.37
34.19 2.09
B.4 Dense Video Captioning
We evaluate on the YouCook2 dense video captioning benchmark 53 where models must detect and describe 8 procedural steps in minute-long cooking videos by outputting, for each step, start time, end time, and caption. Following MMDuet [13], we accumulate per-frame "need response" score (the sum of informative and relevance heads) and emit a caption whenever this sum exceeds a threshold (we set = 2). Since frames themselves do not explicitly mark step boundaries, we heuristically assign the previous and current response times as the start and end of each segment, and merge adjacent steps with identical captions.
18

---

Table compares performance. Even without any DVC-spccific fine-tuning, AHA produces competi- tive captions in real-time, achieving an Fl of 15.1%, demonstrating versatility across auxiliary tasks. However, unlike MMDuet's "rm. prev. resp." trick, which significantly reduces redundancy, our streaming design still tends to repeat captions, reflecting the need for dedicated training or more sophisticated boundary modeling to fully match specialized DVC pipelines.
Table 9: Performance on YouCook2 dense video captioning.
Method
SODA,
CIDEr
F1
TimeChat 50
1.2
3.4
12.6
VTG-LLM 52
1.5
5.0
17.5
LLaVA-OV-TC
1.9
3.3
21.8
LLaVA-OV-V
2.5
6.7
14.0
MMDuet [33
2.4
5.7
19.2
m. prev. resp.
2.9
8.8
21.7
AнA (Ours)
1.4
3.2
15.1
B.5 Robustness to Imperfect Task Conditioning
To assess the framework's ' robustness to variations in task conditioning, we conducted a quantitative analysis on the TVSum dataset using ambiguous and irrelevant prompts. An ambiguous prompt was defined as high-level categorical description of the specific task (e.g., using "Vehicle Maintenance" for video on changing tires). An irrelevant prompt was defined as task description sampled from a video in a completely different category.
Performance was measured by the change in top-5 mAP relative to the baseline score achieved with the original, specific prompt (93.0 mAP). The results, summarized in Table IO demonstrate graceful degradation. With an ambiguous prompt, performance decreased by only 1.1 mAP points, indicating the model can generalize broader task descriptions. When given entirely irrelevant prompt, performance dropped by more significant, yet not catastrophic, 9.7 mAP points. This confirms that while the model is strongly guided by the task objective, its learned visual representations retain strong sense of inherent saliency.
Table 10: Impact of prompt quality on TVSum performance (Top-5 mAP).
Prompt Type
Top-5 mAP
Change (A)
Standard (Specific)
93.0
Baseline
Ambiguous
91.9
-1.1
Irrelevant
83.3
-9.7
C
Supplementary Methodological Details
This section provides additional details and justifications for certain design choices described in the main paper's Methodology section (Section 3), which were condensed for brevity due to page limits.
C.1 Training Objectives: Head Details and Justifications
Relevance Head TV Loss Motivation. The motivation for incorporating the total variation (TV) loss (Eq. ib the main paper) stems from observing the structure of human engagement signals often used for highlight supervision, such as aggregated user replay statistics (see Figure 33. These signals frequently exhibit smooth, bell-shaped distributions centered on replayed segments. The TV loss encourages our relevance predictions 7, to match these smooth trends characteristic of engagement, complementing the point-wise Smooth L1 loss 30 (Eq. Tab while aiming to avoid over-smoothing across genuine sharp transitions in content relevance, The term it acts as a binary mask ensuring the penalty applies only to adjacent valid predictions.
19

---

235
0.22 25 A glimose into the mystery of the Riemann Hypomnesi
Figure 3: YouTube replay distribution, adapted from the Mr.Hisum 1.5 (vertical axis), forming "bell curves," indicate frequently rewatched segments. These high-engagement areas serve as the primary supervision signal for training our relevance prediction head.
Peaks in replay volume
Informativeness Head Rationale and Repurposing. While prior work in dialog-based Vide- oLLMs [13 12
often utilizes informativeness scores to trigger language generation manage conversational turns, we adapt this underlying intuition as a direct learning signal specifically for the highlight detection (HD) task. By supervising the model (Eqq 3 to explicitly recognize tempo- rally novel versus redundant frames, we encourage the development of stronger temporal reasoning capabilities, which beneficial for accurate highlight estimation over extended periods.
Uncertainty Head Rationale and Potential Applications. The introduction of the uncertainty head crucial for addressing the challenges of the online, streaming setting. Since the model must predict relevance fz at time / based only on past and current information (partial observability), its ability to judge the long-term significance of a frame is inherently limited. Training the model to predict its own uncertainty via log variance of the relevance score, using the negative log- log-liiki objective in Eq. explicitly models this limitation. Specifically, the model outputs a raw log- variance i, = W, which is clamped for numerical stability and then exponentiated to obtain the predicted variance o/. During inference, we use the clamped log- oog-variance 4,e as the uncertainty score û, as it is more stable and interpretable for downstream knowledge, the first application of such probabilistic uncertainty modeling in OHD.
As noted in the main text, this is, to
Beyond the immediate model training. the resulting uncertainty scores ún can potentially support downstream applications such as adaptive decision thresholds, mechanisms for deferring judgment on low-confidence frames, or reliability-aware resource allocation when processing multiple video streams.
For a detailed justification of this architecture, including comparisons to alternative uncertainty estimation techniques such Monte Carlo Dropout and Bayesian inference, see Appendix C.3
LM Head- Design Choice Justification. The auxiliary LM head (Eq. 5) aims to foster semantically rich hidden representations. In contrast conversational models that might inject generated text back into the context 12 [3). we deliberately avoid this feedback loop. Our focus is on efficient, unidirectional frame-wise scoring for HD, not multi-turn interaction. This decoupling enhances efficiency and avoids reliance on implicit conversational structures that may not align well with continuous, non-interactive video streams. The LM task serves solely improve multimodal alignment in the representations used by the primary scoring heads.
C.2 Loss Function Weights
The total loss function used for training AHA is weighted sum of the objectives from the different prediction heads, as defined in Eq.
Listal A total A; X XuCunceetanty + ALMLIM
where Crelevanco-tonal itself combines the base relevance loss and the total variation loss: Crelevanoo-total Crelevance + CTV (Eq. 2)
The weights (A) were determined based on the relative importance of each task, considerations of class imbalance, the role of auxiliary objectives, and preliminary experiments. The final fixed weights used throughout training are detailed below, following a general strategy of up-weighting critical or difficult tasks and down-weighting auxiliary regularizing terms:
Relevance Loss Weight (Aysttall 8.0): The total relevance loss (Crrlevanne-totaa), which includes the primary Smoothl.1 regression objective (Eq. Ta is assigned the highest weight (8.0). This emphasizes the main goal of the model: accurately predicting task-conditioned
20

---

highlight relevance. This aligns with multi-task learning principles where primary task losses are often weighted higher [54].
Internal TV Loss Weight (Ary 0.05): Within the Leltevance-ootal term (Eq. 2 the total variation loss component (Eq. lb) is weighted relatively low (ÀTv = 0.05). This ensures it functions as regularizer, encouraging temporal smoothness in predictions without dominating the main regression signal from Lrelevance-
Informativeness Loss Weight (A 0.5): The informativeness head's BCE loss (Eq.) is assigned significant weight (0.5). This decision addresses the substantial class imbalance inherent many HD tasks, where non-informative frames often form the vast majority. By up-weighting this loss, ensure the model remains sensitive to detecting rarer informative frames, drawing inspiration from methods like Focal Loss [55] that effectively give more weight to harder examples or minority classes.
Uncertainty Loss Weight (Aa 0.1): :): The uncertainty head's NLL loss (Eq. is considered auxiliary objective. Its main purpose during training learn predict the variance (af) associated with the relevance prediction, rather than directly driving the relevance value itself. Consequently, receives small fixed weight (0.1), reflecting its supporting role relative to the primary relevance task (approximately 80 times smaller weight).
Language Modeling Loss Weight (ALM = 0.2): The LM head's cross-entropy loss (Eq.6) also serves auxiliary function, primarily aimed at enriching the model's internal multimodal representations, analogous to how models like BLIP (56) benefit from combined vision-language objectives during pre-training. Unlike models where text generation might be a primary output (e.g.. [2)), here supports the main scoring task and is weighted accordingly (0.2).
Variance Diversity Weight (Adiv - 3) Small constant regularizing the uncertainty loss (Eq. 5a
This multi-objective setup, common in complex vision-language tasks, allows the model learn diverse but complementary skills necessary for effective highlight detection. The chosen weights reflect balance aimed at prioritizing the core relevance prediction while leveraging the benefits of auxiliary signals for robustness, temporal understanding, and uncertainty awareness.
C.3 Uncertainty Modeling Design
Motivation. In Online Highlight Detection (OHD), the model observes a video frame-by-frame and must immediately judge whether a frame is task-relevant, without sceing the future. This partial observability inherently limits predictive certainty. For example, the current frame may only gain meaning retroactively (c.g., as a prelude to an event). To address this, we introduce a lightweight uncertainty head that models aleatoric wncertainty (input-dependent uncertainty). i.e., the ambiguity in predictions stemming from incomplete observations. The head outputs a log- variance value 2, = W,,,, at each timestep, predicting the uncertainty of the corresponding relevance score Åt-
We adopt standard heteroscedastic regression formulation 57 treating the tround-truth relevance Y2 as sampled from a Gaussian distribution with mean é (the relevance head output) and predicted variance of = exp(fe.). Here, 4..- is the clamped log-variance for numerical stability. The primary training objective for the uncertainty head the Gaussian negative log-likelihood (NLL) (25]:
Architecture and Loss.
(r, f.)2
1
LNL
20 + log log(2no? 55)
We follow best practices from prior work (58) by predicting log-variance rather than variance directly, ensuring positivity and improving numerical stability.
Preventing Mode Collapse. well-known issue with heteroscedastic models the risk of degen- erate solutions where the network minimizes the NLL loss by predicting arbitrarily high variances, thereby flattening the likelihood (2). To mitigate this, we introduce a regularization term encouraging diversity in predicted uncertainties:
21

---

Caiv - sttd ste(hll,,)iehmaec)
The final uncertainty loss is defined as:
uuncrttinyt max(0, E[Crr]] + Laiv)
This discourages the model from assigning identical uncertainty across all frames and promotes calibration across predictable and ambiguous scenes.
Comparisons to Alternative Approaches.
Monte Carlo Dropout (MC Dropout) 59
Applies dropout during inference to simulate an ensemble. Multiple stochastic forward passes yield a distribution over predictions. While simple and widely used, MC Dropout primarily captures epistemic (model) uncertainty and requires multiple passes per frame, poor fit for real-time streaming.
Bayesian Neural Networks (BNNs) [60]: Learn distributions over weights via variational inference. While theoretically appealing, BNNs incur high computational cost and complex training [61]. Their benefit is mostly in epistemic uncertainty, which is less central than aleatoric uncertainty in the OHD setting.
Combine predictions from independently trained models. This method produces state-of-the-ar uncertainty estimates but is expensive at inference, requir- ing .M forward passes (where M is the number of NNs in the ensemble). Ensembles are known to produce well-calibrated results but are impractical for streaming environments.
Deep Ensembles [1]]
Why Log- Variance Prediction? Compared these alternatives, design:
Requires only a single forward pass, making suitable for high-frequency, low-latency inference.
Provides per-frame alcatoric uncertainty, allowing the model to express ambiguity due to missing future context.
Outputs interpretable uncertainty scores (clamped variance) that are usable down- stream for decision deferral confidence- policies.
Avoids trivial high- variance collapse via diversity promoting regularizer.
This decision is further supported by recent work on heteroscedastic modeling in noisy classification settings 62 which shows improved robustness and calibration when modeling log-variance directly.
Summary. We favor log-variance prediction with NLL loss due to its interpretability, ability to model alcatoric uncertainty under partial observability, and compatibility with efficient online inference. Alternative approaches incur significant overhead focus on epistemic uncertainty, which is secondary in our setting. Our approach allows ÁНА to not only predict whether frame is relevant, but also how confident it in that decision, a vital feature for intelligent agents in real-world deployments.
C.4 Highlight Score Fusion
This subsection details the formulation of our highlight scoring function, the theoretical principles motivating its design, and the empirical validation for our choice of weighting scheme.
C.4.1 Scoring Function Formulation
To compute the final scalar highlight score ÿ per frame fú, fuse the outputs of the relevance (fr). informativeness (i), and uncertainty (ū:) heads. We adopt piecewise linear, uncertainty-aware scoring rule that penalizes predictions made with high uncertainty:
[dd + Afc.
if i To
(low uncertainty)
Út aci + 3f, e(ū, Ta):
û To (high uncertainty)
22

---

Here, a and / weight the informativeness and relevance signals, Tu is an uncertainty threshold, and controls the penalty for predictions exceeding this threshold.
C.4.2 Theoretical Motivation
This scoring function is designed for the specific challenges of Online Highlight Detection (OHD), where the model operates under partial observability. The motivation mirrors principles from selective prediction and risk-aware pnecision-makingg [61] [3]: a system should only act (e.g.. flag highlight) when it is confident. The uncertainty signal de serves confidence gate. Below the threshold Tax scores are computed normally; above it, a linear penalty is applied to down-weight uncertain decisions, implementing simple risk-averse policy.
The piecewise linear design is deliberately chosen because is:
Modular: Each head is trained independently, enabling post-hoe fusion without complex joint optimization.
2. Interpretable: Highlight scores are directly influenced by human-readable weights and a confidence gate.
3. Stable: The threshold provides consistent behavior in streaming conditions, avoiding erratic outputs from minor uncertainty fluctuations.
4. Efficient: It requires minimal computation per frame, making it ideal for real-time inference.
C.4.3 Justification of Static Weighting
To justify choice of static weighting scheme, compared it against complex, dynamic alternatives on the TVSum benchmark. We evaluated three primary strategies, representing spectrum from robustness to Nomain--speiffe optimality: (I) two unstable dynamic methods, (2) robust static zero-shot heuristic, and (3) our top top-terfonee poppeefffrming domain domain-adapted static grid search.
Table 11: Comparison of scoring mechanisms on TVSum (Top-5 mAP).
Method
Top-5 mAP
Notes
Dynamic (MLP Gating)
87.9
Unstable, high variance
Dynamic (EMA Adaptor)
87.5
Unstable, high variance
Static (Zero-Shot Heuristic)
91.6
Most robust, SOTA baseline
Static (Grid Search)
93.0
Data-sensitive, optimal performance
The results in Table II empirically validate the two configurations presented in main results. The dynamic methods proved unstable and underperformed. In contrast, the Static (Zero-Shot Heuristic), which uses the fixed parameters baseline that already surpasses prior state-of-the-ar The Static (Grid Search) method further boosts performance achieve the optimal score, confirming the value of lightweight domain adaptation, though its outcome is sensitive to the validation data.
0.7, 1.0, c 2.9. and T. = 0.3, provides highly robust
The specific domain-adapted parameters found via this grid search, which were used to achieve our highest reported results, are detailed in Table I2
Table 12:
Optimal hyperparameters per dataset. Note: to reproduce these results you will likely need run your own grid search.
Dataset
8
TVSum 14
0.667
1.357
3.571
0.077
Mr.HiSum
0.000
1.778
0.714
0.040
Charades 43
0.888
2.0
-2.143
0.040
SCOUT [6
0.200
1.556
1.000
0.053
C.4.4 Comparison to Alternative Fusion Approaches
Our simple, modular scoring function was chosen over other common fusion techniques for its suitability in streaming context.
23

---

Using neural network to learn the fusion function sacrifices the inter- pretability and modularity that are critical for domain adaptation and risks overfitting.
Learned Fusion:
Attention-Baset Weighting: dynamic attention mechanism over the heads introduces ad- ditional parameters and potential instability in a streaming setting. complicating calibration.
Confidence- Weighted Blending: Using a continuous function (e.g., sigmoid scaling) is more complex tune and less interpretable than clear, thresholded gate.
Our design avoids the common pitfalls of these more complex techniques while enabling fast, stable, and theoretically-gronnee inference.
C.5
Justification for Methodological Design Choices
This section provides additional justification for two main design choices: (1) the use of a fixed-weight loss combination for multi-task training, and (2) the selection of specific model backbones for our AHA framework.
C.5.1 On the Use of Fixed- Weicht Multi-Tast Task Training
Using a fixed, weighted sum of multiple losses is not only de-facto standard in large-scale pre- training (c.g.. BERT [64] adds Masked LM and NSP losses) but also classic scalarization strategy in multi-objective optimization. When each task loss (0) is well-behaved, optimizing
T
Lo(0)
Σ (0),,, ==1
0, =1
Σ
converges to a point on the convex Pareto front without degrading another.
65 guaranteeing that no objective can be improved
Recent empirical studies have rigorously compared simple fixed-weight scalarization against more complex, specialized multi-task optimizers (SMTOs). These studies show that with appropriate normalization and tuming, scalarization can match or even surpass these dynamic methods on diverse benchmarks [66] 677. The primary advantages of this approach are its stability and scalability, as avoids the significant per step computational overhead inherent in dynamic re-weighting schemes. Standard regularization techniques like weight decay and dropout also help mitigate conflicting gradients, reducing the need for more complex optimizers. Thus, our choice is grounded in strong foundation of theoretical guarantees and practical evidence.
C.5.2 On the Selection of Model Backbones
The selection of the Qwen2 [29] and Lig [27] backbones was based on thorough review of high performing, open-source multimodal models at the time this work.
Visual Encoder (SigLIP): We selected SigLIP as it has been shown to offer competitive or superior generalization compared to CLIP, particularly at the smaller batch sizes that are characteristic of our online, per-frame processing setup.
Language Backbone (Qwen2): For the language backbone, adopted the LLLVA-One Vision architecture based on Qwen2. The distilled 7B variant of this model demonstrates SOTA performance while remaining lightweight enough for our framework.
Both Qwen2 and SigLIP are widely used and validated in concurrent streaming vision-language literature [13], showcasing their competitiveness and broad community adoption. While testing on additional backbones is an important direction for future work, these selections represent a well well-grounded starting point for establishing the AHA framework.
D Dataset Curation for Informativeness and Language Modeling Heads
This appendix provides further details on the creation of ground truth labels used to supervise the informativeness and auxiliary LM heads of the AHA framework, For both heads, we leverage existing video-language datasets with segment-level captions, specifically the human-annotater subset of
24

---

Shot2Story [35 and longer procedural videos from COIN [3]. The methodology for generating
supervision signals follows the same strategies employed in the streaming framework MMDuet [33
D.1 Supervision for the Informativeness Head
The informativeness head in AllA is trained to predict whether the current video frame fs introduces new information relative to the preceding context. Our approach for generating ground-trun labels for this task is a heuristic adopted directly from established work in streaming Video-LLMs [13]. The original motivation in that context was to train a model that knows when to speak during a continuous video stream, generating response only after acquiring sufficient context but before the moment becomes stale.
We adapt this principle to derive binary labels (f, E (0, 1}) as follows:
1. Segment Identification: We utilize video segments with corresponding cuman-genneated captions from the Shot2Story and COIN datasets.
2. Point of Sufficient Understanding: For each segment, we simulate a point where enough information has been seen to describe it. This point is randomly sampled to occur between 50% and 75% of the segment's ' duration.
3. Label Assignment: Frames from the 50% mark up to the "point of sufficient understanding" are labeled as informative (ie- 1). All other frames in the segment (before 50% or after the point) are labeled non-informative (i. = 0).
The underlying intuition is that initial frames may lack context, while frames after understanding is achieved are redundant. We hypothesize that this signal, which marks the accumulation of new information, correlates with highlight-worthy moments in an OHD setting. This is hypothesis supported by our strong ablation results (Table 3).
Decoupling Informativeness from Relevance. A key design choice in AНА is the explicit de- coupling of the informativeness head from the relevance head. While related, informational novelty (informativeness) and task -ikpiotttttncee (relevance) are distinct concepts. To validate that our model learns these different signals and that the concept of informativeness generalizes beyond the proce- dural videos used for training, we conducted qualitative analysis on the unconstrained, real-world SCOUT robotics video. Given the task objective (Q) "what objects are in this room?", we observed the following distinct behaviors:
High Informativeness, Low Relevance: When the robot enters a dark room, the drastic scene change correctly triggers high informativeness score due to visual novelty. However, with no task-relevant objects visible, the relevance score remains low.
Low Informativeness, High Relevance: Conversely, if task-relevant "calendar" is visible from afar, both scores are initially high. As the robot moves closer, the informativeness score drops because the visual context no longer novel. The relevance score, however, spikes as the calendar becomes clearly identifiable, confirming its task importance.
Correlated Signals: The scores often peak in unison when the robot enters a new area and immediately encounters task-relevant object (e.g., "a shovel"). Even in these cases, the relevance head typically produces a higher peak, correctly prioritizing the task-specific discovery over the general novelty of the scene.
This analysis confirms that our decoupled design is effective. The informativeness head successfully captures visual novelty in unconstrained environments, while the relevance head remains focused on the specific task objective, allowing more robust understanding of the video stream.
D.2 Supervision for the Auxiliary Language Modeling (LM) Head
To enrich the semantic quality of the hidden representations (h₂) learned by AHA, an auxiliary LM head is incorporated. This head is trained using the same dense captioning annotations from the Shot2Story and COIN datasets as the informativeness head.
The training process is as follows:
25

---

1. At randomly selected timesteps t during training, the LM head is tasked with generating a short, descriptive caption for the current visual context encapsulated by frame ft.
2. This generation is conditioned on the prior context available to the model (i.e., preceding visual tokens and the fixed task prompt Q and system prompt S).
3. The supervision provided via a standard cross-entropy loss for next-token prediction against the ground truth human-annotated captions corresponding to that segment (Eq. 6
It is crucial to reiterate a key design choice for AlIA that distinguishes its use of the LM head from some interactive VideoLLMs like MMDuet:
The captions generated by AHA'S LM head during training not re-injected into the model's input context.
Similarly, the LM head is typically not used during inference for the primary task of highlight detection (unless its hidden states are implicitly part of h4).
This approach strictly preserves AHA'S 's unidirectional, non-dialogue streaming behavior, ensuring it functions purely as a continuous scorer of video frames against a static task objective. The LM task serves solely mechanism improve the overall quality, alignment, and semantic richness of the hidden state representations (h₂) from which the primary highlight detection scores (relevance, informativeness, uncertainty) are derived.
E
Supplementary Quality Dropout Details
This appendix section offers supplementary details and justifications for certain design choices introduced in the main paper's HIHD methodology (Section (3.3 and the robustness experiments (Section 4.2). These elaborations are provided here to expand upon descriptions that were necessarily concise in the main text.
E.1 Video Quality Dropout for Robustness Enhancement
To improve the robustness of Aha against visual artifacts and degradations commonly encountered in real-world video streams, we incorporate video quality dropout mechanism during the training data preparation phase. As described in Section 3.3 for each video in our HIHD data, 5-20% of its duration is randomly selected for augmentation. Frames within these selected segments undergo one of several random perturbation types, detailed below. This process helps the model learn to maintain performance despite noisy or imperfect visual input [41] Let fzz y) denote the pixel values at coordinates (z, y) of an input frame f.
Quality Degradation: This simulates general compression artifacts and loss of detail. The frame f is first downscaled to fixed low resolution (c.g., H!' W' 64 64) using bilinear interpolation, denoted as D(f; H' W'). This downscaled frame fsmall = D(f: H' W') is then upscaled back to the original dimensions U Uffsmatt H, W), to preserve blockiness. Finally, a Gaussian blur G with kernel size k (e.g., (5, 5)) and standard deviation a (e.g., 0) is applied.
x W using nearest-neighhor interpolation,
= Go, H' W'):HW")
This simulates digital transmission errors. The frame f is notionally divided into overlvergg blocks byj of size B, x B, (c.g.. 32 x 32). A fixed random noise pattern N E [0, Rmaz] Ba BB,xB.xe (e.g., Rmaz 49, representing noise intensity up to 49 for an 8-bit channel) is generated. For each block b,j, is replaced by N with a probability Proise (e.g., Proise 0.1). Let maj Bernouli((Pooee) be random variable for each block.
Block Noise:
[N mod B., mod B,) V(x,v)
if ML/BB.///// = if M = 0
f(
This operation is applied over all pixel coordinates (z,z).
26

---

This simulates reduced color depth, leading to visible bands in color gradients. Each pixel channel value P E [5.5 in the frame f quantized using quantization factor Q (e.g., Q 64)
Color Banding:
(x, = f(z. Q
for each channel c.
Blackout: This simulates complete loss of signal. All pixel values in the frame zero
f'(x,y) =
for all channels and coordinates (ar, 3).
During training, one of these dropout types is randomly chosen and applied to frames within the selected 5-20% dropout segments of video. To ensure the model always has some visual context for making predictions and prevent scenarios where all frames in its immediate processing window might be entirely obscured (e.g.. by consecutive 'blackout" frames), we limit consecutive blackout augmentations to maximum of frames (e.g.. X at ffp)). If frame is scheduled for blackout beyond this limit, milder form of degradation (such quality reduction color banding) or no augmentation applied instead for that specific frame, after which the possibility of blackout frames resumes. This prevents the model from being trained on entirely uninformative sequences over extended periods, which could hinder learning. Corresponding dropout masks are also generated alongside the HIHD data. This augmentation strategy critical for preparing AHA handle the unpredictable visual quality often present in real-world online video streams. A visual representation of these methods are shown in Figure 4
No Dropout
Blackout
Quality Degradation
Block Noise
Color Banding
Figure 4: Visualization of dropout modes used during training to simulate real-world sensor degra- dation and robustness challenges. From left to right: No Dropout (clean input), Blackout (entire frame lost), Quality Degradation (strong downsampling and blurring), Block Noise (random black patches), and Color Banding (aggressive color quantization). These corruptions are applied to random segments of video to improve model robustness to visual noise.
E.2 Quality Dropout Results: Kendall 7 and Spearman
This appendix presents supplementary results for the robustness analysis on the TVSum dataset, evaluated using Kendall's and Spearman's rank correlation coefficients. These metrics offer an alternative perspective on the models' ability to maintain ranking performance under various video degradations. As mentioned in the main text, these rank correlation metrics generally showed minor variations across conditions for our model, AHA (Ours), and reinforced the overall conclusions regarding robustness. The detailed scores are provided Table I3
The data Table 13 for AНA (Ours) shows that while there are some fluctuations, particularly with the blackout degradation, the rank correlation scores remain relatively stable across several milder degradation types, supporting the conclusions discussed the main paper.
'Screenshot from the YouTube video "Vlog #509 I'M A PUPPY DOG GROOMER! September 13, 2014" licensed under Creative Commons Attribution 3.0 (CC BY 3.0) via YouTube.
Source:
27

---

Table 13: Robustness to video degradations on TVSum (Kendall 7 applied to 20% of frames.
Spearman p).
Degradations
Model
Clean
+ColorBanding
+BlockNoise
+Quality
+Blackout
AHA (Ours) 0.28/0.40
0.28/0.40
0.29/0.40
0.28/0.39
0.24/0.34
F
Memory Management for Streaming OHD: From SinkCache to Dynamic SinkCache
This section details the evolution of memory management strategy, from adopting the standard SinkCache mechanism to developing our novel, higher-performing Dynamic SinkCache.
F.1 Standard SinkCache Hybrid Memory Baseline
To manage the challenge of unbounded KV cache growth when processing continuous video streams, our initial framework adopted the SinkCache mechanism [2T] This hybrid memory strategy ensures constant memory usage by maintaining two components: fixed set of initial Sink Tokens (k,) for long-term context (like the task objective) and sliding window Recent Tokens (kt-nct) for short-term context. Any tokens outside this combined memory are evicted. An illustration of this standard memory structure is provided in Figure
(a) Default KV Caching
(b) SinkCache
Current Token
Current Token
Ko
K
1
K
Kn ...
K
Ki-n
K,-
cached takens
sink tokens
cvicted tokeas
cached tokens
(c) Dynamic SinkCache
Current Token
Qa
Q,
K,-n
K,-
K
ovicted tokens
sink tokeas())
evieted tokens
eached tnkens
Figure 5: Comparison of memory structures: (a) Default KV Caching, which increases memory linearly. (b) SinkCache, where the current token attends to hybrid memory comprising a fixed set of initial sink tokens and a sliding window of recent tokens. (c) Dynamic SinkCache, where the sink dynamically constructed to contain only the task objective (Q) tokens, combined with sliding window of recent tokens. This preserves long-term context while maintaining constant memory.
E.2 Justification of Sliding Window Size (n=2048)
While the Dynamic SinkCache creates a targeted sink for the task objective, the size the sliding window for recent visual tokens (n) remains key hyperparameter. A larger window provides more short-term context at the cost of higher memory and computational overhead, while a smaller window more efficient but may lose critical immediate context.
To find an optimal balance, we conducted an ablation study on the standard SinkCache over various sink (Ik,) ) and window (n) sizes, with the results summarized in Table [4
28

---

Table 14: Ablation on TVSum for Standard SinkCache sink (A&/) and window (n) sizes.
SinkCache Config. (Ikel,n)
Top-S mAP
Spearman's
Kendall's
(32, 2048)
92.6
0,401
0.280
(16, 2048)
92.0
0,295
0.203
(32, 1024)
90.1
0.412
0.287
(40, 2560)
89.4
0.298
0.205
(16, 1024)
89.1
0,359
0.247
(16, 512)
84.0
0.216
0.145
The results show that a window size of 2048 paired with |k_ 32 sink tokens achieved the highest mAP. While performance degrades gracefully with smaller windows (e.g.. 72 1024 still achieves a strong 90.1 mAP), adopted this window size for our final Dynamic SinkCache implementation, as it provides the best performance by capturing sufficient recent visual context for the OHD task.
2048 proved be the optimal configuration. We therefore
F.3 Dynamic SinkCache: A Task-Focused Improvement
We hypothesized that the standard SinkCache's s method of using the few tokens as generic sink was suboptimal. These initial tokens capture the system prompt, task objective, and sometimes the first few video frames. We proposed that more targeted sink, containing only the essential task information, would provide cleaner and more effective long-term memory.
This led our novel approach, the Dynamic Cinke Instead of using the first tokens of the sequence, this mechanism dynamically constructs the sink to contain exclusively the natural language task objective tokens (Q). This ensures that the model's long-term memory is persistently and exclusively focused on its primary goal, preventing it from being diluted by less relevant initial context.
F.4 Comparative Analysis Memory Mechanisms
To validate final design choice (Dynamic SinkCache) and demonstrate the necessity of hybrid, task-focused memory system, we conducted comprehensive ablation study on TVSum. We compared five different memory management strategies, which are detailed below. The results are summarized in Table
Baseline Mechanisms. We first evaluated two simple, non-hybrid baselines. Sliding Window Only approach, which retains only the most recent visual tokens, performed poorly (69.5 mAP) because it eventually discards and forgets the long-term task objective. Conversely, a Static Window Only approach, which uses only the initial tokens context, performed even worse (63.2 mAP) it completely fails to adapt to new visual events in the video stream.
Unbounded KV Cache. As practical upper-bound, standard unbounded KV cache that retains all previous tokens achieved a strong 91.7 mAP. However, this method is impractical for real-world deployment, its linear memory growth consistently leads to out-of-memory (OOM) errors on the long videos common OHD tasks.
Standard SinkC The standard SinkCache (21). which combines generic sink of the initial sequence tokens with sliding window, proved to be a highly effective hybrid baseline. It achieved 92.6 mAP, outperforming the impractical unbounded cache while maintaining constant memory footprint.
Dynamic SinkCache (Ours). Our proposed method achieves the highest score of 93.0 mAP. By dynamically constructing the sink to contain exclusively the natural language task objective, it creates a more targeted and efficient long-term memory. This confirms our hypothesis that task-focused sink provides the optimal mechanism for context retention in OHD.
29

---

Table 15: Ablation study of memory mechanisms on TVSum (Top-5 mAP).
Memory Mechanism
Top-5 mAP
Notes
Sliding Window Only
69.5
Fails to retain the long-term task objective.
Static Window Only
63.2
Fails to adapt to new visual events.
Unbounded KV Cache
91.7
Strong performance but impractical (causes (OM).
Standard SinkCache
92.6
Effective hybrid memory. strong baseline.
Dynamic SinkCache (Ours)
93.0
Optimal performance with task-focused sink.
E,5
Limitation and Design Trade-off
key consideration of the Dynamic SinkCache is the trade-off between the sink size (determined by the task objective length) and the sliding window size for recent visual tokens. Our implementation assumes a fixed total memory capacity. The strong performance in our experiments is partly due to the concise nature of the task objectives in benchmarks like TVSum, which occupy small, reasonable portion of the cache (-45 tokens), leaving enough capacity for the sliding window tokens.
However, this showcases a limitation: the model's performance could degrade catastrophically if presented with an exceptionally long natural language objective. If task description were long enough to consume the entire memory budget, the sliding window for recent visual tokens would be eliminated. this scenario, the model would retain the task but lose all short-term visual context, rendering it unable perform the OHD task. This trade-off underscores the need for future work in developing more adaptive memory allocation schemes that can handle tasks with highly variable objective lengths.
G Supplementary Details for Real-World Robotic Evaluation on SCOUT Video
This appendix provides additional details that supplement the evaluation of AHA on the SCOUT video presented in Section 4.3
G.1 Additional SCOUT Video Characteristics
Beyond the general description of the SCOUT video [16] in the main text dong-horizon. continuous footage, degraded quality, sparse events), the video present further specific challenges relevant to real-world deployment. These include:
The footage contains periods of near-complete blackout (e.g.. when the robot navigates very dark areas) and intermittent signal static, addition the warping mentioned in the main text.
Severe Visual Degradations:
Domain and Visual Noise: The dataset is characterized by significant domain shift toward indoor navigation compared to common web datasets, and often contains high visual noise and unpredictable robot motion.
G.2 Ground Truth Annotation Specifies for SCOUT Qualitative Analysis
For the 8-minute qualitative analysis discussed in Section 4.3 the ground truth events (i.e. the peaks matching events) were identified by the authors of this paper. This process involved a visual comparison of AHA's 's highlight detection outputs (specifically. the predicted peaks after Savitzky- Golay smoothing
42
against:
1. Moments in the video where the robot was observed to be stationary, often indicating task completion or observation of a point of interest.
2. Timestamps corresponding to human-issued navigation instructions for the robot, as docu- mented the official SCOUT transcripts (16).
This refined how "meaningful actions" were correlated with AHA's 's predictions.
30

---

G.3 Nuanced Analysis of Predicted Peaks in SCOUT Evaluation
Section
reports that 16 of 18 predicted peaks from AHA aligned with human issued commands or follows:
meaningful actions. Further details on the remaining two peaks
One peak corresponded to AHA identifying object of interest (based mission context from SCOUT annotations) while the robot was still in motion executing a prior command.
The other peak did not strongly correlate with new command or clearly defined object of interest from the mission logs for that segment. This might represent model--erceived visual saliency not directly tied to the high-level task commands, or potential false positive.
While this analysis is preliminary and conducted on a single SCOUT video, the results encourage continued exploration of this domain and analysis on additional videos.
G.4 Expanded Implications and Future Work for Robotics Applications
The application of Aha to the SCOUT video suggests further implications for robotics beyond those outlined in Section
Targeted Operator Alerting: AHA could potentially alert human operator specifically if the robot perceives an object of interest that the operator might have missed, particularly it's unexpected finding occurs while the robot is still executing previous command.
Synergy with Human-Robot Dialogue Systems: Combining AHA's perceptual salience with intent aware dialogue systems, such as those explored in prior SCOUT work [68],
could:
Help flag video segments associated with human commands where perceptual ambi- guity (e.g.. unusual saliency detected by AHA that is not aligned with the stated task) might indicate potential misunderstandings execution challenges.
Assist in grounding conversational references (c.g., a human asking "what was that interesting thing we just passed?") to specific video segments highlighted by AHA.
Input for Multimodal Reasoning: АНaAS real-time, frame-level salience scores can serve valuable input signal for more comprehensive multimodal reasoning frameworks, helping to focus computational resources on the most pertinent segments of continuous video data.
H Query Templates for Task Objective Generation
For the Human Intuition Highlight Dataset (HIHD), synthetic task objectives (Q) are generated by programmatically transforming video titles using the following templates. Given video title represented as "[STRING]' query is randomly selected from this list:
query_ terplates
"[STRING]", Repeating the title itself can serve as direct query "What segment of the video addresses the topic [[STRING] ??", "At what timestamp can I find information about [[STRING] the video?". "Can you highlight the section of the video that pertains to [[STRING] ? "Which moments in the video discuss [[STRING] detail?", "Identify the parts that mention ' '[STRING]'."
"Where in the video is "[STRING] demonstrated or explained?",
"What parts are relevant to the concept of [ISTRING'?"
"Which clips in the video relate to the query '[STRING]'?" '?"
"Can you point out the video segments that cover '[STRING]
"What are the key timestamps in the video for the topic [[STRING] ??"
]
This process generates diverse set of queries for each video, enabling task-conditioned supervision.
31

---

I Future Work: Supervised Learning and Validation with MultiVENT-G
While our work establishes strong empirical baseline for OHD, two key areas for future improvement are the unsupervised nature of our uncertainty estimation and the inherent biases of our large-scale HIHD dataset. Our current approach to uncertainty unsupervised due to the profound difficulty of obtaining ground-truth truth confidence labels at scale. Similarly, HIHD relies on YouTube's "Most Replayed" data, a high throughput but imperfect proxy for importance that can be influenced by engagement driven biases like clickbait.
promising path address these limitations involves leveraging the recently released Multi dataset [44]. Focused on high stakes disaster events, MuliVEETTG provides two critical features missing from typical highlight detection datasets: (1) dense, frame-level event role annotations by human experts, and (2) human-annotater confidence scores (I-5 scale) for these annotations, This dataset offers a unique opportunity to advance our work in three key directions:
1. Towards Supervised Uncertainty: The annotator confidence scores can be transformed into ground-truth uncertainty labels (e.g.. 5/5 confidence maps to low uncertainty). This would allow training uncertainty head with a direct supervised loss, moving beyond the current unsupervised NLL objective. This critical step towards improving the interpretability and calibration of our model's confidence estimates, which is essential for the safety-critical applications we target.
2. Mitigation Dataset Bias: MultiVENT-G's expert defined event labels (e.g.. "EMERGENCYREEPONSEE can serve as a gold-standard signal of true relevance. This allows for future work in calibration and debiasing, where our model, pre-trained on the large-scale HIHDD, can be fine-tuned on MultiVENT-G. This process would help correct for systemic biases learned from the raw replay scores, yielding a model that more faithful to expert defined importance.
3. Validation in High Stakes Domains: By providing task-aligned ground truth for disaster events, MultiVENT-G allows for rigorous validation of our model the exact high stakes scenarios for which it is designed. This ensures that the relevance and uncertainty estimates converge toward what human experts deem critical in real- weal-world applications.
Despite its potential, integrating MultiVENTGG presents three primary challenges: Scale (MultiVENT- G's ~1.2k videos vs. HIHD's ~23k), Generalization (its specific ontology may constrain the learned representations), and Subjectivity (labels are from a small team of annotators). Our future work will focus on developing methods to address these challenges, aiming to create model that is not only scalable but also robust, well cell-calibrated, and grounded in expert knowledge.
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction accurately describe AHA, its autoregressive nature, its use of multimodal VI. model, the SinkCache mechanism, and its performance on standard benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes. Section Dactionnxxxpppe explicitly discusses the limitations of our approach and Ap- pendix A.1 details the high computational costs of our method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
33

---

Answer: [NA]
Justification: NA. The paper does not present formal theoretical results; its contributions are primarily empirical and architectural.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes. Section describes the model architecture, dataset characteristics, training procedures, and evaluation setup, providing sufficient detail to reproduce our results. Hyperparameters are listed in Appendix A, and we will release the codebase and the HIHM dataset described in Section 3.3 upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
34

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes. All code and data required reproduce the main experimental results will be made publicly available upon acceptance, including training scripts, evaluation code, and documentation. For the additional real-world evaluation in Section 4.3 use video from SCOUT, an existing public dataset, where a subset image data used to construct the video is already available. We obtained the specific video used in our study through cooperative agreement, and expect this video and others to he released as part of SCOUT 2.0 by the SCOUT authors prior the camera-ready deadline.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtppblic/guiueeeeeeeeeoccpepeee /otpe//uuuidee/oeSuuuuubiiii ssi aasionnPoicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, all experimental settings, including data splits, hyperparameter values and selection strategies, and optimizer details, are described in Section 4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our reported results are averaged over independent runs to reduce the impact of randomness from initialization and sampling (stated in Section 4). We chose to omit error bars for consistency with prior work, which did not report them. Will provide them during the rebuttal phase upon request.
35

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, Appendix A.I details the compute resources used, including GPU types (e.g., 6XNVIDIA A6000 for training), memory, and approximate training times for our main models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
36

---

Answer: [Yes]
broader impacts, including positive applications in, e.g., autonomous systems, and potential negative impacts such as surveillance concerns, along with potential mitigation ideas.
Justification: Yes,
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes. Section discusses the potential for misuse, including surveillance- related privacy concerns and content selection biases. To address these risks, we outline recommended safeguards such pairing the model with privacy- privacy-preserving preprocessing (e.g.. blur filters), enforcing access controls, and conducting commain-ppecffi audits prior deployment. We will also include usage guidelines in the public codebase discourage irresponsible use and encourage ethical application.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
37

---

Justification: Yes, all existing assets (datasets, pre-trained model components) are credited, and the paper does not use existing assets requiring such licensing declarations beyond standard library citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces the Human Intuition Highlight Dataset (HIHDD (Section Its construction, data processing, components, synthetic task objectives, and quality dropout masks are detailed. Appendix I and F.I references query templates and dropout specifics, indicating documentation. These will be released publicly upon acceptance.
Justification:
3.3
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA. The research did not involve crowdsourcing or direct experiments with human subjects for data collection beyond using pre-existing, appropriately licensed datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
38

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: NA. The research did not involve human subjects in a way that required IRB approval, as it utilized pre-existing datasets or data collection methods exempt from such review.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Yes. Appendix A.1 describes the use of Qwen as a core component of our vision- vision-languaen language model for natural language task understanding. Additionally, we evaluate our model on the MAGQA benchmark, which leverages LLMs for question generation and evaluation (see Appendix C).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM for what should or should not be described.
39


Reviews:
[{"review_id": "3gKDiSgUXv", "reviewer": "Reviewer_79jx", "summary": "The paper examines the limitations of existing video highlight detection methods, which typically assume access to the full video during inference. This assumption limits their applicability for real-time streaming and online scenarios that require step-by-step reasoning for real-time decision-making. To address this, the paper introduces AHA (Autoregressive Highlight Detection), a framework for Online Highlight Detection (OHD) that predicts highlight scores for the current frame using only past and present information in a causal manner given a task described in the natural language.\n\nAHA comprises of frozen visual encoder (pretrained SigLip) for extracting frame features, Multimodal projection (linear layer) to map visual embeddings to large language model token space, Autoregressive decoder for decoding during streaming. AHA is trained using four lightweight prediction heads, each optimized for a different objective: task-conditioned relevance, informativeness, uncertainty, and an auxiliary captioning objectives. To prevent linear memory growth during inference, the framework incorporates a SinkCache memory mechanism.\n\nTo effectively supervise the training process, the authors also introduce a large-scale dataset called the Human Intuition Highlight Dataset (HIHD). Extensive experiments demonstrate AHA’s effectiveness in both offline highlight detection (on TVSum and HiSum benchmarks) and as a real-time reasoning module for downstream planning and long-horizon understanding tasks.", "questions": "Please refer to the points listed in weaknesses above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper identifies a relevant research problem in video understanding and highlight detection that existing models fail to support real-time decision making in continuous video streams. To address this, it proposes a novel autoregressive highlight detection framework (AHA) that predicts the relevance of each video frame with respect to a task described in natural language.\n2. The paper is well-motivated and well written, backed by comprehensive evaluations on offline highlight detection using the TVSum and HiSum benchmarks, where it achieves state-of-the-art performance measured by mean average precision. Real-world testing on long-form robotics videos from the SCOUT dataset further demonstrates AHA's ability to identify highly salient moments in real time. \n3. The paper also introduces the Human Intuition Highlight Detection Dataset (HIHD), created to train and benchmark task-conditioned online highlight detection (OHD) models.\n\nWeaknesses\n1. The weights assigned to each loss component relevance, informativeness, uncertainty, and the language modeling head are kept fixed throughout the training. Could this limit the model’s adaptability to different backbone architectures or downstream tasks?\n2. What is the underlying intuition for encouraging variance diversity in the uncertainty loss, as defined in Equations 5a and 5b?\n3. What is the rationale behind the specific choice of SinkCache Memory mechanism to address the unbounded KV cache growth when processing continuous video streams during inference? Are there any alternative methods that were taken into consideration?\n4. Abbreviation mAP (line 19) is used before its full form Mean Average Precision in line 258.", "was_revised": true, "final_justification": "Most of my concerns regarding weights assigned to each loss component, intuition for encouraging variance diversity in the uncertainty loss and specific choice of SinkCache Memory mechanism have been satisfactorily resolved. Therefore, I will raise my current score.", "rebuttal": "We sincerely thank the reviewer for their positive assessment, their high evaluation of our paper's clarity, and their insightful questions. This feedback has been instrumental in helping us improve the clarity, rigor, and overall quality of our work. We provide a detailed response to each of the weaknesses raised below.\n\n## W1 (Fixed Loss Weights and Adaptability)\nThe reviewer raised an excellent and important point regarding our use of fixed loss weights and its potential impact on adaptability. Our choice is grounded in a combination of well established theoretical principles and strong empirical evidence from the multi-task learning literature.\n\n**Theoretical and Empirical Justification:** Using a fixed weighted sum of losses is a classic scalarization strategy in multi-objective optimization. It is not only a standard in large scale pre-training (e.g., BERT[1]) but also theoretically guaranteed to find a solution on the convex Pareto front when individual loss functions are well behaved[2]. This means no single objective can be improved without degrading another. Furthermore, recent empirical work has rigorously compared this method against more complex specialized multi-task optimizers (SMTOs) and found that, with appropriate normalization, simple scalarization can match or even surpass them on diverse benchmarks[3]. The main advantages include its scalability and simplicity, as it avoids the significant per step overhead inherent in dynamic re-weighting schemes.\n\n**Adaptability to Backbones:** Regarding adaptability to different backbone architectures, we agree that this is a crucial aspect of generalization. Our backbone choices were grounded in a thorough literature review of high-performing open-source multimodal models. We selected Qwen2[4] and SigLIP[5] as they represent state-of-the-art open source vision-language models.\n\nSigLIP was chosen as the visual encoder because it has been shown to outperform CLIP at smaller batch sizes (e.g., 4-8k which is reflective of our input in AHA) while offering competitive generalization. For the language backbone, we adopted the LLaVA-OneVision architecture based on Qwen2, which demonstrates strong performance in its distilled 7B variant, which we use for our lightweight model. Both Qwen2 and SigLIP are widely used in concurrent recent streaming vision-language literature[6], further underscoring their competitiveness and community adoption. We will add this additional justification to the paper when describing these components.\n\nWe acknowledge that testing on additional backbones would be valuable as these underlying models continue to advance. In future work, we plan to expand AHA to additional backbones and explore meta learning strategies for dynamic head weighting to further improve generality. This paper represents the first step for AHA, laying the foundation upon which future extensions can generalize across a broader range of backbone architectures. We will include a summary of the rationale for our backbone selection in Section 3 of the main paper, as well as the detailed explanation in Appendix D. \n\n\n\n## W2 (Intuition for Uncertainty Loss (Eqs. 5a, 5b))\nWe thank you for pointing out the lack of clarity here. We apologize for failing to properly cite our detailed justification (Appendix D.3) in the main paper. We have now added a summary to the main text (included below) and will ensure it correctly references the full analysis in the appendix to make the connection clear for all readers.\n\n**Motivation for Uncertainty in an Online Setting:** The core motivation is that for *online, causal systems* like AHA that cannot see the future, uncertainty is not an optional feature but a **necessity** for robust decision making. Our model's uncertainty score is a proxy for its confidence given an unfolding context. A high score signals: *\"Based on what I've seen so far, this frame seems relevant, but I am not confident because crucial context might be missing.\"* While the direct performance gain is modest (a 3.5 mAP drop when removed as indicated in Table 3 on page 8), we view this as a crucial first step in modeling the unknown future, which is vital for the high-stakes applications AHA is designed for.\n\n**Intuition for Variance Diversity:** The uncertainty head is trained without explicit ground truth labels. We instead use a principled, unsupervised approach based on a Gaussian Negative Log-Likelihood (NLL) loss[7]. The specific intuition for encouraging variance diversity is to prevent *mode collapse*. This is a known pitfall where the model learns a degenerate solution by predicting a single, uninformatively high variance for all frames to trivially minimize the NLL loss[8]. Our diversity regularizer, $\\mathcal{L}_{\\text{div}}$, directly counteracts this by penalizing a low standard deviation of the predicted log-variances across a batch. This forces the model to learn an expressive and dynamic signal, allowing it to better distinguish between predictable and ambiguous moments in the video stream. \n\nWe acknowledge that this unsupervised approach has limitations, and in our roadmap we have some preliminary ideas on how to train this on a supervised signal. A promising direction is to leverage the MultiVENT-G dataset[9], which contains human-annotated confidence scores for disaster-related events. These scores can be directly mapped to uncertainty labels, enabling: (1) supervised training of the uncertainty head to improve interpretability and calibration, and (2) evaluation in safety-critical domains aligned with our target applications. Key challenges include the smaller scale of MultiVENT-G (1.1k videos vs. ~22.5k in HIHD), domain generalization beyond its event ontology, and the subjectivity of human-provided labels. These plans are summarized in our response to reviewer Zkgi (R4) and will be detailed in Appendix J of the revised paper.\n\n\n\n## W3 (Rationale for SinkCache and Alternatives) \nThis is an excellent question that prompted a deeper analysis of our memory mechanism and, we are excited to report, led to an improvement in our framework's performance.\n\n**Rationale and Alternatives Considered:** Our initial rationale for selecting the SinkCache mechanism was its unique hybrid approach to memory. We hypothesized that for task-conditioned Online Highlight Detection, a model must simultaneously maintain long-term context (the task objective) and adapt to recent visual information. SinkCache addresses this by retaining a fixed set of initial \"sink\" tokens while also keeping a sliding window of recent tokens.\n\nWe conducted additional experiments and compared the performance reported in our paper (92.6 mAP) to the performance on TVSum against several alternative memory strategies:\n- **Unbounded KV Cache (\"Full History\"):** This standard approach achieved a strong 91.7 mAP but is impractical for long videos due to its unbounded memory growth, which leads to out of memory errors.\n- **Sliding Window Only:** This strategy only retains recent context, eventually losing the vital task conditioning information. As a result, its performance was significantly lower at 69.5 mAP.\n- **Static Window Only:** This method only uses the initial tokens as context, failing to incorporate new visual events and performing poorly at 63.2 mAP.\n\n\nThese results confirmed our initial hypothesis that combining long-term and short-term memory is crucial, with our original SinkCache implementation achieving a 92.6 mAP and providing the best trade off between performance and memory efficiency.\n\n**A New Approach: Dynamic Sink Cache:** The reviewer's question inspired us to refine our hypothesis further. The initial sink tokens capture the system prompt, task objective, and sometimes the first few frames. This prompted us to wonder if we could create a more targeted sink containing *only* the essential task tokens. This led us to perform a new experiment where we dynamically initialize the sink tokens to contain *only* the essential task objective tokens (i.e. $\\mathcal{Q}$'s tokens), disregarding other initial inputs within the sink token.\n\nWe term this new approach **Dynamic Sink Cache**. This method yielded a new state-of-the-art mAP of **93.0** on TVSum, outperforming all previous configurations. This result provides strong evidence that the most critical component for long-term memory in OHD is the persistent natural language objective. Further results on the natural language objective are provided in our response to reviewer NtaH (R3), where we experimented with unrelated and ambiguous task objectives. We observed a substantial performance drop ($\\Delta = -9.7$ mAP) for unrelated objectives and a minor decrease ($\\Delta = -1.1$ mAP) for ambiguous objectives.\n\nWe are very grateful to the reviewer for prompting this line of thought, as it has tangibly improved our paper. We will update the main paper to include the Dynamic Sink Cache as our primary result and provide this expanded ablation study on memory mechanisms in Appendix G.\n\n## W4 (mAP Abbreviation) \nThank you for your careful reading and for catching this oversight. We have corrected the text to define mAP before its first use in the revised manuscript.\n\n\n\n[1]: JDevlin et al., \"BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.\"\n\n[2]: Miettinen, Kaisa., \"Nonlinear multiobjective optimization.\"\n\n[3]: Senushkin et al., \"Independent Component Alignment for Multi-Task Learning.\"\n\n[4]: Bai et al., \"Qwen Technical Report.\"\n\n[5]: Zhai et al., \"Sigmoid Loss for Language Image Pre-Training.\"\n\n[6]: Wang et al., “VideoLLM Knows When to Speak.”\n\n[7]: Nix, David A., and Andreas S. Weigend. \"Estimating the mean and variance of the target probability distribution.\"\n\n[8]: Seitzer, Maximilian, et al. \"On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks.\"\n\n[9]: Sanders et al., \"Grounding Partially-Defined Events in Multimodal Data.\""}, {"review_id": "bm0cR7vr84", "reviewer": "Reviewer_z47u", "summary": "This paper addresses the challenging and highly practical problem of Online Highlight Detection (OHD), where the goal is to assess the relevance of video frames in real-time from a continuous stream, without access to future information. The authors correctly point out that most existing highlight detection research focuses on offline processing, making it unsuitable for real-world applications like autonomous systems and robotics. To fill this gap, they propose AHA, a novel autoregressive framework for task-conditioned OHD. AHA is built upon a frozen vision-language model and employs three lightweight, decoupled prediction heads to score each frame on its relevance, informativeness, and uncertainty with respect to a natural language task. For efficient, long-horizon streaming with constant memory usage, the framework incorporates the SinkCache mechanism. A major contribution of this work is the creation and release of the Human Intuition Highlight Dataset (HIHD), a new large-scale dataset of ~24k videos. HIHD is specifically designed for this task, using YouTube engagement statistics as a proxy for human-perceived relevance and incorporating programmatically generated task queries and data augmentations to improve robustness. The proposed AHA framework achieves new state-of-the-art results on standard benchmarks, remarkably outperforming even offline, full-context methods, and demonstrates its practical utility on a challenging, long-form robotics video.", "questions": "1. **Inference Weighting:** The final highlight score uses fixed weights (α,β,ϵ) determined by a grid search. Have you considered any dynamic weighting mechanisms? For instance, could the model learn to adaptively adjust these weights based on its own uncertainty prediction (u^t), potentially down-weighting the relevance score more heavily when uncertainty is very high?\n2. **Impact of the LM Head:** The auxiliary Language Modeling (LM) head is used during training to enrich the model's representations. Could you please provide an ablation study that quantifies its impact on performance? It would be valuable to know how much the final highlight detection scores are affected if the LLM term is removed from the total loss.\n3. **Generalization of Informativeness:** The informativeness head is trained primarily on procedural videos from COIN and Shot2Story. How well do you think this learned concept of \"informativeness\" generalizes to the more unconstrained domains found in TVSum or the real-world robotics video? Are there scenarios where a frame might be \"informative\" (i.e., introducing new information) but not necessarily a \"highlight\" for a given task, and how does the model handle this?\n4. **On the Generalizability to Different Backbone Models :** The paper demonstrates state-of-the-art performance using the Qwen-7B LLM backbone and the SigLIP vision encoder. This raises a question about the generalizability of the AHA framework with respect to the choice of these backbone models. Have the authors experimented with other popular backbones? For example, it would be valuable to see an ablation study comparing the performance when using an alternative vision encoder like CLIP, or a different family of LLMs such as Llama. Such a comparison would help disentangle the performance gains from the novel AHA architecture itself versus the inherent strengths of the selected SigLIP and Qwen backbones, and would further demonstrate the robustness of your proposed method.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- **Significance and Problem Formulation:** The paper tackles a critical and underserved problem in video understanding: *online*, task-conditioned highlight detection. The authors provide a clear and compelling motivation, effectively distinguishing OHD from the dominant offline HD paradigm and highlighting its importance for real-world intelligent agents.\n- **Major Dataset Contribution:** The creation and release of the HIHD dataset is a substantial contribution to the field. The methodology is innovative, using large-scale user engagement signals as scalable supervision, programmatically generating task-driven queries, and introducing a \"quality dropout\" mechanism to train for real-world robustness. This dataset will be an invaluable resource for future research in OHD.\n- **Novel and Well-Designed Framework:** The AHA architecture is elegant, robust, and well-suited for the task. The inclusion of a dedicated **uncertainty head** is a key novelty for OHD, appropriately addressing the challenge of making predictions under the partial observability inherent in streaming. Furthermore, the adoption of **SinkCache** is a critical and well-justified design choice that enables true, constant-memory, long-form video processing.\n- **Comprehensive and Convincing Evaluation:** The experiments are exceptionally thorough and the results are state-of-the-art. AHA not only surpasses previous OHD attempts but also outperforms strong offline methods on standard benchmarks like TVSum and Mr. HiSum. The extensive ablation studies, robustness checks against video corruptions, and the qualitative evaluation on the real-world SCOUT robotics dataset provide compelling evidence of the model's effectiveness and practical potential.\n\n**Weaknesses:**\n\n- **Fixed Inference Weights:** The final highlight score is a linear combination of the outputs from the three prediction heads, with weights that are determined via a grid search for each dataset. The authors acknowledge this as a limitation. While effective, this static weighting scheme might not be optimally robust across diverse, unseen domains or queries with varying complexity.\n- **Dependence on Engagement Data as Ground Truth:** The relevance head is trained on YouTube replay statistics. While this is a clever and highly scalable proxy for human interest, engagement data can have its own biases (e.g., influenced by clickbait titles or thumbnails) and may not always align perfectly with \"importance\" in high-stakes, task-driven applications (e.g., disaster response or security).\n- **Heuristic for Informativeness Labels:** The method for generating labels for the informativeness head (sampling a \"point of sufficient understanding\") is a reasonable heuristic borrowed from prior work. However, it remains an indirect proxy for true informational novelty and may not perfectly capture all aspects of what makes a frame \"informative\" in a general OHD context.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal, I have confirmed the paper has been strengthened and will maintain my original rating. The rebuttal effectively addressed my most critical concerns, particularly regarding the weighting combination issue and the generalization of 'informativeness', through additional experiments and analysis. While it is noted that experiments on the impact of the auxiliary LM head and backbone model were not conducted due to resource constraints, I consider this an acceptable limitation that does not detract from the paper's core contribution. The rebuttal, therefore, serves to reinforce the potential identified in my initial review, and for this reason, I find it appropriate to maintain the current score. As a final recommendation, I ask that the new weighting method and the analysis on informativeness, as detailed in the rebuttal, be clearly integrated into the camera-ready version of the paper.", "rebuttal": "We are grateful to the reviewer for their exceptionally thorough and positive review. We are thrilled that you recognized the significance of the OHD problem, found our framework to be \"elegant, robust, and well-suited,\" and considered our evaluation \"comprehensive and convincing.\" Your insightful questions have prompted us to conduct further experiments and clarify key aspects of our work, which we believe have made the paper even stronger. We address each of your questions in detail below.\n\n## Q1, W1 (Dynamic Inference Weighting)\nThe reviewer raised an excellent point about our static inference weights and suggested exploring dynamic mechanisms, such as using the uncertainty prediction to adaptively adjust the final score. We followed this suggestion and ran a new set of experiments to investigate this.\n\nWe compared our static grid-search approach against two primary dynamic methods on the TVSum dataset using a standard 80/20 train/test split[1]. We investigated:\n- A small **MLP Gating network** trained to learn per-frame weights ($\\alpha_t, \\beta_t, \\epsilon_t$).\n- An **EMA based Adaptor** that adjusts weights based on each head's recent predictive error.\n\nInterestingly, both of these learned approaches proved to be unstable on the small TVSum training set (standard deviation >7), leading to lower average performance (87.9% mAP for the MLP and 87.5% for the EMA adaptor) as compared to our originally reported 92.6% performance. We attribute this instability to overfitting, a known challenge when training complex fusion mechanisms on smaller datasets.\n\nIn contrast, static weighting methods proved far more effective and robust. In fact, inspired by the reviewer's feedback, our investigation led us to a new, simpler, parameter-free heuristic that performs on par with our original grid search, reaching a stable mAP of **93.0%** when combined with our new Dynamic Sink Cache (detailed in response to reviewer 79jx - R1). This finding suggests that our decoupled prediction heads are already well calibrated and produce robust, directly usable scores, making a complex learned weighting scheme unnecessary and potentially detrimental on existing benchmarks.\n\nBased on this new analysis, we will update the main paper to use this simpler, more generalizable, and higher-performing scoring function. We will also add these new ablations to Appendix D.4. We again thank you for this suggestion, as it has led to a more robust final model.\n\n## Q2 (Impact of the Language Modeling (LM) Head)\nThank you for suggesting this valuable ablation. We agree it is an important analysis. Due to the high computational cost of retraining our model (see Appendix A.1) and the tight rebuttal deadline, we were unable to run this experiment in time. We have added this to our immediate research plan to investigate for future iterations of this work.\n\n## W2 (Dependence on Engagement Data as Ground Truth)\nWe appreciate the reviewer’s point that using YouTube \"Most Replayed\" metric as a ground‑truth relevance signal introduces biases (e.g., clickbait amplification) that may misalign with true importance in safety critical tasks. We deliberately chose this proxy to scale from prior benchmarks of ~50 manually labeled videos to HIHD’s ~24k videos, accepting “popularity” as an imperfect but high‑throughput signal. There are two promising directions of future work that we will add to the discussion in the paper:\n- **Document transparently** We have released a comprehensive report detailing the creation of the HIHD data. By providing this, we aim to empower the community to develop new fairness evaluation frameworks, tooling, and mitigation strategies tailored to engagement based datasets, as well as the responsible AI community improving our current dataset.\n- **Future work in debiasing** For future work, we plan to explore mitigating these biases by first calibrating our model on an expert labeled dataset in high-stakes domains like MultiVENT-G (more below), and then using adversarial training to make the model robust to misleading engagement signals like clickbait, inspired by prior work[2].\n\n**Expert Annotated Signals via MultiVENT‑G:** To move beyond passive engagement proxies, we are actively exploring the recently released MultiVENT‑G dataset[3] which provides dense, frame level event role annotations by humans, into our training and evaluation pipelines. Additionally, uncertainty labels can be extracted from annotator confidence scores where available. Although MultiVENT‑G’s scale (~1.2k videos) is smaller than HIHD, it offers task‑aligned ground truth that can be used to fine‑tune or validate our heads in high‑stakes domains. Currently our issues are addressing challenges of scale mismatch, annotator subjectivity, and domain specialization, which will be a centerpiece of our future work, ensuring that AHA’s relevance and uncertainty estimates converge toward expert defined importance without sacrificing generalization. We will include more details in Appendix J of our revised paper.\n\n\n\n## Q3, W3 (Generalization of \"Informativeness\")\n\nWe thank the reviewer for this insightful question. The reviewer has correctly noted that \"informativeness\" (informational novelty) and \"relevance\" (task-importance) are not always the same, and ask how a concept trained on procedural videos generalizes. Our framework is explicitly designed to address this distinction using decoupled prediction heads. We report results on additional experiments we have run to illustrate these points.\n\nAn extended experiment of the real-world SCOUT robotics video (Section 4.3 in the main paper) reveals three key behaviors that demonstrate this generalization. With the task \"what objects are in this room?\", the model correctly assigns:\n\n\n- **High Informativeness, Low Relevance** to visually novel but task irrelevant events, such as a robot entering a dark room or panning across an empty wall.\n- **Low Informativeness, High Relevance** when focusing on a task-critical object that is no longer new to the scene, for instance, moving closer to a calendar that was already visible in the distance.\n- **Correlated Informativeness and Relevance:** the scores often peak in unison when the robot transitions to a new area and immediately encounters a task-relevant object (e.g., \"a shovel\"). In these cases, the relevance head typically produces a higher peak, correctly prioritizing the task-specific discovery over the general novelty of the scene.\n\nThese new experiments show that our model successfully disentangles these two crucial signals, allowing it to remain focused on the task objective while still recognizing visual novelty. We have expanded Appendix E.1 with this analysis, provided a summary in the main paper, and will include illustrative figures in the revised paper version. We are unable to include the figures in this rebuttal due to NeurIPS' policy against new graphics or links to external sources.\n\n**Justification for the Informativeness Labeling Heuristic:** The reviewer correctly identifies that our method for generating informativeness labels is a heuristic. We apologize for not making the motivation clearer, and will revise the paper to include this explanation. Our approach is adopted directly from established work in the streaming Video-LLM literature[4]. In that context, the goal is to train a model that knows *when to speak* during a continuous video stream, generating a response only after acquiring sufficient context but before the moment becomes stale.\n\nThe underlying intuition we adapt is that initial frames may lack context, while frames after a \"point of sufficient understanding\" are redundant for describing a segment's core content. Our informativeness head is trained using a Binary Cross-Entropy (BCE) loss on labels derived from this principle. We hypothesize that this signal, which marks the accumulation of new information, correlates with highlight worthy moments in an OHD setting, which is supported by our strong results in Table 3 (left) on page 8.\n\n## Q4 (Generalizability to Different Backbone Models)\nThis is an important question about disentangling the contributions of our AHA architecture from the strengths of the chosen backbones. We selected **Qwen2**[5] and **SigLIP**[6] because they represent the state-of-the-art in open-source vision and language models, providing a strong and reproducible foundation. Our choices were grounded in a thorough literature review; for instance, SigLIP outperforms CLIP at the batch sizes relevant to our framework, and our specific Qwen2-based architecture[7] has a higher performance in distilled versions (7B).\n\nWe fully agree that an ablation study using other popular backbones like **LLaMA** or **CLIP** would be valuable. However, adapting AHA to entirely different model families requires substantial engineering and re-training that was unfortunately not feasible within the limited rebuttal period. This is a key priority for our future work. This paper represents the first step for AHA, and we have laid a foundation that we hope others in the community can build upon by integrating our architecture with a broader range of backbones. We have clarified our rationale for the current backbone selection in Section 3 and emphasized this limitation and future work direction in Section 5 of our revised manuscript.\n\n\n[1]: Liu et al., \"UMT.\"\n\n[2]: Zhang et al., \"Mitigating Unwanted Biases with Adversarial Learning.\"\n\n[3]: Sanders et al., \"Grounding Partially-Defined Events in Multimodal Data.\"\n\n[4]: Wang et al., \"VideoLLM Knows When to Speak.\"\n\n[5]: Bai et al., \"Qwen Technical Report.\"\n\n[6]: Zhai et al., \"Sigmoid Loss for Language Image Pre-Training.\"\n\n[7]: Bai et al., \"Qwen-VL.\""}, {"review_id": "KvlE7qtnvB", "reviewer": "Reviewer_NtaH", "summary": "This paper proposes AHA, an autoregressive model for online highlight detection in video streams without relying on future frames. The framework integrates a multimodal transformer architecture enhanced with multiple heads for relevance, informativeness, and uncertainty prediction.  AHA incorporates the SinkCache mechanism, a approach enabling constant-memory inference for handling continuous video streams. The authors further introduce a new large-scale Human Intuition Highlight Dataset (HIHD) to facilitate model training. Experiments illustrate that AHA outperforms current state-of-the-art methods.", "questions": "1. How sensitive is the model's performance to variations in the chosen loss weights?\n2. How robustly does SinkCache retain good context over extended video durations (e.g., hours-long streams)?\n3. How reliable is AHA when provided ambiguous or irrelevant task conditioning?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed AHA model advances in real-time video understanding without the dependence on future frame information, thereby aligning well with practical streaming scenarios.\n2. The proposed method demonstrates strong performance under causal conditions on benchmark datsets.\n3. The new HIHD dataset provides contributions for future research in OHD.\n\nWeaknesses:\n1. Although justified through ablation studies, the multi-head training and fixed weighting strategy could benefit from more theoretical or automated justification.\n2. The practical advantages of uncertainty modeling are unclear.\n3. In additional tasks, the model exhibits redundant or repetitive predictions.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed all my concerns and questions. After reading the rebuttal and other reviewers' comments, I decide to keep my score and recommend accept.", "rebuttal": "We sincerely thank the reviewer for their positive assessment and recognition of our work's strengths, and for providing constructive feedback with insightful questions that help us further improve the paper.\n\n## Q1, W1 (Justification for Multi-Head Training and Weight Sensitivity)\nThe reviewer raised an excellent point about providing more justification for our fixed-weight, multi-head training strategy beyond the initial ablation studies. We provide our response here, and will add it to the main paper.\n\n**Theoretical and Empirical Justification:** Our choice follows established practices in prior work. Using a fixed weighted sum of multiple losses is a common approach in large-scale pre-training (e.g., BERT[1]) and aligns with the scalarization strategy in multi-objective optimization. When each task loss $L_i(\\theta)$ is well-behaved, optimizing $$L_0(\\theta) = \\sum_{i=1}^T w_i\\,L_i(\\theta), \\quad w_i>0,\\;\\sum_{i=1}^T w_i=1$$\n\nconverges to a point on the convex Pareto front[2], guaranteeing that no objective can be improved without degrading another. While we demonstrated theoretically this would represent trade-offs, we agree an empirical sensitivity analysis is an important analysis. Due to the high computational cost of retraining our model (see Appendix A.1) and the tight rebuttal deadline, we were unable to run this experiment in time. We have added this to our immediate research plan to investigate for future iterations of this work.\n\n## W2 (Practical Advantages of Uncertainty Modeling)\n\nWe are happy to clarify the practical advantages of the uncertainty head. Its primary advantage is **improving the robustness of decision-making in a streaming context**.\n\nFor an *online, causal system* like AHA that cannot see the future, expressing uncertainty is a necessity, not an optional feature. A high uncertainty score signals: *\"Based on what I've seen so far, this frame seems relevant, but I am not confident because crucial context might be missing.\"* This is critical in high-stakes applications like disaster response, where a model providing a deterministic score without expressing confidence could be dangerously misleading.\n\nThe practical benefit is empirically demonstrated in our ablation study (Table 3 left, page 8), where removing the uncertainty head results in a concrete **3.5 mAP drop**. This shows that the uncertainty score acts as a valuable regularizer, helping the model to appropriately temper its relevance predictions based on its confidence. However, the uncertainty head is trained without explicit ground-truth labels, which we note as a key limitation in Section 5. This is due to the profound difficulty in obtaining such labels. A \"highlight\" is subjective, and an annotator's confidence is even more so, making it an unsolved challenge to collect these labels at scale. \n\nA promising direction in obtaining these supervised uncertainty scores is to leverage the MultiVENT-G dataset[3], which contains human-annotated confidence scores for disaster-related events. These scores can be directly mapped to uncertainty labels, enabling: (1) supervised training of the uncertainty head to improve interpretability and calibration, and (2) evaluation in safety-critical domains aligned with our target applications. Key challenges include the smaller scale of MultiVENT-G (1.1k videos vs. ~22.5k in HIHD), domain generalization beyond its event ontology, and the subjectivity of human-provided labels. These plans are summarized in our response to reviewer Zkgi (R4) and will be detailed in Appendix J of the revised paper.\n\n\n## W3 (Redundant Predictions in Additional Tasks)\n\nThis is an excellent observation. The repetitive predictions on auxiliary generative tasks (like captioning) are an expected outcome, as our framework was not optimized for this. Our primary research goal was to design and optimize a robust framework specifically for OHD. The evaluation on other tasks was a preliminary investigation into the generalization capabilities of our learned representations.\n\n\nThe issue of repetitive text is a challenge observed in streaming VideoLLMs, and state-of-the-art generative models use specialized techniques to mitigate it (e.g., advanced decoding strategies, KV cache manipulation). Our work provides a strong foundation for future research to build upon by integrating these generative-specific optimizations into our OHD framework, and we see this as a promising direction for future work. The code will be made public upon acceptance, and we encourage others to see if they can adapt AHA to these tasks.\n\n## Q2  (SinkCache's Robustness Over Extended Durations)\n\nWe thank the reviewer for this excellent question that addresses the core challenge of real-world, long-form video analysis. This is an important area, especially for what we have envisioned the works that will come after will be like summarizing long videos such as those in the disaster response or medical domains. We are actively searching for long video highlight datasets. While standard highlight detection benchmarks for hour long videos do not currently exist, we can address this based on the underpinnings of our model and the new ablation studies we have conducted. We believe these are great points that should transfer well to long-term video equally, and which we will explore in future works.\n\n\nOur choice of SinkCache is theoretically grounded to handle this exact challenge. By design, it separates memory into two crucial components:\n- A persistent **sink** that retains the initial, global task-conditioning tokens, acting as a long-term memory for the mission objective.\n- A fixed-size **sliding window** of recent tokens that processes the immediate context.\n\nOur new ablation studies (provided in our response to Reviewer 79jx - R1) validate this design: we found that simpler strategies like a sliding window only approach (which forgets the task) or a static-window-only approach (which fails to adapt to new events) perform significantly worse, with mAP of 69.5 and 63.2 respectively, compared to our reported 92.6 mAP. Furthermore, we performed a new experiment with a new \"Dynamic Sink Cache\" that achieves 93.0 mAP and supports making the sink even more task-focused improves performance, reinforcing the importance of this persistent memory.\n\nHowever, this approach involves an explicit **trade off**. The fixed size of the sliding window ($n=2048$ tokens in our experiments) means that specific visual context that appears and then falls outside this window will be forgotten. If that same object or scene reappears much later, the model would likely perceive it as \"novel\" again, which might not be ideal in specific cases.\n\nThis trade-off motivates a clear direction for future work, future architectures could incorporate a third, content-based memory bank. This bank would not just hold the initial language instructions but could be trained to dynamically cache compressed representations of task-relevant *visual* frames from the distant past. This would allow the model to build a true long-term memory of key events, creating a more robust and scalable solution for hour long streams. We are excited to test these ideas as new long-form benchmarks become available. We will include this in section 5 in the main paper, under limitations.\n\n## Q3 (Reliability with Ambiguous or Irrelevant Task Objectives)\nWe thank the reviewer for this excellent question regarding the framework's reliability under imperfect task conditioning. To provide a quantitative answer, we conducted a new set of experiments on the TVSum dataset to measure the impact of ambiguous and irrelevant task objectives on performance.\n\nFor this experiment, we define an \"ambiguous\" prompt as a higher level definition of the original specific task (e.g., using \"Vehicle Maintenance\" for a video titled \"How to change tires for off road vehicles\"). We also define an \"irrelevant\" prompt as a task description taken from a video in a completely different category (e.g., applying the prompt \"How to change tires for off road vehicles\" to a video having nothing to do with vehicles. We compare the performance in these conditions to our model's optimal top-5 mAP on TVSum, achieved with a standard, specific prompt.\n\n- **With an ambiguous prompt**, the model achieved a minor performance change by $\\Delta = -1.1$ points, indicating that the model successfully captures the broader topic without being overly penalized by the lack of specific details.\n- **With an irrelevant prompt**, performance decreased by $\\Delta = -9.7$ points. This is expected, as the model is guided toward incorrect content.\n\nWe will include a summary of this analysis in the main paper, a full description in Appendix C, and the specific prompts used in Appendix I of the revised paper. \n\n\n[1]: JDevlin et al., BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.\n\n[2]: Miettinen, Kaisa. \"Nonlinear multiobjective optimization.\"\n\n[3]: Sanders et al., \"Grounding Partially-Defined Events in Multimodal Data.\""}, {"review_id": "XVuadHzDC0", "reviewer": "Reviewer_Zkgi", "summary": "This paper presents an online highlight detection framework that processes continuous video streams in real-time without relying on future frames. Unlike existing methods that require the entire video for analysis, this framework works frame by frame, connecting video content with a natural language task description. It uses a multimodal vision-language model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To ensure scalability, it uses a memory-efficient technique called SinkCache, allowing it to handle very long videos with constant memory usage. An empirical study indicates that the proposed framework outperforms existing video highlight detection methods and demonstrates strong potential for real-world, time-sensitive applications like robotics and surveillance.", "questions": "Q1. How do you capture long-horizon video features when using fixed-length windows during video analysis? \n\nQ2. How could the window size affect the framework's performance? Is it possible to conduct experiments on various window sizes?\n\nQ3. When analyzing real-time video streams, is there a report on the framework's latency? What computing resources (e.g., GPUs) are required for real-time video analysis?", "limitations": "L1. The paper includes an uncertainty prediction head for uncertainty estimation. However, the projection head is trained without explicit uncertainty annotations (e.g., a calibration dataset with ground truth labels). This limits the interpretability and calibration of the uncertainty outputs, which could be critical in safety-sensitive domains.\n\nL2. This work is heavily relying on architecture design and empirical studies, but lacks theoretical support. It would be beneficial to include a discussion on formal guarantees on the projection heads' outputs, or some theoretical analysis on loss convergence and hyperparameter selections.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper introduces a framework that can process video streams in real time without processing future frames. This feature allows the framework to process ideally infinite-length videos with constant memory usage, which significantly reduces computation overhead.\n\nEmpirical studies show that the framework outperforms existing methods in highlight detection on benchmark datasets, showcasing its effectiveness with relatively small computation overhead.\n\nThe framework has potential in other streaming video understanding tasks beyond highlight detection, showcasing its broad applicability.\n\n**Weaknesses**\n\nAs mentioned in the limitation section, while it is effective in the inference stage, the training cost is relatively high.\n\nThe uncertainty estimation lacks formal or theoretical support.\n\n*Note to the authors*: Please respond to the **Questions** and **Limitations** sections during the rebuttal period (instead of responding to this section).", "was_revised": true, "final_justification": "I have read the rebuttal and think that the authors have addressed all the concerns I have. Hence, I will keep my score.", "rebuttal": "We thank the reviewer for their positive assessment and for highlighting our framework's real-time capabilities and strong empirical performance. We are grateful for your insightful questions and feedback on the paper's limitations, which we address below.\n\n## Q1 (Capturing Long-Horizon Features)\nOur framework captures long-horizon features via the **SinkCache** mechanism. This is a hybrid approach that separates memory into two critical components: 1) a persistent **sink** that retains the initial, global task-conditioning tokens (i.e., the natural language query), acting as a long-term memory for the mission objective, and 2) a fixed-size **sliding window** of recent tokens that processes the immediate visual context. Our new ablation experiment (detailed in our response to reviewer 79jx - R1) confirm this design is critical: a sliding window only approach fails because it forgets the task (achieving only a 69.5 mAP in comparison to our reported 92.6 mAP), proving that the sink is what retains the essential long-horizon information.\n\n## Q2 (Impact of Window Size)\nWe thank the reviewer for raising this point and apologize that our notation in Table 3 on page 8 was not sufficiently clear. We will revise the table caption in the revised paper version to explicitly state that $|k_s|$ denotes the number of persistent sink tokens and $n$ represents the size of the sliding window for recent tokens, which directly corresponds to the \"window size\" in question. Table 3 (right) details our ablation study over these parameters. Our results show that while our optimal configuration using a window size of $n=2048$ achieves our best mAP of 92.6, the framework's performance degrades gracefully as the window size is reduced. For example, halving the window to $n=1024$ maintains a strong mAP of 90.1, and even a minimal configuration with $n=512$ achieves an 84.0 mAP, demonstrating the model's robustness to smaller context windows. This analysis confirms that our chosen configuration provides an excellent balance between performance and memory efficiency. Should the reviewer find a more extensive analysis beneficial, we would be happy to report results on a wider range of window sizes.\n\n## Q3 (On Latency and Compute Resources)\n\nWe have conducted a detailed performance analysis of our framework on a 1062 second video (~17 minutes) using two NVIDIA A6000 GPUs. The system achieved a sustained throughput of 1 frame per second (FPS), demonstrating high efficiency with 100% peak GPU utilization and 90% peak memory controller utilization. During this process, the framework consumed a peak of 30.49 GB of VRAM across both GPUs and operated well within safe thermal limits at a peak temperature of 65°C, all while maintaining a minimal system RAM footprint of 3.66 GB. While this establishes a strong performance baseline, the 1 FPS rate means that in a live scenario, depending on the GPU, the system would fall behind the incoming video feed (drift), requiring frame-skipping logic for real-time deployment. Given that our framework already leverages the available compute effectively, it is a strong candidate for such optimizations to achieve the higher throughput needed for live applications. We thank the reviewer for bringing up a practical question and will expand Appendix A with this information, and include a summary and pointer to the Appendix in the main paper.\n\n## L1 (On Uncertainty Estimation without Ground-Truth Labels)\n\n\n**Justification for Unsupervised Training:** This is a critical point, and the reviewer is correct that the uncertainty head is trained without explicit ground-truth labels, which we note as a key limitation in section 5. This is due to the profound difficulty in obtaining such labels; a \"highlight\" is subjective, and an annotator's confidence is even more so, making it an unsolved challenge to collect these labels at scale.\n\nRegarding the lack of ground-truth labels, which we identify as a key limitation, we adopted a principled, unsupervised approach using a Gaussian Negative Log-Likelihood (NLL) loss[1]. This leads directly to the intuition for the variance diversity loss.\n\nThe underlying intuition for encouraging variance diversity in Equations 5a and 5b is to **prevent mode collapse**. A known issue with this type of unsupervised uncertainty modeling is that the model can learn a degenerate solution by predicting a single, uninformatively high variance for all frames to minimize the NLL loss[2]. The diversity regularizer, $\\mathcal{L}_{\\text{div}}$, directly counteracts this by penalizing a low standard deviation of the predicted log-variances across a batch. This encourages the model to produce a dynamic and meaningful range of uncertainty values, allowing it to distinguish between predictable and ambiguous moments in a video stream. We will supplement our discussion in the limitations with this additional information.\n\n**Future Work: Towards Supervised Uncertainty with MultiVENT-G:** We are actively exploring datasets that could provide labels for our supervised uncertainty learning in future work. One promising direction we have identified for *supervised* uncertainty-aware OHD involves leveraging the **MultiVENT-G** dataset[3], which focuses on complex, real-world events like disasters and provides a key feature missing in typical HD datasets: **human-annotated confidence scores**. Annotators rated their confidence on a 1-5 scale that a visual entity pertained to an event role.\n\nThis dataset offers three key advantages:\n- **Map Confidence to Uncertainty:** The confidence scores can be directly transformed into ground-truth uncertainty labels (e.g., a 5/5 confidence rating corresponds to low uncertainty).\n- **Generate a New Training Objective:** With these new labels, the uncertainty head could be trained directly with a supervised loss to address the critical issues of interpretability and calibration.\n- **Validate in High-Stakes Domains:** Since MultiVENT-G is focused on disaster events, it allows for the validation of uncertainty-aware models in the exact high-stakes applications our paper targets.\n\nHowever, there are currently three primary challenges with this approach: (1) **Scale**, as MultiVENT-G contains only 1,168 videos compared to the ~22.5k in our HIHD dataset; (2) **Generalization**, as training on MultiVENT-G's specific ontology may constrain the learned uncertainty; and (3) **Subjectivity**, as the labels are derived from a small team of annotators. We will create a new section in the appendix clearly outlining this promising direction and its associated challenges in Appendix J for our revised paper.\n\n**Addressing Safety-Critical Applications and Responsible AI:** Finally, in response to the reviewer's important point on safety-sensitive applications, we acknowledge this is currently limited by available data. A core goal in releasing our **HIHD dataset** is to provide a large-scale resource that can help the responsible AI community develop methods for identifying ground truth and safety critical labels. We are actively exploring collaborations to tackle these issues and, based on the reviewers feedback, will move a stronger summary of this future work to the main paper. Thank you for bringing this important topic to our attention.\n\n## L2 (On Lack of Theoretical Support)\n\nWe agree that our work is primarily empirical, focusing on introducing a new task (OHD), a novel framework, and a large-scale dataset. While we have not derived new formal guarantees, the architectural and training principles we use are grounded in existing literature. Specifically, our multi-head training approach uses a fixed weighted sum of losses. This is a scalarization strategy in multi-objective optimization that is theoretically guaranteed to find a Pareto-optimal solution[4]. It is also an empirically validated standard in large-scale deep learning.\n\nWe have now strengthened the paper's connection to this established literature and explicitly call out the need for future theoretical analysis of OHD systems as an important research direction. We thank the reviewer for encouraging us to make these connections clearer.\n\n[1]: Nix, David A., and Andreas S. Weigend. \"Estimating the mean and variance of the target probability distribution.\"\n\n[2]: Seitzer, Maximilian, et al. \"On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks.\"\n\n[3]: Sanders et al., \"Grounding Partially-Defined Events in Multimodal Data.\"\n\n[4]: Miettinen, Kaisa. \"Nonlinear multiobjective optimization.\""}]

Author rebuttal:

Meta-review: This paper introduces Aha!, a novel framework for Online Highlight Detection (OHD) that processes video streams autoregressively without future context, a significant departure from offline methods. Its core scientific claim is that real-time, task-conditioned highlight detection is achievable using a multimodal vision-language model with lightweight decoupled heads and a memory-efficient SinkCache mechanism. Key strengths include its state-of-the-art performance on benchmarks (e.g., +5.5% mAP on TVSum), the introduction of the large-scale HIHD dataset, and demonstrated applicability in robotics. Primary weaknesses, noted by reviewers, were the initial use of fixed inference weights, the unsupervised nature of the uncertainty head, and a lack of ablation on backbone model generalizability.

Reviewers (79jx, z47u, NtaH, Zkgi) raised pointed questions on loss weighting, uncertainty justification, SinkCache alternatives, and robustness to ambiguous tasks. The authors' rebuttal was exceptional, conducting new experiments that directly addressed these concerns. Notably, they introduced a improved "Dynamic Sink Cache" (inspired by 79jx) boosting performance to 93.0 mAP, provided a theoretical basis for multi-task training, and ran new tests showing graceful performance degradation with irrelevant prompts (-9.7 mAP) and minimal drop with ambiguous ones (-1.1 mAP). They also detailed a clear roadmap for future work on supervised uncertainty using the MultiVENT-G dataset, addressing a major limitation raised by multiple reviewers.

Considering the reviews and rebuttals, I recommend accepting this paper. It makes three substantial contributions: a novel and empirically validated architecture (Aha!), a valuable new dataset (HIHD), and a performance benchmark that surpasses offline models.
Decision: Accept (poster)