VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
Authors: Shenghui Chen
Po-han Li
Sandeep P. Chinchali
ufuk topcu
Venue: NeurIPS 2025 poster ()
Keywords: Human-AI Interaction
Video-to-Text Evaluation
Information Bottleneck
Primary area: evaluation
TL;DR: This paper proposes VIBE, an annotation-free method that selects video summaries for human decision-making by scoring task relevance and visual grounding without retraining.

Abstract: Many decision-making tasks, where both accuracy and efficiency matter, still require human supervision. For example, tasks like traffic officers reviewing hour-long dashcam footage or researchers screening conference videos can benefit from concise summaries that reduce cognitive load and save time. Yet current vision-language models (VLMs) often produce verbose, redundant outputs that hinder task performance. Existing video caption evaluation depends on costly human annotations and overlooks the summaries' utility in downstream tasks. We address these gaps with $\underline{\textbf{V}}$ideo-to-text $\underline{\textbf{I}}$nformation $\underline{\textbf{B}}$ottleneck $\underline{\textbf{E}}$valuation (VIBE), an annotation-free method that scores VLM outputs using two metrics: $\textit{grounding}$ (how well the summary aligns with visual content) and $\textit{utility}$ (how informative it is for the task). VIBE selects from randomly sampled VLM outputs by ranking them according to the two scores to support effective human decision-making. Human studies on $\texttt{LearningPaper24}$, $\texttt{SUTD-TrafficQA}$, and $\texttt{LongVideoBench}$ show that summaries selected by VIBE consistently improve performance—boosting task accuracy by up to $61.23$% and reducing response time by $75.77$% compared to naive VLM summaries or raw video.

VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR
Shenghui Chen,"
Po-han Li; Sandeep Chinchali, Ufuk Topeu
The University of Texas at Austin pohanli, sandeepc, utopau)@utexas.edu
{shenghui. chen, chen,
Abstract
Many decision-makinn tasks, where both accuracy and efficiency matter, still require human supervision. For example, tasks like traffic officers reviewing hour-long dashcam footage or researchers screening conference videos can benefit from concise summaries that reduce cognitive load and save time. Yet current vision-language models (VLMs) often produce verbose, redundant outputs that hinder task performance. Existing video caption evaluation depends on costly human annotations and overlooks the summaries' utility in downstream tasks. We address these gaps with Video-to-text Information Bottleneck Evaluation (VIBE), annotation-free method that scores VLM outputs using two metrics: grounding (how well the summary aligns with visual content) and utility (how informative it is for the task). VIBE selects from randomly sampled VLM outputs by ranking them according to the two scores to support effective human decision makinn. Human studies on LearningPaper22
TTTD-TrafficQA, and LongVideoBench show that summaries selected by VIBE consistently improve verformance-boosttie task accuracy by up to 61 23% and reducing response time by 75. 75.77% compared to naive VLM summaries or raw video.
Introduction
Efficiently extracting relevant information from extensive video is major bottleneck for human decision-making where both accuracy and efficiency matter [1-4]. Tasks demanding human supervi- sion, such traffic officer analyzing hours of dashcam footage determine fault researcher distilling key insights from lengthy oral presentation, are often limited by the time and cognitive load required process raw video streams. In this work, we aim to improve the quality and brevity of video summaries to boost human task performance compared to existing vision- laiiunl model (VLM) outputs and raw video, especially for longer clips where summarization offers greater utility.
Existing video caption evaluation metrics, however, rely heavily on reference-based comparisons to human-annotated summaries [5-8]. These metries face two main issues. First, these works require human annotators to watch video clips and write gold-standard captions, which contradicts the goal of redueing human response time and limits generalization to unseen video clips. Second, they are oblivious to downstream tasks and fail to measure how well captions support the tasks.
We propose Video-to-tex. Information Bottleneck Evaluation (VIBE), annotation- freee method for selecting task-rela mesk-relevant video summaries without model retraining. As shown in Figure 1, VIBE defines metrics-grounding grounding and utility scores-based on the information bottleneck principle [9]. uses pointwise mutual information to quantify how well a summary reflects video evidence and supports the downstream task. We leverage next-token prediction in VLMs access the probability
"Equal contribution (Order determined by coin toss).
Prroject Website, Code, and LearningPaper24 Dataset.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Video (V)
Summaries
Candidates (T)
Task (Y)
VLM Captioning
white webpage
background. says
Physics the Everyday' with yellow icon next to text. There bird next
vidoidoo peplores the unique physical odapections of Somingcs fcouding on their obllity to standion ure log censerce body heat ... teatiumms steuada itamingss througheut the video.
lico.
Grouncing = -149, Ubile
What kind of feethers
This viceo tcochcs the prycics principic regorcing birds vinserving, body heot due to beodyesseb. inferonces webologe called "Physic of the Everyday" for loaning rini physcco of everpday octhvitis.
does this bird have?
Grounding -2774; Unilyt
(A) Gmy
Black
This video from SraShow covwes the un quee cne-legged alaeping stancw of fammingo compunte
(C) Black & white
to a geomi tettaiicr, using diagrams and tootage to wepben their anatooy. c-cnnclutte with prometian for Enllant's Phypace coursee.
(D) Pink v
Grounding = 10;; Unilty m 0.6
IEI White
"Annotation-Free Rejection Sampling"
Grounding Score I(V:T) Utility Score I(Y:7)
L Prolific
VIBE
Response Time vs.
'Raw Video"
Accuracy
Figure 1: VIBE for Video-to- Text Summary Selection. Given a video, a task, and VLM-generated summaries, VIBE ranks the summaries using the proposed grounding and utility scores, which assess video alignment and task relevance. It selects the summary most conducive to helping human users achieve higher task accuracy and lower response time compared to watching the full video.
of generating summaries task answers. By comparing these probabilities with and without key information, we measure how well one modality (text or video) compensates for missing information in the other assess grounding and task relevance, shown later in Figure 2. Using these scores, VIBE selects the most decision-supporrive summary for humans from randomly sampled VLM outputs via annotation-free rejection sampling, which filters candidates without human labels for faster task completion than watching full video clips.
To evaluate VIBE, we conduct between-subjects user studies with 243 participants across three Latastts-LearningPaapp
(self-curated), LongVideoBench [10], and -TTD-TraffiCQA [[]]- measuring human performance in terms of accuracy, response time, and inverse efficiency score, the ratio of response time to accuracy [12]. Results show that summaries selected with maximal utility score improve task accuracy by up to 40%, while those chosen by maximizing grounding score yield up to 27.6% gains, both on the LongVideoBench dataset. VIBE-selected summaries also significantly reduce response time compared raw video. We also observe strong positive correlation between utility score and human accuracy, and a strong negative correlation between summary length and response time per word. These patterns highlight the value of concise, relevant information for efficient human decision-making.
Contributions. Our contributions are threefold: (a) We identify the need and propose the problem of annotation-free, task-aware evaluation for video-to-text summarization, improving human response time and accuracy without relying on gold-standard captions. (b) We propose VIBE, an annotation- free evaluation framework that combines grounding and utility scores of video clips to rank and select high-quality summaries from VLM outputs without requiring retraining. (c) We demonstrate through user studies that VIBE summaries significantly boost humans' task accuracy by up to 61.23% and reduce response time by up to 75.77% compared to standard VLM outputs.
Critique and Open Problems. This work reframes the video-to-text evaluation problem through the lens of human decision support, offering an annotation- scalable alternative to costly reference- based comparisons. VIBE's s ability to score and select summaries without training makes it a practical plug-in for both closed- and open-source VLMs. Looking ahead, VIBE opens several promising directions. One is exploring the joint optimization of summary generation and selection through self- supervised fine-tuning of VLMs. Another is extending beyond human supervision tasks to investigate whether task- lask-aware captions can improve VL.M performance on downstream reasoning tasks. These directions point to the broader applicability of VIBE beyond evaluating video summarization.

---

2
Related Work
Reference-baset Video Caption Evaluation. Evaluating video caption quality is crucial for tasks like video question answering [13, 14], text-to-vidoo retrieval [5, 15, 16], and multimodal language model training. The quality of captions greatly impacts the downstream performance of tasks and models. While existing benchmarks compare VLM-generatte captions to gold-standard references using metrics like ROUGE [17], BLEU 8], and CIDEr [19], these are costly to curate and focus solely on captions rather than video context. In contrast, we propose VIBE, new evaluation metric that incorporates video context, requires no gold standard, and can be applied to unseen video- caption pairs. VIBE extends the information-theoree information to assess news summaries for language model tuning. We adapt for video captions and examine their effect on human task accuracy and response time.
approach of [20], which uses pointwise mutual
Human-Centric Evaluation via Response Time and Accuracy. the context of -ggett interaction involving natural language [21], traditional human evaluations have focused on assessing summaries based on fluency and informativeness [22, 23], but these methods are often subjective and hard to scale. To address this, we adopt an extrinsic evaluation approach that measures how summaries affect human performance on downstream tasks [24, 25]. Specifically, we evaluate captions based on human response time and accuracy on multiple-choiee questions grounded in video content. To account for the speed-accuracy tradeoff [26], we also report the inverse efficiency score [12], which normalizes response time by accuracy.
3 Preliminaries
Mutual Information. Our work heavily relies on the concept of mutual information. For the ease of readers, we briefly introduce it here. Mathematically, for two random variables, X, Z, we can calculate their mutual information:
P(X, Z) I(x; Z) E log P(X)P(Z)
P(X[Z)] = log P(X)
(Mutual Information)
(1)
where the expectation operator is over the joint probability distribution of X and Z.
Intuitively, mutual information measures the reduction in uncertainty about X after observing Z. It is zero if and only if X and Z are independent, meaning knowledge of 2 provides no information about X. Higher mutual information indicates stronger dependency between the two variables. In this work, we leverage mutual information to quantify how much the summaries retain relevant information from the raw video clips and how well they support the target downstream task.
Information Bottleneck. The information bottleneck (IB) framework extracts relevant information from input data while compressing irrelevant details. Tishby al. [9] first introduced it to formalize the trade-off between accuracy and complexity in lcarning systems. Later, researchers applied it to domains such neuroscience [27], natural language processing [28, 29], and computer vision [30, 31]. To ensure clarity, we modify the notation from [9] to align with our later sections, where V is the raw video input, T is the summary of the video, and Y is the downstream task of our interest.
We now introduce the IB framework, which seeks to learn a representation T that discards sensitive and irrelevant information from the input V while preserving information useful for predicting the target task Y. IB relies on minimizing the mutual information between input V and compressed representation T, while preserving as much information possible about a target variable Y:
min I(V2) 3 I(T:Y)
(Information Bottleneck)
(2)
P()j))
compression
informativeness
where I(.:-) denotes mutual information, and 3 2 0 controls the tra between compression and prediction. The optimization variable is the conditional distribution P(tle), which defines a
stochastic encoder that maps v E V to tot& E T. The IB principle in eq. (2) formalizes the trade- off between compression and prediction. Minimizing I(; encourages stronger compression, while maximizing I(T; Y') promotes the preservation of task-relevant information. The parameter B balances these two competing objectives: a higher prioritizes retaining more information about Y. while a lower 3 encourages stronger compression of V.
3

---

Researchers use the IB principle to analyze generalization, regularization, and the role of hidden representations of neural networks [32, 33], where V is the imput of neural networks, the output of intermediate layers, and Y is the classification label. It also drives recent advances in representation learning, where models aim for compact, informative encodings [34, 35].
Video-to-Text Information Bottleneck Evaluation (VIBE)
Video (V)
Masked Summary (T masked)
Fill in Grounding Score
VLM
<MASK>
PT masked: V)
The video explores the unique physical <MASKo
log
I(VTT)
<MASK>
P ((TTmaaked
vooldo
Masking
Summary (T)
Masked Video (V masked)
VLM
Answer Task
Utility Score
P Maakedd T) I(Y: T)
The sideo explores the physical adaptations flamingos..
log
P(YIVaasked)
Computing VIBE Scores via Masked Inference. VIBE estimates grounding and utility scores using the next-token prediction mechanism of VLMs. The grounding score measures how well the video helps reconstruct masked summary, while the utility score captures how much the summary improves task prediction given masked video.
Figure 2:
We now formally describe our video-to-text information bottleneck evaluation (VIBE) method. Inspired by the IB formulation in (2), we adapt its two terms for the purpose of evaluating video-to-text summary quality. We denote the raw video clips that are fed into the VLMs as V, the summaries from the VLM response as T, and the task target we care about as Y. For instance, let V a video of a paper presentation, T be the summary of the presentation, and the task Y be to determine the primary area to see if it is of one's interest.
Grounding Score. In eq. (2), the mutual information I(V between raw data and representation the compression term. In our setting, reinterpret it as the grounding score, which quantifies how grounded the textual summary is to the video. The higher this score is, the more closely the summary anchored to the video.
However, one cannot access the joint and marginal probability distributions over the occurrences of video clips or text summaries. Therefore,
only calculate the empirical mutual information log P(X)P(2) without the expectation operator to approximate the I(V; term. Literature refers to the empirical term without the expectation as pointwise mutual information [36], which we denote as (::)). Unlike mutual information, which must be non-negative as it is the expectation of pointwise mutual information, pointwise mutual information ranges from [-x0, oo).
P(X.2)
We now explain the approximation technique in VIBE to calculate the grounding score. Recall from eq. (1), we rewrite pointwise mutual information as:
P(TIV, Tiaakea)) (Grounding Score) IVV log P(TV)) log (TT P(T)
(3)
We condition both the numerator and denominator with Tmasked, masked version of the text, to approximate I" ("(V:T) The approximation holds if T Tmasked is independent of both the video V and the original text T. Masking key content makes Tinsked effectively uninformative and thus independent. We put the detailed derivation in Appendix I. Following Jung et al. [20]. we use tf-idf [37] to identify keywords and replace them with a "<MASK>" token. This masking helps ensure that eq. (3) reflects the true dependency between V and T,, rather than exploiting shorteuts from the text

---

itself. Thus, the grounding score measures how much the TL;DR remains genuinely grounded in the video beyond what the masked text alone would suggest.
To compute eq. (3), we run two separate inferences with the same VLM to define the ratio of two conditional probabilities P(.I-). The first, ((TiV,Tmaseed) mmasked), is the product of next-token prediction probabilities when reconstructing the masked text given both the video and the masked input. The second, P makked) is computed similarly but without providing the video. Only by conditioning on the masked input can we leverage the VLM's s next-token prediction probabilities to estimate these likelihoods. By dividing these two conditional probabilities and taking the logarithm, we approximate the pointwise mutual information, allowing us to estimate the grounding score.
Utility Score. The second term in eq. (2), the informativeness term, is the mutual information Y) between the representation and the target task. We refer to it the utility score, which measures how useful the summary is for solving the downstream task. A higher utility score indicates that the summary preserves more information relevant to predicting or completing the target task. Similar to the grounding score, directly computing [(;; Y) is intractable because we cannot access the true probability distributions. Again, approximate it using the pointwise mutual information with conditional probabilities:
IT;TY)) = log PTY P(Y)
P( V Viaaseed) P(Y|V masked)
2o log
(Utility Score)
(4)
Similar to the grounding score in eq. (3), Vinaseed denotes video clips with key information removed to reduce their direct informativeness about both the summary T and the task label Y. To compute the utility score in eq. (4), we compare two model inferences: one conditioned on both the summary T and the masked video Vmaakee. and the other on Vmaaked alone. Since both use the same masked visual input, any improvement in predicting Y must come from the information provided by T. Intuitively, if summary is useful, it should compensate for the missing content in the masked video and help recover task-relevant information. Thus, the utility score quantifies how well the summary restores information necessary for predicting the task, directly measuring the summary's contribution to downstream decision-making.
Rejection Sampling over Grounding and Utility Scores. We now describe how VIBE optimizes grounding and utility scores to select more informative and task-relevant video-to-tex summaries for humans, as shown in Figure Given multiple summaries T sampled from the VLM's output distribution P(TIV), VIBE selects the candidate that maximizes a weighted sum of the two pointwise mutual information terms:
+ 3 arg a I' (VV
I (TL:DR Selection with VIBE)
(5)
Here, a, 3 are hyperparameters controlling the trade-off between grounding and utility in eq. (5). A higher emphasizes alignment with video, favoring summaries that faithfully reflect the video, while a higher 3 prioritizes task relevance, selecting summaries that improve downstream performance.
VIBE (ours)
*
Naive VLM
TLDR (OpenReview)
Abstract (OpenReview)
Unisty Score be
1.0
Unility Score togl/V/77
0.2
Uniltey score loo 009gl000
0.0
0,6
-.2
1.0
0.4
0.2
-0.6
2
10
200
20
30
Grounding Score log (L::T)
Grounding Score logit?: 7)
Grounding Score log (T
(a) LongVideoBench
(b) SUTD CUTD-TrafficQA
(c) LearningPaper24
Figure 3: Pareto front of VIBE. Summaries selected by VIBE form Pareto frontier across different (a., .) per Equation (5), demonstrating optimal trade-offs between grounding and utility. In contrast, Naive VLM summaries, author written TLDRs, and abstracts from OpenReview in LearningPaper24 fall inside the frontier, indicating suboptimal scores.
5

---

Our formulation differs from traditional IB, where compression serves a regularizer during model training to mitigate overfitting. In contrast, VIBE does not train a model selects summary. Overfitting is therefore not a concern, and maximizing both mutual information terms is desirable to produce summaries that are both useful and faithful. To this end, we consider o, / 2 0 to jointly maximize both scores. Then, to explore potential trade-offs between the two scores, we perform convex combination sweep over a E {0, 0.05, 0.1, ..., 1.0], with 3 = a. Each (a, () pair is applied consistently across all datasets. This linear scalarization technique, commonly used in multi- objective optimization [38], produces the convex ("Pareto curve") shown Figure 3, empirically revealing the inherent trade-off between grounding and utility scores.
Our results based on grounding and utility scores across sampled summaries from multiple datasets reveal consistent trade-off between the two objectives, As shown in Figure 3. summaries selected by VIBE with varying a and B form a Pareto front, capturing the inherent trade-off between the two VIBE scores. In contrast, randomly sampled summaries (Naive VIM) consistently fall inside this Pareto curve across datasets, indicating suboptimal grounding and utility. We observe this pattern when evaluating the two VIBE scores of VLM-generated summaries tasks from LearningPaper24, SUTD-Traff cQA, and LongVideoBench. Summaries with high grounding scores tend to mirror the video content closely but often include redundant irrelevant details for the task. In contrast, summaries with high utility scores boost task accuracy and response efficiency but may overlook important visual context. This trade-off reflects the core tension in the IB framework. By tuning and 8, VIBE adjusts this balance to match the needs of different downstream applications. We detail the experimental settings in the next section.
As shown empirically in the experiments, higher utility scores correlate with higher human task accuracy. Thus, one can use VIBE to select summaries with higher utility scores and present only the most informative summaries to humans, filtering out less useful ones. Notably, calculating the utility score requires access to task labels Y. but not gold-standard, human-annotated labels for summary T as in previous works. When task labels are unavailable, VIBE can select summaries based on the highest grounding score, an unsupervised measure of alignment between the video and the generated summaries. Although the grounding score alone yields lower human performance than the utility score, our user study shows it still improves task accuracy compared unfiltered VLM summaries.
From Evaluation to Real-Worke Use. VIBE evaluates the quality of summaries only from the VLM perspective, but its connection to human decision makiin performance remains unanswered solely by the formulation. To bridge this gap, we conduct user studies measuring how different summary qualities, as evaluated by VIBE, impact human decision-making performance in Section 5. VIBE relies solely on text generation and next-token probability access, using black-box access to VLMs like the OpenAl API [39]. It enables VIBE work with both closed- and open-source VLMs and scale well to real-world scenarios.
5
Experiment
We conduct user studies across three diverse datasets to evaluate how VIBE-generated summaries support human decision-making. The study measures participants' accuracy and response time in answering questions about video content. Here, we use -een2.5- VIBE for all datasets. A representative qualitative example from the study is provided in Appendix A. We also show that VIBE works across various VLMs through ablation studies.
[40] to evaluate
5.1 Datasets
For evaluation, we select three datasets varying in domain, duration, and task nature.
LearningPaper24 We introduce LearningPaper24. a curated dataset of 2,287 video presentations from ICLR 2024 and NEURIPS 2024, filtered based on key criteria: valid OpenReview ID, an
accessible SlidesLive video, an author-provided TL;DR and abstract, and a clearly defined primary area. Details on the curation process and dataset statistics are provided in Appendix C. The task associated with this dataset is to identify the primary area of each paper from 12 options. See instruction details in Appendix E.

---

LongVideoBence & SUTD-TraffiQQA LongVideoBench [10] features long instructional clips with QA pairs for extended reasoning, while -UTD-TraffiQQA [11] contains short traffic clips with multiple-choice questions on causal and temporal understanding. Participants answer one question per each LongVideoBench clip; four questions per each SUTD-TrafficQA clip.
Preprocessing In datasets, we mask text by removing keywords with high tf-idf scores. For LearningPaper24. which features slide videos, extract and mask keywords in the slide using EasyOCR [41]. For the other two datasets, we apply random 1/16 cropping to all video frames. We subset of each dataset for VIBE evaluation and user studies (details provided in Appendix D).
5.2 User Study Design
For each dataset, 10 video stimuli with multiple-choice questions are shown in randomized order. We adopt a between-subjects design where participants are assigned to one of the four conditions listed below, and the format which the stimulus is presented varies by the condition.
Independent Variables. Participants are assigned to one of four conditions. the Video Only condition, they watch the original video without text. In the remaining conditions, they view only VL.Mgennrated summary: a randomly selected VLM summary (Naive), the top-ranked summary from k response candidates by utility score (Max-U) or by grounding score (Max-G).
Dependent Measures. We report three metrics: accuracy, the proportion of correct responses across 10 stimuli; response time, the time (in seconds) spent reading or watching each stimulus and answering its corresponding questions; and inverse efficiency score (IES), the ratio of response time to accuracy, to account for speed-accuracy trade-offs [12].
Hypotheses. We evaluate the following hypotheses: (H1) Participants in the Max-U and Max-G will achieve higher accuracy than those in Naive VLM. (H2) Participants in the Max-U U and Max-G will respond more quickly than those in the Video Only. (H3) Max-U and Max-G will yield lower (i.e., more efficient) IES scores than Video Only, reflecting better
accuracy trade-off.
Participants. We recruit 243 participants across three datasets: 92 for LearningPaper24, 82 for BUTD-Trafficn and 69 for LongVideoBench For LearningPaper24, participants are primarily CS graduate students or prescreened degree holders on Prolific [42]: participants for the other datasets recruited generally. The average age is 37.59 + 11.06 years, with a gender distribution of 63.37% male, 35.80% female, and 0.82% non-binary. Further details are provided Appendix F.
5.3 User Study Results and Analysis
We assess statistical significance using the independent t-test, reporting the t-statistic t(df) (with degrees of freedom df), the significance level p. and the effect size measured by Cohen't 's d.
On HI (Accuracy). As shown in Table and (al, bl) of Figure (with Max-U in yellow and Max-G in purple above Naive VLM in pink), both Max-U and Max-G consistently outper- form the Naive VLM across all datasets, confirming H1. Max-U achieves the largest gains, with statistically significant improvements over Naive VLM in LearningPaper24 (I(36) 2.486. p = 0.009, d = 0.806), LongVideoBench (t(26) = 3.215, p = 0.002, d = 1.261), and TUUDDTraffin relevant content that helps users quickly locate key information. However, computing the util- ity score requires task labels, which may limit its general applicability and scalability. Max-G, while yielding smaller gains, still significantly outperforms the baseline (e.g.. LearningPaper24 t(33) 1.750, p = 0.045, d 0.594; LongVideoBench: t(25) 2.691. p 0.006, d = 1.077; BUTD-TrafficOA. ((28) 1.759, p = 0.045, = 0.667). Crucially, calculating the grounding score is fully self-supervised and does not require task labels, making an alternative that still delivers meaningful accuracy gains.
(t(25) 3.311.p p 0.001, = 1.325). likely due to its focus task-
k 5 in this study, and the k responses generated with various temperatures.
7

---

Conditions
Dataset
Duration (s) Metric
Video Only
Naive
Max-G (ours)
Max-U U (ours)
Acc f
23.50 10.62
28.42 12.68
35.00 = 7.91
37.89
10.04
LearningPaper24 250-325
RT +
192.73 = 108.01
47.48 + 20.31
61.13 39.33
46.69
25.06
IES
9.85 + 6.63
1.81 + 1.51
1.85 = 1.20
1.28
0.66
Acc
74.44 14.23
46.43 13.94
59.23 + 9.17
65.00
15.47
LongVideoBench 221-489
RT
202.35 87.26
86.50
63.33 1.83 -
71.93
35.96 1.25
65.86 = 41.34
IES
2.93 1.48
1.14 = 0.80
Acc
82.86 5.08
76.96 + 6.89
80.47 + 3.21
84.81 4.65
SUTD- Trafficff
2-10
48.63 21.30
79,46 25.77
80.97
41.11
137.27
81.64
IES
0.59 0.27
1.05 0.37
1.01
1.66 =
Table 1: Human performance (mean standard deviation) across three datasets under different IV conditions (detailed in Section 5.2). Metries: Accuracy (Acc, %. higher is better), Response Time (RT, seconds, lower better), and Inverse Efficiency Score (IES RT/Acc, lower is better). Bolded values indicate the best performance among the IV conditions for each metric.
(al)
Max-G
Max-U
(a2) (,330) =n 0.686, 8.0.03
90.0
60
Accuracy
-7.5
-5.8 -2.5
Utility Score NT:Y)
(a3)
50.0
Time WWrdd
(CoT) Accuracy
0.018
0.029
0,030
0.040
1/ Response Time (s-1)
100 Word Count
(a) LongVideoBench
(b1)
Max-G
(b2)
1,388 - 0.395, p-0001
60
40
Accuracy
Utility Score NT:
30.0
00
10. Am
(b3)
Word
VLM (CoT) Accuracy (10%)
6.010
0.026
8.630
0,048
1/ Time (s-1)
20
Word
(b) LearningPaper24
Figure 4: (al, b1) Accuracy versus inverse response time. Each point represents individual participant; large markers indicate group means with standard error of the mean. Shaded areas denote 2D kemel density estimates (threshold 0.45). (a2, b2) Correlation between accuracy and utility score. (a3, b3) Correlation between response time per word and word count.

---

On H2 (Response Time). Table I shows that both Max-U and Max- significantly reduce response
times compared to the Video baseline in LearningPaper24 (t(37) - .001,
-1.794) and LongVideo (1(34) = -4.5 y < .001, d = - 1.5 However, this trend does not extend to SUTD-TrafficQA where response time shows no consistent improvement. surprising, the significantly shorter video durations (2-10 seconds, compared to over 3 minutes in the other datasets) inherently limit the benefits of summarization. These results align with the expectation that VIBE is most effective in reducing cognitive load for long video clips.
is not
On H3 (IES), As shown in Table 1, Max-U significantly outperform the Video Only baseline in LearningPaper22 (f(37) = -5.458, p < .001, d = -1.749) and LongWideoBench (t(21) =
-3.592, .001, d 1.617), with Max-G G showing similar gains [LearningPaper24: t(34)
-4.631, P < .001, d = -1.533; LongVideoBench: t(20) = 3.422., = .001, d = -1.667).
Furthermore, Max-U also outperform Naive VLM (4(26) 1.940,, p .032, d -0.761) in LongVideoBence These results highlight VIBE's effectiveness in settings where processing full video clips is costly. In contrast, WOUTD-Taafficoo accuracy improvements over Naive VLM and Video Only, the brevity of the clips and the visual nature of fine-grained actions reduce the benefit of text-based summaries.
shows limited IES improvement: despite Max-U's
Correlation Analyses. We explore the correlation between utility score, grounding score, word count, and human performance (accuracy and response time) using Spearman's 's rank coefficient [43], which is robust to outliers and non- non-linear relationships. Full plots are provided in Appendix G. Two consistent trends hold across datasets: First, utility score positively correlates with accuracy: strongly in LongVideot deoBenchhechhhccc (r, = 0.684, p < .001), and moderately in LearningPaper24 (r, 0.399, p .011) and WUTD-Trafficie
(r, = 0.463, p .004). Second, word count is strongly negatively correlated with response time per word (LongVideoBench: r, = -0.761, LearningPaper -0. -0.684, WUTD-Traffic(4. r, -0.658, all p < .001), implying that longer summaries lead to faster processing per word, possibly due to reduced engagement and more shallow reading.
Key Takeaways. Max-U and Max-G outperform Naive VLMs in accuracy, response time, and inverse efficiency score, with Max-U providing the most significant improvements, especially for longer video clips. Max-G, though offering smaller gains, remains self-supervised alternative that improves accuracy without task labels. The benefits of summarization are less pronounced in shorter video clips, like those in SUTD TraD-TrfffiAA, where brief durations limit gains. Correlation analyses show the VIBE utility scores align with higher accuracy, while longer summaries tend increase unit response time, highlighting the importance of concise, relevant information.
5.4 Ablation over VLM Variants
VIBE scores can inherit bias from the underlying VLMs due to their training data and conditional probability estimates used in mutual information computation. To assess robustness under such model-induced bias, we compare VLMs of varying size and source. As shown in Appendix B, different backbones produce consistent IB score trends and scales. This consistency suggests that lightweight models can efficiently verify outputs from larger ones. To further reduce bias, one can evaluate single summary using multiple VL.Ss-an ensemble-style strategy aligned with mixture- of-experts [44, 45] and model selection techniques [46]. This reflects the intuition that good outputs are often hard generate, but easy verify.
Limitations. Despite being training-free, VIBE requires multiple VLM inferences for grounding score evaluation, with the number of masked tokens influencing inference steps. Its pointwise mutual information estimate may be biased by two main factors: (a) the text and video masking strategy, and (b) the conditional probability modeling of the VLMs. For (a), the tf-idf score we use to select keywords for text masking requires hyperparameter tuning, and the optimal masking strategy remains an open question. For (b), VLMs introduce bias through their training data and modeling assumptions in computing eq. (5). Using separate models for generation and evaluation, as discussed in Section 5.4, can help mitigate this bias.

---

Conclusion
In this work, we introduce VIBE, an annotation-free framework for evaluating and selecting vidco-to- text summaries to support human maniin.. Unlike traditional caption evaluation metrics that rely on human- written references, VIBE uses information-theoree theoretic scores scores-utility and grounding to assess how well a summary supports a downstream task and aligns with video evidence. This enables scalable, task-aware summary selection without the need for retraining or annotations. Through large-scale user study spanning three diverse datasets, we show that summaries selected by VIBE significantly improve both task accuracy and response time, especially for longer video clips where information overload is more likely.
Acknowledgement
This work was supported in part by the National Science Foundation grants No. CNS-1836900, 2148186, the Office of Naval Research (ONR) under Grant No. N00014-22-1-2254, N00014-24-1 1-1 2097. the Defense Advanced Research Projects Agency (DARPA) contract FA8750-23-C-1015 and DARPA ANSR: RTXCW22311 10. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
10

---

References
[1] S. Amershi, D. Weld, M. Vorvoreanu, et al. "Guidelines for Human-AI Interaction". In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. CHI '19. Glasgow, Scotland Uk: Association for Computing Machinery, 2019, pp. 1-13. ISBN: 978 978145035.702.
[2] Holzinger. "Interactive machine learning for health informatics: when do we need the -nttinio-the-oopp"" In: Brain informatics 3.2 (2016), pp. 119-131.
[3] D. Xin, L. Ma, J. Liu, et al. "Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities". In: Proceedings of the Second Workshop on Data Management for End-To- End Machine Learning. DEEM' 18. Houston, TX, USA: Association for Computing Machinery,
2018. ISBN: 978145038.
[4] A. Holzinger, K. Zatloukal, and H. Müller. "Is human oversight to Al systems still possible?"
In: New Biotechnology 85 (2025), pp. 59-62. ISSN: 1871-6784.
[5] J. Xu, T. Mei, Yao, and Y. Rui. "Msr-vtt: A large video description dataset for bridging video and language". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp- 5288-5296.
[6] X. Wang. J. Wu, J. Chen, et al. "VaTeX: Large-Scale, Nigh-Quality Multilingual Dataset for Video-and-Language Research". In: The IEEE International Conference on Computer Vision (ICCV). Oct. 2019.
[7] W. Chai, E. Song, Y. Du, et al. "AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark", In: The Thirteenth International Conference on Learning Representa- tions. 2025.
[8] M. Monfort, Jin, A. Liu, al. "Spoken moments:
Learning joint audio-visual representations from video descriptions". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 14871-1488)
[9] N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. 2000, arXiv: Dhysics/000400 [physics data-an].
[10] H. Wu, D. Li, B. Chen, and J. Li. "Longvideobench:: A benchmark for long-context interleaved video-language understanding". In: Advances in Neural Information Processing Systems 37 (2024), pp. 28828-28855
[11] L. Xu, H. Huang, and J. Liu. "Sutd-trafficqa: A question answering benchmark and efficient network for video reasoning over traffic events", In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 9878-9888 9888.
[12] J. T. Townsend and F. G. Ashby. Stochastic modeling of elementary psychological processes.
CUP Archive, 1983.
[13] D. Xu, Z. Zhao, Xiao, al. "Video Question Answering via Gradually Refined Attention over Appearance and Motion". . In: ACM Multimedia. 2017.
[14] Y. Zhong, Xiao, W. Ji, et al. Video Question Answering: Datasets, Datasets,Algoringee Algorithmss and Challenges.
2022. arXiv: 2203. 01225 [cs.CV].
[15] P.-h. Li, Y. Yang, M. Omama, al. "Any2Any: Incomplete Multimodal Retrieval with Confor- mal Prediction". In: arXiv preprint arXiv:2411.105// (2024).
[16] M. Omama, P.-h. Li, and S. P. Chinchali. "Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval". In: The Thirteenth International Conference on Learning Representations. 2025.
[17] C.-Y. Lin. "Rouge: A package for automatic evaluation of summaries". In: Text summarization branches out. 2004, pp. 74-81.
[18] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. "Bleu: method for automatic evaluation machine translation". In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002, pp. 31 311-318.
[19] Vedantam, C. Lawrence Zitnick, and D. Parikh, "Cider: Consensus-bass image descrip- tion evaluation".. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 4566 4575.
[20] J. Jung, X. Lu. L. Jiang, et al. "Information-Theee Distillation for Reference-less Summa- rization". In: First Conference Language Modeling. 2024.

---

[21] S. Chen, D. Fried, and U. Topeu. "Human- agent cooperation in games under incomplete information through natural language communication.. In: International Joint Conference on Artificial Intelligence, Hluman-Centred Al (2024).
[22] A. Belz and E. Reiter. "Comparing automatic and human evaluation of NLG systems". In: Itth conference of the european chapter of the association for computational linguistics. 2006,
pp. 313-320.
[23] Y. Graham, T. Baldwin, A. Moffat, and J. Zobel. "Can machine translation systems be evaluated by the crowd alone". In: Natural Language Engineering 23.1 (2017), pp. 3-30.
[24] A. Nenkova, K. McKeown, et al. "Automatic summarization", In: Foundations and Trends in Information Retrieval 5.2-3 (2011), pp. 103-233.
[25] X. Pu, M. Gao, and X. Wan. "Is summary useful or not? an extrinsic human evaluation of text summaries on downstream tasks". In: arXiv preprint arXiv:2305. 255::2305.15044 (2023).
[26] R. P. Heitz. "The speed-accuracy tradeoff: history, physiology, methodology, and behavior".
In: Frontiers in neuroscience 8 (2014), p. 150.
[27] M. Kleinman, T. Wang. D. Xiao, et al. "A cortical information bottleneck during decision-
making".. In: bioRxiv (2023).
[28] P. West, A. Holtzman, J. Buys, and Y. Choi. "Bottlesum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle". In: arXiv preprint arXiv:1909, 074455 (2019).
[29] C. Zhang, X. Zhou, Y. Wan, et al. "Improving the adversarial robustness of NLP models by information bottleneck". In: arXiv preprint arXiv:2206.0551/ (2022).
[30] Gordon, Greenspan, and Goldberger. "Applying the information bottleneck principle to unsu- pervised clustering of discrete and continuous image representations", In: Proceedings Ninth IEEE International Conference on Compuler Vision. 2003, 370-377 vol.1.
[31] N. Tishby and N. Zaslavsky. "Deep learning and the information bottleneck principle". In: 2015 IEÉE Information Theory Workshop (ITW). 2015, pp. 1-5.
[32] K. Kawaguchi, Z. Deng. X. Ji, and J. Huang. "How does information bottleneck help deep learning?" In: International Conference on Machine Learning. PMLR. 2023, pp. 16049-16096. [33] Z. Goldfeld and Y. Polyanskiy. "The information bottleneck problem and its applications in machine learning". In: JEEE Journal on Selected Areas in Information Theory 1.1 (2020), pp. 19-38.
[34] S. Mai, Y. Zeng, and H. Hu. "Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations" In: IEEE Transactions on Multimedia 25 (2023), pp. 4121-4134.
[35] R. Islam, H. Zang, M. Tomar, et al. "Representation Learning In Deep RL Via Discrete Information Bottleneck". In: A/STATS 2023. May 2023.
[36] G. Bouma. "Normalized (pointwise) mutual information in collocation extraction". In: Pro- ceedings of GSCL 30 (2009). pp. 31-40.
[37] G. Salton and C. Buckley. "Term-weightinn approaches in automatic text retrieval". In: Infor- mation Processing & Management 24.5 (1988), pp. 513-523. ISSN: 0306-4573.
[38] S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[39] OpenAL. OpenAl Python Library. https: / //gttttb. com / openai / /penaii python.
Accessed: 2025-04-28. 2025.
[40] S. Bai, K. Chen, X. Liu, et al. Qwen2.5-VL Technical Report. 2025. arXiv: 2502, 13923
[41] JaidedAI. EasyOCR: Ready-to-use OCR with 80+ supported languages. htgg:ggge..ttpss//ihhhhh com/ SraidedAIEEasyDCC Accessed: 2025-05-15 00-5--5-15. 15. 2020.
[42] S. Palan and C. Schitter. "Prolific. ac- A subject pool for online experiments". In: Journal of Behavioral and Experimental Finance 17 (2018), pp. 22-27.
[43] C. Spearman. "The proof and measurement of association between two things." In: (1961).
[44] I. Ong. A. Almahairi, V. Wu, et al. "RouteL.LM, Learning to Route LLMs from Preference Data". In: The Thirteenth International Conference on Learning Representations. 2025.
[45] Y. Zhou, T. Lei, H. Liu, et al. "Mixture-of -exeeerss with expert choice routing". In: Advances in Neural Information Processing Systems 35 (2022), pp. 7103-7114.
12

---

[46] P.-h. Li, O. S. Toprak, A. Narayanan, et al. Online Foundation Model Selection in Robotics. arXiv:2402.08570 [es]. Feb. 2024.
[47] Z. Chen, W. Wang, Y. Cao, et al. Expanding Performance Boundaries of Open-Source Multi- modal Models with Model, Data, and Test-Time Scaling. 2025. arXiv: 2412. 05271 [cs.CV].
[48] J. Zhu, W. Wang, Z. Chen, et al. Intern V1.3: Exploring Advanced Training and Test-Time Recipes for Open-Soure Multimodal Models. 2025, aiv: 2504. 10479 [cs.CV].
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the last page in the main paper. There is a section of limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
14

---

We do not have theoretical results in this work. We cited all previous theoretical work we gained insights from.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all details in the appendix and the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Refer to the supplemental material for code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
cublic/guides/ Stwiidde//odSSumiissseee wwwsssssslllooooooooooooooo plyycy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.cooo wttpccccppppcceeeecececccccccccc httpa///ggiieee/eeeuuuubiiiiiiia /Codeeeooiissseeeeee wwtpaubmmsssoonni ssisnnol onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See the user study section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See the user study section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The setup is mentioned in the Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : //neurrrss... cc httpppllcc/thhhhcculeleeele
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sepprrswithood thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification: Yes, we will provide comments and datasets to reproduce our results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Yes. Please refer to Appendix E.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Yes, do have the IRB's approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
19

---

Justification:
Yes, we use VLMs as part of our proposed method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hattps://neurips htte:p////arrrpe..c///ootreeeee/ / for what should or should not be described.
20

---

Appendix
A Qualitative Result
To illustrate differences across summary generation methods, we present a qualitative example. In Table 3, each condition-Naiie Max-U, Max-G, and CoT distinct response based on the same video input (keyframe thumbnails shown in Figure 5). For better comparison, we also provide the original TL;DR and abstract from the authors on OpenReview in Table 2. We highlight key terms derived from OpenReview metadata and tf-idf analysis by masking them in gray.
Aiy dien (fffornood (ra) Fal)))
Saliin
Proo Mealling twilkhc
1960
Figure 5: Keyframe thumbnails from the qualitative example SlidesLive talk. The correct answer is (I) Human-Al Interaction and Ethics (Privacy, Fairness & Safety), while the VLM prediction (F) Reinforcement Learning & Robotics.
Type
Paper Summary
TL;DR
We study the challenges that arise when learning reward functions with human feedback from partial observations
Abstract
Past analyses of reinforcement learning from human feedback human evaluators fully observe the environment. What happens when human feedback based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the human as Bolizmann-rational w.r.t. a belief over trajectories, we prove conditions under which RLHF guaranteed to result in policies that deceptively inflate their performance, overjustify their behavior make an impression, or both. Under the assumption that the human's partial observability known and accounted for,
RRLIF)) assume that the
then analyze how much information the feedback process provides about the return function. We show that sometimes, the human's feedback determines the return function uniquely up to an additive constant, in other realistic cases, there is irreducible ambiguity. We propose exploratory rescarch dircetions help tackle these challenges and experimentally validate both the thcoretical concerns and potential mitigations, and caution against blindly applying RLHF in partially observable settings
Table 2: Author-provided TL:DR and abstract on OpenReview.
21

---

Condition
Summary Response
Naive
This paper explores the challenges of partial observability reinforcement learning from
human feedback, focusing on deceptive inflation and overjustification in A1
systems
Max-U
This rescarch investigates challenges in ensuring A1
safety and addressing partial
observability through advanced frameworks in human machine reinforcement learning.
interactions and
Max-G
The presentation discusses the challenges of partial
observability in reinforcement
learning from human feedback, focusing on deceptive inflation and overjustification in AI systems
CoTT
The slides discuss challenges in reinforcement learning from human feedback particularly focusing partial observability and the issues of deceptive inflation and overjustification The content delves into the framework of Markov Decision Processes (MDPs) and the human choice model, which are central to reinforcement learning. The presentation also addresses safety problems and alignment concerns, which are critical the context of human-
AI interaction. Given the focus on reinforcement learning and the interaction between AI and human evaluators, the most suitable area for this paper is one that encompasses both reinforcement learning and the ethical considerations of human- AI interaction.
Table 3: Responses under each condition for the qualitative example, keywords from Open and if-idf masked in grey.
B Ablation Study on Evaluating with Various VLMs
VIBE (ours)
Naive VLM
TLDR (OpenReview)
Abstract (OpenReview)
1.2
Ukility Scooe log (0g
Uniliy Scor loginvem
Uniliity Score (00
0.6
0.6
0.4
0.2
0.5
0.2
0.2
10
20
30
10
15
20
25
30
10
15
20
25
Grounding Score log NT:7)
Grounding Score (V:
Grounding Score log KYY
(a) 1wen-2.5-72B-A.
(b) InternVL-2. 2-----
(c) InternVL-3-368
Figure 6: VIBE generalizes to various VLMs. Summaries selected by VIBE form Pareto frontier not only across various (a, B) but also across various VLMs from different sources.
We conduct an ablation study to examine how different VLM variants affect the reliability VIBE, Specifically, we compare Qwen -L-2..5-72--AWO [40] (used in the main paper) with VL2.5.
8B-MPO [47] and InternVL3-38B [48]. These models vary in size, architecture, and pretraining sources. We apply VIBE to the same set of summaries to compute their utility and grounding scores. Results in Figure 6 show consistent VIBE score trends across models, indicating that smaller or alternate VLMs yield similar evaluation signals while reducing inference cost and potential bias. Notably, various VLMs give similar ranges both utility and grounding scores. The consistency across models further supports the generalizability of VIBE training-free framework.
C LearningPaper24 Dataset Curation
We source the dataset from the public PaperCopilot Paperl lists repository, which indexes all accepted papers at ICLR 2024 and NEURIPS 2024. To ensure data quality and completeness, apply the
22

---

following filtering criteria: (1) the paper must have a valid OpenReview ID to guarantee access to full metadata and public reviews; (2) the associated SlidesLive presentation video must be accessible and playable, verified via headless browser; (3) TL;DR summary must be present on OpenReview; and (4) a primary area must be specified. After applying these filters, we retain 2,287 entries from an initial set of 2,556.
Infrastructure, Benchmarks & Evaluation (77, 3.0%)
Models
5.0%)
Graph-Based Representation Neurosymbolic (95, 3.7%)
Probabilistic & Causal Methods (149, 5.8%)
Learning Theory (334, 13.1%)
Human-Al Interaction and [241, 9.4%)
Optimization (200, 7.8%)
Natural Language, Vision Muftimodal Learning
Reinforcement Learning Robotics (260, 10.2%)
Applications to Sciences Engineering (254, 9.9%)
Figure 7: Distribution of papers in LearningPaper24 by consolidated primary area.
OvenReview's primary areas often too specific overlapping, remap them into 12 broader and semantically coherent categories. For example, "self-supervised learning" and "representation learning for vision" become Representation Learning, while "probabilistic methods" and "causal inference" are grouped as Probabilistic & Causal Methods. The final taxonomy is shown Figure The final dataset includes the following required fields: OpenReview ID, SlidesLive talk ID and URL, TL;DR, abstract, and primary area.
D VIBE Masking and Evaluation
We now report how we select data to calculate the VIBE score in Figure 3. We further select 10 stimuli for our user study.
LearningPaper24 We randomly sample 80 papers each from ICLR 2024 and NeurIPS 2024
following the curation process in Appendix C. For each presentation, we generate VLM responses and CoT response. We compute the tf-idf score separately for each of the six response corpora. We discard words and phrases (n-grams with range 1-3) appearing over 10% of responses and retain only those with tf-idf scores above 0.0025. We also leverage the keywords of papers provided by the authors on OpenReview for masking. We uniformly sample 20 frames from each video as VLM input, with without random cropping applied.
SUTD-TraffiQQA We randomly sample 100 video clips from the dataset. For each video clip, we generate 5 VLM responses and CoT response. All responses, including CoT, form the corpus for tf-idf computation. We discard words and phrases (n-grams with range 1-3) appearing in over 30% of responses and retain only those with tf-idf scores above 0.01. We uniformly select frames from each video the input of VLM with or without random cropping. We uniformly sample 20 frames from each video as VLM input, with or without random cropping applied.
BongVideoBench We sample 150 video clips, each 30 500 seconds long, from categories ["E20", "E2E" "030", "S2A", "S2E", "S20", "sOS"]. For each clip, we generate 5 VLM responses and CoT response. All responses form the tf-idf corpus. We discard words and phrases (n-grams with range 1-3) appearing in more than 50% of responses and keep only those with tf-idf scores above 0.006. We uniformly select 20 frames from each video as the input of VLM with or without random cropping. We use the Long VideoBench package to obtain 32 frames, which is also uniformly sampling the video, from each video as VLM input, with without random cropping applied.
23

---

E
User Study Instruction and Interfaces
All participants provide informed consent before participation and are compensated a rate consistent with Prolific guidelines and institutional standards. The user interface is presented in Figure 8. Below are the instructions provided to participants for each dataset. Text in parentheses indicates variations between video and text conditions.
1011 places
Survey
Tadkctess vtuude screesing
LastL.COO
shudy
Nook: Wettc 10 shart talk videos ond, ofter soch one, serect the research oreo you beloogs to froor fot of aptions.
Veniftontio: Complese plrpitss a briel CAPTCHSt corfirm human perteipent.
Compenantion You compenanted occs ding-ding to the rone Sited on
Devicos
Dedktep
Mobille
Toablet
You dis need
Audio
(a) Vidco Only
this oskedd
Teals poper tes each andect research unou bollonga from options.
Veriflcution Complese o brief CAUTCHA confiin you hunor portecipane
Compenuction: wi opanuneee according
Praffi.
Dewices take thee
Desktop
Metile
Toblet
(b) Text-based (Naive, Max-U, Max-G, CoT)
Figure 8: Prolific recruitment interfaces for LearningPaper2: dataset showing video condition (top) and text conditions (bottom). Similar interfaces are used for the other two datasets.
Instructions for LongVideoBence
In this study, you (watch/read) (short videos/short summaries) featuring a variety of everyday content. Topics may include:
Cooking
STEM education
Art and creativity
Daily vlogs and lifestyle scenes
After each (video/summary), you will answer multiple-choice question assess your recall and understanding of the content.
Please avoid leaving the page idle, as we are also measuring your response time.
Instructions for SUTD OUTD-Traffich
In this study, you'll (watch/read) (short videos/sets of descriptions) of real traffic videos, which may involve an accident.
After cach (video/descriptiion), you'll answer 4 multiple-choice questions. These questions may involve:
Understanding what happened in the video
Considering what might have happened under different conditions
Reflecting on your own interpretation or reasoning
Please avoid leaving the page idle, as are also measuring your response time.

---

Instructions for LearningPaper24
In this study, you will (watch/read) 10 (short research talk videos/paper summaries). For each (video/summaryy, select the rescarch area you think belongs to from:
(A) Learning Theory
(B) Representation Learning
(C) Generative Models
Optimization
Probabilistic & Causal Methods
(F) Reinforcement Learning & Robotics
(G) Graph-Based Based & Neurosymbolic Learning
(H) Natural Language, Vision & Multimodal Learning
(I) Human-AI Interaction and Ethics (Privacy, Fairesss & Safety)
(J) Applications to Sciences & Engineering
Infrastructure, Benchmarks & Evaluation
(L) Others
Please avoid leaving the page idle, also measuring your response time.
F
Participant Recruitment and Demographics
Table 4 summarizes demographic information for participants recruited across the three datasets.
Dataset
Participants
Gender Distribution
Age (years)
LearningPaper24
70.65% male, 28.26% female, 1.09% non-binary
32.86 8.03
SUTD-Tree CQQ
59.76% male, 40.24% female
40.67 = 12.69
LongVideoBench
69
57.97% male, 40.58% female, 1.45% non-binary
40.22 + 12.40
Total
243
63.37% male, 35.80% female, 0.82% non -inbaiary
37.59 = 11.06
Table Demographic breakdown participants by dataset.
Participants were assigned to IV conditions as shown Table 5. In addition to the four primary conditions-Max-U Max-G, Naive, and Video Only (see Section 5.2) each dataset also includes group evaluating plots in the main text due their fundamentally different generation process: unlike other conditions, CoT has access to answer options of the task, making direct comparisons unfair and potentially misleading. However, include CoT responses in the correlation analyses, as their grounding and utility scores remain valid for assessing alignment with human performance.
(CoT) responses. CoT responses are excluded from the scatter
Dataset
Max-U
Max-G
Naive
Video Only
CoT
LearningPaper24
19
16
20
SUTD-Trafficih
14
17
15
16
20
LongVideoBench
15
14
15
10
15
Table 5: Number of participants IV condition across datasets.
G
Additional User Study Plots
Figure 9 presents the full set of scatter plots and Spearman correlation results for all datasets. The left panels show scatter plots of accuracy versus inverse response time, while the right panels display correlations between accuracy and utility score, response time and utility score, accuracy and grounding score, response time and grounding score, accuracy and word count, and response time per word and word count.
25

---

H
Computation Resource
All experiments were conducted on four NVIDIA RTX 6000 Ada GPUs (48GB VRAM) using the vLLM backend. The system was equipped with an Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz, featuring 64 cores (x86_64 64-bit). This setup handled all models Qwen- Owen-2.5-72B-AA. 2.5-72B-AWQ. Intern 2.5-8B-MPO, and InternVL-3333
efficiently for both generation and evaluation.
I Derivation of Mutual Information Approximation
derive case where the approximation holds under the stated assumption, starting from the right-hand side of Eq. and Eq. 4. Since both approximations follow nearly identical steps, present the derivation of grounding score as an example:
P(T maaked)))) log P(T T Tmasked)
Since T masked is masked sentence with all keywords removed, assume it contains information and is independent from any other random variables. Thus, we can rewrite the expression as:
P(T V. T maskedd) log P(T) T manked)
P(T, V, maskeed) / (V, VWV.Taane Tmasked)
log
P(T, T Tmasked) ) P(Tmaskedd
PTT, V) P[Tmasked )/ (VV) P(Tmasked))
log
P(T). P P(Tmasked) PPP (TTaacked)
PT,, V)/P(V)
log
P(T)
P(T (V)
log
P(T)
All steps follow from Bayes' theorem and the assumption of independence.
26

---

3.00
Molve VLM
52.5
-14
50
241
34
0.
Accumsy 176
wOll JAIIA EHEAST
23.0
Grounding Score (IV)
Groundinn Score AVV,
WCW -AAN,
Response Time (5-))
Whrrd Connet
Whvrd
(a) LearningPaper24
Video
300
Max-G rours
Accumey
200
03.0
Ligtt op
comy- L.M.E-OAA
Accurncy t50
125
00.0
-10
13
Greunding Score <190l proias
VLM (GoT) Accurecy (40%)
2gez
Resporaae Time (3/s)
Word Count
Ward
(b) LongVideoBench
Viteo
250
4.M.
Accorwyy Fil
Awam
Mas (oun))
-L5
-1.
-0.5
0,8
0.
-1.5
-L.8
-8.5
0.9
65
pore 2o44-0231.. --000
<deite -4 proms
Accunscy 158
Ansency ENA
200
03.0
2
300
280
Creanting Sooee (W
Greusting Scere XVS
103
6414
9,008
0.46
300
(c) SUUTD-Traffich
Figure 9: Scatter plots and Spearman correlations across all datasets. Trendlines shown for forp- < 0.05.
27


Reviews:
[{"review_id": "ysIJoTHK10", "reviewer": "Reviewer_iMoZ", "summary": "Considering the existing video caption evaluation metrics either require human annotators to write gold-standard captions or fail to measure how well captions support the downstream tasks. The paper introduces VIBE (Video-to-text Information Bottleneck Evaluation), an annotation-free method for evaluating and selecting video-to-text summaries to support human decision-making. VIBE uses grounding and utility scores, based on the information bottleneck principle to evaluate video-to-text summaries. The grounding score measures how well the summary aligns with the visual content, while the utility score assesses how informative the summary is for a downstream task. By conducting between-subjects user studies with 243 participants across three datasets, the results demonstrate that summaries selected by VIBE significantly improve human task accuracy.", "questions": "As provided in Section Strengths And Weaknesses, W1-W6.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "S1: VIBE introduces an annotation-free method for evaluating video-to-text summaries, using grounding and utility metrics, which innovatively introduced Mutual Information and Information Bottleneck to score VLM-generated summaries.\nS2: The authors conduct extensive user studies across three diverse datasets involving 243 participants. The results demonstrate that summaries selected by VIBE significantly improve human task accuracy by up to 61.23% and reduce response time by up to 75.77% compared to naive VLM summaries or watching the full video.\nW1: Inadequate coverage of VLMs in related work. The study heavily relies on VLMs, yet the related work section lacks a comprehensive review of the state-of-the-art in VLM research. The author should expand the related work to include a detailed discussion of VLM development and their application in caption.\nW2: The technical novelty presented in this paper is somewhat limited. The proposed VIBE heavily draws from existing literature. The novelty of this paper can be illustrated better. Specifically, the challenges authors want to address should be highlighted. The techniques to illustrate challenges should be more detailed.\nW3: The description of the approximation using pointwise mutual information is insufficient.: VIBE introduces the use of masked information to approximate the grounding score and utility score. The author should provide a deeper explanation and provide a detailed derivation of the formula to justify the approximation, which will give the reader a better understanding of the reasons for this operation.\nW4: As mentioned in abstract, the author argue that the VLMs would have generated verbose caption. The authors should incorporate additional quantifiable metrics, such as average output token lengths to provide a more comprehensive evaluation of redundancy reduction. This would better validate VIBE’s impact on optimize the length of the output.\nW5: More explanations on experimental results. The authors should provide potential reasons, e.g. the similar performance of Video-Only and Max-G on SUTD-TrafficQA. This would enhance the interpretability of the findings.\nW6: The authors should provide a clearer explanation of the applicable scenarios for their work. While VIBE is designed to support human decision-making, the discussion of limitations in traditional caption evaluation metrics may be confusing. To better illustrate the problem addressed by this work, the authors could offer an alternative example that demonstrates the need for an approach like VIBE instead of the examples of evaluation metrics.", "was_revised": true, "final_justification": "The authors' rebuttal was constructive and has satisfactorily addressed most of my initial concerns. I appreciate the authors' detailed responses.", "rebuttal": "We thank the reviewer for the thoughtful and detailed feedback. Below, we respond to each concern individually.\n\n---\n**Weakness 1: Missing coverage of VLMs in Related Work**  \nWe have added a subsection in the related work to address recent advancements in video-language models (VLMs). This new section surveys key developments in VLM architecture, training, and evaluation benchmarks--including models not directly used in the paper [1, 2, 3].\n\nWe also discuss how these models have been applied in related tasks such as video captioning [4], video understanding, and multimodal reasoning [5], to contextualize our approach within the broader landscape of VLM research. In the final version, we will further expand this subsection to include additional relevant works beyond the citations listed here.\n\n[1] Nguyen, Thong, et al. \"Video-language understanding: A survey from model architecture, model training, and data perspectives.\" arXiv preprint arXiv:2406.05615 (2024).\n\n[2] Tang, Yunlong, et al. \"Video understanding with large language models: A survey.\" IEEE Transactions on Circuits and Systems for Video Technology (2025).\n\n[3] Li, Zongxia, et al. \"A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges.\" arXiv preprint arXiv:2501.02189 (2025).\n\n[4] Abdar, Moloud, et al. \"A review of deep learning for video captioning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).\n\n[5] Li, Yunxin, et al. \"Perception, reason, think, and plan: A survey on large multimodal reasoning models.\" arXiv preprint arXiv:2505.04921 (2025).\n\n---\n**Weakness 2: Clarification on the challenge and novelty**  \nWe appreciate the opportunity to clarify this. Our key challenge lies in evaluating video-language alignment in unannotated, real-world videos without human-annotated captions—a setting poorly addressed by existing metrics. Unlike prior work that assumes access to human-annotated gold-standard text (e.g., captions or scripts), our approach enables scalable evaluation directly from raw video and model-generated text. \n\nThis setting introduces both conceptual and technical innovations. Conceptually, it redefines the evaluation of \"video captioning\" without relying on human annotations. Technically, it grounds evaluation in mutual information approximations between modalities, as detailed in Section 4.\n\nIn contrast, prior works in the related work section applied mutual information with the token masking technique only to assess text-to-text summary quality.\n\n---\n**Weakness 3: Mathematical derivation to justify approximation using pointwise mutual information**\n\nWe derive the case where the approximation holds under the stated assumption, starting from the right-hand side of Eq. 3 and 4. Since both approximations follow nearly identical steps, we present the derivation of the grounding score as an example:\n$$\n\\log \\frac{\\mathbf{P}(T \\mid V, T_{\\mathrm{masked}})}{\\mathbf{P}(T \\mid T_{\\mathrm{masked}})}.\n$$\n\nSince $T_{\\mathrm{masked}}$ is a masked sentence with all keywords removed, we assume it contains no information and is independent of any other random variables. Thus, we can rewrite the expression as:\n\n$$\n\\log \\frac{\\mathbf{P}(T \\mid V, T_{\\mathrm{masked}})}{\\mathbf{P}(T \\mid T_{\\mathrm{masked}})}\n= \\log \\frac{\\mathbf{P}(T, V, T_{\\mathrm{masked}}) / \\mathbf{P}(V, T_{\\mathrm{masked}})}{\\mathbf{P}(T, T_{\\mathrm{masked}}) / \\mathbf{P}(T_{\\mathrm{masked}})}\n= \\log \\frac{\\mathbf{P}(T, V) \\cdot \\mathbf{P}(T_{\\mathrm{masked}}) / (\\mathbf{P}(V) \\cdot \\mathbf{P}(T_{\\mathrm{masked}}))}{\\mathbf{P}(T) \\cdot \\mathbf{P}(T_{\\mathrm{masked}}) / \\mathbf{P}(T_{\\mathrm{masked}})}\n= \\log \\frac{\\mathbf{P}(T, V)/\\mathbf{P}(V)}{\\mathbf{P}(T)}\n= \\log \\frac{\\mathbf{P}(T \\mid V)}{\\mathbf{P}(T)}.\n$$\n\nAll steps follow from Bayes’ theorem and the assumption of independence.\n\n---\n**Weakness 4: Metrics on redundancy reduction**  \n\nThank you for the suggestion. Recent works [1,2] on long video understanding with VLMs often generate clip-level captions and aggregate them for downstream tasks, resulting in a lengthy and verbose summary composed of many sentences. \nWhile we briefly mention this in the abstract, it is not the central focus of our study. Our current experiments emphasize downstream task performance rather than length reduction.\n\nThat said, we believe VIBE offers a principled approach to addressing verbosity—not by simply truncating output, which can compromise grounding and utility—but by selectively preserving the most informative content. We have noted verbosity evaluation as a promising direction in the future work section.\n\n[1] Hur, Chan, et al. \"Narrating the Video: Boosting Text-Video Retrieval via Comprehensive Utilization of Frame-Level Captions.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Wang, Ziyang, et al. \"Videotree: Adaptive tree-based video representation for LLM reasoning on long videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n---\n**Weakness 5: Interpretation of SUTD-TrafficQA results**  \nThank you for raising this point. If the paper is accepted, we will expand the interpretation in the final version to further contextualize the results.\n\nRegarding the specific findings on SUTD-TrafficQA: As shown in Table 1, while the accuracy of Video-Only and Max-G is similar, the key differentiator is response time (Video-Only: 48.63s; Max-G: 80.97s; Max-U: 137.27s). As discussed in Lines 272–275 and 297–298, this pattern is due to the short video durations (2–10 seconds) in SUTD-TrafficQA. In such settings, participants can extract relevant information directly from the video with low working memory demand. In contrast, text summaries require parsing language and mentally reconstructing events—particularly challenging for temporally and spatially grounded traffic event questions.  \n\nThis result suggests a promising future direction: systematically identifying the boundary conditions under which summarization enhances human task performance versus when direct video consumption is more effective. Exploring this trade-off more systematically could inform adaptive interfaces that tailor content presentation to context. Despite this open question, the consistent utility-accuracy correlation trends across datasets (see Appendix Fig. 9 (b)) support the generalizability of our core findings. Moreover, video-to-text summarization methods offer clear value in scenarios where visual attention is constrained—such as accessibility applications, mobile multitasking, or hands-free settings—where direct video inspection may not be feasible and text can be converted to audio for easier accessibility.\n\n---\n**Weakness 6: Elaboration on Applicable Scenarios**  \nWe appreciate the chance to elaborate on our evaluation setting. Our work focuses on real-world scenarios where videos often lack human-generated captions. This is typical in surveillance, driving, sports, and other domains where scalable annotation is infeasible. As such, conventional reference-based metrics (e.g., CIDEr, METEOR) are not applicable.  \nThe datasets we selected reflect this challenge: they contain raw video and questions but no canonical ground-truth summaries. Our approach—VIBE—avoids reliance on expensive or biased annotations by directly measuring alignment between video and model-generated summaries. We believe this setup is not only practical but essential for general-purpose video understanding evaluation going forward."}, {"review_id": "hv6p692cLw", "reviewer": "Reviewer_SJcs", "summary": "This paper presents an annotation-free framework for evaluating video-to-text summaries generated by VLMs. Grounding and utility score are introduced to evaluate the VLM outputs. Grounding score measures the alignment of summary semantics with visual signal, while utility score measures the task relevance of the generated summary. User studies on several benchmarks demonstrate the method's effectiveness.", "questions": "Mentioned in the Weaknesses section.", "limitations": "See my cons.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. A novel video caption evaluation method. The idea of decoupling the evaluation score to grounding and utility makes sense to me.\n2. The manuscript clearly communicates its key contributions and ideas. \n3. The user studies are comprehensive, indicating the method's effectiveness.\n\nWeaknesses: \n\n1. It seems that the grounding score can not prevent the summary from the hallucination problem without additional visual cues, as discussed in previous works [1][2]. The grounding score may be high even when there are factual errors between the summary and the video sequence (just with high semantic similarity).\n2. In lines 282 thru 284 mention \"the brevity of the clips and the visual nature of fine-grained actions reduce the benefit of text-based summaries.\" Does this mean the proposed method may suffer from object-centric bias?\n\n[1]Li, Chaoyu, Eun Woo Im, and Pooyan Fazli. \"Vidhalluc: Evaluating temporal hallucinations in multimodal large language models for video understanding.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.\n\n[2]Yin, Shukang, et al. \"Woodpecker: Hallucination correction for multimodal large language models.\" arXiv preprint arXiv 2310.16045, 2023.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns, and many of those of other reviewers. Therefore, I have decided to maintain my positive score.", "rebuttal": "We thank the reviewer for the insightful comments and for pointing to relevant related work. Below, we address the specific concerns raised.\n\n---\n**Weakness 1: On grounding score and hallucination**  \nThe grounding score measures the logarithm of the \"lift\" in next-token probability with and without the video, given a masked summary (Eq. 3). \nRoughly speaking, hallucination can arise from two sources:\n\n1. **Internal model bias** — originating from pretraining or fine-tuning;\n2. **External input** — due to misleading or misinterpreted content from the video or summary.\n\nIf hallucination stems from **(1)**, the model’s predictions likely remain unchanged regardless of the video, and the grounding score becomes ($\\log 1 = 0$), indicating the summary is ungrounded.  \n\nIf it arises from **(2)**—due to misleading content or VLM misinterpretation or mis-attribution of key frames—the summary is still grounded, because the video influences generation. The masked summary, with removal of meaningful keywords, should carry no new information and thus should not contribute to hallucination\n\nThe works referenced by the reviewer ([1], [2]) primarily examine the fundamental limitations of VLMs,  which can be categorized into case **(1)**. Since the grounding score also considers the accuracy (probability of answering the correct option) without video, this term acts as a normalization term to mitigate the internal model bias.\n\n---\n**Weakness 2: Object-centric bias and framing of short clips**  \nWe agree that object-centric bias is a known limitation of some VLMs, but this issue stems from model pretraining, not from our proposed evaluation method. Recent VLMs are typically trained to mitigate this bias, and we do not observe it to significantly impact our results.\n\nAs noted in Lines 282–284, traffic events in SUTD-TrafficQA unfold over just a few seconds. In such settings, directly viewing the clip is often more efficient and informative than reading textual summaries—even when the sampled frames inputted by VLM capture the full event and the generated summary is accurate. This emphasizes VIBE’s value in scenarios where processing full videos is costly or time-constrained."}, {"review_id": "i7lXFWBwpm", "reviewer": "Reviewer_2iXL", "summary": "This paper proposes an **annotation-free** method for selecting video summaries generated by vision-language models (VLMs). Inspired by the Information Bottleneck framework, the authors introduce two evaluation metrics: a **grounding score**, which measures the alignment between the summary and the video content, and a **utility score**, which reflects the summary’s usefulness for downstream tasks. To make the computation tractable, the method replaces the intractable joint and marginal probability distributions with **masked-content-conditional probability distributions**, enabling the use of VLMs’ next-token prediction capabilities. The paper is well-written, and both the motivation and method are easy to understand. Human studies demonstrate that the proposed method helps people to improve task accuracy and reduce response time compared to both randomly selected summaries and human-written summaries in benchmark video summarization datasets.", "questions": "1. Could you provide a comparison between summaries selected by traditional metrics and those selected by your proposed method? Specifically, it would be helpful to include statistics on the agreement or disagreement between the rankings, as well as the values of your selected summaries under these traditional metrics. This would offer more insight into how your method differs from or complements existing approaches.\n2. It seems that, the grounding score is normalized to the range [0, 1], while the utility score is not in Figure 3 of main paper. If these two scores are combined via a weighted sum to perform trade-offs, would it be better to normalize both for consistency and interpretability?\n3. Have you considered or analyzed the lengths of the selected summaries? As shown in Tables 2 and 3 of Appendix A, the summaries selected by the proposed method appear to be longer than TL;DR but shorter than abstract, which suggests that length may be a critical factor influencing both task accuracy and response time.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The method eliminates the need for costly manual annotations as references.\n2. It leverages powerful pre-trained vision-language models (VLMs) as evaluators.\n3. The proposed method is computationally efficient and easy to implement.\n\n**Weaknesses**\n1. Although the paper mentions **traditional metrics** such as ROUGE, BLEU, and CIDEr, it does not provide a thorough analysis or comparison between these established metrics and the proposed grounding/utility scores.\n2. The proposed metrics emphasize informativeness but may overlook the **correctness** of the summaries.", "was_revised": true, "final_justification": "The rebuttal clarified my main questions. I have no remaining concerns and will maintain my original score.", "rebuttal": "We thank the reviewer for the thoughtful feedback and detailed technical questions. Below, we respond to each point.\n\n---\n**Weakness 1: Comparison with traditional caption evaluation metrics**  \nTraditional metrics such as ROUGE, BLEU, and CIDEr require gold-standard captions, which are unavailable and prohibitively expensive to produce—particularly for the datasets we use. More importantly, these metrics evaluate sentence-to-sentence (caption-to-caption) similarity, whereas VIBE evaluates caption-to-video alignment in a multimodal setting. Hence, VIBE focuses on a fundamentally different problem.\n\nGold-standard captions also introduce annotator bias and often lack the conciseness and utility needed for downstream tasks. Therefore, previous works need 3~5 human annotators per video. We believe comparing VIBE with caption-based metrics is not a meaningful apples-to-apples comparison.\n\n---\n**Question 1: Why no direct comparison to BLEU or CIDEr?**  \nWe cannot provide a direct comparison to BLEU or CIDEr because our datasets do not include gold-standard captions, which these metrics require. This is precisely why we propose VIBE: to support evaluation in the absence of reference annotations.\n\nFurthermore, even when available, high BLEU scores between candidate summaries do not guarantee correctness—if all captions are inaccurate, their mutual similarity is irrelevant. Our method instead evaluates how well a generated summary aligns with the video content, a dimension traditional caption-to-caption metrics fail to capture.\n\nAs suggested by **Reviewer P4an**, we have added an additional baseline using LLM-as-a-judge for caption selection.\nSpecifically, we experimented with GPT-4o (via OpenAI API) as a meta-evaluator on 150 examples from LongVideoBench, used in Fig. 3. For each video, GPT-4o was asked to rank five generated captions in terms of grounding and utility (the relevance to video content and questions).\n\nHowever, the results revealed significant issues:\n- GPT-4o indicated that **32%** of the video clips contained captions completely irrelevant to the video content, making grounding-based ranking unreliable. This issues happens even if the model is not asked to acknowledge the \"unrankability\".\n- Similarly,  GPT-4o flagged **92%** of captions as irrelevant to the associated task questions, making utility-based ranking equally unstable.\n\nThis suggests that even strong VLMs like GPT-4o fail to reliably evaluate captions on LongVideoBench—likely due to known challenges in long-video understanding. Although humans can often infer correct answers from these captions, current VLMs cannot. Due to this inconsistency, we decided not to include GPT-4o as a baseline in our experiments.\nInterestingly, despite the poor performance of VLMs in this task, they can still be used in VIBE to evaluate captions since we do not require complicated visual reasoning, but only asking the VLM to \"guess\" certain masked tokens and measure their confidence in next-token probability.\n\nFor the subset of examples where ranking was possible, the Mean Absolute Error (MAE) of the ranking was:\n- **Grounding MAE:** 1.59  \n- **Utility MAE:** 1.92\n\nThese numbers reflect disagreement with VIBE as there are only 5 captions (ranking 1~5).\nTo sum up, the experiments do not justify including GPT-4o as a reliable oracle.\n\n---\n**Weakness 2: On summary correctness**  \nRegarding task correctness, we demonstrated in Fig. 4(a) and (b) that VIBE scores correlate with human task performance, as reflected by accuracy and utility ratings.\n\nOur grounding score assesses factual correctness by measuring how token probabilities shift when the video is removed. Tokens whose probabilities remain unchanged are likely ungrounded (Eq. 3), suggesting the model generated them without relying on visual evidence. As a result, hallucinated content typically receives grounding scores near zero. Similarly, when a caption is factually incorrect—either irrelevant or contradictory—inputting the video tends to suppress the probability of those tokens, leading to zero or negative scores. This mechanism allows our method to identify ungrounded or inaccurate content.\n\n---\n**Question 2: Clarification on score range and normalization**  \nWe believe the reviewer is referring to the utility score, as it has the closest range to [0, 1]. We also acknowledge that Fig. 3 had the axes flipped: the X-axis should be “Grounding Score $\\log I(V; T)$” and the Y-axis should be “Utility Score $\\log I(Y; T)$”. This has been corrected.\n\nAs noted in Line 132, pointwise mutual information (PMI) can range from \\([−∞, ∞]\\). While the scores may appear bounded in Fig. 3, this is not mathematically the case—both grounding and utility scores are unbounded in theory.\n\nRegarding score behavior: the grounding score (Eq. 3) is computed at the token level, while the utility score considers the joint probability over multiple masked tokens. This means utility scores tend to be higher when more relevant keywords appear. We considered normalizing by the number of masked tokens, but doing so would break consistency with mutual information theory and would make $\\alpha$ a function of the number of masked tokens.\n\n---\n**Question 3: Influence of summary length on response time and accuracy**  \nWe computed the mean and standard deviation of word count for each method across datasets:\n| Dataset         | Naive VLM            | Max-U                | Max-G                |\n|-----------------|----------------------|-----------------------|----------------------|\n| LearningPaper   | $27.90 \\pm 4.79$     | $30.20 \\pm 11.38$     | $29.90 \\pm 5.36$     |\n| LongVideoBench  | $48.60 \\pm 7.18$     | $38.20 \\pm 9.05$      | $42.40 \\pm 10.27$    |\n| SUTD-TrafficQA  | $65.60 \\pm 10.02$    | $67.60 \\pm 7.09$      | $68.00 \\pm 7.60$     |\n\nAs shown, the summary lengths are relatively consistent across methods within each dataset. This is expected, as all methods select from the same candidate pool with comparable length distributions. Therefore, differences in task accuracy and response time found in Table 1 are unlikely to be explained by length alone.\n\nAs shown in Appendix Fig. 9(f), we conducted a correlation analysis between summary length and task accuracy across all three datasets:\n* **LongVideoBench**: No significant correlation\n* **SUTD-TrafficQA**: No significant correlation\n* **LearningPaper24**: A weak positive correlation was observed (Spearman $r\\_s(38) = 0.429,\\ p = 0.006$)\n\nThese results suggest that while summary length may have some influence, particularly in the LearningPaper dataset, it is not a consistent or dominant factor. Instead, accuracy appears to be better predicted by the utility score we proposed. We conjecture that **task accuracy depends more on the presence of informative content (e.g., task-relevant keywords)** than on overall word count.\n\nRegarding response time, we did not observe a significant correlation with summary length in any dataset. This may be due to the narrow length distribution across conditions, as all summaries were drawn from the same candidate pool. While it is intuitive that longer summaries require higher response time, we consider this effect an inherent property of human information processing and hence not a focus of our analysis."}, {"review_id": "2Spcc78SbI", "reviewer": "Reviewer_P4an", "summary": "The paper presents a new method to select the best video summary for downstream tasks using video-to-text information bottleneck, without any human annotation. To achieve it, the authors propose to calculate two scores - a grounding score, which measures alignment between a summary and a video, and an utility score, which demonstrates how good summary is for the further task. There conducted experiments across three benchmarks showing that their method significantly improves people speed and accuracy on video-related tasks in most cases.", "questions": "1. I like the baselines you propose (video watching and random summary), but why don't you include a baseline where a meta LLM (or VLM) selects the best summary given the downstream task? Alternatively, what about a baseline where a human chooses the best summary? Also, it would be interesting to compare the performance of such a meta LLM against human annotator.\n2. Could you clarify how you obtained the five different summaries used in the experiment? Did you use multiple VLMs, a single VLM with varied prompts, or some specific API? If so, please list used ones.\n3. In Figure 4, there is a mention of \"VLM (CoT) Accuracy (10%)\", but this term is not defined in the main text. Please correct this.\n4. I have a concern regarding your NeurIPS Paper Checklist response (\"No\") to question #8. The guidelines state: \"The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.\" Since your experiments involve generating five summaries (using some VLMs or APIs) and calculating utility and grounding scores via VLM inference, this clearly requires compute. Please report the compute usage in the paper and revise your answer to question #8, or clarify if my understanding is incorrect.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. Well-written and easy-to-follow text with excellent illustrations.\n2. Introduces a novel problem and provides an elegant solution that does not require human annotation.\n3. Well designed experiments involving humans, demonstrating the practical utility of the proposed method in downstream tasks. \n4. Two proposed metrics enable the method to be effective both when the downstream task is known and when it is not. \n\n\n**Weaknesses:**\n1. Lack of an easy baseline. \n2. Requires multiple inferences of the VLM for each summary.", "was_revised": true, "final_justification": "I consider this paper to be a strong piece of work. All my concerns have been addressed. Therefore, I maintain score 5.", "rebuttal": "We thank the reviewer for thoughtful comments and detailed technical questions. Below, we respond to each point.\n\n---\n**Weakness 1 & Question 1: Suggested additional baselines**  \nDue to rebuttal time constraints, we were unable to conduct a full-scale human preference study. Additionally, such studies are often subject to annotator disagreement and noise, and more importantly, they contradict our core motivation: building an annotation-free, scalable video-caption evaluation framework that avoids the cost and subjectivity of manual labeling.\n\nPer reviewer suggestion, we experimented with GPT-4o (via OpenAI API) as a meta-evaluator on 150 examples from LongVideoBench, used in Fig. 3. For each video, GPT-4o was asked to rank five generated captions in terms of grounding and utility (the relevance to video content and questions).\n\nHowever, the results revealed significant issues:\n- GPT-4o indicated that **32%** of the video clips contained captions completely irrelevant to the video content, making grounding-based ranking unreliable. This issue happens even if the model is not asked to acknowledge the \"unrankability\".\n- Similarly,  GPT-4o flagged **92%** of captions as irrelevant to the associated task questions, making utility-based ranking equally unstable.\n\nThis suggests that even strong VLMs like GPT-4o fail to reliably evaluate captions on LongVideoBench—likely due to known challenges in long-video understanding. Although humans can often infer correct answers from these captions, current VLMs cannot. Due to this inconsistency, we decided not to include GPT-4o as a baseline in our experiments.\nInterestingly, despite the poor performance of VLMs in this task, they can still be used in VIBE to evaluate captions since we do not require complicated visual reasoning, but only ask the VLM to \"guess\" certain masked tokens and measure their confidence in next-token probability.\n\nFor the subset of examples where ranking was possible, the Mean Absolute Error (MAE) of the ranking was:\n- **Grounding MAE:** 1.59  \n- **Utility MAE:** 1.92\n\nThese numbers reflect disagreement with VIBE as there are only 5 captions (ranking 1~5).\nTo sum up, the experiments do not justify including GPT-4o as a reliable oracle.\n\n---\n**Weakness 2: Multiple inference limitation**  \nWe acknowledge this limitation, as noted in the paper. However, we wish to clarify that VIBE operates at the token level—each evaluation is a single forward pass for one token, rather than a full sequence generation. Since sentence generation involves many such token-level steps, the relative cost of VIBE is comparable and not prohibitive in practice.  \nWe will add a brief clarification to the limitations section to make this point more explicit.\n\n---\n**Question 2: Implementation details**  \nIn all experiments (including the ablation in Appendix B), we generated **five summaries per model** using **five temperature values**: $0$, $0.5$, $1$, $1.25$, and $1.5$.  \nFor each run, we held the prompt, model, and vLLM backend constant. The primary model used in the main paper was **Qwen-2.5-72B-AWQ**. For ablation studies, we also evaluated:\n- **InternVL-2.5-8B-MPO**\n- **InternVL-3-38B**\nThese results are reported in Appendix B.\n\n---\n**Question 3: Clarification on “VLM (CoT) Accuracy (xx%)”**  \nThank you for pointing this out. “VLM (CoT) Accuracy (xx%)” refers to a baseline where we prompt the same VLM—Qwen-2.5-72B-AWQ—to answer the 10 task questions from the user study using only the video input and **chain-of-thought (CoT)** prompting. The percentage reflects its accuracy on those questions.  \nWe will clarify this definition in the main text in future revisions.\n\n---\n**Question 4: Hardware details for reproducibility**  \n\nThank you for the reminder. We have updated the paper checklist to reflect this and added the following system information:\n\nAll experiments were conducted on **four NVIDIA RTX 6000 Ada GPUs** (48GB VRAM) using the **vLLM backend**.  \nThe system was equipped with an **Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz**, featuring **64 cores (x86_64, 64-bit)**.  \nThis setup handled all model variants—Qwen-2.5-72B-AWQ, InternVL-2.5-8B-MPO, and InternVL-3-38B—efficiently for both caption generation and VIBE evaluation."}]

Author rebuttal:

Meta-review: This paper introduces an annotation-free framework for evaluating video-to-text summaries generated by VLMs, proposing two key metrics: grounding score and utility score. Prior to the rebuttal, the paper received mixed reviews from reviewers, the main concerns are about the absence of comparisons with traditional evaluation metrics and insufficient discussion of related inferences from VLMs. The authors have addressed the majority of these concerns in their rebuttal, providing clarifications and additional results that mitigate the initial criticisms. Based on this, the final recommendation is to accept this paper. We encourage the authors to incorporate the addressed points into the camera-ready version to further strengthen the work.
Decision: Accept (poster)