SPIKE-RL: Video-LLMs meet Bayesian Surprise
Authors: Sahithya Ravi
Aditya Chinchure
Raymond T. Ng
Leonid Sigal
Vered Shwartz
Venue: ICLR 2026 Poster (2026)
Keywords: Video LLMs
Video reasoning
Bayesian Surprise
Belief tracking
GRPO
Frame Sampling
Video Captioning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We make Video-LLMs surprise-aware by tracking belief shifts in videos.

Abstract: Real-world videos often show routine activities punctuated by memorable, surprising events. However, most Video-LLMs process videos by sampling frames uniformly, likely missing critical moments that define a video's narrative. We introduce SPIKE, an inference-time framework that quantifies Bayesian Surprise as the belief update triggered by new visual evidence in the video stream, identifying moments where new visual evidence conflicts with prior beliefs. SPIKE effectively localizes surprise in videos, correlated with humans on positive (FunQA) and negative (Oops!) surprise benchmarks. SPIKE-RL further improves on SPIKE's ability to detect surprise, leveraging GRPO to refine its belief hypotheses based on a reward signal from the video caption. SPIKE and SPIKE-RL guide query-agnostic surprise-weighted frame sampling, which allocates more frames to interesting moments in the video. With this strategy, we achieve consistent performance gains on five downstream benchmarks. By enabling Video-LLMs to track beliefs and register surprise, our work paves the way for more robust models that can revise their understanding in response to new information.

Published as a conference paper at ICLR 2026
SPIKE-RL: VIDEO-LLMS MEET BAYESIAN SURPRISE
Sahithya Ravi 1,2 Aditya Chinchure Raymond T. Ng' Leonid Sigal" 1.2 Vered Shwartz
'Tee University of British Columbia, Vector Institute for AI (sahiravi, adityal0, rng, isigal, vshwartz ecs. ubc. ca
ABSTRACT
Real- world videos often show routine actívities punetuated by memorable, sur- prising events. However, most Video-LLMMs process videos by sampling frames uniformly, likely missing critical moments that define a video's narrative. We in- troduce SPIKE, an inference-time framework that quantifies Bayesian Surprise the belief update triggered by new visual evidence in the video stream, identifying moments where visual evidence conflicts with prior beliefs. SPIKE effec- tively localizes surprise in videos, correlated with humans on positive (FunQA) and negative (Oops!) surprise benchmarks. SPIKE-RL further improves SPIKE's ability to detect surprise, leveraging GRPO to refine its belief hypotheses based on a reward signal from the video caption. SPIKE and SPIKE- RL guide query-agnostic surprise-we weighted frame sampling, which allocates more frames to interesting moments in the video. With this strategy, we achieve consistent perfor- mance gains on five downstream benchmarks. By enabling Video-LLMs to track beliefs and register surprise, our work paves the way for more robust models that can revise their understanding in response to new information
INTRODUCTION
Humans navigate the world not passive observers, but active predictors of the future who infer the hidden causes behind events and update their predictions (Millidge et al. 2022). This process, formalized within the Bayesian Theory of Mind (ToM) framework (Baker al. 2017), suggests that our brain continuously builds and updates an internal model of the world, using discrepancies between expectation and reality, surprise, the primary signal for learning and attention. This allows us to efficiently process a constant stream of sensory data, focusing our cognitive resources on moments that are novel and informative, and ignoring redundant, expected information. For instance, in the Mr. Bean video shown in Figure I our cognitive focus is on the moment the man unexpectedly falls, because deviates from the established routine.
However, current Video-LLMs are fundamentally disconnected from this sequential, belief-driven process. Most models treat videos as 'bag of frames', where a subset is uniformly sampled from the video (OpenAl 2024 Bai al.] 2023 2025 Cheng al., 2024: Liu et al.) (023). Lacking an evolving belief about the video's s story, uniform sampling is much more likely to sample highly frequent mundane moments over rare surprising (and therefore memorable) events. This can poten- tially overwhelm Video-LLMs with redundant information, over pivotal moments human observer would focus on, such as the fall in Figure 1
To overcome this, some methods select or retrieve frames retroactively for a given textual query Yo et al. 2025 Wang et al.) 2025 2024 Liang et al. 2024 Tang et al.] 2025b). However, in dynamic, open-world settings, we often don't know in advance what questions will be asked. What we need instead is a model that reasons proactively, anticipating what is surprising, and paying attention to these shifts, similar to a human observer. In this work, we study two fundamental questions to bridge this gap: (1) How can Video-LLMs proactively track and update their beliefs as new visual evidence presents itself? and (2) Can detecting semantically surprising events proactively and ahead of downstream queries improve video understanding?
!Code, data and models will be made public
htttps://github.com htapl
SPIKK-RY

---

Published as a conference paper at ICLR 2026
Original Video
Humen
Sump
BPIKE
Burprite Store
Weiggtte roig
Surpetoe Localization Accuracy
x
X
Figure I: (a) Uniform sampling misses key moments. (b) Our surprise-based sampling focuses on high-surprise regions, strongly aligning with human laughter. (c) Our method achieves significantly better surprise localization than zero-shot Qwen2.5-VL baseline.
To answer these, introduce SPIKE, inference-time framework that represents model's be- liefs as explicit probability distributions over human-interpretablee textual hypotheses, and quantifies Bayesian Surprise as the divergence between prior and posterior beliefs (Itti & Baldi 2005), giving us a surprise score. As shown in Figure [b), this surprise score pinpoints moments that contra- diet the model's prior beliefs. We further improve the surprise scoring by introducing SPIKE-RL, trained using reinforcement learning objective that teaches the model to prioritize beliefs that lead more accurate video captions. SPIKE achieves 65.7% on FunQA (Xie al., (025). surprise localization benchmark, and SPIKE-RL improves on it further, with 68.2%, significantly outperforming the zero-shot performance of Qwen2.5- (Figure]cc)) Our experiments show that SPIKE-RL delivers two complementary benefits: it improves the diversity of generated belief hy- potheses, and boosts surprise localization accuracy beyond what the inference-time scorer alone can achieve. Finally, we leverage this signal by replacing the standard uniform frame sampling with surprise-weighted sampling in Qwen2.5-VI. and demonstrate that this leads to consistent improve- ments five downstream video understanding tasks.
Our approaches allow Video-LLMs to focus on the most salient parts the video, akin to human notions of surprise. In the future, surprise-aware Video-LLMs can be used to improve the robustness real-time applications such as streaming, surveillance, robotics, and interactive agents that need to adapt new information on-the-fly.
BAYESIAN BELIEF TRACKING
2.1
SURPRISE SCORING
The architecture of SPIKE is shown in Figure SPIKE quantifies Bayesian surprise by tracking how the model's belief distribution over human-interpretable textual hypotheses shifts when new frame observed. Fach incoming frame updates this belief distribution, and the magnitude of the change defines the surprise score. SPIKE produces surprise scores for each step, across the complete video. For simplicity, we describe this process using fixed-length videos. However, our method can be adapted to streaming video setup by applying the update online.
Setup. A video is composed a sequence of frames XI.Ts where T is the length of the video. To compute surprise at timestep t, prior window of W frames immediately preceding the current I, W; = X,- W.t-1 (ii) historical summary, H₁, textual summary of what happened far in the video, derived from the C frames, Xz-Ca--* W that occurred before W; and (iii) the newly observed frame O₂ = X,. This setup al- lows the model to form belief' based on both long-term context and recent events, and then measure surprise with respect to the new information
three key inputs as shown in Figure [b:): (i) the
Hypothesis Generation. First, timestep I, we generate of belief hypotheses, B. (b.......,b,.. }. where each hypothesis b is textual description of what might happen next, gen-
"See Appendix for further information on how the textual summary is obtained. 'See Appendix A.1 for the prompts for the hypothesis generation and scoring.
2

---

Published as a conference paper at ICLR 2026
Vidco Segment
X
SPIKE
Hatorical Frames
Prior Window 3P,
Video-LUM
Hictony 11
Hypothesis Generator (M)
Burprise-Weighteer Sampling
Hypothasis (M)
Sampled Fremes
Higee canepiig ligh
Hyoothesze S6,
Pace
Question
Video-LLM LLM for Downstream Taaka
Score d, KL Divergence
Answer
Caption
h
Figure 2: (a) Overall architecture: SPIKE computes surprise scores, which guide weighted frame sampling for downstream tasks. (b) SPIKE Given history He, prior window W, and observed frame O₁, the hypothesis generator produces belief set Bg. The hypothesis scorer computes Pprior and P posts yielding surprise score S; as KL divergence.
erated by model M by conditioning on the historical summary H, and the prior frame window W. (Fig. 2). We use Video-LLM our model M and generate diverse beliefs B3 using nucleus sampling (Holtzman et al. 2020).
Bayesian Surprise. Next, we establish prior and posterior belief distributions over the generated beliefs B.. We define score for each hypothesis br,; based on its plausibility, which inversely proportional to its negative log-likelihood (NLL) as computed by the Video-LLMM M. This score reflects how well the hypothesis aligns with the given context.
The prior distribution Prior is calculated based the historical context (H₂) and the recent prior window (We), before the new frame O₂ is observed:
cxp (- NLL( He, W.)
Pptor(be (be,s He, We)
(1)
=== cxp (- -NLL(b., He, Wo)
where NLL(b; 1.) = log (b, 1)) is the negative likeeiioddood of the hypothesis tokens given the context, and т is temperature parameter. We apply softmax to normalize the scores into probability distribution.
After observing the new frame Oz, we update our beliefs to form the posterior belief distribution, Poost by incorporating this new visual evidence into the model's context:
cxp (-2 NLL(b,, He, W. O.)
Ppnal (br.s He. We, Oi)
(2)
En 1 exp (- NLL(br, He W., 0,))
Following the Bayesian formalization of surprise by & Baldi (2005), we quantify our surprise score to be the information gain induced by Oz, as the Kullback Leibler (KL) divergence between posterior and prior beliefs over hypotheses:
S = DKL (Ppoal: (: He, W., O₂) Ppttor(: H2, Wc)
(3)
(bo log Ppon (bou(
(4)
Pprior(be,s)
Using Equation 3 at each timestep we compute a scalar surprise score S, as well as belief set at containing hypotheses and their prior and posterior probabilities, B, {(br,,, (bn,. Ppeaa)) (b, (b..)))) 1 B, is human-readable and interpretable, enabling insight into why video segment is surprising.
3

---

Published as a conference paper at ICLR 2026
2.2 SURPRISE- WEIGHTED FRAME SAMPLING
Since it is computationally infeasible and impractical to process all frames of a video, Video-LLMs sample frames - by default, uniformly. Only the selected frames are then processed by the model while the rest are discarded. We define frame budget, F, as the maximum number of frames that a Vidco-LLM uses. Our goal is to effectively select those frames among the video frames X1.T by recognizing surprising regions of the video, which may be especially important for downstream tasks such as captioning and question answering.
Computing a Guided Probability Distribution. As shown in Fig 2a), for given video XI.T, we first uniformly sample timesteps fi....,tx. for K S F. Each timestep repre- sents the end of video segment, which measure surprise; this is akin dow over the frames of the video. We use SPIKE to compute surprise scores for each seg- ment, and obtain scores Si....,Sp Sk [0, 1] for the corresponding timesteps t1,...,tk. We can now modify the frame sampling to be proportional to the surprise scores. Specifically, we compute the probability of sampling from segment ending at t as the softmax over scores, exp(e./T.)
sliding win-
P: softmax 1
(T 0), and use p;=1/K if all s; are equal. Ta is the
Σソ= exp(sy/T.)
temperature of this softmax function.
Sampling. Given the frame budget F for the Video-LLM, we sample F frames by repeatedly choosing a segment with probability p; (with replacement) and drawing uniform timestamp within that segment; each timestamp is mapped to a frame index via the video frame rate. Choices independent, high-surprise segments can contribute multiple frames. We use Ta in Eq. 2.2 to control sampling: a small т; concentrates the budget on surprising regions, whereas larger T, spreads the frame budget more uniformly. We set T5 0.7 for experiments.
2.3 COMPLEXITY ANALYSIS
Let video contain 7 frames. We uniformly sample a fixed budget of F' frames, so the video is divided into W T/F segments and one frame is drawn from each segment. For each sampled frame we generate N text hypotheses and compute their prior and posterior likelihoods.
Time Complexity. The method requires F hypothesis- geneeratonnn steps and two batched likelihood evaluations per step. The total cost is therefore O(F N), which is linear in the chosen frame budget F (and therefore at most linear in T if F grows with T). In practice, GPU parallelization allows batching the N hypotheses at each step, amortizing the generation cost and reducing the total complexity from O(F - N) O(F) when sufficient parallel compute is available.
Relation to Inference- Time Scaling. Our overhead is comparable to recent inference-time scaling methods for Video-LLMs: controllable number of extra forward passes improves where the model allocates its fixed frame budget, without changing its architecture.
Interpretability. Because SPIKE represents beliefs as textual hypotheses, its Bayesian surprise scores are interpretable: one can inspect the generated hypotheses to understand what the model "expected" versus what the new frames revealed.
REINFORCEMENT LEARNING FOR BELIEF OPTIMIZATION
Motivation. The effectiveness of SPIKE relies on the model's 's ability to generate belief hypothe- ses that are accurate, diverse, and representative of the video segment shown. However, since VLMs, are not tailored to perform belief tracking on frame windows, the model has no incentive to refine its intermediate hypotheses. However, training SPIKE with direct supervision on this reasoning process is intractable, as it is impractical to collect ground truth hypotheses across every segment of a video, for large set of videos. Instead, we leverage GRPO Shao et al.) 2024) to optimize SPIKE using reinforeement learning. SPIKE-RLE is based the insight that strong final caption i.e. of what happened in the complete video - is built upon accurate intermediate belief hypotheses ie. about what is likely to happen after having watched portion of the video.
4

---

Published as a conference paper at ICLR 2026
SPIKE-AL
Surphte Meee--WWliggg Sampling
Video- LLM
Caption [Rollout ']
GT Caption
8,
Hypotheais Ganerator
Scorer
Caption (Rollout 2]
LLM-Match
Caption (Rollout 3)
GPPO Optimization
Figure 3: SPIKE-RL explores multiple hypothesis trajectories, whose surprise scores guide frame sampling. Captions from these rollouts are scored with LLM-Match, and GRPO propagates the reward improve hypothesis generation.
Figure 3 demonstrates our approach. To train the hypothesis generator, our policy model, we com- pute reward signal based on the quality of the final caption. This reward signal is then propagated backward, assigning credit to the sequence of beliefs that led to the successful outcome. In this way, supervision on the final result implicitly transformed into training feedback for the model's internal reasoning process. Our rewards are derived from an LLM--assed metric that computes the similarity between the generated caption and the ground truth caption.
Rollout. We design the GRPO-based training procedure by generating group of captions, based on different trajectories beliefs and frame allocations. For each video, we draw trajecto- ries {-(r)] Jr=1- Each trajectory 7(r) runs SPIKE over segments of the video. At every timestep t, samples textual beliefs B(r)
and scores prior and posterior beliefs to prior, post,e) and the surprise scores s(7), We then use the surprise scores to inform sampling of frames that are inputted into a Video-LLM to generate a single final video cap- tion, c(r) using our surprise-based frame allocation 2.2.. Thus each input induces a GRPO group: G ( S c(r) Ad
obtain (p
p(r)
Reward. At the end of rollout, the caption c(r) is scored using LLM-Match, where LLM judge measures how similar it is to the ground truth caption, obtain scalar reward R The prompt for the L.M judge is in Appendix A.2 We 7-score the LLM rewards within the group, and use the normalized scores as advantages in the policy objective, A R
dR
Loss. We treat the full set of hypotheses in a trajectory as sequence-level action. Let po (be. H, W,) denote the policy for generating a hypothesis given the video context. We define our belief- optimization objective as,
M
K ΣΣ
Lhelief optimizationn (0)
= M
((r)
log pa Po6
H (w())
(5)
1=1
which increases the likelihood of hypotheses along high-advantage trajectories and suppresses those along low-advantage ones. Optimizing Equation EquationStrriin trains the model to generate hypotheses that reli- ably support strong captions, improving both the intermediate belief trajectory and the final output.
Training. For training SPIKE-RL, we curated a video captioning dataset of 2,000 videos with 30% surprising and 70% unsurprising videos. The goal is expose the policy both routine events where beliefs remain stable and to inflection points that induce belief shifts. For the unsurprising portion, used ActivityNet Captions (Caba Heilbron al. 2015), which predominantly includes videos depicting everyday activities. For the surprising videos, we sample from from the training set of Oops! (Epstein al. 2020), a collection of short clips centered on unintentional human failures. We use Qwen2. 1122255I.. 5-VL- 7B- Inst as the Video-LLM model (M) and 0lmo-7B-hh as the LLM-Match reward model. See App. C forr the training hyperparameters.
4 SURPRISE LOCALIZATION
We first evaluate how well SPIKE and SPIKE-RL can identify surprising segments video. Hyperparameters for surprise scoring are described in App.
5

---

Published as a conference paper at ICLR 2026
4.1 EXPERIMENTAL SETUP
Benchmarks. We evaluate surprise localization on three benchmarks: Oops! (Epstein et al. 2020), FunQA (Xie et al.) 2025) and Mr. Bean (App. E Oops! is surprise detection task, whose test set contains 4,791 videos with precise timestamps marking the exact transition point to surprise. FunQA has 424 videos with annotations for the most surprising segment in each video, given by a start and end time. While these established benchmarks, they only annotate single surprising event per video. Since our method is capable of detecting multiple surprising segments in the video, curate our own benchmark, Mr. Bean, using 48 clips from the live-action TV show. Mr. Bean's audio laughter track serves as standard surprise annotations - segments of the video with laughter considered surprising.
Metries. Following the protocols of Oops! and FunQA, we report Acc@0.25s and Acc@ 1.0s for Oops!, and loU for FunQA. The accuracy metries (Acc) measure whether the predicted surprise peak falls within 0.25 or 1.0 seconds of the ground truth peak surprise, while loU measures the overlap between the predicted surprising windows and the ground-truth surprising windows. For details on the implementation of the metrics, see App. D
Baselines. We establish lower bound with a Random baseline that selects surprising frames random. We also report the zero-shot performance of our base Qwen2. 5-VL-7B-Inst model. which directly scores each uniformly sampled frame on whether it is surprising or not, with- out our proposed belief tracking mechanism (See Appendix AA.3] for the prompt and setup). On Oops!, we compare against: (i) VideoSpeed (Epstein et al., 2020p, the strongest reported baseline for this dataset; (ii) Motion Magnitude (Epstein et al., (020)) an optical-flow-batt based approach; and (iii) F2C2V (Duka et al. 2022). self-supervised method. As an upper-bound reference, we also re- port the human consistency or agreement from the original dataset. On FunQA, we compare against TimeChat (Ren et al. 2023), UniVTG (Lin et al. 2023), a specialized video temporal grounding framework, and
a baseline provided by the FunQA benchmark that applies the clipping and rating (CR) technique from UniVTG to LLaVA-NeXT
(Liu et al. 2024).
4.2 RESULTS
Table [ the performance of SPIKE and SPIKE-RL on the surprise localization task. On the Oops! benchmark, our SPIKE.RL model achieves an score of 62.9% on Ace@ 0.25s, re- markably close to the human performance (6)..%) Notably, both SPIKE and SPIKE-RL. show about a tenfold improvement over the performance the zero-shot version of the same model (Qwen2. 5-V--- BB) Compared to VideoSpeed, F2C2V, we observe that SPIKE and SPIKE-RL are better at accurate localization, with 23.4% higher Acc@0.25s, and achieve similar Acc@ Is scores. On the FunQA benchmark, SPIKE-RE RL once again demonstrates superior performance with an IoU of 68.2, surpassing both prior approaches and the zero-shot model by a substantial margin. It is worth noting that this significant boost is despite the fact that FunQA which is composed of positive surprises related to humor and creativity - is out----distiitation for SPIKE-RI.
Mr. Bean shows similar trend to the other benchmarks, but the absolute Acc@0.25s is lower. This dataset is particularly challenging. In contrast to the other benchmarks, some of the surprising moments in Mr. Bean arise from subtle, fine-grained nuances in his facial expressions rather than easily noticeable unexpected events. Finally, we observe significant 6.3% gain in IoU score with SPIKE-RL RL over SPIKE. Since IoU on Mr. Bean evaluates detection across multiple surprising segments, this gain highlights the ability of our scorer to capture nuanced surprises within video.
Overall, the inference-time method, SPIKE, achieves superior performance across all benchmarks and generalizes to different types of surprises, while SPIKE-RL further boosts performance through reinforcement-guicee refinement.
4.3 BELIEF SET EVALUATION
We evaluate the hypotheses generated by SPIKE and SPIKE-RL using a combination of automatic metrics and human evaluation.
Diversity. We are interested whether models generate multiple conceptually- diverse hypotheses or different lexical variations of the same hypothesis. For given video, we measure the diversity

---

Published as a conference paper at ICLR 2026
Table 1: Performance of SPIKE and SPIKE-RL surprise localization.
Oops!
FunQA
Mr. Bean
Method
A.c00.25s Acc@ Is
IoU
Acc@0.25s
Ace@1 Is
loU
Baselines
Random
6.8
2,6
7.5
0.6
3.5
0.9
Motion
23.1
50.7
Video Speed
36.6
65.3
F2C2V
39.5
69.5
TimeChat
9.6
UniVTG
45.3
LLaVA-NeXT-CR
62.3
Qwen2.5-VL
6.6
9.6
11.6
11.2
23.2
13,8
SPIKE
60.0
67.3
65.7
53,2
70.2
54.8
SPIKE-RS RL
62.9
69.1
68.2
57.4
78.7
61.1
Human
62.1
88.0
Table 2: Performance of Qwen2.5-VL with uniform vs. surprise-weighted and other query-free frame sampling methods. MCQ tasks are evaluated with accuracy; generative tasks with LLM- Match. Comparable open-source Video-LLMs are shown for context.
Model
Size
Sampling
BlackSwan FunQA ExFunTube
VideoMME-S NexIQA
VideoChat2
7B
Uniform
49.7
17.9
45.6
VideoLLama2
Uniform
52.9
7.7
56.0
FunMentor
7B
Uniform
33.2
LLaVA-Vide
7B
Uniform
70.4
46.6
62.7
Qwen2.5-Vv VL
7B
Uniform
67.2
66.8
68.7
59.8
68.6
Qwen2.5-Ve LL
7B
RGB Histogram
49.6
55.4
Qwen2.5-VL
7B
ECR
49.7
58.2
Qwen2.5-VL
Katna
QWen2.5-VL
7B
Optical Flow
58.6
58.1
Qwen2.5-VL.
7B
SPIKE
68.8
70.3
73.2
60.8
69.8
Qwen2.5-VL.
SPIKE-RI
69.5
71.4
75.7
62.5
70.3
Qwen2.5-VL
32B
Uniform
69.4
72.7
71.9
69.9
72.3
Qwen2.5-VL
32B
SPIKE-RI RL
71.7
75.8
75.8
73.5
74.1
of hypothesis set with the average inverse cosine similarity (1 cos(b, (,1,,,;)) b;)) across all hypothesis pairs. SPIKE-RL achieves 40.3%. higher than SPIKE's 33.5%, showing that the RL training improves diversity.
Correlation with human judgments. We measure how well our surprise score aligns with human judgments by showing human annotators random sample 100 videos from Oops! along with the generated hypotheses and asking them to rank the hypotheses by dragging them onto 0-100 scale. Each video segment is evaluated twice: first using only the prior frames (Ozr), and then again after revealing the observed frame (O,). This setup emulates the prior and posterior probabilities in Eq from which we compute human-derived surprise scores. Comparing these to SPIKE and SPIKE. RL's surprise scores yields Spearman correlation of 0.84 and 0.87 respectively, indicating very strong correlation and confirming that our method effectively captures belief shifts. The template for human evaluation provided in App.
5 DOWNSTREAM TASKS
Having shown that SPIKE and SPIKE-RL can perform surprise localization, now explore how identifying surprising segments of the video and allocating more frames to such regions can improve Video-LL.M's performance downstream tasks described in 2.2

---

Published as a conference paper at ICLR 2026
Uniform
shows thrse people riding signaaaa grasey path. peraon wearing halmet riding amonthly while persor behind them also wearing a helmet but appeors be struggling with balance. As they continue down the path, the perzon behind falls off their segway and lands the ground. The other riders continue on without stopping.
SPIKE
The video shows three people riding segways grassy parh. One of them, woman wearing pink shirt and red helmet, loses her balance and la off her segway onto the grass. The other two continue niing without stopping. The woman remains on the ground for moment before getting and walking
people ride their segmays forword
riding when woman with pink shirt stays cantyrery.
the
0.00
0DD1
seguays
The lieey fall causing cheos
0.45
sher fa her segway
regsin
SPIKE -RL
The women the pink shirt hur
woman join and they continue
Pam
woman the segiaay will athempt
The video threa ind viduals segways grassy The riders enjoying their time ouidbors, halmats black jecket, stops segway.
safety, they
sogwey onto grass. path, riders, woman woman, seem
react quickly
@ Bellef Hypotheses Sets
Soores
+ Frames
Explanation
32 frames used
Figure 4: Qualitative Results. We compare Uniform, SPIKE and SPIKE-RL sampling methods. Errors in the explanation generated using uniform sampling reduce with SPIKE and are resolved with SPIKE-RL. We show belief hypotheses (B,) at various timesteps, and observe how the divergence Pprtor and P Post accurately captures the surprising moment the video.
5.1 EXPERIMENTAL SETUP
Benchmarks. We evaluate sampling method diverse selection of tasks, spanning surprise explanations, question answering, and temporal reasoning. The Reporter-MCQ portion of Black- SwanSuite (Chinchure al., 20255) tests models' ability to describe unexpected event in MCQ setup. FunQA's Task (Xie et al.) 2025) and ExFunTuub (Dayoon Ko 2023) ask models to generate an explanation of why events are surprising. Moving beyond surprising videos, we test our models on two MCQ tasks VideoMME (Fu et al. 2024), which probes general multimodal reasoning (we focus on short videos without subtitles), and NexiQA (Xiao et al. 202I), which targets temporal, commonsense, and causal reasoning.
Following prior work (Majumdar al. 2024 Xie et al. 2025), evaluate the generative tasks using LLM-Match, prompting GPT-4o to rate the similarity between model-generated and ground-truth answers. Multiple-choice tasks are evaluated using accuracy.
Metrics.
Video-LLM Baselines. We consider widely adopted open-source Video-LLMs capable of video explanation and QA, including VideoChat2 (Li et al.] 2024). Vide2 (Cheng et al. 2024), and L.AVA-VVide (Liu et al., 2023). We also include FunMentor (Xie et al. 2025), model specifically designed for humor understanding. Our base model is Qwen2.5-VL. (7B), which we use to evaluate alternative sampling strategies under fixed frame budget on BlackSwan and VideoMME-S. F.nnMM..E....l.ll........ we test whether SPIKE-RL improves performance on a larger model, Qwen2.5-VL (32B).
Query-free Frame Sampling Baselines. To assess the effectiveness of our sampling, bench- mark against shot boundary detection methods on BlackSwan and Tideo-MME-S. Specifically, we tested RGB Histogram differences (V & Narayanan 2015). Edge Change Ratio (ECR; Mann & Kaur 2015), and motion-based detection (Wolf) 1996). which capture changes in texture, structure, motion, and similarity. In all of these approaches, salient peaks are detected via smoothed scores and frames are distributed proportionally to peak strength, ensuring that the frame budget F is met. We also include Katna clustering method which applies K-means frame histograms and
8

---

Published as a conference paper at ICLR 2026
selects the frame closest to each centroid. We use maximum frame budget F of 64 frames for all our baselines, regardless of the sampling approach.
5.2 RESULTS
Table 2 shows the performance of SPIKE and SPIKE-RL on downstream benchmarks. On tasks with surprising videos (BlackSwan, FunQA, ExFunTube)) surprise-aware sampling provides sub- stantial gains over uniform selection. Relative to uniform sampling, SPIKE improves accuracy by +1.6% on BlackSwan, +3.5% on FunQA, and +4.5% on ExFunTube. We observe that SPIKE-RL further extends performance these tasks, with gains of +2.3% and +4.6% on BlackSwan and FunQA, and +7.0% on ExFunTube, marking our largest gains over uniform sampling. These results not only show the effectiveness of SPIKE in prioritizing surprising frames, but also credit the im- proved hypothesis quality in SPIKE- RL. On Qwen2.5-VE. VL 32B, we see 2.3%, 3.1% 5 and 3.9% gains respectively with SPIKE-RI, showing that our methods benefit larger models as well, extending their video understanding capability.
In general QA tasks (VideoMME-S, NexIQA), we moderate but consistent improvements over uniform sampling. SPIKE boosts scores by +1.0% on VideoMME-S and +1.2% on NexIQA, while SPIKE-RL achieves +2.7% and +1.7% respectively on the 7B variant. The 32B variant with SPIKE-RL shows larger improvements of 3.6% and 1.8% on these tasks. These results show that surprise-awee aware sampling is broadly beneficial.
SBD strategies such as RGB Histogram, ECR, Katna, and Optical Flow consistently underperform uniform sampling. Their reliance on raw visual change makes them sensitive camera motion and scene cuts, which rarely align with semantically important events. In contrast, our method offers principled guidance for identifying critical moments. Overall, we demonstrate that Bayesian Sur- prise provides a powerful inductive signal for adaptive frame selection: SPIKE delivers immediate gains by reallocating a fixed frame budget toward more informative segments, while SPIKE-RL further improves robustness through reinforcement-guiie guided belief optimization.
5.3 QUALITATIVE EXAMPLE
Figure 4 illustrates the differences between uniform sampling, SPIKE, and SPIKE-RL. Under uniform sampling, the Video-LLM generates a caption that notes someone falling off a segway but misidentifies the person and the actions of the other riders (error highlighted in red). With the same frame budget, SPIKE and SPIKE-RL reallocate samples toward segments with high surprise scores, guided by observed belief shifts as demonsrated by the hypotheses. SPIKE correctly captures that the woman in the pink shirt and helmet loses balance and falls, though it still makes an error by stating that the other riders continue without stopping. SPIKE-RL. RL improves on this. By more accurately localizing surprising segments with one peak at the main fall and another smaller peak later SPIKE RL increases sampling density around both critical events. This leads to a more precise description of both the fall and the subsequent reactions of the other riders.
RELATED WORK
Belief Tracking and Updating. Recent research in NLP has explored the idea of maintaining and updating beliefs, often with Bayesian inspired methods. Studies show that, with sufficient evidence, LLMs can approximate Bayesian updates rather than simply relying on pattern matching (Gupta al., 2025). Closest to our work, Kim et al. (2025) explicitly maintain and re-weight hypotheses about agents' mental states as new information becomes available, mirroring Bayesian Theory of Mind. This principle of explicit tracking also improves model robustness complex scenarios involving multiple characters and higher-order Theory of Mind (Sclar al., (023). This process is closely related to the concept of defeasible reasoning, where conclusions are revised by new evidence (Rudinger al. 2020)). More broadly, the principle of Bayesian Surprise has been used as a powerful driver for exploration in other domains, such as guiding open-ended scientific discovery (Agarwal al. 2025). We extend this idea of discovery the domain of video understanding.
Adaptive Frame Sampling Strategies for Video-LLMM. Prior work on frame selection for Video- LLMss is primarily based on the relevance the question. Query-conditioned methods rank frames

---

Published as a conference paper at ICLR 2026
with respect textual prompt minimize redundancy while preserving task-relevant evidence. Frame- Voyager( Yu et al. 2025). Flexible Frame Selection (FFS; |uch et al. 2025) and Hu et al. (2025) learn to select informative frame sets conditioned on the query using lightweight train- ing strategies. Adaptive Keyframe Sampling (AKS: Tang et al. 2025b) formulates selection as a plug-and-play module optimizing relevance the prompt and Guo al. (2025) propose dynamic keyframe search driven by visual chain-of-thought. VideoTree (Wang et al. (0255 organize a video into a hierarchical tree and traverse it in question adaptive manner. contrast, our method drops in as replacement for the Video-LLM's uniform sampling layer, reallocating the frame budget towards surprising moments while remaining query-agnostic.
Video Saliency and Attention. Several works have explored video saliency and attention mecha- nisms to focus on important frames. Hu et al. (2025) uses differential keyframe selection to choose salient frames and differential feature merging to compress non-keyframes, focusing on query- relevant information while reducing redundancy, thus improving long-form QA. Ma et al. (2025) introduces Video Token Sparsification (VTS) is CNN-based approach to reduce visual tokens for efficient MLLM use autonomous driving. Lee et al. (2025) introduces LLMVS, a video sum- marization framework using LLMs to evaluate frame importance based captions and refines it based on global attention mechanism. Tang et al. (2025a) introduces Adaptive Keyframe Sampling (AKS), that performs keyframe selection maximize useful information within token limits, opti- mizing for relevance to the prompt and coverage of the video. Azad et al.] (2025) introduces HierarQ. hierarchical Q-Former framework that processes video frames sequentially using short and long- term memory banks for enhanced temporal modeling and task-aware video comprehension. Koala Tan et al. (2024) key frame-conditioned long Video-LLM, which uses learnable spatiotemporal queries to adapt pretrained VLMs for longer videos. The key distinction in our work is that we explictly peform belief tracking, and show that our surprise is correalted with humans. Frame selec- tion is our downstream application, but the belief tracking process could be useful for explainable reasoning, video streaming, and training models with grounded belief trajectories in the future.
Bayesian Theory of Mind and Prediction Error. Bayesian models of social cognition frame hu- man reasoning about others through Bayesian Theory of Mind (BToM), where observers infer latent beliefs and goals by inverting generative model of action Baker et al. (2009) 2017). Comple- menting these Bayesian approaches, predictive coding theory proposes that the brain continuously generates predictions about sensory input and uses prediction errors to update internal models Rao & Ballard (199)); Millidge et al. (2022). Empirically Koster-Haa Hale & Saxe (2013), extended predic- tive coding to theory of mind, with brain activity showing reduced responses for predictable versus unpredictable beliefs. In this integrated view [Thornton et al.] (2018). BToM provides the content of the generative model (c.g., "She is thirsty"), while predictive coding describes the process of maintaining and updating this model. SPIKE follows a similar approach of maintaining futures and updating its belief scores.
7 CONCLUSION
We introduced SPIKE, a framework that lets Video-LLMs proactively register surprise. We further showed that SPIKE-RL can refine intermediate belief generation, improving both belief diversity and surprise localization. This enables surprise-driven frame sampling, yielding consistent gains across downstream tasks, especially when critical information is sparse. Modeling surprise offers a path toward understanding video narratives, adapting to violated expectations, and anticipating events. Future work could investigate extending SPIKE to real-time streams and combining with task-specific relevance signals.
REPRODUCIBILITY STATEMENT
We intend to make all our data, code and models open-source, SPIKE based on an open source Video-LLM, Qwen2.5-VL, and our training code for SPIKE-RL will be made available on GitHub. We note that all our prompts are included in Appendix A and hyperparameters in Appendix C For the Mr. Bean evaluation set create, we plan to share the video clips, along with annotations containing their original source. We also share the instructions and template used in our human evaluation in Appendix H.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Dhruv Agarwal, Bodhisattwa Prasad Majumder, Reece Adamson, Megha Chakravorty, Satvika Reddy Gavireddy, Aditya Parashar, Harshit Surana, Bhavana Dalvi Mishra, Andrew Mc- Callum, Ashish Sabharwal, and Peter Clark. Open-ended scientific discovery via bayesian sur- prise, 2025. URL htttp::////axxx
Shehreen Azad, Vibhav Vineet, and Y. S. Rawat. Hierarq: Task-aware hierarchical q-former for enhanced video understanding. 2025 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 4545-8556, 2025. URLhttps://aat http:////ap..smmtt....... ieemanticschan
org/Corpusic: 276919922276992288
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv preprint, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
URL hhttps://arxttt wwwp:ivv.orvaaaooooo abs 2502. 13923
Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum. Action understanding inverse plan- org
www.ntisscccnallaa.....aa ning. Cognition, 113:329-3444 2009. URL 20t.r///oo//////ooooao......oo..
CorpusTD:
64
Chris L. Baker, Julian Jata-Ettinger, Rebecca Saxe, and Joshua B. Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1, 2017. URL hhttps://aaaa htwp://tppp..saa...emeeemmmaa..e
semant www.iccchhcoooooooooccoocooo...o ott.ppp....aaa.....ccoc....ooooe Carp/or 33383320
S. Buch, Arsha Nagrani, Anurag Arna,, and Cordelia Schmid. Flexible frame selection for ef- ficient video reasoning. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 29071-29082, 2025. URL https: /api. semanticscholest
Corpus 28806554222
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-lims. arXiv preprint arXiv:2406.07476, 2024.
Aditya Chinchure, Sahithya Ravi, Raymond Ng, Vered Shwartz, Boyang Li, and Leonid Sigal. Black swan: Abductive and defeasible video reasoning in unpredictable events. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24201-24210, 2025.
Gunhee Kim Dayoon Ko, Sangho Lee. Can language models laugh at youtube short-form videos? In The 2023 Conference on Empirical Methods Natural Language Processing, 2023.
Enea Duka, Anna Kukleva, and Bernt Schiele. Leveraging self-supervised training for unintentional action recognition. In European Conference on Computer Vision Workshop SSLWIN (ECCVW). Springer, 2022.
Dave Epstein, Boyuan Chen, and Carl Vondrick. Oops! predicting unintentional action in video. In The IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims in video analysis. arXiv preprint ar arXiv:2405.2 2024.
Weiyu Guo, Ziyang Chen, Shaoguang Wang, Jianxiang He, Yijie Xu, Jinhui Ye, Ying Sun, and Hui Xiong. Logic-in-franne inrmmie---aes: Dynamic keyframe search via visual remantic-logican verification for long video understanding, 2025. https: www.aawwwww/wwaaaa.aaa///a//
org abs /203 13139
11

---

Published as a conference paper at ICLR 2026
Ritwik Gupta, Rodolfo Corona, Jiaxin Ge, Eric Wang, Dan Klein, Trevor Darrell, and David M. Chan. Enough coin flips can make LLMs act Bayesian. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7634-7655, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 01033//1/222255552 1cc5 acl-long.377, URL httpp::/ppellaaaooooo.rrr/////// 1221-ong.3111
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https:
//openreview.
Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, and Trishul M. Chilimbi. M-llm based video frame selection for efficient video understanding. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13702-13712, 2025. URL https::/ www.sssscatccccool..oooo Btc//Coppus1D:27641]]
Laurent Itti and Pierre Baldi. Bayesian surprise attracts human attention. In Y. Weiss, B. Schölkopf, and J. Platt (eds.). Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005. URL
Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B. Tenenbaum, and Yejin Choi. Hypothesis-driven theory-of-mind reasoning for large language models. In Second Conference on Language Modeling, 2025. URL pettttssss//ppeeeevieee bet//oxum?id=yGGeee
Jorie Koster-Hale and Rebecca Saxe.
Theory of mind: A neural prediction problem. Neuron, 79(5):836 848, 2013. ISSN 0896-6273.
doi: http:///oi.gg//0.0006j.eeumm
2013.08.020. URL https:://// /twp://wwwwc....tt www.::/ceencdiiecccm http://ciiecccccccoccccc/e/ee/ta 01896627313007555
Min Jung Lee, Dayoung Gong, and Minsu Cho. Video summarization with large language models. 2025 IEEE/CVF Conference on Computer Vision and Pattrnn Recognition (CVPR), pp. 18981- 18991, 2025. htthtttp///appp www.steesmceneinsee...e.e
org/ Corpppsd:
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22206 2024.
Hao Liang, Jiapeng Li, Tianyi Bai, Xijie Huang, Linzhuang Sun, Zhengren Wang. Conghui He, Bin Cui, Chong Chen, and Wentao Zhang. Keyvideollm: Towards large-scale video keyframe selection. ArXiv, abs/2407.03100 2024. URL httph::////// /apii. semanticscholle org
CorpussD:27092200
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick. Difei Gao, Alex Jin- peng Wang. Rui Yan, and Mike Zheng Shou. Univtg: Towards unified lideo-language temporal grounding, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
30- -lava-vl.co vi..avvv.ggii... httpoo//////--0000/0/0 10w--20-----------a------------- llava-next////
Yunsheng Ma, Amr Abdelraouf, Rohit Gupta, Ahmadreza Moradipari. Ziran Wang, and Kyungtae Han. Video token sparsification for efficient multimodal Ilms in driving visual question answering. In 2025 IEEE Intelligent Vehicles Symposium (IV), pp. 2235-22222 2242,22, 2025. doi: 10.1 109//6 2025.1 1097438.
12

---

Published as a conference paper at ICLR 2026
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the of foundation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16488-16498, 2024.
Jaspreet Kaur Mann and Navjot Kaur. Key frame extraction from a video using edge change ratio. 2015. httppp//wggg....aaaaaaaaaaaaaaaa
:0:p:/oorpppii::::2222:33 52062935
Beren Millidge, Anil Seth, and Christopher I. Buckley. Predictive coding: a theoretical and experi- mental review, 2022. URL
Taisei Omine, Kenta Akita, and Reiji Tsuruno. Robust laughter segmentation with automatic diverse data synthesis. In Interspeech 2024, pp. 4748-4752, 2024. doi: 10.214337/mierspeechh2222 1644.
OpenAl. GPT-4o system card, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Maleyv and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-28518, PMLR, 2023.
Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpre- tation of some extra classical receptive- field effects. effecs.. Nature Neuroscience, 2:79-87, 1999. URL
https://aph www.smmettttt.. icscho ar www.o//oroooooooooooo ID: 221 208500
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive mul- timodal large language model for long video understanding. 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14313 14323, 2023. URL https:
//api. weemmattcsccoolase oow.sssssaaa..ooooooooooooaaao0.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ro- nan Le Bras, Noah A. Smith, and Yejin Choi. Thinking like a skeptic: Defeasible inference in natural language. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pp. 4661-4675, Online, November 2020. Association for Computational Linguistics. doi: https://acl antho wertchhoogy...o 2020. fi fi www.niiig--ooo emn 1. 418/
URL
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 13960- 13980, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ vl/2023.acl-long.773 URL https: settts::/aacaattolooot
http///aalalll.crr.///ooo// wwtpooooooo..gg///2o0 wttpe//gg/........ acl-long. 780/
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe-
matical reasoning open language models, 2024. URL TtRLhttps://arc org/ abs/2403
03300
Reuben Tan, Ximeng Sun, Ping Hu, Jui hsien Wang, Hanieh Deilamsalehy, Bryan A. Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-Ilm. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13581-13591 2024. URL ltttp://Leeeppooxe ieee. . org/ st ampp st http:://paaaaaanunseee-000e..11.
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sam- pling for long video understanding. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29118-29128, 2025a. URL https://app...... http:///ppp..emmaattcccccooooooe
Exc//oo:pusld:27672555
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. arXiv preprint 272122250 2025b.
Mark A. Thortton, Miriam E. Weaverdyck, and Diana I. Tamir. The social brain automatically predicts others' future mental states. The Journal of Neuroscience, 39:140 - 148, 2018. URL
htttss//eap www.nticschiiieee ar at...rgg/oopssss::55522
13

---

Published as a conference paper at ICLR 2026
Sheena C V and N.K. Narayanan. Key-frame extraction by analysis of histograms of video frames using statistical methods. Procedia Computer Science. 70:36-40, 2015. URL https: //api. api.
semanticscholle ott.:/oooo..a.ooooooooooooaaaaa.
org/
Haibo Wang, Chenghang Lai, Yixuan Sun, and Weifeng Ge. Weakly supervised gaussian con- trastive grounding with large multimodal models for video question answering. Proceedings of the 32nd ACM International Conference on Multimedia, 2024. URL https://api.
Semanticscholaz: ott.://........./..o.....aa.....
Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp- 3272-3283, June 2025.
Wayne H. Wolf. Key frame selection by motion analysis. 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, 2:1228-1231 vol. 2, 1996. URL hatpps::/api..smaaiiissssss
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pailern Recognition (CVPR), pp. 9777-9786, June 2021.
Binzhu Xie, Sicheng Zhang. Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. In European Conference on Computer Vision, pp. 39-57. Springer, 2025.
Sicheng Yu, CHENGKAI JIN, Huanyu Wang, Zhenghao Chen, Sheng Jin, ZHONGRONG ZUO, XU XIAOLEI, Zhenbang Sun, Bingni Zhang. Jiawei Wu, Hao Zhang, and Qianru Sun. Frame- voyager:
Learning to query frames for video large language models. The Thirteenth Interna- lional Conference Learning Representations, 2025. URL /ttttts::////eeeee/ wttpp/eeeeeeee.e.e.eeee/
forum? Forum?id=INN.7. N1.. 2kym
14

---

Published as a conference paper at ICLR 2026
A PROMPTS
A.1 HYPOTHESIS PROMPTS
Generation. We prompt the model with memory of prior events and recent frames, asking for concise next-frame prediction (8-10 words):
Given textual summary of the video far and the most recent prior window framess predict what will most likely happen in the next frame.
Context far: memory text
Prior window (video inputs); A sequence of images corresponding to the last W frames. Output format: Hypothesis: 8-10 words
Prior. We use the following prompt to score each hypothesis.
Context far: memory _ext
Prior window (video inputs): A sequence of images corresponding to the last W frames.
Current frame: The observed frame immediately following the prior window.
Here is what will happen next: [hypothesis statement]
Posterior. We use the following prompt score each hypothesis and compute the probability of yes as the posterior likelihood of that hypothesis.
You are given a textual summary of the video so far, a prior window of frames, and the current frame that follows. Your task is to evaluate whether each hypothesis generated from the prior context still holds in the current frame.
Context far: memory text
Prior window (video inputs): A sequence of images corresponding to the last W frames. Current frame: The observed frame immediately following the prior window.
Hypothesis: [hypothesis statement]
Question: Is this hypothesis true in the current frame? Answer with a single word: yes or no.
A.2 LLM REWARD PROMPT
Rate how closely the content of the prediction matches the content the reference description in terms of meaning and how well it captures important details regarding events in the video. Ignore the difference in length. Score 0.0-1.0 where:
0.0-0.3: Poor match (key details in the reference missing in the prediction) 0.4-0.6: Mod- erate match (a few key details the reference are captured in the prediction) 0.7-0.9: Good match (most key details are present in the prediction) 1.0: Perfect match (all key details in the reference are accurately captured in the prediction) Output only the numerical score (e.g., 0.75).
Reference: gt
Response: response
Score:
A.3 ZERO-SHIOT SCORER PROMPT
You are analyzing video frames for surprisingness. For each frame, assign label of 1 if it is surprising and if not. I: surprising content expected content
Video frames: Original Video Frames

---

Published as a conference paper at ICLR 2026
B HISTORICAL SUMMMARY
In our implementation, the memory of what happened since the beginning of the video i.c the His- torical summary, is maintained as rolling textual summary that updates with each newly observed frame. Before use, the memory is compressed using the BART-Large-CNI summarization model whenever it exceeds approximately 200 word. For each step, the model receives the condensed memory, short window of prior frames, and the most recent observed frame, and generates cap- tion describing the new event. This caption appended to the memory, yielding a continuously updated narrative of "what has happened so far", which is then used for hypothesis generation and surprise computation.
C HYPERPARAMETERS
C.I TRAINING
We train using 4 H100s on a single node with DeepSpeed ZeRO-3 offload. All runs use Qwen2.5- VL-7B- B-Insturuct the backbone, with FlashAttention-22 bfloat16 precision, and PEFT enabled.
Table 3: Key hyperparameters for GRPO training.
Hyperparameter
Value
Learning rate
x 10
GRPO
0.1
Number of GRPO Rollouts
Number of Hypotheses per window
Max prompt length
8192 tokens
Training samples
2000
Epochs
Per-device batch size
Effective global batch size
Random seed
42
C.2 INFERENCE
For both SPIKE and SPIKE-RL. RL, we maintain a hypothesis set N = per time step. We use a prior window of W 4 frames, and the frames for surprise scoring allocated in proportion to the video duration, F = f (duration)). Videos up to a minute are assigned base budget of 8 frames. For longer videos, the budget continues double with each additional minute.
D SURPRISE LOCALIZATION METRICS
Accuracy@a) @ó. Let be the predicted time (in seconds) obtained by converting the frame with the highest surprise score to time, and let t* be the ground-truth transition time. We use the transition time provided Oops! directly. For FunQA and Mr.Bean, center the most surprising window used transition time. The instance-level score is
Accuracy @i - t+| < 6].
and the reported metric is the mean of this indicator over the evaluation videos. Typical choices include å (0.25, 1.0) seconds.
IoU. Let Wpred {a, b] s(t) fort E Ja, b}} be the predicted surprising windows and Wgr be the given set of ground truth surprising windows. The Temporal IoU is:
Temporal loU intersection coverage
V Pred n W
union coverage
U W pred U W
2

---

Published as a conference paper at ICLR 2026
where |-| denotes temporal coverage (total duration). We define predicted surprising windows set of maximal contiguous intervals where the surprise score exceeds a threshold T 0.8 max; s(t) for that video.
E MR. BEAN
We collect 48 videos from Mr. Bean compilation videos on YouTube. Specifically, we follow this process:
1. Each clip is divided into its scenes using a scene detector model, PySceneDetect, using its ContentDetector with threshold of 30.
2. Scenes shorter than 12 seconds and longer than 60 seconds are filtered out, reduce incor- rect scene cuts or have videos that are too short for our analysis.
3. We extract the audio from these scenes, and use a laughter segmentation model from Omine et al. (2024) identify where laughter present. We filter scenes obtain only those that have to 3 laughter segments.
4. Because we rely on laughter tracks as our silver-standard surprise annotation, we transcribe the audio in these clips. We use OpenAl's Whisper (Radford al.) 2023), with the turbo model. If clip has too many words in its transcription (> 8), it is discarded. Through empirical observation, we found that laughter occurs in small peaks. We ensure that at least one such loud peak (> -2dB) of at least second occurs.
5. As a final step, we manually filter through the video set to discard scenes which contain ad- ditional noises (e.g. bells) or scenes that are not semantically meaningful (e.g. the opening credits) that may have passed the other filters. This leaves us with 48 video clips.
The full list of clips, a link to their original source, along with video scenes which we use, will be provided with the code and data release.
F JSD
For bounded and symmetric reporting, we convert KL to the Jensen-Shanmn divergence (JSD), where,
S. -JSD(Ppmt Ppost Pprior) = DKro(PPodl + 1 DKI. (Ppiod|M),
(6)
where M (Ppoat + Pprio),, which maps naturally to [0, 1] after log, normalization.
G HYPERPARAMETER SENSITIVITY ANALYSIS
HYPOTHESIS SET SIZE
We fixed our hypothesis set size to N 3, it worked well for surprise detection across our benchmarks, and the beliefs are restricted to the most likely beliefs given the context the video. To understand the sensitivity to hypothesis set size, we conducted experiments with varying values of on the Oops! and Mr. Bean datasets, reported in Tables and respectively.
Table 4: Effect of hypothesis set size on the Oops! dataset.
Approach # Hypotheses (N)
Acc@0.25s
Ace@ 1s
SPIKE SPIKE
3
60.00
67.30
5
60.36
67.73
SPIKE
60.34
67.69
3

---

Published as a conference paper at ICLR 2026
Table 5: Effect of hypothesis set size N the Mr. Bean dataset.
Approach # Hypotheses (N)
Acc@0.25s
Acc@ Is
SPIKE
3
53.20
70.20
SPIKE SPIKE
5
53.67
70.82
10
53.70
70.86
As shown in Tables and increasing from N to N hypotheses yields marginal improve- ments (0.5-0.6%% on Acc@ 1s), while further increasing to N = 10 shows diminishing returns. This occurs because the probability mass already well captured by the top 5 hypotheses, and adding more hypotheses does not meaningfully change the KL divergence. We therefore adopt N = 3 as the default for an optimal balance between accuracy and computational efficiency, though N can be used for additional gains in performance and depending on the length and complexity of the video.
SENSITIVITY TO WINDOW SIZE W AND HISTORICAL SUMMARY
We conducted ablation study to assess the contribution of the historical summary component.
Table reports results on the Oops! dataset with and without the historical summary.
Table 6: Ablation of the historical summary component the Oops! dataset.
Approach
Acc@0.25g
Ace Is
SPIKE
60.0
67.37
SPIKE w/o Historical Summary
53.7
61.29
The results Table demonstrate that the historical summary is critical for semantic understanding of surprise. To recognize surprising event such as a fall, the model must first establish the narrative context (e.g., a person riding bicycle smoothly). Without the textual summary, the model relies solely on the visual buffer W, which captures motion but lacks the narrative intent that makes KL. divergences semantically meaningful. The historical summary enables the model to detect violations of established narrative expectations, contributing substantially to performance.
We further ablate the window size W, with results reported Table
Table 7: Effect of window size W on the Oops! dataset.
Approach W
Acc@ Acc@0.25s
SPIKE
2
59.55
67.04
SPIKE
60.00
67.37
SPIKE
6
59.97
67.33
SPIKE
59.33
67.32
As shown in Table 7
performance peaks at for the Oops! dataset.
FRAME BUDGET AND LATENCY/THROUGHPPI
Our design ensures that SPIKE is practical for real-world applications. All hypothesis generation and scoring operations run in parallel. Compute time scales linearly with video length (number of timesteps), making performance predictable.
To understand the trade-off between frame budget and surprise localization accuracy, we vary B on the Oops! dataset, with results reported in Table
As shown Table at B = 32 frames (half our default budget), performance drops by approxi- mately 1.1% in Acc ( 0..55 25s compared to B 64. At the extremely low budget of B 8, SPIKE

---

Published as a conference paper at ICLR 2026
Table 8: Effect of frame budget B on surprise localization accuracy the Oops! dataset.
Approach
B
Acc@0.25s
Acc ls
SPIKE
8
55.60
63.76
SPIKE
16
58.44
66.41
SPIKE
32
58.91
66.83
SPIKE
64
60.00
67.37
SPIKE
128
60.25
67.65
achieves 55.6% Acc@0.25s, corresponding to r-4% drop relative to the default. Performance sat- urates beyond B 64, with B 128 frames yielding only marginal gains (+0.25% Acc@ suggesting that the model is approaching the performance ceiling on this benchmark.
H HUMAN EVALUATION TEMPLATE
Fig
A1 and Fig
A2
show the template and instructions used for human evaluation.

---

Published as a conference paper at ICLR 2026
Previous
Frame
Next
Drag hypothesis scale (0-100): top likety (100), bottom Is Impossible (0).
the paddleboard and occupant further the they away.
100:
Optional note (why you placod thers)
Hypothesis
50-70: Defiviely plousitle
paddizboard continue
parson now submerged the water balow
Ukety
Score:
Optional note (why you placed there)
10-30: Unileary
Hypothasis 3:
the person may try right paddleboard or float in the water.
1-10: Temiir Impoeable
Score:
Optional note (why you placed thens)
Hyo 1: the pardkehcord and ts ocoupent will sink further under the water's surfice as they crff away.
Hyp the pacclebzord
aide
water
Figure A1: We ask human evaluators to score the hypotheses by dragging and dropping them into likelihood bands between 0 - 100. This repeated twice - by scoring the hypothesis with and without the observed new frame.

---

Published as a conference paper at ICLR 2026
Task Instructions
Goal: You will be shown the frames of video one after another: For each frame, score each hypothesis based only on what frames have been seen far.
Use the slider or Next/Previous to move between frames.
Only
evidence from
you seen far (from the start up current frame).
For each hypothesis shown at the current frame, assign likelihood score (0-100) by dragging Hyp1,
Hyp2 and Hyp3 boxes into the colored band shown on the left:
0-10-
Terrible/Ippossible:: Contradicted by what you have seen far.
10-30
Unlikely: Little support; seems implausible given the evidence so far.
30-50 Likely: Supported several cues; plausible given the evidence far.
50-70 Definitely plausible to 70-100 Most likely: Strongly supported.
Optional: add short explaining why you chose the
Figure A2: Instructions shown to human evaluators.


Reviews:
[{"review_id": "ndozkwFvXB", "reviewer": "Reviewer_2MSg", "summary": "This paper introduces SPIKE-RL, a frameworks that enables Video-LLMs to identify surprising moments in videos through Bayesian belief tracking. The core approach maintains explicit probability distributions over textual hypotheses about what might happen next in a video, computing surprise as the KL divergence between prior and posterior beliefs when new frames are observed. SPIKE-RL further refines this through reinforcement learning (GRPO) that optimizes hypothesis generation based on video captioning quality. The authors demonstrate that their surprise scores correlate strongly with human judgments on surprise localization benchmarks and show that replacing uniform frame sampling with surprise-weighted sampling—which allocates more frames to high-surprise moments—leads to consistent improvements across five downstream video understanding tasks including question answering and temporal reasoning. The method is query-agnostic, computationally efficient, and provides interpretable results through human-readable belief hypotheses, offering a principled alternative to uniform sampling that better mimics human attention to memorable, expectation-violating events.", "questions": "Why is KL divergence the right measure of surprise for this task? JSD mentioned in appendix but not compared", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and principled approach: Introduces a theoretically grounded Bayesian framework for surprise detection in videos, formalizing belief tracking through explicit probability distributions over human-interpretable textual hypotheses rather than relying on heuristics or purely visual features.\n2. The proposed method achieves near-human performance on surprise localization (62.9% vs 62.1% human accuracy on Oops! Acc@0.25s) and significantly outperforms baselines across multiple benchmarks, with 10-fold improvement over zero-shot Qwen2.5-VL.\n3. Unlike query-conditioned methods, SPIKE works proactively without requiring downstream questions, and provides human-readable belief hypotheses that make the surprise scores interpretable and debuggable.", "weaknesses": "1. The training set is small. SPIKE-RL is trained on only 2,000 videos, which raises concerns about overfitting and generalization. No ablation studies on training set size or analysis of what the model actually learns during RL training.\n2. The zero-shot Qwen2.5-VL baseline appears poorly designed (simple binary scoring prompt). Missing comparisons with recent video understanding methods that use attention mechanisms or saliency detection. Shot boundary detection baselines (RGB Histogram, ECR) seem strawmen as they're not designed for semantic surprise.\n3. The scope of the method is limited. Only tested on short videos were used and evaluation focuses heavily on unintentional failures (Oops!) which may not generalize to other surprise types\n4. Limited discussion of attention mechanisms in video transformers, video saliency detection, or cognitive science literature on prediction error beyond the single Bayesian ToM citation.\n5. On VideoMME-S and NextQA, gains are only +2.7% and +1.7%, suggesting the approach primarily benefits surprise-heavy scenarios, limiting broader applicability claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7n6lrCvgbR", "reviewer": "Reviewer_FCCu", "summary": "The paper proposes **SPIKE**—an inference-time, **Bayesian-surprise** framework for Video-LLMs—and **SPIKE-RL**, a reinforcement-learning refinement of SPIKE’s hypothesis generation. SPIKE tracks **human-interpretable textual beliefs** about upcoming events and defines surprise as the **divergence between prior and posterior belief distributions** when a new frame arrives; this surprise signal then drives **query-agnostic, surprise-weighted frame sampling** that reallocates a fixed frame budget toward informative moments. SPIKE-RL uses **GRPO** with an **LLM-Match** caption reward to improve the diversity and usefulness of belief trajectories. The approach substantially improves **surprise localization** on **Oops!** and **FunQA**, and delivers consistent gains on **five downstream benchmarks** (e.g., BlackSwan, ExFunTube, VideoMME-S, NextQA) by simply replacing uniform sampling in Qwen2.5-VL with surprise-weighted sampling.", "questions": "1. What additional **human studies or inter-rater reliability analyses** would you require to trust **LLM-Match** as a faithful reward for belief optimization (vs. stylistic matching)? \n2. Which **non-humor / non-slapstick** domains and **alternative surprise annotations** (beyond laughter or single-event labels) should be added to convincingly establish general surprise tracking? \n3. What **latency/throughput breakdowns** (per-step hypothesis generation, scoring, sampling) and **ablation of frame budget vs. quality** would you need to see to judge practicality for streaming or long-video settings?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Surprise is formalized as **KL/JSD** between posterior and prior over **textual hypotheses**, yielding a transparent score and human-readable beliefs that explain *why* a segment is surprising.  \n* Surprise-weighted sampling consistently outperforms uniform and classic shot-boundary heuristics across diverse tasks, with additional improvements from SPIKE-RL. \n* SPIKE-RL approaches human [Acc@0.25s](mailto:Acc@0.25s) on **Oops!** and improves **FunQA** IoU, showing robust belief tracking beyond zero-shot baselines. \n* Drops in as a **sampling layer** for existing Video-LLMs, leaving architectures untouched and keeping complexity linear in frame budget.", "weaknesses": "* SPIKE-RL optimizes with an **LLM-based caption reward** (LLM-Match); potential evaluator bias and reward hacking risks are only partially audited. \n* Surprise localization relies on **Oops!**, **FunQA**, and a **Mr. Bean** set with **laughter-based silver labels**; broader domains and richer human annotations would strengthen claims.  \n* Inference adds hypothesis generation and prior/posterior scoring at multiple timesteps", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5rOJisHya", "reviewer": "Reviewer_vBvH", "summary": "This paper introduces SPIKE, an inference-time framework that enables Video-LLMs to detect semantically \"surprising\" events by quantifying Bayesian Surprise. The core mechanism computes surprise as the KL divergence between belief distributions over textual hypotheses before and after observing new visual evidence. The paper further proposes SPIKE-RL, which cleverly uses reinforcement learning (GRPO) with a video captioning reward signal to optimize the hypothesis generation process, by passing the need for direct hypothesis supervision.", "questions": "1.\tHow reliable is the LLM-Match reward function across diverse linguistic styles?\n2.\tDoes the reinforcement signal propagate stably when rewards are normalized within small rollout groups?\n3.\tFor the Qwen2.5-VL baseline in Table 1, the prompt in Appendix A.3 asks for a binary 0/1 label . How was this binary output converted into a continuous surprise score suitable for calculating the IoU and Acc metrics, which require finding a \"peak\" or a \"window\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe use of  Bayesian surprise for frame selection introduces an interpretable probabilistic belief-tracking mechanism distinct from prior query-based or heuristic methods.\n2.\tHuman-readable hypothesis sets enable qualitative analysis of belief shifts.\n3.\tGains on five video understanding tasks show that surprise-aware sampling transfers beyond the surprise domain.\n4.\tThe method is explained very clearly, and the figures (especially Fig 2 and Fig 4) provide excellent visualizations of the architecture and qualitative improvements .", "weaknesses": "1.\tThe model's performance seems highly dependent on the quality and diversity of the belief set. The number of hypotheses is fixed at N=3, which seems very low to capture a complex belief space.\n2.\tThe sensitivity to hyperparameters (W) is not explored. It is unclear how performance changes with different $W$ sizes or if the complex historical summary $H_t$ is truly necessary compared to just using the $W_t$ window.\n3.\tThe entire SPIKE-RL optimization relies on a proxy reward signal from an LLM (LLM-Match, using Olmo-7B-hf). The paper does not provide validation for the reliability or potential biases of this specific LLM judge. If the reward model is flawed, the optimization may lead to a policy (hypothesis generator) that is simply good at \"gaming\" the Olmo-7B judge, rather than generating genuinely better hypotheses.\n4.\tThe RL training procedure appears computationally prohibitive. For each training sample, it requires $M$ full rollouts , each involving $K$ steps of hypothesis generation/scoring, plus a final caption generation, plus a call to the reward model LLM. This high training cost (multi-model, multi-step) may limit the method's scalability, reproducibility, and adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S35gsYrDsP", "reviewer": "Reviewer_PKB1", "summary": "The paper presents a novel and well-motivated framework that integrates Bayesian surprise into video-language models, enabling belief-driven reasoning and adaptive frame selection. The proposed SPIKE and SPIKE-RL frameworks not only introduce interpretable belief tracking but also deliver consistent improvements across surprise localization and general video understanding benchmarks. The idea of quantifying model surprise to guide frame sampling is elegant and cognitively inspired, bridging the gap between passive perception and proactive understanding in Video-LLMs.", "questions": "Have you consider the other modality included? (e.g.: Audio).\n\nAs you mentioned that the Oops is the negetive side, it seems related the anomaly detection? Have you tried the video anomaly understanding as the downstream application?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of connecting Bayesian surprise to video understanding is elegant and intuitive. Whle the proposed method is simple to plug in and does not require retraining large models. As authors report, results are strong and consistent across both surprise-related and general reasoning tasks.\n\n2. The paper is clearly written and easy to follow; the belief visualizations make the concept concrete.\n\n3. The reinforcement learning extension (SPIKE-RL) adds genuine value and improves diversity and accuracy.", "weaknesses": "The computational cost of generating multiple belief trajectories isn’t fully discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)