Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding
Authors: Weiyu Guo
Ziyang Chen
Shaoguang Wang
Jianxiang He
Yijie Xu
Jinhui Ye
Ying Sun
Hui Xiong
Venue: NeurIPS 2025 poster ()
Keywords: vision-language models
key frame search
long video understanding
Primary area: applications

Abstract: Understanding long video content is a complex endeavor that often relies on densely sampled frame captions or end-to-end feature selectors, yet these techniques commonly overlook the logical relationships between textual queries and visual elements. In practice, computational constraints necessitate coarse frame subsampling, a challenge analogous to “finding a needle in a haystack.” To address this issue, we introduce a semantics-driven search framework that reformulates keyframe selection under the paradigm of Visual Semantic-Logical Search (VSLS). Specifically, we systematically define four fundamental logical dependencies: 1) spatial co-occurrence, 2) temporal proximity, 3) attribute dependency, and 4) causal order. These relations dynamically update frame sampling distributions through an iterative refinement process, enabling context-aware identification of semantically critical frames tailored to specific query requirements. Our method establishes new state-of-the-art performance on the manually annotated benchmark in keyframe selection metrics. Furthermore, when applied to downstream video question-answering tasks, the proposed approach demonstrates the best performance gains over existing methods on LongVideoBench and Video-MME, validating its effectiveness in bridging the logical gap between textual queries and visual-temporal reasoning. The code will be publicly available.

Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding
Weiyu Guo
Ziyang Chen
Shaoguang Wang' Jianxiang He' Ying Sun Hui Xiong 1.2
Yijie Xu² Jinhui Ye
Thrust of Artificial Intelligence, HKUST (Guangzhou), China DDpartment Computer Science and Engineering, HKUST, Hong Kong SAR, China cn; cn; yingsthkust- -z. edu. en; zionghui@ust,
vguo3395cconeett. www.:aeeeettttttteeeeeeee wwust-gz. edu.
Abstract
Understanding long video content is a complex endeavor that often relies on densely sampled frame captions or end-to-end feature selectors, yet these tech- niques commonly overlook the logical relationships between textual queries and visual elements. In practice, computational constraints necessitate coarse frame subsampling, challenge analogous to "finding needle in a haystack." To address this issue, we introduce a semanties-dieg driven search framework that reformulates keyframe selection under the paradigm of Visual Semantic- Loggiiaiii Search. Specifically, we systematically define four fundamental logical dependencies: 1) spatial co-occurrence, 2) temporal proximity, 3) attribute dependency, and 4) causal order. These relations dynamically update frame sampling distributions through an iterative refinement process, enabling context-aware identification of semantically critical frames tailored to specific query requirements. Our method establishes new SOTA performance on the manually annotated benchmark in key-frame selec- tion metrics. Furthermore, when applied to downstream video question-ant -anteionnnnn tasks, the proposed approach demonstrates the best performance gains existing methods on LONG ViDDOBeEN ENCH and VIDEO-MME validating its effectiveness in bridging the logical gap between textual queries and visual- tempooll reasoning. The code is available at httpps: //github. com www gww.gguwowuuueo Loggiiic in- Frames
1 Introduction
Iision-Language Models (VLMs) Yin al. (2024) have achieved remarkable progress in video understanding Zou et al. (2024); Tang et al. (2023), particularly in video question answering Wang et al. (002d)) Zhang et al. (023). demonstrating potential for modeling real- world scenarios. However, existing methods can only simultaneously process a limited number of frames due to the inherent token limit and extremely high dimension of spatio-temporal video data, especially for long videos. Furthermore, uniformly sampled keyframes are query- query-agnostic and insufficient to represent query-related contents. To tackle these challenges, this paper addresses pivotal research question:
How can we efficiently and accurately select keyframes that are semantically critical for answering video-based queries?
We hypothesize that deconstructing visual semantic and logical cues (e.g., target objects, logical relations including temporal, spatial, attribute, and causal relationships between visual entities) from textual queries enables effective identification of task-relevan frames through heuristic sampling and search. Building on this insight, we propose Visual Semantic -ooanticLLogical Search (VSLS), novel keyframe search framework that incorporates target object confidence estimation and joint verification of visual semantic logic into the iterative update of frame sampling distribution and
"ooresponning authors. 'Equal Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Temporal
(Bext, time, pen)
Attribute
(man, attribute, white shirt)
pen the use wrote time? B3 Pink
Qn niger map on there weoring
"ggatori"
doing? B) speoking
A) Brown
A) gozing et cicuit boand
Histery of
The Hastory
Curfar
Cultar
Spatial
Egyption Pyramide)
(man, baskedball)
Q:At arimation, building dirplane fly over?
Q Af man wearing red short short-sleeved block farishedd speaking front black background, what this man do?
A) The Eiffel Tower.
B) The Egyption Pyranids
A) picked up mobile phone.
B) picked up basketball.
RED
Baseline Answer GREEN- -Or Arswer
Figure 1: Examples of four types of visual semantic-logical relationships in video QA detected by VSLS framework: Temporal (text, time, pen), Attribute (man, attribute, white shirt), Spatial (copilot, spatial, Egyptian Pyramids), and Causal (man, causal, basketball). Green boxes indicate correct answers, while red hoxes show baseline errors.
selects the most informative frames with the highest confidence. Experimental results show that our approach requires only sparse sampling (1.4% of frames per video on average) to identify critical frames, significantly reducing computational complexity compared to conventional dense sampling strategies while maintaining performance on downstream video understanding tasks.
Compared to conventional methods, VSLS shows three distinct advantages.
First, the framework is training- free and highly efficient in comparison with dense captioning Chen et al. (2024c); Kim et al.
(2024); Wang et al. (2024c) video clustering Wang al. (2024T); Rajan and Parameswaran (2025) strategies, sampling only 1.4% of frames on average in LV HAYSTACK. Second, it explicitly models logical binary relations (namely spatial, temporal, attribute and causal) in the query beyond simple target detection Ye et al. (2025b), utilizing additional visual semantic feature and enhancing logical consistency throughout the reasoning process. Third, VSLS a plug-and-play module, which can be seamlessly integrated into existing VLM pipelines without cross-component dependencies.
We further examine VSLS several public datasets, including LoNG V VIDEOBENCH Ye al. 2025a
a comprehensive benchmark for long video understanding: LIDEO-MMED Fu et al. (2024). widely adopted multimodal video question answering dataset; and HAYSTACK BAACCKLLLBENCC Ye al. (225a) with meticulously annotated keyframes based on human feedback for more precise analysis. Extensive experiments demonstrate significant improvements in both the semantic similarity and temporal coverage between the retrieved keyframes and the ground truth labels, as well as the accuracy in downstream video question answering tasks. More importantly. with only 1.4% of video frames (EGo4D Grauman et al. (2022)) sampled in the search iteration, our method achieve improvement in GPT-40 Hurst et al. (2024)'s 's long video QA accuracy. This performance gain attributed to our simple yet powerful observation: query-guided visual semantic logic retrieval can mitigate the gap between potential visual logic video frames and the logic expressed in the query. To be specific, constructing ternary logic triplets with visual elements (e.g., objectl, logic type, object2) can enhance downstream reasoning capabilities when performing textual-visual retrieval.
8.7%
To the best of our knowledge, we are arguably the first to search for keyframes in long videos by detecting visual semantic logic, with potential extensions to other textual-visue retrieval tasks. Our main contributions are follows:
We define four fundamental types of semantic logic relations in video QA tasks, including temporal, causal, attribute, and spatial relations, which can be accurately detected across various datasets.
We sample only 1.4% of frames on average of frames average during keyframe search through heuristic sampling and distribution updating by different visual semantics and logical relations.
We comprehensively evaluate retrieval efficiency, semantic similarity, temporal coverage, and video question answering accuracy across several widely used video understanding datasets, demonstrating significant improvements in downstream tasks.

---

Keyframe Bearching
Video
Sampler
Object Detection and Scoring
Selected Frames
Spare
Great
Distribution
Update
Contiore
map the willl theee is man souring verie
LLMVLM
Clss Objject
Object
Time
Attribue
VLM ACTEPNE
"Speakiing
Sputial
Calasal
Query
Query
Visual Semantic Logic
QA
Our VSLS Framework for Efficient Keyframe Selection. VSLS sparsely samples frames and selects key ones via object detection and logic verification: 1) Use &LM& VL. to extract cue/target objects and four logic types (spatial, temporal, attribute, causal); 2) Adaptive sampling with evolving confidence; 3) Detect objects via YOLO-WORLD WORLD; 4) Fuse scores with spline to identify frames for downstream tasks.
Figure 2:
2 Method
Although existing long-context VLM frameworks implement keyframe search for video QA tasks Liang et al. (2024); Park et al. 2024); Tan al. (2024); Wang et al. e024ble); Yu al. (2024). their computational efficiency and search accuracy remain suboptimal. To address this needle-in maddle-i--cc aeeee-nna-haaaa in-a-haystack challenge Wang al. (025 ; Zhao al. (2024), we propose a novel method VSLS that aligns the semantic relations between the text modality and video modality, improving the
plausibility of logical reasoning and the performance of downstream tasks.
2.1 Task Formulation
Given a video sequence V = ( with N₂ frames and a query Q, the ideal temporal search framework aims to retrieve the minimal keyframe subset y = {fmm 1K S with keyframes that satisfies the following:
Conservation: The keyframe subset V C V must satisfy the answer consistency condition:
A(V* Q) = A(V, Q), where A(-) denotes the video QA function.
Compactness: /K must minimal subset that preserves completeness, which means that no frame in V can be removed without hindering the accuracy and efficiency of video QA.
2.2 Visual Semantic Logic Extraction
Starting from question Q and uniformly sampled frames V from video V, our goal is to extract key visual elements to answer Q. We first classify the detected objects in Q and VN into two categories:
Key Objects: The main participants or references in the scene that the question explicitly or implicitly focuses on (e.g., "person", "microphone").
Cue Objects: Secondary or contextual entities that help locate or disambiguate the Key Objects (e.g., "book", "tiger painting")).
To further leverage semantic and logical links among these objects, we define a set of relations R O x A x O, where each relation r (o,, oy) R, with o,, 0j E O denoting detected objects in the key and cue objects dataset and E A representing one of the following types of relations:
Spatial Co-occurrence
Attribute Dependency
o and 0j appear in the same frame, indicating 0 and oj share visual properties. e.g., color or co-occurrence or proximity. Example: "A person standing beside vase." Example: "A person wears black shirt." (person, attribute, black shirt)
size.
(person, spatial, vase)
Temporal Proximity
Causal Order
0, and 0j occur in close frames, linking se- 0 and o, follow cause-effect or prerequisite quences or transitions.
order.
Example: "After dog entered the room, cat Example: "A little girl broke the vase." entered." = (dog, temporal, cat)
(little girl, causal, pieces)
3

---

Algorithm 1: Visual Semantic-Logical Search
Function Bemeaniiiilgiiialaaaaaaaaaaaaaa Q, T,,,))
O, <- ParseQuestion(()
If Extract keylcue objects and relations
P +- Uniform, B +- [V]. S + N, + |
If Initiaiize distribution and state
while B > 00 and [ > 0 do
A + LvB], G + Grid(Sample(()
If Adaptive grid sampling
S 4 (etect(bjecta(C)
R Detect objects in sampled frumes
foreach / E do
Ct t CalcuclateBaseScore(()
If Base detection confidence
foreach rtype E R do
8 - Processrelation(Tnpp: Ω, T,,
relations require distinct processing
C + +
(pdateeScoresSS.4.CC
#f Update global score registry
DiffuseScores(S, w) P' +
If Temporal context propagation
B < B - 2
I Update sampling distribution
foreach g TSp K) do
if [ n O * 0 then [2()]
I Remove identified key objects
return KS)
Return top-K keyframes
The choice of these four relations draws on core concepts in linguistics and logic Cohen (198)); (Sowa (2000); Talmy (2000), which identify spatial, temporal, attributive, and causal aspects fundamental for structuring, perceiving, and communicating information about events and states. For more details on this selection, please see appendix A for reference. As shown in Figure [ we construct semantic- logical relations that support a broad range of queestion-answerigg involving temporal queries (when does X happen?"), causal reasoning (why did Y occur?"), attribute dependence (What is the person wearing sunglasses doing?"), or spatial constraints (Who is standing next to the red car?") can be answered more reliably by incorporating these structured relations and contextual cues.
tasks. Specifically, questions
2.3 Iterative Semantic-Logica Temporal Search
Based on the extracted key and cue objects and their logic relations, our algorithm iteratively searches for keyframes through semantic and logical reasoning. including four main stages: Frame Sampling (Sec. 2.3.1) Object Detection and Scoring (.3.2). Visual Semantic Logic Detection (Sec. 2.3.3) and Distribution Update (Sec. (Sec.2...44) The pseudocode shown in Algorithm and Algorithm provides more detailed version.
2.3.1 Frame Sampling
Given video with N₂ frames, employ probability sampling strategy instead of exhaustively scanning all frames to improve searching efficiency. Let P denote uniform distribution over all frames, then the sampling process is defined as:
I Sa,
where Sample(:::.)) is a function that draws N₂ indices from the population No, based on the probability distribution P. To further leverage the detection capability of YOLO, we stack the sampled frames on a k x k grid, which requires that the sample size N₂ be square number. The benefits of such practice are analyzed in detail in Appendix C.I Although P is initially uniform, can be adapted over multiple rounds of sampling to focus on frames of greater interest in the video.
2.3.2 Object Detection and Scoring
In this stage, construct the detection search space by taking the union of both key objects and cue objects. For each iteration, we detect objects in the N, sampled frames using a lightweight model like YOLO-WORLLS W.ooohhnhhhheng et al. (2244) for high efficiency and score the frames based detection confidence. Specifically, let 22 be the set of detected objects in the frame at time I, c, the confidence of each detected object, and w, the corresponding weight. We define the frame score as:
C max ( wo) oEN,

---

m MaMao
20
Feratinn Faralana
Oherre Trep
250
Figure 3: Sample weight evolution under VSLS optimization for keyframe selection. Top: 16 iterations show progressive convergence toward Ground Truth (red). Bottom: 15 iterations demonstrate similar alignment. Yellow highlights indicate precise matches between algorithm outputs (green) and manual annotations.
If the confidence score of any key object exceeds predefined threshold, we will add it to list to maintain record of frames where crucial targets have been identified for subsequent processing.
2.3.3 Visual Semantic Logic Detection
Beyond individual object detection and frame-level scoring, we refine each frame's confidence score by modeling higher-level object relations. Let R be the set of relations, where each R involves pair (01, 02) and is labeled by a type Ttype- Denote C₁ as the confidence score at time t, with a global scaling factor a and relation-specific weight "rage The confidence refined cP considering the relation r' is defined as:
c. C. + a Trpp"
Spatial Relation. A spatial relation enforces that two objects and 02 must co-occur in the same frame. Let S4 be the set of detected objects frame t. both 0r E Qr and 02 E 2r then the corresponding frame confidence is updated by:
C C. Tispatill
Attribute Relation. An attribute relation is satisfied when the bounding box of 0₁ and 02 share significant overlap in the same frame. Let overlap be the ratio of their intersection area to the minimum of their individual areas of bounding box. If the overlap ratio exceeds a predefined threshold 7 (r 0.5 in our experimental setting), we increase the frame confidence by:
C + C. + a Tattribure
(5)
Time Relation. A time relation checks whether two objects appear in temporally close frames. Suppose 4, and 1j (4, S (2)) are sampled such that [1 - 4] < A., where A, a threshold (e.g. frames in our experimental setting), if o1 occurs in frame 1; and in frame Lj. then the confidences of both frames are updated by:
C2 + Ct. + a Viime C2 - Ct.y + a Time-
Causal Relation. A causal relation models ordering constraint, enforcing that 0₃ must appear earlier than 02. Specifically, if 01 E Ωr and 02 E St with f; < tj, we update the confidences the frames t and t by:
C₁ Cr. + Teausal. G + Yeausal.
Through this scoring mechanism, frames with detected relations will have greater confidence and are more likely to retrieved as keyframes for the given query and video. We have also performed hyperparameter search experiments and find that a 0.3 (from 0.3, 0.5, 0.7, 1.0) and Trype = 0.5 achieve the best results across different datasets.
2.3.4 Distribution Update
After each iteration of frame sampling, we merge the newly obtained frame confidences into the global score distribution {Sy} spanning all frames f 1, N... When frame f is selected for detection, its score is set as the confidence value C, and marked as visited. To incorporate temporal context, we diffuse this updated score to neighboring frames within a window of size w. Denoting each nearby index by f + 8 (for& å E [-w, w)), we apply the following:
S
Sy48 f max S 1+ 181
(8)
5

---

In this way, high-confident frames will increase the scores of close-by frames and ensure temporal continuity. Following these local updates, the sampling probability distribution P refined using spline interpolation and then normalized. This iteration continues until either the search budget B reached or all key objects have been successfully identified. The visualization of the probability distribution in different iterations can be seen in Figure 3 Finally, the method outputs the top K frames according to their integrated scores.
3 Experiment
3.1 Benchmark Datasets
The proposed VSLS is systematically evaluated across four benchmark datasets: a) LONGVIDEOBENCII Ye al. (2025a) for assessing long-context video-language comprehension capa- bilities; b) VIDEO-MMET analytics; c) HAYSTACK LV BENCH, extended from
et al. (2024) as the first comprehensive benchmark for multimodal video with numan-annotarn frame index answers; and d) HAYSTACK -ooCcKDDDDD,..- derived from EG04D with similar annotations. While LONGVIDEOBENCE and VIDEO-MME measure performance enhancement in QA accuracy, HAYSTACK-EGO4) and HAYSTACK-LV
quantitatively evaluate keyframe selection accuracy through recall and precision metrics. Further details of datasets are provided in Appendix
3.2 Evaluation Metrics
3.2.1
Evaluation Metrics for Search Utility
Our assessment framework emphasizes both effectiveness and efficiency. For search effectiveness, we use three metrics to compare model-predicted keyframes with human annotations, considering both individual frames and full sets addressing the possibility of multiple valid keyframe sets per query. For frame-level comparison, we evaluate the alignment between a predicted frame fpr and a numan-annotann frame fgi from two perspectives:
Temporal coverage evaluates the coverage of ground truth frames by predicted frames in the temporal perspective, which be described as:
Nal 1l
min
ty
f/p
6
T Toover (Tp:: Tgt)
N
where Tp and Tgr denote the of predicted and ground truth timestamps, respectively. Here, [/ggil is the number of ground truth frames, f and tur are the i-th ground truth and j-th predicted timestamps, respectively. 5 is the temporal similarity threshold defining the maximum allowed time deviation, and I[-] is the indicator function, returning
the condition holds and 0 otherwise.
Visual Similarity is measured by the Structural Similarity Index (SSIM) Brunet al. (2012). capturing structural detail, luminance, and contrast between fn and fg. For set-to-set comparison, the key challenge is defining inter-set similarity. We adopt Precision P and Recall R complementary metrics: Precision checks whether each predicted frame matches any reference frame, while Recall ensures that all reference frames are represented. Given the ground truth set Fgt = f'gt", j = 1 and the predicted set Fpi f'ppt' i = 1, we define the multimodal retrieval quality metrics as follows:
P(Fms Fe))
1
Σ
max Of PH), HEF
(10a)
PP
FEF
1
R(Fpu Fe.)
Σ
&f HER
fp),
(10b)
Fg
REEF
where ((:,.) represents extensible multimodal similarity metric function. 3.2.2 Evaluation Metrics for Search efficiency
Existing studies Fan et al. (2024); Park et al. (2024); Wang et al. 2024b e): Wu and Xic (2023) have mainly concentrated optimizing task-specific performance metrics while neglecting computational efficiency in temporal search operations. To systematically analyze this dimension, our evaluation framework incorporates two criteria: 1) FLOPs representing arithmetic operation complexity, and 2) Latency recording real-world execution duration.

---

Mithod
Training Roquired
Monching
Searching Efficiency
lisation
TRLOPs
Latency (see)
Ovcrall Efficinnyy Lanency (kee)
Statie Frame Sampling
UNIOOIMM Ye 20125
Training Based
NIA
NA
NA
0.2
3.8
53.7
Danse Reurival
VIDEOAGENT Fan
2024
Training Based
Radiocd 2021
536.5
49.2
To RETRIEVAL Tectal
2023b
Troining Based
WOLD TIM
840
216.1
28.6
32.2
51.3
Temporal Search
Te-AFTENTION et alaal. 20.5
Training lased
NIA
N/A
88.9
13.7
17.3
59.3
Te-DETECTOR
Training-Frow
O-WORLD- OOLO-WORL--TONN
VSLB (OURS)-DETXC
Trsininn-Frre
11.6
Table 1: Evaluation of performance metries across the LV-HAYSTACK benchmark, presenting both search efficiency and end-to-end processing overhead (combining search and inference stages).
3.3 Evaluation of Search Framework efficiency
Current approaches for keyframe selection can be broadly categorized into three paradigms: statistic- based frame sampling, dense feature retrieval-based selection, and temporal search-based methods. As shown in Table while uniform sampling achieves the fastest processing speed, its ignorance of frame semantics severely limits downstream task effectiveness. Although dense feature retrieval methods attain moderate accuracy improvements (57.3%), their exhaustive frame processing demands 4.2x more TFLOPs and introduces 4.5x higher latency than temporal search approach. Crucially, our method introduces four visual semantic logic detectors during temporal search while maintaining comparable execution time T* methods. This strategic design elevates downstream task accuracy to 61.5%, achieving the best performance-efficiene. trade-off.
3.4 Visual Semantic Logic Search Performance
As demonstrated in Table we evaluate VSLS on from two critical perspec- tives: visual similarity (measured by precision and recall) and temporal coverage. Our method achieves stateoofthh--ad performance across all metrics. Specifically, under the 32-frame setting, VSLS attains precision of 74.5% and recall of 92.5%, outperforming
Table 2:
Search utility results on LONgVIDEoBENCh. Best scores in the 8- frame setting underlined. and in the framme setting are bold. Gray indicates results original paper.
LONgVIDEOBE
Method
Frame
Precision
LONG VIDeOBENC
Recall
baselines in visual sim- ilarity. More notably, the temporal coverage of VSLS reaches 41.4%, surpassing the second- best method (T* at 36.5%) by 13.4%- the largest mar- gin among all comparisons. This significant im- provement highlights the effectiveness of our vi- sual semantic logic detection modules in identify- ing query-relevane keyframes with both semantic alignment and temporal completeness.
Static Frame Sampling Method
UNIFORM Ye
2025%
56.0
72.0
63
UNIFORM
60.7
80.4
4.7
UNIFDRM
12
58.7
81.6
24.9
60
85.0
8.1
Dense Rerrieval Method
Fan
2014
10.1
55.8
75.2
85
RETRIEVAL
63.1
RETRIIVAL--ASS
32
$9.9
803
2188
Temporal
Method
Yo 2025b
58.4
72.7
7.1
T4
75.3
88.2
26.2
75.6
32
58.3
83.2
28.2
These results empirically support our core hypoth- esis: leveraging semantic and logical cues from text queries enables precise detection of relevant video frames. Improvements in visual similarity and temporal coverage confirm that VSLS effectively captures keyframes while preserving temporal coherence through visual-logical alignment.
32
74.0
90.3
36.5
3.5 Downstream Video QA Performance
To demonstrate the advantages of VSLS, evaluate downstream video QA performance BONGOEBEO and VIDEO-MME. As shown in Table 3) videos are grouped by length into Short, Medium, and Long (15-3600s, up to 60 mins). VSLS consistently achieves the highest accuracy in the long-video category across different frame counts and QA models. Compared to the baseline T*, incorporating our visual semantic logic relations (Figure yields substantial gains. These results confirm that modeling visual-logical relations is key to effective QA on long videos.
4 Analysis
4.1
Coverage Analysis of Semanttc--ogge Relations
To ascertain the practical applicability and coverage of our defined semantic-logical relations (spatial, temporal, attribute, and causal), conducted an analysis of their detection across all queries in the
7

---

LoNG YIEEBEENC
VIDEO- MIE
Video Length Medium
Video Length Mudium
Model Size
Frume
Short
Model Size
Frume
Long
GPT
67.3
GPT-40
GPT-40
56.3
68.0
GPT--4
68.9
GPT-4a + VELS (ou3)
51.2
58.9
74
GPT-4o a VELB (ous)
56.9
66077
68.2
INTERNVL 2.5 78B Chen dl.
202ld
55.7
57.3
74.0
INTERS VL 73B
52,6
58.5
55.9
INTERNVE 3 5-788 - VELS
58.0
61.5
74,0
ENTERSVLL 2:5-78B VSL (Mm))
57.7
57.5
59.0
LLAYA-ViDRO-
200th
42.0
46.3
500
LLAVA-ViRRE Nar-o-ieoo--- 7B-QwEx2+1
39.6
48.0
LLAVA. VIDEO OwEN2
M.AVA-VIDR--7.
50.0
QWENZ.5 VL-7B INsTBUCT (Wang
62.0
QWEN22
INSTRICT
55.4
QWINZ VVL-28--NSSTUOO C1
41.0
422
54.0
12wEN2 -
00.3
50.0
$6.9
QwaN2 5. DI-SINUUU (our))
45.8
49.2
54.0
OwEN2. 1...L-77---NSTIICCT INSTRUCT VBLB (ouzs)
43.3
49.5
60.8
GPT-40
32
53.8
56.5
74.0
GPT-4o
32
55.2
61.0
71.4
CIPT-40+77 +
12
A5.1
58.
71.0
GPT--0-
87
55.2
61.5
22.6
GPT 40 VBL3 (ours)
32
54.2
60.0
76.0
GPT do VEL8 (ours)
32
57.5
61.9
74.5
LLAVA--Vinna-8-- 7B-OWN22
32
42.3
45.
54.0
LLLVA-VVDDO--BB
32
35.9
36.5
37.4
60.2
44.
500
L. ALLAVAVVVON MAA--VV00--1---0000 (B-QWEN))
35.8
39.6
11.8
LLAWA Viveo OWENZ (ouss)
41.7
48.1
54.0
LLAVA VIDEO 7B QWENZ VSLB
QwTN2. 5-VI-78-INSTRUCT
50.0
5-VEL-7B-DNSTRUU
QwaN2.5 VL- 1
QwEN? 3-- INSIKULL
QwiNt VL--7---SSTRRO cT+ Steum)
36.
42.3
54.0
QweN2 5-V VBLS (ouns)
37.9
35.8
BLLVA-UNNVVSSINN Qwwe2--188--0
32
59.3
638
77.
LLaVV--neViionn-00
00.0
62.2
66.3
PLLAVA-344 34B
32
49.1
50.8
66.5
VIDROLLAMM
32
57.6
59.9
62.4
128
59.5
63.9
77.4
DRYX-115
128
98.3
63.3
67.9
MPLUG-OWWL3-7
125
53.9
58.8
73.7
ARIA-8X3.58
256
58.8
67.0
67.6
GPT-40 (0513))
61.5
667
76.8
GRMINI-1 1-5PPR (0615)
160.5
Table 3: Downstream task evaluation results on two benchmarks. All accuracy scores (%) black are from our replication. We also cite reported SQTA accuracy in gray (noting that their settings may differ and results may not be reproducible),, along with the number of frames used for QA inference, for full transparency.
Long VideoBench and VideoMME datasets. Our findings reveal a crucial insight: for every question posed within these extensive VQA benchmarks, our query analysis module successfully identified and mapped the query to at least one of the four defined logical relation types. This empirical result supports the completeness of our proposed relation set for interpreting the semantic and logical intent inherent in these VQA tasks.
4.2 Time Complexity
The proposed framework consists of two stages. First, VLMs such as LLAVA-7B and GPT-40 extract a semantic set S from video V with n frames. S includes target objects, cue objects, and their relations, with their size constrained by prompt design. the second stage, keyframe iden- tification is performed via a heuristic search: k candidates are iteratively selected using a scoring function S). The score distribution scores[n] [n] dynamically refined using outputs from the YOLO- WORLD detector.
124
Aecrp Www.BBunrcee Dookk
4.12
Ki
Speiill
Ceaei
Our analysis focuses on YOLO-WORD detec- tions, the main computational bottleneck due to their reliance on deep neural networks. Redue- ing the number of detections improves efficiency without sacrificing accuracy. At each iteration, the detector processes & selected frames to match ob- jects and relations S. yielding ie detections. The search stops when all targets are found or the iteration budget minn1000. 0.1 x V.) (with V; as the video duration in seconds) is exhausted. In the worst case (e.g.. videos with > 10,000 frames and no matches), the cap is 1,000 iterations. Ideally, the evaluation function S) assigns high confidence to target frames, making the algorithm resemble top-k selection over n candidates in ((S) log n) iterations Ye et al. (2025b), resulting in an average of O(S)) logn) n) YOLO- FOLO-WORLD inferences.
Figure 4: Average occurrences of detected semantic- logical relation types per question on VideoMME and Long VideoBench. Spatial relations most frequent, while queries both datasets trigger at least one of the four relation types.
Experimental results also demonstrate that integrating relational information into the search algo- rithm incurs negligible computational overhead compared to the baseline T* approach. On the LV- LV-HAYSTACK benchmark, the average iteration count increases from 42.94 (T*) to 48. iterations, representing a modest 13.69% rise in the time cost.
8

---

4.3 Ablation Study of Four Relations
VENGEBEBBNN
Logic Type
Method
Figure illustrates the distribution of four logic relation types across LONGVIDEOBENCII and VIDEO-MME datasets, where spatial relations predomi- nate, followed by attribute relations. In Ta- ble 4 we extract samples containing differ- ent relation types from LONG BIDEOBENCH to compare the object detection-basee T* method with our VSLS approach. Ex- perimental results demonstrate that VSLS achieves significant improvements across both image similarity metrics (SSIM Pre- cision and SSIM Recall). Additionally, temporal coverage shows marked enhance- ment for altribute, spatial, and causal relations, with spatial relations exhibiting the most substantial improvement (21.3% increase over T*). For the time relation category, we observe a slight decrease in temporal coverage, which may be attributed to the relative scarcity of time relation samples in the dataset, limiting the opportunity to demonstrate the advantages of VSLS. Nevertheless, Figure [] provides visual evidence of how effectively leveraging time relations facilitate downstream drestion-answeringg tasks.
Precision t Recall
TC
T*
72.9
88.7
37.5
Spatial
VSLS (ours)
73.6
91.4
45.5
T+
71.8
87.6
38.5
Attrihute
VSLS (ours)
72.7
90.9
42.1
T+
76.7
89.2
37.3
Time
VSLS (ours)
77.5
92.5
36.1
T+
74.7
92.4
38.6
Casual
VSLS (ours)
74.7
93.8
39.6
Table 4: Comparison of method (VSLS) with the base- line across four logic relation types on LONGVIDEoBENCH. Precision: SSIM Precision; Recall: SSIM Recall; TC: Tem- poral Coverage.
Related Work
Challenges in Long Video Understanding: Long video understanding is inherently more chal- lenging than short-video or image-basee based tasks due to its rich temporal dynamics and massive redun- dancy Qian et al. (2024) Zeng et al. (2024): Yu et al. (2019). The large number of frames increases both memory and computational requirements, making straightforward dense sampling infeasible. Moreover, crucial events may span distant timestamps, demanding high-capacity models to capture long-range dependencies Ranasinghe et al. (2025): ); Shi et al. (2024); [Chen et al. (2024b); WWnn et al. (2024). Meanwhile, the diverse and continuous visual content raises noise and distractors; thus, strategies to effectively locate or distill essential parts of the video are of primary importance /Zhang al. (023): Cheng et al. (2024b); Xu et al. (2023); Ye et al. (2025b).
Existing Solutions based on VLMs typically share three core ideas: 1) video sampling or retrieval for efficiency, 2) multi-stage interactive reasoning to handle complex questions, and 3) compact representation to accommodate the VLM's limited context window. For instance, retrieval- -based pipelines partition video into segments and employ learned rule-based retriever to identify the relevant chunks before passing them to a VLM [Pan et al. al.] (202)); (022)))(Chouuurr et al. (2023 2025). Other lines of research compress each frame into minimal tokens reduce computational overhead ei al. (2024): Chen et al. (2024a); Song et al. (2024), or adopt a streaming mechanism to propagate memory
representations along the temporal axis Qian al. (2024); : Wu al. (2022) Liu al. (2024). Beyond these
approaches, LLM/VLM WWV----a--planee frameworks factorize the process into series of perception queries, enabling an agent fetch additional frame-level details needed Wang
et al. (2024c); Zhang et al. (2024a): Liao et al. (2024).
6 Conclusion
In this paper, we present Visual Semantic-Logical Search (VSLS), novel framework that efficiently selects semantically keyframes for long video understanding by decomposing logical relationships between textual queries and visual elements. VSLS based on four defined logical dependencies (spatial co-occurrence, temporal proximity, attribute dependency, and causal order). significantly outperforms existing methods while sampling only 1.4% of video frames. The 8.7% improvement in GPT-40's 's long video QA accuracy demonstrates that query -urded visual semantic logic search effectively bridges the gap between textual queries and visual content. VSLS's plug-and- play nature enables seamless integration with existing pipelines, making it practical for real-worid applications. Future work could consider more logical relations, learnable search methods, enhancing interpretability, and exploring more downstream tasks.

---

7 Acknowledgment
This work was supported in part by the National Key R & D Program of China (Grant No.
in part by the National Natural Science Foundation of China (Grant No. No.92370,04), part by the Guangdong Basic and Applied Basic Research Foundation (Grant 1555581151100005 in part by the Education Bureau of Guangzhou. Yijie Xu acknowledge the support from the modern matter laboratory, HKUST(GZZ.
References
Dominique Brunet, Edward R. Vrscay, and Zhou Wang. On the mathematical properties of the structural similarity index. IEEE Transactions on Image Processing, 2012.
Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yame Yang Wang. Daniel Khashabi, and Alan Yuille. Llavolta: Efficient multi-modal models via stage-wise visual context compression. In arXiv preprint arXiv:2406.20000 2024a.
Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Yu-Chiang Frank Wang. ReXTime: A benchmark suite for reasoning across-time in videos. In The Tiiht Conference on Neural Information Processing Systems Dutaxets and Benchmurks Track, 2024b.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang. Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. NeurIPS, 37:19472-19999 19495, 2024c.
Zhe Chen, Jiamnan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23185-24199 2024d.
Tianheng Cheng, Lin Song. Yixiao Ge. Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. CVPR, 2024a.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in iime 2024b.
Rohan Choudhury, Koichiro Niiuuma, Kris M Kitani, and László A Jeni. Zero-shot video question answering with procedural programs. arXiv preprint arXiv:2312.00937, 2023.
Rohan Choudhury, Koichiro Ninuma, Kris M. Kitani, and László A. Jeni. Video question answering with procedural programs. In ECCV, 2025.
David Cohen. Universals in linguistic theory, 1968.
Yue Fan, Xiaojian Ma. Rujie Wu, Yuntao Du Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. ArXiv, abs/2403. 1481,, 2024.
Charles Fillmore. The case for case. Bach and Harms (Ed.): Universals in Linguistic Theory, 1967.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark multi-mode Ilms video analysis. ArXiv, abs/2405.21075. 2024.
Kristen Grauman, Andrew Westbury, Fugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours egncentric video. In Proceedings of the IEEF/CVF conference on computer vision and patter recognition, pages 18995-19012, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Minkuk Kim, Hycon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Scong Tae Kim. Do you remember? dense video captioning with eross- memory retrieval. In CVPR, 2024.
Yanwei Li. Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024.
10

---

Jianxin Liang, Xiaojun Meng, Yueqian Wang, Chang Liu, Qun Liu, and Dongyan Zhao. End-to-end video question answering with frame scoring mechanisms and adaptive sampling. ArXiv, abs/2407.15047, 2024.
Ruotong Liao. Max Erler. Huiyu Wang. Guangyao Zhai, Gengyuan Zhang. Yunpu Ma, and Volker Tresp. Videoinsta: Zero-shot long video understanding via informative spatial-tempont reasoning with Ilms. In EMNLP Findings, 2024.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, dlava-plus: Learning to use tools for creating multimodal agents. In European Conference on Computer Vision, pages 126 142. Springer. 2024.
William C Mann and Sandra A Thompson. Rhetorical structure theory: Toward a functional theory of text organization. fex--interisciplinaary Journal for the Snudy of Discourse, 8(3):243-281) 281, 1988.
Leland Gerson Neuberg. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19(4)6675-685, 2003.
Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, and Hongsheng Li. Retrieving- to-answer: Zero-shot video question answering with frozen large language models. In ICCV Workshops, 2023.
Jong Sung Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryoo, Donghyun Kim, and Michael Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa. ArXiv, abs/2406.00936. 2024.
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang 7ang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. In NewrlPS, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
Manjusha Rajan and Latha Parameswaran. Key frame extraction algorithm for surveillance videos using an evolutionary approach. Scientific Reports, 15(1):536, 2025.
Kanchana Ranasinghe, Xiang Li. Kumara Kahatapitiya, and Michael S Ryoo. Understanding long videos with multimodal language models. In ICLR, 2025.
Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie. Unlocking vidco-llm via th-ttughss distillation. arXiv preprint arXtv:2412.01669 2024.
Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi-modal llms. arXiv preprint arXiv:2409. 10994, 2024.
John F. Sowa. Knowledge Representation: Logical, Philosophical, and Computational Foundations. Brooks/Cole Publishing Co., Pacific Grove, CA, USA, 2000.
Leonard Talmy. Toward a Cognitive Semantics (Volume 1: Concept Structuring Systems; Volume 2: Typology and Process in Concept Structuring). MIT Press, Cambridge, MA, USA, 2000.
Reuben Tan, Ximeng Sun, Ping Hu, Jui hsien Wang, Hanich Deilamsalchy, Bryan A. Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long vidco-llm. CVPR, 2024.
Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, al. Video understanding with large language models: A survey. arXiv preprint arXiv:2312. 1732,, 2023.
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang. Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long-context capability multimodal large language models, 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan. Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang. Mengfei Du Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 2024,
Xiaohan Wang, Yuhui Zhang, Oir Zohar, and Serena Yeung-Levy, Videoagent: Long-form video understanding with large language model agent. In ECCV, 2024b.

---

Xiaohan Wang. Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model agent. In ECCV, pages 58-76. Springer, 2024c.
Zhanyu Wang, Longyue Wang. Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A unified multimodal large language model for Instruction- followed understanding and safety-awan generation. ACM MM, pages 3907-3916, 2024d.
Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jachong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree- video representation for IIm reasoning on long videos. ArXiv, abs/2405. 19209, 2024e.
Ziyang Wang, Shoubin Yu, Elias Stengel-Eske Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. arXiv preprint arXiv:2405. 19209, 2024f.
Yuetian Weng. Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Vision, pages 450- Springer, 2024.
Chao Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13587-13597, 2022.
augmented multiscale vision transformer for efficient long-term term video
Penghao Wu and Saining Xic. V*: Guided visual search as core mechanism in multimodal Iims. CVPR, 2023.
Jiaqi Xu. Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Setrieval-basee video language model for efficient long video question answering. arXiv preprint arXiv:2312.04931, 2023.
Jinhui Yc. Zihan Wang, and Haosen Sun. Longvidcohaystack. https::/bugg:nntaccc.o/// cttp:/aaaccccc/aaatttt///o
LWhhysstcck/loo""Vddooaaaayaaaa 2025a. v1.0.
Jinhui Ye, Zihan Wang. Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguiree, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding. In CVPR, 2025b.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun. Tong Xu, and Enhong Chen. survey on multimodal large language models. National Science Review, 2024.
Sicheng Yu, Chengkai Jin, Huan Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xioalei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, and Qianru Sun. Frame-voyagen Learning to query frames for vidco large language models. ArXiv, abs/2410.03226, 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu. Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Xiangyu Zeng. Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang. Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang. Yali Wang, Yu Qiao, and Limin Wang. Timesuite: Improving milms for long video understanding via grounded tuning. 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction -undd audio-visual language model for video understanding. In EMNLP, 2023.
Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma. and Kyusong Lee. OmAgent: A multi-modal agent framework for complex video understanding with task andiie-anddcoqquee In EMNLP, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024b.
Zijia Zhao, Haoyu Lu. Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in video baystack: A scalable synthetic evaluator for video milmss. arXiv preprint arXiv:2406.00 093667 2024.
Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv. Guangcong Wang, Junyang Chen, Zhuochen Wang. Hansheng Zhang, Huaijian Zhang, et al. From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding. arXiv preprint 2409 18938, 2024.
12

---

Partl I
Appendix
Table of Contents
A
Theoretical Underpinnings of Relation Categories
14
A.1
Linguistic Grounding
14
A.2
Logical Grounding.
14
A.3
Pragmatic Completeness for VQA
A.4
Accuracy of Extracting Logical Relations
Semantic Logics Specialized Applications
B
Performance
16
Hyperparameter Setting
16
Stacking Multiple Frames
16
Contidence Threshold
D) Analysis of the Impact of Search Frame Count
17
E Details of Datasets
17
E.1 Details of VIDEO-MMP
17
Details of LONG IDEOBENNG
18
E.3
Details HVYSSTAK
18
Details EGO-4D
F
Detailed Algorithm
19
F.1
Algorithm Overview and Core Components
19
F2
Implementation Considerations
20
F3
Computational Complexity Analysis
21
F.4
Technical Implementation Detail
21
F5
Practical Application Examples
22
System Specifications for Reproductivity
G
Case Study of VSLS Keyframe Selection
23
H
Itcration Analysis
24
Prompt
25
Prompt Template for Query Grounding
25
Prompt Template Question Answering
25
Limitations
26
K Broader Impacts
26
K. I Positive Impacts
26
Potential Considerations
26
13

---

A Theoretical Underpinnings of Relation Categories
Our choice of the four relation categories- -spatall temporal, attribute, and causal- is grounded in foundational concepts from linguistics and logic. While achieving absolute "completeness" in describing the infinite complexity of the real world is formidable challenge, this selection aims to describe core aspects of events, states, and the way humans conceptualize and communicate them.
A.1 Linguistic Grounding
Semantic Roles and Case Grammar: Theories like Fillmore's Case Grammar Fillmore (1967)
analyze sentences in terms of semantic roles that nominals play in relation to the verb (the event).
Spatial relations directly correspond roles like Locative (the location of an event state)
Parh (the trajectory of motion).
Temporal relations align with Temporal roles, specifying when an event occurs or its duration.
Attributes describe the properties entities (participants) involved in these roles. While direct
case roles for verbs, they are fundamental for identifying and characterizing the "who" and "what" (e.g., Agent, Patient, Theme, Instrument) that possess these attributes during an event.
Causal relations are central understanding agency and event structure. Roles like Agent (the instigator of an action) or Cause (the non-volitional trigger of an event) highlight the importance of causality in linguistic descriptions of events.
Works in lexical semanties (e.g. following Pustejovsky Cohen (1968) on the generative lexicon, or Talmy Talmy (2000) on cogmitive semantics) often decom- pose event meaning into fundamental components. Talmy Talmy (2000), for instance, extensively discusses how language structures concepts like space, time, and force dynamics (which inherently relate causality). Events are situated in space and time, involve entities with specific attributes, and are often linked through causal chains (e.g., one action causing another, or an agent causing a change of state).
Lexical Semanties and Event Structure:
Discourse Relations: Theories like Rhetorical Structure Theory (RST) Mann and Thompson (1988) identify relations that bind textual units together. Many of these fundamental relations are inherently temporal (e.g., Sequence), causal (e.g., Cause, Result, Purpose), or involve describing entities and their settings (which encompasses spatial and attributive information, often under relations like Elaboration or Background). This suggests that these four categories capture essential elements for constructing coherent descriptions and explanations, a core function of Video Question Answering (VQA).
A.2 Logical Grounding
Predicate Logic and Knowledge Representation: In formal logic and AI knowledge representation (e.g., Sowa Sowa (2000)), events and states are often represented using predicates with arguments that specify participants, locations, times, and properties. typical event representation might implicitly or explicitly include Location (event, place), Time (event, time interval), HasProperty (entity, attribute_ _aauee,, and relations like Causes (event1, event2). Our four categories provide high-level abstraction over these common predicate types.
Modal and Specialized Logics: * Temporal Logic is specifically designed to reason about propo-
sitions qualified in terms of time.
Spatial Logic deals with reasoning about spatial properties and relations between entities.
Logics of Action and Causality (e.g., situation calculus, event calculus, Pearl's work causality Neuberg (2003) explicitly model how actions bring about changes and the causal dependenctes between events.
A.3 Pragmatic Completeness for VQA
From pragmatic standpoint, particularly for VQA, these four relations address the core "Wh- questions" humans often ask understand a scene or event:
What/Who? (Identifies objects/entities, often distinguished by their attributes)
Where? (Answered by spatial relations)
When? (Answered by temporal relations)
14

---

Why/How did it happen? (Often answered by causal relations or a sequence of events linked temporally and spatially)
While more fine-gre aine-graiedd relations (as in Action Genome) undoubtedly provide deeper semantic detail, our chosen set aims to provide foundational, yet computationally manageable, framework for keyframe selection based on the most common semantic and logical inferences required for a broad range of video queries. They represent a level of abstraction that is both meaningful for human queries and feasible for current visual-language models to parse and verify.
In essence, these categories are not arbitrary but reflect fundamental dimensions along which events and states are structured, perceived, and communicated in language and reasoned about in logic. We believe they offer robust and broadly applicable framework for the task at hand.
A.4 Accuracy of Extracting the Logical Relations
Evaluating the LLM's ability accurately extract logical relations is crucial point for validating our framework's 's reliability. To quantitatively address this, we conducted a verification study. We randomly sampled 500 query instances from each question category across the Video-MME and Long VideoBench datasets. We then used LLMs to perform the logical relation and object extraction as described in our paper. Subsequently, ensure the quality of the results, we performed rigorous manual audit of all extracted logical relations and objects. Our analysis yielded the following high accuracy rates for the extraction task:
GPT-4o: 92%
Qwen- VI. 72B: 88%
These results demonstrate that state-of-the-an LLMs are highly proficient at this task, confirming that the logic extraction ability does not serve as performance bottleneck for our framework at this stage. Due to committee guidelines, we are unable to provide a direct link to this data in the rebuttal. However, we are committed to transparency and will release the full set of our manually audited data and LLM outputs as part of our public code release.
A.5 Semantic Logics in Specialized Applications
The four logics we presented-spatee temporal, attribute, and causal- are intended as a foundational set designed to cover a broad range of general queries. We see VSLS not as a system with a fixed set of logics, but an extensible framework that can be adapted to various domains.
To derive new logics for different scenarios, we propose the following systematic, three-step approach:
The first step is collaborate with domain experts to identify the critical relationships and events they analyze. For medical application like surgical video analysis, this would involve identifying key instrument-tissu interactions (e.g., 'cutting', 'suturing', 'retracting') which are far more specific than our general 'causal' or 'spatial' relations. This process translates expert knowledge into set of target logical relations.
Domain Knowledge Elicitation:
Operationalization into Verifiable Rules: The conceptual relationship must then be translated into a computable, verifiable rule that the VSLS search can execute. This involves defining the specific visual and temporal evidence required. For example, a new logic like suturing could be operationalized as:
(i) Detecting a needle- holder instrument and suture material.
(ii) Observing the instrument in periodic contact with tissue edges.
(iii) Verifying that the tissue edges become approximated in subsequent frames.
Modular Integration: Finally, this new, operationalized logic can be integrated as new module into the VSLS framework. Our design allows new logic-checking functions to be added alongside the existing four. The query parser would be extended recognize domain-specifer keywords (e.g.. "suture") and trigger the corresponding verification function during the iterative search process.
This process ensures that the VSLS framework can effectively adapted to specialized fields, making it more general and powerful. We believe this discussion significantly strengthens our paper's contribution. We will add section covering the framework's 's extensibility and these guidelines for deriving new logics to the final version of the paper. Thank you again for this valuable suggestion.
15

---

B Performance
Long-form video understanding presents unique challenges due to the complexity of temporal dy- namics and cross-modal interactions in extended durations (900-3,600 seconds), Our comprchensive evaluation of the LVB-XL benchmark reveals significant performance gaps between existing ap- proaches. While large-scale models like GPT-40 (32 frames) and INTERNVL 2.5-78B (16 frames) have demonstrated competence in short-video tasks, their direct application to long-form content (marked by circle sizes proportional to model parameters) yields suboptimal results (53.8% and 56.5% accuracy respectively).
Search (VSLS) framework addresses these limitations. This ad- vancement enables consistent performance improvements across different architecture scales, clevat- ing GPT-40 to 54.2% (+0.4pp) and achieving remarkable 62.4% (+5.9pp) for INTERNVL 2.5-78B on this benchmark. The comparative analysis further suggests that VSLS's gains become particularly pronounced when processing longer visual sequences, highlighting its effectiveness in modeling extended temporal contexts.
Our Visual Semantic-Logia
C Hyperparameter Setting
While several parameters in our framework are manually set, they are designed to be interpretable and correspond clear, logical trade-offs, allowing users configure the system based their specific needs for efficiency versus accuracy.
C.1 Stacking Multiple Frames
Our rationale for this design is that it offers deliberate trade-off between search efficiency and downstream task accuracy. The grid size is configurable parameter allowing users to balance these two competing factors based on their needs. The table below shows a supplementary experiment we conducted. It presents the required steps complete the keyframe search and the performance the downstream QA task of GPT4o. As demonstrated, increasing the grid size substantially decreases the search cost (from 770 steps down to few 48 steps). However, it simultaneously leads moderate reduction in performance (from 56.7 down to 52.7). Notably, increasing the number of images in grids leads to a decline detector accuracy, which in turn increases the overall search cost.
Table 5: Effect of the number of images in grids on search cost and QA performance.
Num. of Images in Grids
Search Cost (steps)
QA Performance
1
770
56.7
4
160
55.5
8
48
53.5
12
730
53.2
16
860
52.7
Therefore, this stacking approach presents clear flexibility to choose optimal balance based on their specific requirements resource constraints. Users prioritizing detection accuracy may prefer smaller grid sizes or individual frames, while those with limited computational resources stringent runtime constraints may opt for larger grid configurations to achieve greater efficiency.
trade-off, offering users the
C.2
Confidence Threshold
For example, two key parameters are the confidence threshold and the number of images in grids. The confidence threshold allows user to balance search cost against performance; shown in our analysis, increasing the threshold from 0.3 to 0.8 improves performance from 50.2 to 56.4, while the required search cost increases from 18 to 162 steps. Similarly, the number of images in grids parameter controls the granularity of the search. A smaller grid size (e.g., 1) yields the highest performance (56.7) at high search cost (770 steps), whereas a medium grid size (c.g., 8) minimizes
16

---

Table 6: Effect of confidence threshold on search cost and performance.
Confidence Threshold
Search Cost (steps)
Performance
0.3
18
50.2
0.5
42
54.1
0.7
49
55.3
0.8
162
56.4
the search cost to just 48 steps, offering more efficient search. This demonstrates that the parameters offer predictable control over the search behavior rather than being arbitrary settings.
D Analysis of the Impact of Search Frame Count
Baseline Without Search
Orade: Selection
Frome Searcting
60
50
Performance
40
20
10
Number Frames
Figure 5: Performance improvement with increasing search frames. VSLS consistently enhances accuracy and reaches near-human oracle performance 64 frames.
This section investigates the impact of the number of search frames on the performance of our Visual Language Models (VLMs) in the context LoNg BENCH
Figure in the T* framework study empirically demonstrates the non-monotonic relationship between input frame quantity and model accuracy on the LONG IDEoBENCh XL benehmark. Through systematic experimentation across 18 state-of-the-art VLMs, this visualization reveals critical phenomenon: excessive frame inputs degrade performance for models lacking temporal redundancy mitigation mechanisms.
E
Details of Datasets
E.1
Details of VIDEO-MME
The VIDEO-MME (Video Multi-Modal Evaluation) dataset represents the first comprehensive bench- mark tailored assess the capabilities of Vision-Language Models (VLMs) in video understanding. Aiming to address limitations in existing benchmarks, it emphasizes diversity, temporal complexity. and multi-moda. integration while ensuring high-quality human annotations. The dataset contains 900 carefully curated videos across six primary domains Knowledge, Film and Television, Sports Competition, Artistic Performance, Life Record, and Multilingual- with 30 fine-grained subcate- gories such as astronomy, esports, and documentaries. These videos vary significantly in duration, ranging from short clips (1 seconds) to long-form content (up to hour), enabling robust evaluation across temporal scales.
domains--Knoweddee
17

---

Each video is paired with expert-annotated multiple-choice questions (2,700 QA pairs in total), rigorously validated to ensure clarity and reliance on visual or multi-modal context. Questions span 12 task types, including action recognition, temporal reasoning. and domain-specific knowledge, with a focus on scenarios where answers cannot inferred from text alone. To quantify temporal complexity, the dataset introduces certificate length analysis, revealing that answering questions often requires understanding extended video segments (c.g., median lengths 26 seconds for short videos and 890.7 seconds for long videos), surpassing the demands of prior benchmarks like EGOSCHEMA.
VIDEO-MME serves as a universal benchmark, applicable to both image- and video-focused MLLMs. and exposes key challenges for future research. These include improving architectures for long- sequence processing, developing datasets for complex temporal reasoning, and enhancing cross- modal alignment. By providing a rigorous evaluation framework, VIDEO- MMM aims to drive progress toward MI.LMs capable of understanding dynamic, real-world scenarios.
E.2 Details of LONG VIDEOBENCH
The LONG BINEOBENCH benchmark pioneers the evaluation of long-context interleaved video- language understanding in VLMs, addressing critical gaps existing benchmarks through its focus on detailed retrieval and temporal reasoning over hour-long multimodal inputs. Designed to overcome the "single- frame bias" prevalent in prior video benchmarks, the novel referring reasoning paradigm enables models to locate and analyze specific contexts within extended sequences. The data set comprises 3,763 web-sourced videos that span various themes movies, news, life vlogs, and knowledge domains (including art, history, and STEM) with durations progressively grouped into four levels: 8-15 seconds, 15-60 seconds, 3-10 minutes, and 15-60 minutes. Each video is paired with aligned subtitles, forming interleaved multimodal inputs that mimic real-world viewing scenarios.
The benchmark features 6,678 human-annotated multiple-choice questions categorized into 17 fine- grained task types across two levels: Perception (requiring object/attribute recognition in single scenes) and Relation (demanding temporal/causal reasoning across multiple scenes). Questions incorporate explicit referring queries (c.g.. "When the woman descends the rocky hill...") that anchor reasoning to specific video moments, with an average question length of 43.5 words to ensure precision. Temporal complexity is quantified through duration duration-grouped analysis, where models must process up to 256 frames (at 1 fps) for hour-long videos, significantly exceeding the demands of predecessors like EGOSCHEMA (180s videos).
E.3 Details of LV-HAYSTACK
The LV-HAYSTACK benchmark establishes the first comprehensive evaluation framework for temporal search in long-form video understanding, addressing critical limitations in existing synthetic needle- in-haystack benchmarks through real-world video annotations and multi-dimensional evaluation metrics. Designed assess models' ability to locate minimal keyframe sets (typically 1-5 frames) from hour-long videos containing tens of thousands of frames, the dataset comprises 3,874 human- annotated instances spanning 150 hours of video content across two distinct categories: egocentric videos from EGO4D (101 hours) and allocentric videos from LONG VIDEOB ENCH (57.7 hours).
subsets, the benchmark features videos averaging 24.8 minutes in length (max 60 minutes) with 44,717 frames per video. Each instance contains:
Organized into HAYSTACK- BAYSDD and HAYSTACK-LVBEIR
Expert-curated multi-choice questions requiring temporal reasoning (15.9 questions/video); fluman-annotater keyframe sets (4.7 frames/question for egocentric, 1.8 frames/question for allocentric);
Temporal and visual similarity metrics for precise search evaluation.
E.4 Details of EGO-4D
The EGO4D (Egocentric Computer Vision Benchmark) dataset establishes a transformative founda- tion for advancing research in first-person visual perception through unprecedented seale, diversity, and multi-modal integration. Designed to overcome limitations in existing egocentric datasets, it captures 3,670 hours of unseripted daily activities from 931 participants across 74 global locations and 9 countries, spanning houschold, workplace, leisure, and outdoor scenarios. The dataset features
18

---

30+ fine -ee-ggaindd activity categories including carpentry, social gaming, and meal preparation, with videos ranging from brief interactions (8 (8-minute clips) to extended continuous recordings (up to 10 hours), enabling comprehensive analysis of long-term behavioral patterns.
Each video is enriched with multi-modal annotations totaling 3.85 million dense textual narrations (13.2 sentences/minute), coupled with 3D environment meshes, eye gaze tracking, stereo vision, and synchronized multi-camera views. Rigorous privacy protocols ensure ethical data collection, with 612 hours containing unblurred faces/audio for social interaction studies, The benchmark suite introduces five core tasks organized across temporal dimensions:
Episodic Memory: Temporal localization of natural language queries (74K instances) and 3D object tracking using Matterport scans;
Hand-Object Interaction: State change detection (1.3M annotations) with PNR (point-of-no- return) temporal localization;
Social Understanding: Audio-visual diarisation (2,535h audio) and gaze-directed communication
analysis:
Action Forecasting:
Anticipation of locomotion trajectories and object interactions.
Quantitative analysis reveals the dataset's complexity: hand-object interactions involve 1,772 unique verbs and 4,336 nouns, while social scenarios contain 6.8 participant interactions per minute on average. Multi-modal fusion experiments demonstrate performance gains, with 3D environment context improving object localization accuracy by 18.7% compared to RGB-only baselines. State- of-the-art models achieve 68.9% accuracy in action anticipation tasks, yet struggle with long-term forecasting (41.2% accuracy for 5s predictions), highlighting critical challenges in temporal reasoning.
EGO4D's unique integration of egocentric video with complementary modalities (IMU data 836h, gaze tracking in 45h) enables novel research directions in embodied AI and augmented reality. The dataset exposes fundamental limitations in current architectures, particularly processing hour-long video contexts and synthesizing cross-modal audio-visual synchronization cues. By providing standardized evaluation protocols and curated challenge subsets, EGO4D serves as a universal testbed for developing perceptive systems capable of understanding persistent 3D environments and complex human behaviors.
23% of tested models effectively utilized
F
Detailed Algorithm
The detailed VSLS algorithm is represented in Algorithm 2
F.
Algorithm Overview and Core Components
The algorithm operates (represented as set V) to locate frames matching semantic-logical query requirements (Q). Unlike traditional linear search methods, it employs probabilistic sampling strategy that dynamically adjusts based on confidence scores from multiple relationship types.
adaptive search framework that intelligently explores video content
Initialization Phase
The process begins by parsing the input query Q into two fundamental
components:
O: A set of key objects or entities to identify
RR: collection of relationships (spatial, temporal, causal, and attribute) that must satisfied
The algorithm initializes with uniform probability distribution (P) across all video frames, estab- lishing a budget (B) equivalent to the total number of frames (VV), and creating an empty score registry (S) to track confidence values. This approach ensures unbiased initial exploration before evidence-guided refinement.
Adaptive Sampling Strategy Rather than exhaustively processing every frame, the algorithm employs a square-root scaling sampling strategy where k = [vB] determines the sampling density. This provides a mathematical balance between exploration breadth and computational efficiency. The Grid function organizes sampled frames into a structured representation that preserves spatial- temporal relationships, facilitating subsequent relationship analysis.
19

---

Algorithm 2: The completed Visual Semantic-Logical Search
Function aamanni
Q, 7,,,,))
2, R +- ParseQuestion(())
// Extract key/cue objects and relationships
P t Uniform, B + [V + 0, N. + [V|:
// Initialize distribution and state
while B > 0 and O > 0 do
k - LvB G + (rrid(Samlle(PPk3))
// Adaptive grid sampl ing // Detect objects in sampled frames
Ω + DetectObjecta(());
foreach 9 G do
Cg (al.uulltteaaseSore(((())))
Base detection confidence
foreach r do
if r..ype = Spatial then
C9 - Cg + 0' whw.heaaaacaalllllaaaaaaaaaapppp
else r.tye Temporal then
Cg &- C. + O'jimme Cht.xceepprralllaaaaaiaaaaaapapa A,)
if r.ype Causal then
Cg + Cg + O'cawall
else r. r.type = Autribute then
C. + Cg + O')amr
UpdateSCcoress.gg..C..
// Update global score regiatry // Temporal context propagation
DiffuseScores(S, w)
< formmalliiDDststbbuttooomo//// foreach g E TopK(S. S, K) do if [ n O of then
k²
// Update sampling distribution
+- [[g]:
// Remove identified key objects
return (S)
// Return top-K keyframes
Multi-modal Object Detection The DetectObjects function applies state-of-theearr computer vision techniques to identify objects within each sampled frame. This step leverages deep neural networks pre-trained on diverse visual datasets, enabling recognition of a wide range of entities with their corresponding confidence scores and spatial locations within frames.
Score Propagation and Distribution Update The DiffuseScores function implements temporal context propagation mechanism that spreads confidence values to neighboring frames, acknowledging that relevant content likely extends beyond individual frames. This diffusion creates a smoothed confidence landscape that guides subsequent sampling.
After each iteration, the algorithm normalizes the accumulated scores form an updated probability distribution, focusing future sampling on promising regions while maintaining exploration potential in unexamined areas.
Convergence Criteria and Termination The search continues until either:
The sampling budget (B) is exhausted, indicating comprehensive coverage of the video content
All target objects (O) have been successfully identified at satisfactory confidence levels
This dual-termination approach balances thoroughness with efficiency, preventing unnecessary computation once objectives are met.
The algorithm concludes by returning the top-K frames with the highest confidence scores, representing the most relevant video segments that satisfy the semantic-logical query requirements. These keyframes provide concise summary of the content matching the user's information needs.
Result Generation
F.2
Implementation Considerations
The algorithm's performance depends several configurable parameters:
A, Temporal window size for relationship analysis
Confidence threshold for attribute matching
20

---

a: Global relationship influence factor 7: Type-specific relationship weights
These parameters can be tuned based on application requirements, video characteristics, and com- putational constraints. The algorithm's modular design allows for straightforward substitution of specific component implementations (e.g.. different object detectors or relationship checkers) without altering the overall framework.
F.3
Computational Complexity Analysis
The time complexity scales with O(VN) where N is the total number of frames, significantly improving upon linear approaches. Space complexity remains O(N) to maintain the probability distribution and score registry. The algorithm intelligently balances exploration and exploitation through its adaptive sampling approach, making particularly suitable for large-scale video analysis tasks where exhaustive processing would be prohibitive.
F.4 Technical Implementation Details
Object Detection and Feature Extraction To achieve real-time performance, the object detection module utilizes pre-trained deep convolutional neural network architectures, particularly variants based on FAST R-CNN and YOLO series. The system employs a two-stage detection strategy:
Using lightweight models to rapidly identify potential regions: Fine-grained Classification: Applying more sophisticated models for detailed classification high-confidence regions.
Preliminary Detection:
The feature extraction process leverages self-attention mechanisms from Visual Transformers (ViT), generating rich semantic embeddings robust various visual variations such scale, rotation, and illumination. Each identified object is associated with a feature vector f. E RR, where 512 represents the dimensionality of the embedding space.
Mathematical Formulations for Relationship Assessment The evaluation of various relationship types is based on precise mathematical definitions:
Spatial Relationships Given bounding boxes B, = (Ix, Vs, Wt, h2) and B, = (x Uy. Wy, hy) for two objects, the confidence for spatial relationship T'spatall calculated as:
Csparial (Bi, Bj.r) (B, B,) v (B.) (BB)),
(11)
where Φ, is a relationshippspecificc compatibility function and the object detection confidence. For example, the compatibility for a "contains" relationship is defined as:
Doontains (B,, B,) = IoU(B,, Bj) Area(B,)
(12)
Temporal Relationships Temporal relationships are calculated by evaluating object behavior patterns across a sequence frames F.t...,,..,,
A,-1 Ciemporal (O,, = II T ofth1),
(13)
k=0
where T, is relationship-specifie temporal transition matrix and O represents the state of object i time f.
Causal Relationships Causal relationships utilize Bayesian network framework to compute
conditional probabilities:
21

---

P(E Ed)
Ceansal (Bi, P(E)(E)) log P(F.)
(14)
where E, and E, represent the presumed cause event and effect event, respectively.
Attribute Relationships Attribute evaluation employs cosine similarity metrics between feature vectors and attribute prototypes:
Caur (O a) max(0, cos([s, -))
(15)
where is the prototype vector for attribute and is the minimum similarity threshold.
Score Propagation Algorithm Temporal score propagation is implemented through weighted diffusion process, analogous to heat diffusion on graph structure:
5)( = S(1)
WKEt S(k), kEN())
Σ
(16)
where
(t)) (((e) represents the temporal neighborhood of frame I, and WK,t is weight based temporal distance, defined as:
(- 1k 1k-10 1|2
Wk,t exp
(17)
20
where a controls the diffusion range.
The sampling strategy is further improved through dynamically adjusted Thompson sampling method, modeling the probability distribution as Beta distribution with shape parameters updated through previous observations:
Adaptive Sampling Optimization
P(() Beta(ou, + Σ S.(1..B+ +n Σ S(O)).
(18)
where and 3r are prior hyperparameters and is the total number of observations.
F5 Practical Application Examples
In practical visual search scenarios, the algorithm processes complex queries such as "a person wearing blue shirt sits down table and then picks up coffee cup":
Query parsing identifies key objects (person, shirt, table, coffee cup) and relationships (blue attribute, sitting action, temporal before- relation, spatial proximity);
Adaptive sampling selects representative frames from the video;
Multi-relationship evaluation integrates various sources of evidence;
Score propagation establishes unified confidence landscape across related frame sets;
Result generation provides concise summary of the most relevant segments in the video.
This semantic-logical loepoollellllaorallll search framework represents a significant advancement in multimodal content retrieval, enabling natural language queries that incorporate complex relationships across objects, time, and causal chains.
F.6 System Specifications for Reproductivity
Our experiments were conducted on high-perfonee performance servers, each equipped with either an Intel(R) Xeon(R) Platinum 8378A CPU 3. 3.00GHz or an Intel(R) Xeon(R) Platinum 8358P CPU 2.60GHz, 1TB of RAM, and 4/6 NVIDIA A800 GPUs with 80GB memory. Machines with 4 GPUs are configured with the SXM4 version, while those with GPUs use the PCIe version. The software environment included Python 3.11, PyTorch 2.4, and NCCL 2.21.5 for reproductivity.
22

---

G Case Study of VSLS Keyframe Selection
Quastion:: In the video, what color did outhor when he Vguitar\ for second time?
Baseline Arswer: "Brown"
Our Answer: "Pink"
Searched Poslion Indicas Uniform Sampling
100
(secoro))
Figure 6: Qualitative comparison of frame selection strategies demonstrates VSLS's ability to pinpoint query-critical moments (e.g., the subject presenting pink objects) with temporal precision, while baseline approaches exhibit color misinterpretation (brown) due to suboptimal frame choices. VSLS maintains superior temporal diversity and content relevance, effectively avoiding the redundant selections observed in comparative methods.
As shown in Figure 6 the VSLS framework demonstrates its effectiveness through video question- answering case study involving temporal handwriting analysis. The experiment focuses on dis- tinguishing between two sequential events: brown pen writing "guitar" at seconds and a pink pen rewriting the same word at 3 seconds, with the query requiring identification of the second occurrence's 's pen color.
VSLS's analytical process unfolds through three interpretable phases:
Identifies core visual entities (handwritten text, pen. paper) and constructs temporal relationships through triplet formulation: (lext, time, pen), establishing the framework for tracking writing instrument changes;
Semantic Logic Extraction:
Temporal Relevance Scoring: The gray relevance curve reveals precise temporal localization. with peak scores aligning perfectly with ground truth positions at 2s and 3s, contrasting sharply with baseline methods' random fluctuations;
Search Pattern Visualization: Demonstrates VSLS's focused inspection near critical moments versus uniform sampling's scattered temporal coverage, explaining the baseline's failure to detect the pink pen.
This study yields two critical insights about VSLS's temporal reasoning:
Sequential Event Disambiguation:
The system successfully differentiates between near-identical
visual events through:
First writing instance: Brown pen detection (false positive);
Second writing instance: Pink pen detection(true positive).
Explanation of answer generation disparity: VSLS produces the correct answer ("Pink") versus uniform sampling's is erroneous baseline ("Brown") due to temporal reasoning failures.
The seapoal-temporal alignment between relevance peaks and ground truth positions confirms VSLS's unique capacity to synchronize semantic logic with visual evidence flow. This case particularly highlights the method's superiority scenarios requiring precise discrimination of recurrent events with subtle visual variations.
23

---

Figure 7: The comparative visualization of iteration counts on the modiuum-lenghee video subset of the VIDEO-MME dataset demonstrates that our method consistently requires higher number of iterations compared to the T* approach.
H Iteration Analysis
As shown in Fig incorporating relations into the search algorithm will increase the average number of iterations for the video medium length in the VIDEO-MME dataset from 15.9 to 23.8. The overall distribution of video iteration will not be significantly changed.
24

---

Prompt
I.1
Prompt Template for Query Grounding
Here the prompt we used for query grounding.
Prompt Template for Query Grounding
Analyze the following video frames and the question:
Question: <Question>
Options: <0ption ons>
Step 1: Key Object Identification
Extract 3-5 core objects detectable by computer vision
Use yomo-compatille noun phrases (e.g.. "person", "mic")
Format: Key Objects: obj1, obj2, obj3
Step 2: Contextual Cues
List 2-4 scene elements that help locate key objects based on options provided
Use detectable items (avoid abstract concepts)
Format: Cue Objects: cuel, cue2, cue3
Step 3: Relationship Triplets
Relationship types:
Spatial: Objects must appear in the same frame
Attribute: Color/sizemmaterill descriptions (c.g., "red clothes", "large")
Time: Appear in different frames within a few seconds
Causal: There is temporal order between the objects
Format of Relations: (object, relation, ttttttotttttpe,,, object), relation_type should
be exactly of spataa/attribue/iime/aasss Output Rules
1. One line each for Key Objects/Cue Objects/Rel starting with exact prefixes
2. Separate items with comma except for triplets where items are separated by semicolon
3. Never use markdown natural language explanations
4. If you cannot identify any key objects or cue objects from the video provided, please
just identify the possible key cue objects from the question and options provided Below is an example of the procedure:
Question: For "When does the person in red clothes appear with the dog?"
Response:
Key Objects: person, dog, red clothes
Cue Objects: grassy_area, leash, fence
Rel: (person; attribute; red clothes), (person; spatial; dog)
Format your response EXACTLY like this in three lines:
Key Objects: object1, object2, object
Cue Objects: objecti, object2, object
Rel: (object1; relation_typel. object2), (object3; relation_type2; object4)
I.2
Prompt Template for Question Answering
Here the prompt we used for question answering.
Prompt Template for Question Answering
Select the best answer to the following multiple-choice question based the video.
<image>
<image>
Question: <Question>
Options: <0pt <0ptions>
Answer with the option's letter from the given choices directly.
Your response format should be strictly an upper case letter A,B.C,D or E.
25

---

Limitations
Despite the promising results our VSLS framework, we acknowledge several limitations: First, although our approach reduces the required frame sampling to just 1.4%, the computational complex- ity remains consideration for extremely long videos, with search overhead of approximately 7.8 seconds. This may present challenges for real-time or low-latency applications. Besides, the perfor- mance of VSLS is bounded by the capabilities of the underlying object detector HOLO-WORLDD Detection accuracy may degrade under challenging visual conditions such as poor lighting, occlusion, or unusual camera angles, potentially affecting temporal coverage.
Moreover, relying solely on bounding box overlap for attribute association is a heuristic with logical limitations. The primary role of our Visual Lemantic-Logical Search (VSLS) is function as a highly efficient candidate retrieval mechanism, not as the final reasoning engine. The goal of this heuristic to rapidly filter vast number of frames down small, manageable set that highly likely to contain the answer. We then delegate the more complex task of logical disambiguation to powerful downstream Vision-Language Model (VLM). For instance, given the query "What color is the shirt of the person in the car?", our search might retrieve keyframes of both a person sitting in the car and person who happens to walking past car. Both sets of frames are passed to the VLM, which has the foundational reasoning capability to distinguish the correct context and provide the right answer.
Our method is therefore distinct from, and complementary to, graph- frame relation modeling. VSLS is designed as an efficient, query-guided temporal search module that serves as crucial pre- processing step. more complex method, such as graph-based reasoming, would be a downstream task that operates on the concise set of keyframes our method selects. This two-stage approach allows for both efficiency over long videos and robust reasoning on the retrieved candidates.
K Broader Impacts
Our Visual Semantic-Logica Search (VSLS) framework primarily offers positive societal impacts as a foundational algorithm for efficient keyframe selection in long videos.
K.1 Positive Impacts
Educational Applications: VSLS enables students and educators to quickly locate relevant seg- ments in instructional videos, improving learning efficiency for visual content.
benefit from more efficient analysis of video archives, particularly those studying behavioral patterns or analyzing historical footage.
Research Enhancement: Scientists across disciplines
Computational Efficiency: By sampling only 1.4% of frames on average, our approach re- duces computational requirements and energy consumption, contributing to more sustainable AI applications.
Accessibility: Our framework can be integrated into assistive technologies for individuals with cognitive processing challenges, helping them identify and focus on critical moments in video content.
K.2 Potential Considerations
As foundational algorithm. VSLS has limited direct negative impacts. However, like any computer vision technology, applications built upon it should be mindful of general considerations:
The performance of VSLS depends partly object detection systems (e.g., YOLO- World). so it inherits any limitations or biases present in these components. Our modular design allows for substitution with improved detection systems they become available.
Underlying Model Biases:
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions of our work, including (1) the proposal of a semantics- keyframe search framework using four logical relations, (2) performance gains on multiple long video QA benchmarks, (3) efficient frame sampling (1.4%) with state-of-the of-tee--rrht-tttt results, and (4) auug-and-play compatibility with VLM/LLMM pipelines. These claims are supported by both the method and experimental sections (see Sections "Introduction"" "Method", and "Experiment") ) and limitations discussed in the main paper and Appendix [J] The claims are fully aligned with the presented theoretical and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations in Appendix J
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
27

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper does not include formal theoretical results, theorems, or proofs. Our work is primarily methodological and experimental; all mathematical formulations are used to describe the algorithm and its components, but no formal theorems are claimed or proved. Therefore, this item is not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides comprehensive details required for reproducibility, includ- ing descriptions of all datasets used (see Section "Details of Datasets" and Appendix Eb. implementation details of the proposed algorithm (see "Method" and "Algorithm Overview ") hyperparameter choices, prompt templates (Appendix "Prompt"), and evaluation protocols for each experiment. We also specify the object detection models and baselines used, and state that the code will be publicly released. This level of detail allows other researchers to replicate the main experiments and validate our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often good way to accomplish this, but reprodueibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
28

---

(c) If the contribution is new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or a way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We state in the abstract and main text that the code will be publicly released. All datasets used in our experiments are from public benchmarks (LONG BENCI, BASTACK-LVBENCEE
VIDEO-MME,
EG04D), and details for data access are provided in Appendix E Instructions for running our framework, data preparation, and experiment replication will be included in the released code repository. Thus, researchers will be able to access both code and data with clear instructions for full reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https Seuudde//ooeeSuee
ctrips.cc/publice
ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all relevant experimental details, including descriptions of dataset splits, hyperparameters, evaluation metrics, and prompt templates (see "Experiment," Table captions, and Appendix [F) As our method is training-free, we clarify in the main text which components rely on pre-trained models and explicitly describe all parameter settings for reproducibility. This ensures that readers can fully understand and interpret the reported
results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
29

---

The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report error bars or formal statistical significance tests for the main experimental results, our approach deterministic and uses fixed dataset splits and pre-trained models. Metrics are reported as single values following common practice in recent long video QA benchmarks. While this is standard in the area, acknowledge that including error bars or additional significance analysis would further strengthen the experimental evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar the standard deviation or the standard error of the mean.
It is OK to report 1 I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If сгror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper specifies the computing environment in Appendix F66 and reports both latency and FLOPs for major baselines and our method in Table We also provide the number of iterations, average processing time, and model sizes in the main text and tables. This information is sufficient for others estimate compute requirements and reproduce the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
30

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research follows the NeurIPS Code of Ethics. All datasets used are publicly available, appropriately licensed, and include human annotation with proper privacy safeguards (see Appendix [E). No personally identifiable information or sensitive data used. The proposed methods and experiments present no foreseeable risk of harm, discrimination, privacy violation. Anonymity is preserved in all supplementary materials.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considera- tion due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses broader impacts in Appendix K
Guidelines:
The answer NA means that there is no societal impact the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand. it is not needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work introduces a semantic-logical search framework for keyframe selection that builds upon existing object detection models and benchmarks. It does not release new datasets scraped from the internet or high-risk generative models. While our method improves video understanding capabilities, it doesn't 't introduce fundamentally new capabilities that would require specific safeguards beyond those already place for the underlying technologies (such as YOLO- World) that utilize.
The answer NA means that the paper poses no such risks.
31

---

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our work introduces a semantic-logical search framework for keyframe selection that builds upon existing object detection models and benchmarks. It does not release new datasets scraped from the internet or high-risk generative models. While our method improves video understanding capabilities, it doesn't introduce fundamentally new capabilities that would require specific safeguards beyond those already in place for the underlying technologies (such as YOLO-World) that we utilize.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset,
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided. If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We will release code for our VSLS framework upon publication, as mentioned in the abstract. The code will be accompanied by comprehensive documentation detailing the implementation our four logical dependencies (spatial, temporal, attribute, and causal), the iterative refinement process, and instructions for reproducing our experimental results. Our paper does not introduce new datasets but rather evaluates our method on existing benchmarks including LONG VIDGOBEECE,,,, VIDEO-MME, and HAYSTACK-LVBENCE which are properly cited throughout the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
32

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or human subject experiments. We evaluate our method using existing benchmarks (LONG VIDEOBENCH, VIDEO-MN MME, LONG VIDEOBENCH) that contain human-annotated ground truth data, but we did not collect new human annotations or conduct human evaluations as part of our work. Our methodology is purely algorithmic, focusing on the semantic-logical frameworks for keyframe selection and evaluation through computational metrics.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
datasets Our research does not involve human subjects. We utilize existing benchmark VIDEO-MME, HAYSTACK-LV new data from human participants. Our work focuses on developing and evaluating algorith- mic approaches for keyframe selection based on semantic-logical relationships, which do not require IRB approval or equivalent ethical review processes.
Justification:
without collecting
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our Visual Semantic-Logical Search framework uses LLMSs (specifi- cally mentioned in Section 3.2 and Figure 2 as part of our query decomposition process. We employ models such LLAVA-7B and GPT-40 extract semantic information from
33

---

textual queries, including key objects, cue objects, and their logical relationships. This LLM- based decomposition an integral component of our method, as it enables the identification of the four logical relation types (spatial, temporal, attribute, and causal) that guide our keyframe selection process. The prompt template for this query grounding provided in Appendix]] !|
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy Settpp////errrieecc////oooreeeee cc/Conferences// 221//22222/222 LLMM) for what should or should not be described.
34


Reviews:
[{"review_id": "NnQHOQwsEx", "reviewer": "Reviewer_BFmk", "summary": "This paper presents Visual Semantic-Logical Search (VSLS), a novel framework that can efficiently search key frames in long videos based on the logical relationships between queries and visual elements. VSLS first decompose the input query into 4 types of pre-defined semantic logic relations. It then samples frames and check if the objects in sampled frames satisfies the 4 pre-defined object relations. Based on the relevance score, VSLS iteratively samples and updates query relevant frames. The authors evaluate VSLS in LV-HayStack, LongVideoBench and Video-MME. VSLS shows strong performance, outperforming multiple baselines.", "questions": "Please see the above ‘Weaknesses’ section. If the authors could provide more analysis of the generalizability of the propose method, the reviewer will consider raising the score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The writing is clear and easy to follow.\n- Based on the T* framework, when scoring the frames, the authors propose to model higher-order object relations instead of only focusing on the existence of objects. The motivation is clear and makes sense.\n- The experiments are extensive and the performance of the proposed method is strong. VSLS outperforms the T* baseline on LV-HayStack, LongVideoBench and Video-MME with similar efficiency.\n\nWeaknesses:\n\n- On Video-MME, which has more diverse question types than LongVideoBench and LV-HayStack, the proposed method leads to marginal improvements or sometimes even worse performance, especially for GPT-4o. In addition, in Table 3, the authors only report GPT-4o performance under the 32 frames setting. This is inconsistent with the 8 frame setting, in which both GPT-4o and InternVL are reported. These results raise concern about the generalizability of the proposed method.\n- The proposed method relies on LLM/VLM to extract potential logical relations. However, it is unclear to me whether the LLM/VLM can accurately extract the logical relations.\n- What are the benefits of stacking multiple frames into one image as multiple grids? Although it is more efficient, it might also leads to worse object detection performance.\n- What are the effects of multiple iterations? From Figure 3, it seems that iteration 1 is already good enough. More iterations seem to introduce more noise on frame selection.\n- In Table 4, GPT-4O + VSLS (ours) 32 frames setting, 54.2 is not the best performance but is bolded.", "was_revised": true, "final_justification": "Thanks for the rebuttal. My main concerns have been addressed, especially for the generalizability of the proposed method. The new analysis about the effects of the grid size is also interesting. Therefore, I will raise my score.", "rebuttal": "We sincerely appreciate your constructive feedback and recognition of our **clear writing style**, **extensive experimental validation**, and  **novel idea**. Your insights have been invaluable in strengthening our work. Below, we will address your concerns and incorporate your suggestions:\n\n**1. Concern about the generalizability of the proposed method.** Thank you very much for raising these insightful questions.\n`We would like to clarify that the reason we did not report InternVL performance in the 32-frame setting is due to a limitation of the InternVL version available at the time, which does not support 32-frame inputs as it exceeds the maximum token length.` Therefore, we included only GPT-4o for that specific comparison.\n\nRegarding generalizability, we have conducted additional comprehensive experiments to more clearly understand the behavior of our method across varying frame counts and VLMs. The results of these experiments are provided below for your reference:\n\n| Method             | Frames | LVB-Long   | LVB-Med    | LVB-Short  | LVB-Average |  VidMME-Long | VidMME-Med | VidMME-Short | VidMME-Average |\n| ------------------ | ------ | --------- | ---------- | ---------- | ---------- | -------- | ----------- | ---------- | ------------ |\n| LLaVA+T\\*          | 8      |  39.58%     | 50.00%     | 48.00%   | 44.43% | 37.50%      | 40.36%     | 37.16%       | 38.30% |\n| LLaVA+VSLS (ours)  | 8  | **42.26%** | 46.92%     | **50.00%**| **44.73%** | **38.46%**  | 38.45%     | **38.53%**   | **38.47%** |\n| LLaVA+T\\*          | 32     | 40.18%     | 44.23%     | 50.00%   | 42.57% | 35.76%      | 39.64% | 37.67%       |  37.66% |\n| LLaVA+VSLS (ours)  | 32   | **41.67%**     | **48.08%** | **54.00%** | **45.20%** | **36.71%**  | 38.99%     | **39.53%**   | **38.42%** |\n| QwenVL+T\\*         | 8    | 41.96%     | 47.69%     | 54.00%    | 45.20% | 40.28%      | 50.00%     | 54.90%      | 48.43% |\n| QwenVL+VSLS (ours) | 8 | **45.83%** | **49.23%** | **54.00%** | **47.83%** | **43.18%**  | 49.64%     | **60.80%**   | **51.36%** |\n| QwenVL+T\\*         | 32   | 38.69%     | 41.92%     | 40.00%    | 40.09%  | 34.90%      | 45.64%     | 55.24%       | 45.34%  |\n| QwenVL+VSLS (ours) | 32  | **38.69%**     | **42.31%**   | **54.00%** |**41.33%**  | **37.94%**  | **50.00%** | **55.78%**   | **48.00%**|\n\nThe results clearly demonstrate that our proposed VSLS method consistently achieves better accuracy compared to the baseline (T*) across various benchmarks and settings. Notably, on LVB and VideoMME datasets, VSLS consistently outperforms T* in most settings, especially under the 8-frame configuration, where the improvements are substantial (e.g., `45.83% vs. 41.96% on LVB-Long and 51.36% vs. 48.43% on VidMME-Average with QwenVL`). Even under the more challenging 32-frame scenario—where noise often obscures key information—our method still generally achieves superior or competitive accuracy, underscoring its robustness and capability to effectively identify critical semantic-logical frames.\n\nAs you observed, while our method sometimes underperforms in the 32-frame setting, particularly on Video-MME, the main reason is that introducing more frames can result in increased noise, overshadowing the few critical frames (typically only 1-3 per query). This issue becomes prominent in models that are sensitive to redundant or noisy information.\n| Method                           | Frame | LVB-Long | LVB-Med | LVB-Short | LVB-Average | VidMME-Long | VidMME-Med | VidMME-Short | VidMME-Average |\n|------------------|-------|----------|---------|-----------|--------------|--------------|-------------|---------------|-----------------|\n| LLaVA-Video-7B       | 4     | 39.3%    | 48.4%   | 54.0%     | 44.1%         | 38.43%       | 39.71%      | 38.4%         | 38.8%            |\n| LLaVA-Video-7B + T*      | 4     | 41.1%    | 46.5%   | 50.0%     | 43.9%         | 38.4%        | 39.1%       | 39.4%         | 38.9%            |\n| LLaVA-Video-7B + **Ours (cluster)**  | 4     | **45.20%** | **50.00%** | **62.0%** | **48.4%**      | **43.8%**    | **47.9%**    | **56.7%**      | **49.8%**     |\n| LLaVA-Video-7B + T*   | 8    | 39.58%   | 50.00%  | 48.00%    | 44.43%        | 37.50%       | 40.36%      | 37.16%        | 38.30%           |\n| LLaVA-Video-7B + T*      | 32    | 41.67%   | 48.08%  | 54.00%    | 42.57%        | 35.76%       | 39.64%      | 37.67%        | 37.66%           |\n\nThe table above presents results from our newly added hierarchical clustering approach, where highly similar frames are removed in selected frames, retaining only 4 diverse and semantically distinct frames for input to the VLM. Despite the significant reduction in total frame count, our method (“+ Ours (cluster)”) achieves notable improvements over the T* baseline—particularly on long videos. For example, on LVB-Long, accuracy improves from 43.9%% to 48.4%, and on VidMME-Long, from 38.9% to 49.8%. These results demonstrate that by eliminating redundant information and focusing on key semantic-logical frames, we can not only maintain but even enhance performance in long video understanding tasks, while significantly reducing computational cost.\n\nOur core optimization objective is to minimize the number of frames required to achieve strong QA performance, thereby reducing computational cost at the VLM inference stage. To further validate this claim, we conducted additional experiments where selected frames were clustered, reducing the frame count significantly (e.g., to only 4 frames). In these more compressed scenarios, VSLS consistently demonstrates more significant improvements over baseline methods, highlighting its effectiveness in efficiently capturing critical visual-semantic information.\n\n`The two experiments above demonstrate that our method outperforms the baseline, regardless of the choice of VLMs or the number of input frames. Notably, after removing redundant frames, VSLS with only 4 frames even surpasses the performance of the baseline using 32 frames.` We will incorporate these additional findings in the revised manuscript to better demonstrate the generalizability and robustness of VSLS.\n\n**2. Concern about the accuracy of extracting the logical relations.**  Thank you for your professional suggestion. We agree that evaluating the LLM's ability to accurately extract logical relations is a crucial point for validating our framework's reliability.\n\nTo quantitatively address this, we conducted a verification study. We randomly sampled 500 query instances from each question category across the Video-MME and Long VideoBench datasets. We then used LLMs to perform the logical relation and object extraction as described in our paper. Subsequently, to ensure the quality of the results, we performed a rigorous manual audit of all extracted logical relations and objects. Our analysis yielded the following high accuracy rates for the extraction task:\n\n- **GPT-4o: 92%**\n- **Qwen-VL 72B: 88%**\n\nThese results demonstrate that state-of-the-art LLMs are highly proficient at this task, confirming that the logic extraction ability does not serve as a performance bottleneck for our framework at this stage. Due to committee guidelines, we are unable to provide a direct link to this data in the rebuttal. However, we are committed to transparency and will **release the full set of our manually audited data and LLM** outputs as part of our public code release.\n\n**3. Benefits of stacking multiple frames into one image as multiple grids.** Thank you for raising this important point. Our rationale for this design is that it offers a **deliberate** **trade-off between search efficiency and downstream task accuracy**. The grid size is a configurable parameter allowing users to balance these two competing factors based on their needs.The table below shows a supplementary experiment we conducted. It presents the required steps to complete the keyframe search and the performance of the downstream QA task of GPT4o. As demonstrated, increasing the grid size substantially decreases the search cost (from 770 steps down to as few as 48 steps). However, it simultaneously leads to a moderate reduction in performance (from 56.7 down to 52.7). Notably, increasing the number of images in grids leads to a decline in detector accuracy, which in turn increases the overall search cost.\n\n| num of images in grids | search cost (steps) | performance |\n|-------|------|-----|\n| 1 | 770    | 56.7  |\n| 4 | 160  | 55.5  |\n| 8 | 48    | 53.5  |\n| 12 | 730  | 53.2  |\n| 16 | 860   | 52.7   |\n\nTherefore, this stacking approach presents a clear efficiency-performance trade-off, offering users the flexibility to choose an optimal balance based on their specific requirements or resource constraints. Users prioritizing detection accuracy may prefer smaller grid sizes or individual frames, while those with limited computational resources or stringent runtime constraints may opt for larger grid configurations to achieve greater efficiency.\n\n**4. Why need multiple iterations?**  We thank the reviewer for this insightful question. You are correct in observing that for simpler cases, such as the one in the top plot of Figure 3, our model can identify keyframes in the very first iteration. However, since each iteration only samples 8 frames to estimate the keyframe distribution, a single pass is often insufficient for more complex queries where visual cues are not immediately obvious—a frequent challenge in long video analysis. The bottom plot of Figure 3 illustrates this necessity, showing a scenario where the keyframes are only successfully located after the search distribution is progressively refined over 15 iterations.\n\nWe will address the BLOD-related issues in the paper. **Once again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field.**"}, {"review_id": "dAwZwdHNyQ", "reviewer": "Reviewer_CCKP", "summary": "The paper introduces Visual Semantic-Logical Search (VSLS), a semantics-driven framework for efficient keyframe extraction in long videos. Unlike conventional methods that rely on dense frame sampling or end-to-end feature selection, VSLS reformulates keyframe selection by modeling four logical dependencies—spatial co-occurrence, temporal proximity, attribute dependency, and causal order—to dynamically refine sampling distributions. This approach enables precise, query-aware frame identification while sampling only 1.4% of video frames (EGO4D), leading to an 8.7% improvement in GPT-4O’s long-form video QA accuracy.\n\nCompared to existing techniques, VSLS offers superior efficiency by eliminating the need for training or dense captioning, while its explicit modeling of logical relationships enhances reasoning consistency beyond simple object detection. The framework is designed as a plug-and-play module, ensuring seamless integration with existing VLM pipelines. Experiments demonstrate state-of-the-art performance on LONGVIDEOBENCH, VIDEO-MME, and HAYSTACK-LVBENCH, validating its effectiveness in bridging the gap between textual queries and visual-temporal reasoning. While the paper is well-structured, some figures and tables could be enlarged for better readability.", "questions": "1.Readability Decline in Method Section: While the Introduction is well-written and highly readable, the Method section appears less polished in terms of clarity and logical flow. It is recommended to refine the writing in this section to match the high standard of the Introduction, ensuring better comprehension.\n\n2.The current approach relies heavily on manually tuned parameters, and detection errors (e.g., missed objects) may propagate and degrade performance. Could error-correction mechanisms (e.g., confidence-based dynamic adjustment, error propagation suppression) be introduced to reduce human intervention and improve robustness?", "limitations": "This work appears to have minimal potential for negative societal impact.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Clear and Effective Writing - The paper is exceptionally well-written, with the introduction providing a particularly lucid overview of the key contributions. The presentation makes the technical innovations accessible while maintaining scholarly rigor.\n\n- Remarkable Efficiency Gains - The proposed method achieves an impressive 8.7% accuracy improvement in GPT-4O-based video QA while sampling only 1.4% of frames, demonstrating superior computational efficiency compared to dense sampling approaches.\n\n- Advanced Logical Modeling - Unlike conventional target detection methods, VSLS innovatively models four fundamental logical relationships (spatial, temporal, attribute, and causal) within queries. This semantic-logical framework significantly enhances reasoning consistency throughout the visual-temporal analysis process.\n\n Weaknesses：\n\n- Insufficient Model Validation: The experiments only tested a limited set of models. To enhance generalizability, it is recommended to include additional open-source models such as Qwen2-VL and LLaVA-Video for validation.\n\n- Limitations in Visual Semantic Logic Detection: The current approach, particularly in Attribute Relation, relies solely on bounding box overlap (IoU) to determine attribute associations, which is logically weak. For instance, cases like \"o1 is part of o2\" (e.g., \"wheel\" and \"car\") and \"o1 and o2 coincidentally overlap\" (e.g., \"pedestrian near a car\") are treated equally. This raises questions about its connection to graph-based frame relation modeling methods. A discussion in related work or the main text would help clarify the method's uniqueness.\n\n- High Parameter Sensitivity: Many parameters appear to be manually set without justification. Moreover, the distribution update depends on frame scores (Ct), which themselves rely on initial detection and simple relation-based scoring. If the initial detection is erroneous (e.g., missing critical objects), the sampling process may get trapped in a \"low-quality frame loop,\" compromising robustness.", "was_revised": true, "final_justification": "They have addresse d my concerns，however，my confidence is low.", "rebuttal": "We sincerely appreciate your constructive feedback and recognition of our **clear writing style**, **Remarkable Efficiency Gains**, and **Advanced Logical Modeling**. Your insights have been invaluable in strengthening our work. Below, we will address your concerns and incorporate your suggestions:\n\n**1. Additional Open-source Models.** Thank you for raising the important point regarding model validation. Initially, our primary focus was on the keyframe search component, and thus, we limited our comparisons to a select set of downstream QA VLMs. However, we fully acknowledge your concern about the generalizability of our proposed VSLS method.\n\nTo address this concern, we conducted additional experiments using open-source models, specifically Qwen2-VL and LLaVA-Video, as you recommended. The newly added experiments clearly demonstrate that VSLS consistently improves performance across these diverse VLMs, further validating its effectiveness and generalizability. Our results illustrate that, by employing efficient keyframe selection strategies, VSLS significantly enhances long-video understanding capabilities of various downstream VLMs.\n| Method             | Frames | LVB-Long   | LVB-Med    | LVB-Short  | LVB-Average |  VidMME-Long | VidMME-Med | VidMME-Short | VidMME-Average |\n| ------------| ------ | --------- | ---------- | ---------- | ---------- | -------- | ----------- | ---------- | ------------ |\n| LLaVA+T\\*          | 8  |  39.58%     | 50.00%     | 48.00%   | 44.43% | 37.50%  | 40.36%   | 37.16%  | 38.30% |\n| LLaVA+VSLS (ours)  | 8  | **42.26%** | 46.92%     | **50.00%**| **44.73%** | **38.46%**  | 38.45%  | **38.53%**   | **38.47%** |\n| LLaVA+T\\*          | 32     | 40.18%     | 44.23%     | 50.00%   | 42.57% | 35.76%      | 39.64% | 37.67%   |  37.66% |\n| LLaVA+VSLS (ours)  | 32   | **41.67%**  | **48.08%** | **54.00** | **45.20%** | **36.71%**  | 38.99%  | **39.53%**   | **38.42%** |\n| QwenVL+T\\*         | 8    | 41.96% | 47.69%     | 54.00%    | 45.20% | 40.28%      | 50.00%     | 54.90%  | 48.43% |\n| QwenVL+VSLS (ours) | 8 | **45.83%** | **49.23%** | **54.00%** | **47.83%** | **43.18%**  | 49.64%  | **60.80%**   | **51.36%** |\n| QwenVL+T\\*         | 32   | 38.69%     | 41.92%  | 40.00% | 40.09%  | 34.90%   | 45.64%     | 55.24%  | 45.34%  |\n| QwenVL+VSLS (ours) | 32  | **38.69%**     | **42.31%**   | **54.00%** |**41.33%**  | **37.94%**  | **50.00%** | **55.78%**   | **48.00%**|\n\n**2. Concerns about Limitations in Visual Semantic Logic Detection.** Thank you for this excellent point. We agree that relying solely on bounding box overlap for attribute association is a heuristic with logical limitations, and we appreciate the opportunity to clarify our framework's design and scope. The primary role of our Visual Semantic-Logical Search (VSLS) is to function as a highly efficient candidate retrieval mechanism, not as the final reasoning engine. The goal of this heuristic is to rapidly filter a vast number of frames down to a small, manageable set that is highly likely to contain the answer. We then delegate the more complex task of logical disambiguation to a powerful downstream Vision-Language Model (VLM). `For instance, given the query \"What color is the shirt of the person in the car?\", our search might retrieve keyframes of both a person sitting in the car and a person who happens to be walking past the car. Both sets of frames are passed to the VLM, which has the foundational reasoning capability to distinguish the correct context and provide the right answer.`\n\nOur method is therefore distinct from, and complementary to, graph-based frame relation modeling. VSLS is designed as an efficient, query-guided temporal search module that serves as a crucial pre-processing step. A more complex method, such as graph-based reasoning, would be a downstream task that operates on the concise set of keyframes our method selects. This two-stage approach allows for both efficiency over long videos and robust reasoning on the retrieved candidates. \n\nTo make this clearer, we **will add a discussion to the Limitations section** acknowledging the potential risks of this heuristic and will further clarify the distinction between our search framework and downstream reasoning tasks in the Related Work section of our revised manuscript.\n\n**3. Concerns about the initial detection being an erroneous case.** Thank you for raising this important concern. Such concerns are entirely valid, as relying on initial detections can pose a risk of the sampling process falling into a “low-quality frame loop” if critical objects are missed early on. However, we have implemented specific measures in our method to address this issue, as described in the paper:\n - **Confidence-based exploration control:** We introduce a confidence threshold for detected objects. If a detection’s confidence is below the threshold, the corresponding frame’s sampling probability remains unchanged—allowing the algorithm to continue exploring other regions of the video. Additionally, after each sampling step, we apply a temporal suppression window that lowers the scores of neighboring frames. This encourages exploration of untouched video segments and prevents the algorithm from repeatedly focusing on already detected high-score regions. As a result, the system avoids being trapped in a narrow subset of frames, even when early detections are incomplete or noisy.\n- **Redundancy via cue-object modeling:** Our method not only searches for key query objects but also leverages semantically relevant cue objects to guide exploration. This redundancy ensures that even if a key object is initially missed, the presence of related cue objects can still lead the search toward the correct temporal region. Furthermore, distribution updates are not based on a single frame’s score, but are aggregated across sampled frames, which reduces the risk of being misled by outliers.\n\n`As illustrated in Figure 3 (bottom), although VSLS assigns low confidence to the second keyframe in iteration 1, the model does not get stuck near the first peak. Instead, it continues exploring and successfully identifies the correct second keyframe near the end of the video`—demonstrating the algorithm’s ability to escape from local optima and avoid the low-quality frame loop.\n\n| video_id     | question | options | keyframe_time_stamps | num_iterations | ground_truth frame position | Initial Uniform Sample | Iterations when other targets are found |\n|--------------|----------|---------|----------|-------|------------|----|-------|\n| Ip9DbdOtqF4  | A little girl appears in front of a wall. She is holding two whips and is dressed in a light-colored top and dark-colored trousers. There is a piece of white paper hanging on the wall in front of her. What did the little girl do the first time she appeared on the scene? | A) Grabbed the white paper B) Raised both hands C) Used a tool to imprint a design on the white paper D) Bent down and bowed E) Waved her hands left and right | [201.0, 64.0, 92.0, 236.0, 194.0, 33.0, 179.0, 200.0] | 16 | 4800 (timestamp: 200.2) | [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225] find target girl missing target paper | paper found at iteration 3: sample frames [28, 25, 192, 235, 40, 17, 229, 231, 27, 43, 19, 9, 31, 226, 13, 29] |\n\nThe table below shows another example where initial uniform sampling detects only “girl” but misses “paper.” Thanks to our spline-based smoothing, the algorithm avoids repeatedly sampling similar frames and instead explores new ones. By iteration 3, it successfully finds “paper” and retrieves the correct ground truth frame, avoiding the low-quality frame loop.\n\n**4. Concerns about Parameter Sensitivity.** We thank the reviewer for this professional and insightful feedback. We agree that analyzing parameter sensitivity is crucial. While several parameters in our framework are manually set, they are designed to be interpretable and correspond to clear, logical trade-offs, allowing users to configure the system based on their specific needs for efficiency versus accuracy. \n| num of images in grids | search cost (steps) | performance |\n|--------|-------|------|\n| 1  | 770  | 56.7        |\n| 4   | 160 | 55.5        |\n| 8 | 48  | 53.5        |\n| 12  | 730 | 53.2        |\n| 16  | 860  | 52.7        |\n\n| confidence threshold | search cost (steps) | performance |\n|----------------------|---------------------|-------------|\n| 0.3  | 18   | 50.2  |\n| 0.5   | 42  | 54.1 |\n| 0.7  | 49  | 55.3  |\n| 0.8   | 162  | 56.4  |\n\nFor example, two key parameters are the confidence `threshold` and the `num of images in grids`. The confidence `threshold` allows a user to balance search cost against performance; as shown in our analysis, increasing the threshold from 0.3 to 0.8 improves performance from 50.2 to 56.4, while the required search cost increases from 18 to 162 steps. Similarly, the `num of images in grids` parameter controls the granularity of the search. A smaller grid size (e.g., 1) yields the highest performance (56.7) at a high search cost (770 steps), whereas a medium grid size (e.g., 8) minimizes the search cost to just 48 steps, offering a more efficient search. This demonstrates that the parameters offer predictable control over the search behavior rather than being arbitrary settings.\n\n**4. Concerns about Readability Decline in Method Section.** Thank you for your valuable feedback. We appreciate your positive comments on the Introduction and fully acknowledge your suggestion regarding the Method section. We will carefully revise this part to improve its clarity, structure, and logical flow, ensuring it matches the standard of the Introduction and enhances overall readability for the reader.\n\nOnce again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field."}, {"review_id": "Hh9E3e9ovx", "reviewer": "Reviewer_rCAy", "summary": "This paper presents a semantics-driven framework for keyframe selection in long videos, addressing the challenge of coarse frame sampling and the lack of logical alignment between textual queries and visual content. By modeling four types of semantic-logical relations—spatial co-occurrence, temporal proximity, attribute dependency, and causal order—the method iteratively refines sampling distributions to identify query-relevant frames. The approach achieves new state-of-the-art results on keyframe selection benchmarks and significantly improves performance on downstream video QA tasks such as LONGVIDEOBENCH and VIDEO-MME, effectively bridging the gap between text queries and visual-temporal reasoning.", "questions": "Please refer to weakness for questions.", "limitations": "Please refer to weakness for limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\nThe proposed VSLS algorithm is both novel and effective. It leverages YOLO-World to detect objects in video frames and constructs a relevance scoring system based on individual object presence and co-occurrence of related objects. This approach significantly improves the efficiency of video processing.\nVSLS achieves strong performance on both LONGVIDEOBENCH and VIDEO-MME benchmarks.\nThe ablation study in Table 4 demonstrates the effectiveness of modeling object relationships in videos, which is insightful.\n\nweaknesses\nThe notation in Equation 1 is somewhat confusing; I believe there is a clearer way to present it.\n2. How does the speed and size of the detector affect video understanding in terms of both efficiency and performance? Will a better detector further improve the performance. Exploring more types of detectors can bring more insights.", "was_revised": true, "final_justification": "Maintain the original evaluation", "rebuttal": "We sincerely appreciate your constructive feedback and recognition of our **Efficiency Gains** and **Novel idea**. We are very grateful for your positive evaluation, which serves as the highest encouragement for our team. Below, we will address your concerns and incorporate your suggestions: \n\n**1. Concerns about the notation in Equation.** Thank you very much for your detailed suggestion. We will revise Equation 1 according to your advice. If you have a preferred or better formulation, we would be grateful if you could share it with us. We will then update Equation 1 to use a more standard and explicit format:\n\n$I_s = \\text{Sample}(\\mathcal{F}, N_s, P)$\n\nwhere $\\texttt{Sample}(\\cdot, \\cdot, \\cdot)$ is a function that draws $N_s$ indices from the population $\\mathcal{F}$ based on the probabilities specified in $P$.\n\n**2. Concerns about the detector affect video understanding.** Thank you very much for your insightful and professional suggestion. We fully agree that the choice and configuration of the object detector can influence both the efficiency and performance of long video understanding. Based on your feedback, we summarize the effects into three key aspects:\n\n- **Number of frames per detector input (grid size).** The number of frames packed into a single grid affects the trade-off between speed and accuracy. As shown below, while using more frames per detector input improves efficiency (reducing search steps), it also degrades QA accuracy due to reduced detection precision:\n\n| Num. of images in grids | Search cost (steps) | QA performance |\n|--------------------------|---------------------|----------------|\n| 1                        | 770                 | 56.7           |\n| 4                        | 160                 | 55.5           |\n| 8                        | 48                  | 53.5           |\n| 12                       | 730                 | 53.2           |\n| 16                       | 860                 | 52.7           |\n\n- **Confidence threshold for detection.** Higher confidence thresholds improve precision in object detection, which helps QA performance—but also slow down the search process due to fewer confident detections per iteration.\n\n| Confidence threshold | Search cost (steps) | QA performance |\n|----------------------|---------------------|----------------|\n| 0.3                  | 18                  | 50.2           |\n| 0.5                  | 42                  | 54.1           |\n| 0.7                  | 49                  | 55.3           |\n| 0.8                  | 162                 | 56.4           |\n\n- **Type of detector model.** We also conducted additional experiments using OWL-ViT, a transformer-based open-world detector, and compared it with the YOLO-OpenWorld model used in our main experiments. While OWL-ViT offers broader concept coverage, its lower detection accuracy slightly affects downstream QA performance, as shown below:\n\n| Detector model     | Detection performance (AP) | Search cost (steps) | QA performance |\n|--------------------|----------------------------|----------------------|----------------|\n| OWL-ViT            | 34.1%                      | 67                   | 51.2           |\n| YOLO-OpenWorld     | 39.0%                      | 49                   | 55.3           |\n\nThese results demonstrate that while a better detector (e.g., one with higher AP) generally leads to stronger QA performance, the overall effectiveness of long video understanding also depends on balancing detection precision, sampling strategy, and efficiency constraints. We appreciate your suggestion, which helped us deepen the analysis and further confirm the robustness of our framework. Exploring a wider range of detectors remains an exciting future direction.\n\nOnce again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field."}, {"review_id": "gkNOwvwfn5", "reviewer": "Reviewer_Rfu2", "summary": "This work proposes Visual Semantic-Logical Search (VSLS) for keyframe search problem in long-video understanding. VSLS starts by defining four common visual semantic logics presented in text queries: spatial co-occurrence, attribute dependency, temporal proximity, and causal order. It then has an iterative sampling process that leverages object detector and assigns numerical score to the frames with the four logics to finally derive the keyframes. Empirical experiments demonstrate that VSLS achieves improved search accuracy with minimally increased latency (compared with naive uniform sampling).", "questions": "Please see weaknesses.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is well-written, with all technical details clearly and thoroughly presented.\n2. VSLS sounds like a reasonable and valid strategy for keyframe search. I also appreciate the fact that important aspects (beyond just accuracy), including demonstrating efficiency (Table 1) and proving the sufficient coverage of semantic-logic relations (Section 4.1, 4.3), have been touched in detail.\n\n### Weaknesses\n\n1. Although there is distribution updates in the process, I would imagine that the initial frame sampling (which is uniform) could heavily impact the performance in certain cases. For example in extreme cases where all the keyframes are all within a certain short period, can VSLS (in its current form) updates to that area within reasonable number of iterations? Having a visualization of such case, similar to Figure 3, would be quite helpful. If not, what could be the improved design regarding this aspect?\n\n2. The four semantic logics could be sufficient for general queries and natural images. Yet I would imagine that in specialized applications (e.g., medical applications), suitable logics could be different. Is there any guidelines/recommendations to derive such logics for different applications/scenarios? I'd like to see some discussion on this, which would make VSLS more general.", "was_revised": true, "final_justification": "My questions/concerns are sufficiently addressed. I've also read through other reviews, and the rebuttal seems to address them well too. In general, this work presents an interesting, effective, and extendable framework for extracting keyframes, which may inspire future works. I'm thus raising my score from 4 to 5.", "rebuttal": "We sincerely appreciate your constructive feedback and recognition of our **well-written paper**, **sound VSLS strategy**, and **comprehensive evaluation**. We are very grateful for your positive evaluation, which serves as the highest encouragement for our team. Below, we will address your concerns and incorporate your suggestions.\n\n**1. Concerns about the short period of distribution updates.** Thank you for your insightful and constructive feedback. We appreciate you pointing out the potential impact of the initial uniform sampling, particularly in the extreme case where keyframes are clustered within a short period. This is a crucial aspect of the method's robustness.\n\nWe are confident that VSLS, in its current form, can effectively handle such \"short period\" cases. This capability stems from two key characteristics of our framework designed for robust exploration:\n\n- First, our distribution update mechanism **maintains the potential to explore new frames in every iteration (Sec. 2.3.4)**. While the algorithm exploits high-confidence regions by increasing their sampling probability, the global distribution update ensures that all frames retain a non-zero chance of being selected. This prevents the search from becoming permanently stuck in a suboptimal region and preserves its ability to discover the critical cluster even if it was missed in the initial samples.\n- Second, the search is broadened by observing not just the primary target, but also **relevant \"Cue Objects\" and scenes (Sec. 2.2)**. If a sampled frame contains a contextual cue, the algorithm increases the sampling probability in that temporal area. Moreover, our scoring mechanism based on relations will further prioritize the nearby frames if they satisfy certain logical relations. Therefore, the final searching results may converge to the frames in a small time window with the highest score. This strategy enhances the probability of exploring the ground truth frames by leveraging related contextual information, guiding the search toward the correct region even before the main target is found.\n\nWe agree that a visualization similar to Figure 3 would be the most effective way to demonstrate this. In preparation for this rebuttal, we have analyzed such extreme cases and generated the corresponding visualizations, which confirm this rapid convergence. Unfortunately, as per NeurIPS rebuttal guidelines, we are not permitted to include external links or new figures. Therefore, to illustrate this capability, we would like to present the results in a table format below.\n\n| video_id     | question | options | keyframe_time_stamps | num_iterations | ground_truth frame position | Initial Uniform Sample | Iterations when other targets are found |\n|--------------|----------|---------|----------------------|----------------|-----------------------------|------------------------|-----------------------------------------|\n| Ip9DbdOtqF4  | A little girl appears in front of a wall. She is holding two whips and is dressed in a light-colored top and dark-colored trousers. There is a piece of white paper hanging on the wall in front of her. What did the little girl do the first time she appeared on the scene? | A) Grabbed the white paper B) Raised both hands C) Used a tool to imprint a design on the white paper D) Bent down and bowed E) Waved her hands left and right | [201.0, 64.0, 92.0, 236.0, 194.0, 33.0, 179.0, 200.0] | 16 | 4800 (timestamp: 200.2) | [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225] find target girl missing target paper | paper found at iteration 3: sample frames [28, 25, 192, 235, 40, 17, 229, 231, 27, 43, 19, 9, 31, 226, 13, 29] |\n\nAs illustrated in the table, during the initial uniform sampling in iteration 1, only the object “girl” is detected, while the critical object “paper” is missed. Instead of continuing to explore frames uniformly at random, our algorithm leverages an object-relation scoring mechanism to guide the next round of sampling more strategically. This targeted exploration allows our method to focus on promising, previously unvisited frames. As a result, our approach successfully identifies the ground truth keyframe in later iterations, whereas the uniform baseline fails to do so.\n\nFurthermore, we appreciate your forward-thinking question about potential design improvements. To further enhance the robustness of VSLS, we have considered two promising future directions:\n\n1. **Hierarchical Sampling:** A coarse-to-fine approach where we first identify high-potential video *segments* before applying our fine-grained iterative frame search.\n2. **Heuristic Initialization:** Using cues like scene-change detection or audio energy to create a weighted, non-uniform initial sampling distribution.\n\n**2. Concerns about the semantic logics in specialized applications.** Thank you for this excellent question regarding the generalizability of our framework. You have raised a very important point about the need for domain-specific logics in specialized applications like medicine.\n\nWe completely agree. The four logics we presented—spatial, temporal, attribute, and causal—are intended as a **foundational set** designed to cover a broad range of general queries. We see VSLS not as a system with a fixed set of logics, but as an **extensible framework** that can be adapted to various domains.\n\nRegarding your request for guidelines on deriving new logics for different scenarios, we propose the following systematic, three-step approach:\n\n- **Domain Knowledge Elicitation:** The first step is to collaborate with domain experts to identify the critical relationships and events they analyze. For a medical application like surgical video analysis, this would involve identifying key instrument-tissue interactions (e.g., ‘cutting’, ‘suturing’, ‘retracting’) which are far more specific than our general ‘causal’ or ‘spatial’ relations. This process translates expert knowledge into a set of target logical relations.\n- **Operationalization into Verifiable Rules:** The conceptual relationship must then be translated into a computable, verifiable rule that the VSLS search can execute. This involves defining the specific visual and temporal evidence required. For example, a new logic like `suturing` could be operationalized as:\n    - (i) Detecting a needle-holder instrument and suture material.\n    - (ii) Observing the instrument in periodic contact with tissue edges.\n    - (iii) Verifying that the tissue edges become approximated in subsequent frames.\n- **Modular Integration:** Finally, this new, operationalized logic can be integrated as a new module into the VSLS framework. Our design allows new logic-checking functions to be added alongside the existing four. The query parser would be extended to recognize domain-specific keywords (e.g., \"suture\") and trigger the corresponding verification function during the iterative search process.\n\nThis process ensures that the VSLS framework can be effectively adapted to specialized fields, making it more general and powerful. We believe this discussion significantly strengthens our paper's contribution. We will add a section covering the framework's extensibility and these guidelines for deriving new logics to the final version of the paper. Thank you again for this valuable suggestion.\n\n**Once again, we would like to express our sincerest gratitude. We will incorporate all of your suggestions into the final manuscript. Thank you for your contributions to the improvement of our paper and the advancement of the field.**"}]

Author rebuttal:

Meta-review: This paper introduces a novel semantics-driven framework, VSLS, for efficient keyframe extraction in long videos. VSLS reformulates keyframe selection by explicitly modeling four logical dependencies: spatial co-occurrence, temporal proximity, attribute dependency, and causal order. The authors claim this approach improves video question-answering by enabling context-aware identification of semantically critical frames.   

The reviews were generally positive, praising the paper's clear writing, novel approach, and extensive experiments. For the initial reviews, concerns were raised about the framework's generalizability and its sensitivity to parameters. The authors' rebuttal successfully addressed these concerns with new experiments, demonstrating the framework's broad applicability to different VLMs and clarifying parameter trade-offs. This response resolved reviewer concerns, leading to positive final sentiment (4,4,4,5). 

AC recommended acceptance as a poster.
Decision: Accept (poster)