Title: Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision-language models

key frame search

long video understanding
Abstract: Understanding long video content is a complex endeavor that often relies on densely sampled frame captions or end-to-end feature selectors, yet these techniques commonly overlook the logical relationships between textual queries and visual elements. In practice, computational constraints necessitate coarse frame subsampling, a challenge analogous to “finding a needle in a haystack.” To address this issue, we introduce a semantics-driven search framework that reformulates keyframe selection under the paradigm of Visual Semantic-Logical Search (VSLS). Specifically, we systematically define four fundamental logical dependencies: 1) spatial co-occurrence, 2) temporal proximity, 3) attribute dependency, and 4) causal order. These relations dynamically update frame sampling distributions through an iterative refinement process, enabling context-aware identification of semantically critical frames tailored to specific query requirements. Our method establishes new state-of-the-art performance on the manually annotated benchmark in keyframe selection metrics. Furthermore, when applied to downstream video question-answering tasks, the proposed approach demonstrates the best performance gains over existing methods on LongVideoBench and Video-MME, validating its effectiveness in bridging the logical gap between textual queries and visual-temporal reasoning. The code will be publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Visual Semantic-Logical Search (VSLS) for keyframe search problem in long-video understanding. VSLS starts by defining four common visual semantic logics presented in text queries: spatial co-occurrence, attribute dependency, temporal proximity, and causal order. It then has an iterative sampling process that leverages object detector and assigns numerical score to the frames with the four logics to finally derive the keyframes. Empirical experiments demonstrate that VSLS achieves improved search accuracy with minimally increased latency (compared with naive uniform sampling).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The paper is well-written, with all technical details clearly and thoroughly presented.\n2. VSLS sounds like a reasonable and valid strategy for keyframe search. I also appreciate the fact that important aspects (beyond just accuracy), including demonstrating efficiency (Table 1) and proving the sufficient coverage of semantic-logic relations (Section 4.1, 4.3), have been touched in detail.\n\n### Weaknesses\n\n1. Although there is distribution updates in the process, I would imagine that the initial frame sampling (which is uniform) could heavily impact the performance in certain cases. For example in extreme cases where all the keyframes are all within a certain short period, can VSLS (in its current form) updates to that area within reasonable number of iterations? Having a visualization of such case, similar to Figure 3, would be quite helpful. If not, what could be the improved design regarding this aspect?\n\n2. The four semantic logics could be sufficient for general queries and natural images. Yet I would imagine that in specialized applications (e.g., medical applications), suitable logics could be different. Is there any guidelines/recommendations to derive such logics for different applications/scenarios? I'd like to see some discussion on this, which would make VSLS more general.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents a semantics-driven framework for keyframe selection in long videos, addressing the challenge of coarse frame sampling and the lack of logical alignment between textual queries and visual content. By modeling four types of semantic-logical relations—spatial co-occurrence, temporal proximity, attribute dependency, and causal order—the method iteratively refines sampling distributions to identify query-relevant frames. The approach achieves new state-of-the-art results on keyframe selection benchmarks and significantly improves performance on downstream video QA tasks such as LONGVIDEOBENCH and VIDEO-MME, effectively bridging the gap between text queries and visual-temporal reasoning.

STRENGTHS AND WEAKNESSES:
strengths
The proposed VSLS algorithm is both novel and effective. It leverages YOLO-World to detect objects in video frames and constructs a relevance scoring system based on individual object presence and co-occurrence of related objects. This approach significantly improves the efficiency of video processing.
VSLS achieves strong performance on both LONGVIDEOBENCH and VIDEO-MME benchmarks.
The ablation study in Table 4 demonstrates the effectiveness of modeling object relationships in videos, which is insightful.

weaknesses
The notation in Equation 1 is somewhat confusing; I believe there is a clearer way to present it.
2. How does the speed and size of the detector affect video understanding in terms of both efficiency and performance? Will a better detector further improve the performance. Exploring more types of detectors can bring more insights.

QUESTIONS:
Please refer to weakness for questions.

LIMITATIONS:
Please refer to weakness for limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Visual Semantic-Logical Search (VSLS), a semantics-driven framework for efficient keyframe extraction in long videos. Unlike conventional methods that rely on dense frame sampling or end-to-end feature selection, VSLS reformulates keyframe selection by modeling four logical dependencies—spatial co-occurrence, temporal proximity, attribute dependency, and causal order—to dynamically refine sampling distributions. This approach enables precise, query-aware frame identification while sampling only 1.4% of video frames (EGO4D), leading to an 8.7% improvement in GPT-4O’s long-form video QA accuracy.\n\nCompared to existing techniques, VSLS offers superior efficiency by eliminating the need for training or dense captioning, while its explicit modeling of logical relationships enhances reasoning consistency beyond simple object detection. The framework is designed as a plug-and-play module, ensuring seamless integration with existing VLM pipelines. Experiments demonstrate state-of-the-art performance on LONGVIDEOBENCH, VIDEO-MME, and HAYSTACK-LVBENCH, validating its effectiveness in bridging the gap between textual queries and visual-temporal reasoning. While the paper is well-structured, some figures and tables could be enlarged for better readability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Clear and Effective Writing - The paper is exceptionally well-written, with the introduction providing a particularly lucid overview of the key contributions. The presentation makes the technical innovations accessible while maintaining scholarly rigor.\n\n- Remarkable Efficiency Gains - The proposed method achieves an impressive 8.7% accuracy improvement in GPT-4O-based video QA while sampling only 1.4% of frames, demonstrating superior computational efficiency compared to dense sampling approaches.\n\n- Advanced Logical Modeling - Unlike conventional target detection methods, VSLS innovatively models four fundamental logical relationships (spatial, temporal, attribute, and causal) within queries. This semantic-logical framework significantly enhances reasoning consistency throughout the visual-temporal analysis process.\n\n Weaknesses：\n\n- Insufficient Model Validation: The experiments only tested a limited set of models. To enhance generalizability, it is recommended to include additional open-source models such as Qwen2-VL and LLaVA-Video for validation.\n\n- Limitations in Visual Semantic Logic Detection: The current approach, particularly in Attribute Relation, relies solely on bounding box overlap (IoU) to determine attribute associations, which is logically weak. For instance, cases like \"o1 is part of o2\" (e.g., \"wheel\" and \"car\") and \"o1 and o2 coincidentally overlap\" (e.g., \"pedestrian near a car\") are treated equally. This raises questions about its connection to graph-based frame relation modeling methods. A discussion in related work or the main text would help clarify the method's uniqueness.\n\n- High Parameter Sensitivity: Many parameters appear to be manually set without justification. Moreover, the distribution update depends on frame scores (Ct), which themselves rely on initial detection and simple relation-based scoring. If the initial detection is erroneous (e.g., missing critical objects), the sampling process may get trapped in a \"low-quality frame loop,\" compromising robustness.\n\nQUESTIONS:\n1.Readability Decline in Method Section: While the Introduction is well-written and highly readable, the Method section appears less polished in terms of clarity and logical flow. It is recommended to refine the writing in this section to match the high standard of the Introduction, ensuring better comprehension.\n\n2.The current approach relies heavily on manually tuned parameters, and detection errors (e.g., missed objects) may propagate and degrade performance. Could error-correction mechanisms (e.g., confidence-based dynamic adjustment, error propagation suppression) be introduced to reduce human intervention and improve robustness?\n\nLIMITATIONS:\nThis work appears to have minimal potential for negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Visual Semantic-Logical Search (VSLS), a novel framework that can efficiently search key frames in long videos based on the logical relationships between queries and visual elements. VSLS first decompose the input query into 4 types of pre-defined semantic logic relations. It then samples frames and check if the objects in sampled frames satisfies the 4 pre-defined object relations. Based on the relevance score, VSLS iteratively samples and updates query relevant frames. The authors evaluate VSLS in LV-HayStack, LongVideoBench and Video-MME. VSLS shows strong performance, outperforming multiple baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The writing is clear and easy to follow.\n- Based on the T* framework, when scoring the frames, the authors propose to model higher-order object relations instead of only focusing on the existence of objects. The motivation is clear and makes sense.\n- The experiments are extensive and the performance of the proposed method is strong. VSLS outperforms the T* baseline on LV-HayStack, LongVideoBench and Video-MME with similar efficiency.\n\nWeaknesses:\n\n- On Video-MME, which has more diverse question types than LongVideoBench and LV-HayStack, the proposed method leads to marginal improvements or sometimes even worse performance, especially for GPT-4o. In addition, in Table 3, the authors only report GPT-4o performance under the 32 frames setting. This is inconsistent with the 8 frame setting, in which both GPT-4o and InternVL are reported. These results raise concern about the generalizability of the proposed method.\n- The proposed method relies on LLM/VLM to extract potential logical relations. However, it is unclear to me whether the LLM/VLM can accurately extract the logical relations.\n- What are the benefits of stacking multiple frames into one image as multiple grids? Although it is more efficient, it might also leads to worse object detection performance.\n- What are the effects of multiple iterations? From Figure 3, it seems that iteration 1 is already good enough. More iterations seem to introduce more noise on frame selection.\n- In Table 4, GPT-4O + VSLS (ours) 32 frames setting, 54.2 is not the best performance but is bolded.\n\nQUESTIONS:\nPlease see the above ‘Weaknesses’ section. If the authors could provide more analysis of the generalizability of the propose method, the reviewer will consider raising the score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}