Title: iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: video understanding

driving video analysis
Abstract: Grounding large language models (LLMs) in domain-specific tasks like post-hoc dash-cam driving video analysis is challenging due to their general-purpose training and lack of structured inductive biases. As vision is often the sole modality available for such analysis (i.e., no LiDAR, GPS, etc.), existing video-based vision-language models (V-VLMs) struggle with spatial reasoning, causal inference, and explainability of events in the input video. To this end, we introduce iFinder, a structured semantic grounding framework that decouples perception from reasoning by translating dash-cam videos into a hierarchical, interpretable data structure for LLMs. iFinder operates as a modular, training-free pipeline that employs pretrained vision models to extract critical cues—object pose, lane positions, and object trajectories—which are hierarchically organized into frame- and video-level structures. Combined with a three-block prompting strategy, it enables step-wise, grounded reasoning for the LLM to refine a peer V-VLM's outputs and provide accurate reasoning.
Evaluations on four public dash-cam video benchmarks show that iFinder's proposed grounding with domain-specific cues—especially object orientation and global context—significantly outperforms end-to-end V-VLMs on four zero-shot driving benchmarks, with up to 39% gains in accident reasoning accuracy. By grounding LLMs with driving domain-specific representations, iFinder offers a zero-shot, interpretable, and reliable alternative to end-to-end V-VLMs for post-hoc driving video understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a structured semantic grounding framework, iFinder, to improve zero-shot vision-based reasoning of dash-cam driving videos by utilizing the reasoning capabilities of large language models. Instead of depending on end-to-end vision-language models (that are capable of video processing), this framework separates perception from reasoning. It uses pretrained vision models to extract driving-related information, such as object detection, lane position, and depth estimation. The framework then organizes the driving-related information and feeds this information to an LLM via structured prompts. This approach enables step-wise, grounded reasoning, leading to more accurate and interpretable driving scenario analyses than conventional VLMs, particularly in accident causation and hazard detection.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper provides structured, symbolic cues (such as object orientation, lane context, and object distance) to the LLM, enabling transparent reasoning, making the analysis process more interpretable and trustworthy for complex driving scenarios.\n- Sufficient empirical analysis shows that the framework significantly outperforms both general-purpose and driving-specific VLMs on multiple zero-shot driving benchmarks.\n- The framework uses a modular design based on pretrained vision models, which allows easy integration and parallelization of new perception modules without retraining.\n\n**Weaknesses**\n- The framework heavily relies on the quality of the existing deep learning models.\n- The framework cannot provide any guarantees for their outcomes.\n\n*Note to the authors*: More details of the weaknesses are in **Questions** and **Limitations**, so please respond to those sections ONLY during the rebuttal.\n\nQUESTIONS:\nQ1. How do you determine the prompts? I.e., how do you know the current prompts are \"optimal\"? Are there any ablation studies with varied prompts?\n\nQ2. How does the choice of the LLM (in Fig. 2) affect the performance?\n\nQ3. Can this framework be generalized to other domains? Or, will the framework still work if I remove a subset of modules (e.g., remove 2D object detection + tracking and keep the remaining modules)? What if I add more modules (e.g., global latitude & longitude)?\n\nQ4. A follow-up for Q3: Do you need to refine the prompts or retrain the LLM when adding or removing modules?\n\nQ5: Is this framework adaptable to multi-view cameras? E.g., self-driving cars have six cameras at different angles.\n\nLIMITATIONS:\nL1. The authors mention that real-time deployment is not a concern of this work. However, the authors could have a discussion on the applications of this framework beyond real-time deployment. I'm not clear where this framework can be applied except for real-time deployment.\n\nL2. While the paper includes some analysis on error propagation (showing graceful degradation when object detection confidence is low), it would be valuable to see a deeper discussion about how sensitive the entire pipeline is to failures or inaccuracies in upstream modules (e.g., tracking, depth estimation). How does ı̇Finder handle compounding errors, and are there mechanisms for uncertainty quantification or correction? Is there a way to provide probabilistic guarantees to the framework's outcomes?\n\nL3. The framework's performance heavily relies on the accuracy of multiple off-the-shelf vision models (e.g., for detection, tracking, lane estimation, depth, etc.). The overall reasoning can degrade if these modules perform poorly due to unusual conditions or domain shift.\n\nL4. The paper lacks a discussion of limitations and future work in the main body.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nı̇Finder is a zero-shot framework that grounds LLMs for driving video analysis by decoupling perception and reasoning. It converts dash-cam videos into structured, interpretable representations, capturing object motion, lane position, and orientation, etc. It also uses them to guide LLMs in accurate, causal reasoning, outperforming baselines on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper is well-written and easy to follow.\n2. This method requires no training and has good interpretability. \n3. This method achieves competitive results among several benchmarks.\n\nWeakness\n1. My main concern is that the paper’s contribution is not clearly defined or significant. There is no explicit contribution statement, and the introduction only outlines three experimental findings without a clear methodological or conceptual innovation. This makes it difficult to assess the novelty and impact of the work.\n2. This paper introduces seven additional intermediate processing steps for dash-cam video reasoning, but nearly all of them rely on separate off-the-shelf models to extract specific cues. While the pipeline is training-free, the use of multiple models makes the system quite heavy. I am curious about the overall memory usage during inference, as excessive memory demands could make the approach impractical for real-time or resource-constrained driving scenarios.\n3. From Lines 252 to 255, the paper states that real-time performance is not the focus, emphasizing interpretability and accuracy instead. However, in driving scenarios, inference efficiency is just as critical as accuracy. The proposed method relies on seven intermediate modules and a peer V-VLM for final LLM processing, which raises concerns about latency and scalability. If the improved accuracy comes at the cost of significantly slower inference, the practical value of the approach becomes questionable. At least, the paper should report and compare inference time with existing baselines. Additionally, it would strengthen the work to propose a possible direction for improving efficiency based on the current design.\n4. In Tables 1–3, the driving-specialized baselines (DriveMM and WiseAD) perform significantly worse than generalist models. Notably, DriveMM achieves only 49% accuracy in Table 3, which is worse than random guessing (50%). This raises concerns about whether these methods are truly strong baselines or representative of the state-of-the-art driving-specialized  methods in driving video understanding.\n\nQUESTIONS:\n1. What is the contributions of this paper?\n2. What is the inferencing time and memory usage during inference?\n3. What is the potential direction to improve the efficiency based on the current design?\n4. Why driving-specialized baselines performance pretty bad?\n\nLIMITATIONS:\nI think the largest limitation of the method is the inference latency, which is not mentioned in the limitation in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents iFinder, a framework for zero-shot post-hoc reasoning on dash-cam videos. It decouples perception and reasoning by converting raw video into structured, hierarchical data (including object orientation, lane position, and ego motion), which is then passed to a general-purpose LLM. Such an approach enables interpretable, grounded reasoning and significantly improves performance over existing V-VLMs on 4 benchmarks. Despite its strong results, the possible use of the proposed framework is most likely limited \"just\" to offline analysis.\n\nSTRENGTHS AND WEAKNESSES:\n**Note**: _Despite I'm a little bit \"negative\" in the review, I believe this is a strong and well-written paper. The design, structured reasoning approach, and strong zero-shot results make it an extremely valuable contribution. While there are areas for improvement, I view these more as opportunities for future work than fundamental flaws._\n\n### **Strengths**\n- **STRUCTURE and LANGUAGE**: The paper follows a standard structure, which makes it easier to follow and read. Overall, the paper is well-written. The level of English is great. Figures are of good quality and appropriately support the clarity. Formally, everything is on a high level.\n\n- **REPRODUCIBILITY**: Although the code is not provided, the paper and supplementary materials present outstanding implementation details, making it potentially \"easy\" to reproduce. Despite the large complexity and inherently non-deterministic nature of some LLMs, I believe there is no better way how to do it.\n\n- **METHOD DESIGN**: The core idea is well-designed and implemented. The framework is modular, does not require additional fine-tuning/training, and introduces a principled way to integrate domain knowledge into LLM reasoning via hierarchical scene representations and well-thought-out prompting.\n\n- **STRONG PERFORMANCE**: The method achieves consistent and significant improvements across four benchmarks, outperforming both general-purpose and domain-specific video-language models in zero-shot settings. Notably, the \"performance\" gains are much higher in tasks requiring spatial and causal reasoning.\n\n- **RELEVANCE**: The paper addresses a highly relevant problem in vision-language understanding for not just autonomous systems. Post-hoc reasoning from dash-cam videos is a challenging yet increasingly important task, especially given the limitations of real-time data collection in real-world driving environments.\n\n### **Weaknesses**\n\n- **SYSTEM COMPLEXITY**: The framework relies on many pretrained models (e.g., for object detection, depth estimation, 3D orientation, lane detection, ego-motion), which increases complexity and potential points of failure. Besides, the paper does not provide any system-level runtime or memory analysis, making it more challenging to assess practical deployment costs.\n\n- **RELIANCE ON RULE-BASED STRUCTURE**: The scene representation is handcrafted and domain-specific. While effective for driving, it’s unclear how easily this structure transfers to other domains (e.g., indoor navigation, robotics), or how it would perform under noisy or ambiguous visual inputs.\n\n- **UNCLEAR PEER CONTRIBUTION**: The peer V-VLM is treated as a noisy first-pass guess, but its actual contribution to the final performance is not deeply analyzed. A deeper ablation (e.g., replacing the peer) would clarify whether it meaningfully supports the reasoning or serves as a distractor.\n\n\n**[Additional Comments]:**\n- The Related Work section is rather short. While key methods are covered,  recent works on structured reasoning and OOD evaluation [1, 2, 3], particularly those addressing zero-shot hazard identification under domain shift, are missing, while it could be beneficial to include them. Evaluating iFinder on such a dataset/benchmark would significantly strengthen the paper’s narrative around robustness and generalization. Given the method’s zero-shot, modular nature, this would be a natural and valuable extension.\n- The qualitative examples are useful and illustrate the reasoning capabilities well. However, it would be valuable to also include some failure cases with a brief analysis, especially where the LLM misinterprets the structured input or the visual modules produce conflicting cues. \n- The authors apparently misunderstand what is expected regarding statistical significance. Reporting results from a single seed is insufficient, even if followed by a qualitative error analysis. Statistical significance typically requires multiple runs with different random seeds and corresponding variance or confidence intervals. The current justification (\"we provide an error analysis in the Supplementary Material\") does not address it. Although I understand the computation cost is rather heavy, a few repeated runs with 2–3 random seeds would make the results more trustworthy and support stronger conclusions.\n\n**[Grammar & Typos & Formating]**:\n- Line 240: The JSON example is overflowing the line width.\n- Table 1-7: The random verticall allignement does not look apealing. While minor, it feels like a huge ink stain on an otherwise good-looking and polished submission.\n\n___________________________________\n\n[1] @article{alshami2024coool,\n  title={Coool: Challenge of out-of-label a novel benchmark for autonomous driving},\n  author={AlShami, Ali K and Kalita, Ananya and Rabinowitz, Ryan and Lam, Khang and Bezbarua, Rishabh and Boult, Terrance and Kalita, Jugal},\n  journal={arXiv preprint arXiv:2412.05462},\n  year={2024}\n}\n\n[2] @inproceedings{picek2025zero,\n  title={Zero-shot hazard identification in Autonomous Driving: A Case Study on the COOOL Benchmark},\n  author={Picek, Lukas and Cermak, Vojtech and Hanzl, Marek},\n  booktitle={Proceedings of the Winter Conference on Applications of Computer Vision},\n  pages={654--663},\n  year={2025}\n}\n\n[3] @article{shriram2025towards,\n  title={Towards a multi-agent vision-language system for zero-shot novel hazardous object detection for autonomous driving safety},\n  author={Shriram, Shashank and Perisetla, Srinivasa and Keskar, Aryan and Krishnaswamy, Harsha and Bossen, Tonko Emil Westerhof and M{\\o}gelmose, Andreas and Greer, Ross},\n  journal={arXiv preprint arXiv:2504.13399},\n  year={2025}\n}\n\nQUESTIONS:\n* _Q1:_ Could the authors provide rough estimates of pipeline inference time and resource requirements? Even _coarse_ timing would be useful.\n* _Q2:_ Is there a plan to extend ı̇Finder toward semi-real-time scenarios (e.g., batch processing at the edge)? If not, what would be the major technical problem?\n* _Q3:_ How adaptable is the pipeline to other domains beyond driving? Could the same framework be applied to indoor navigation or surveillance video with minor changes?\n* _Q4:_ Would the authors consider evaluating ı̇Finder on more challenging or domain-shifted settings such as those proposed in [1]? These benchmarks focus on out-of-distribution hazard identification and symbolic reasoning, and would align well with ı̇Finder’s strengths.\n* _Q5:_ Since many LLMs are inherently non-deterministic, could the authors clarify whether multiple runs of the reasoning stage produce consistent outputs? Given the structured symbolic input and multi-step prompt design, some variability is expected, but it would be helpful to know if the system’s answers are stable in practice. Did the authors use any measures (e.g., fixed temperature, response filtering, or prompt anchoring) to manage output variability?\n\nLIMITATIONS:\nMostly yes. The authors acknowledge the offline nature of iFinder and briefly mention its lack of real-time deployment support. However, several additional limitations deserve discussion. \n1. The system relies on many pre-trained models, each of which may introduce noise, yet, no formal analysis of this sensitivity is provided. \n2. The scene representation is handcrafted, which limits the method’s generalization. \n3. While the system claims interpretability and robustness, no formal evaluation or consistency analysis is conducted to support it.\n4. The results are most likely based on single-seed evaluation without statistical significance testing", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ı̇Finder, a training-free pipeline for post-hoc reasoning over dash-cam videos. It decouples perception from language reasoning by (i) extracting driving-specific cues—object pose, lane location, distances, ego motion, 3D orientation and scene context—using off-the-shelf vision models, (ii) packaging those cues into a hierarchical JSON structure, and (iii) feeding the structure, together with a peer VLM’s first-pass answer, to a GPT-4-class LLM via a three-block prompt (key explanations, step instructions, peer instruction). On four zero-shot driving benchmarks (MM-AU, SUTD-TrafficQA, LingoQA, Nexar), experimental results show that ı̇Finder outperforms both general-purpose Video-VLMs and driving-specialised models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- It has solid engineering for all vision modules and connect them successfully with GPT-4o.   \n\n- The paper has rigorous zero-shot evaluation on 4 public ego-view driving benchmarks. Each of them contain problems that current VLMs can not resolve alone.    \n\n- The paper combines peer-LLM self-correction with structured visual grounding—a fresh twist on “LLM-agent” ideas for autonomous driving.     \n\n**Weaknesses**\n\n- Real-time latency/cost of running eight vision modules is not analysed. This might be a big problem if one or more module take a long time during inference. Besides, It relies exclusively on existing pretrained components. What if some module make mistakes? This is not discuss in the ablation study.   \n\n- It seems that gains come largely from adding extra model-derived priors (3D orientation, depth) rather than novel learning; may be less compelling once future VideoVLMs close the gap.\n\nQUESTIONS:\n1. What is the end-to-end runtime of ı̇Finder per 5s/10s/30s/60s video? A breakdown would clarify deployment feasibility.\n\n2. If stronger/weaker object trackers or depth estimators are substituted, do the gains persist? Ablation study can be conducted by using weaker object trackers and check their performance with iFinder.     \n\n3. Discuss two weaknesses.\n\nLIMITATIONS:\nThey discussed it in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-08"}