iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
Authors: Manyi Yao
Bingbing Zhuang
Sparsh Garg
Amit Roy-Chowdhury
Christian R. Shelton
Manmohan Chandraker
Abhishek Aich
Venue: NeurIPS 2025 poster ()
Keywords: video understanding
driving video analysis
Primary area: applications

Abstract: Grounding large language models (LLMs) in domain-specific tasks like post-hoc dash-cam driving video analysis is challenging due to their general-purpose training and lack of structured inductive biases. As vision is often the sole modality available for such analysis (i.e., no LiDAR, GPS, etc.), existing video-based vision-language models (V-VLMs) struggle with spatial reasoning, causal inference, and explainability of events in the input video. To this end, we introduce iFinder, a structured semantic grounding framework that decouples perception from reasoning by translating dash-cam videos into a hierarchical, interpretable data structure for LLMs. iFinder operates as a modular, training-free pipeline that employs pretrained vision models to extract critical cues—object pose, lane positions, and object trajectories—which are hierarchically organized into frame- and video-level structures. Combined with a three-block prompting strategy, it enables step-wise, grounded reasoning for the LLM to refine a peer V-VLM's outputs and provide accurate reasoning.
Evaluations on four public dash-cam video benchmarks show that iFinder's proposed grounding with domain-specific cues—especially object orientation and global context—significantly outperforms end-to-end V-VLMs on four zero-shot driving benchmarks, with up to 39% gains in accident reasoning accuracy. By grounding LLMs with driving domain-specific representations, iFinder offers a zero-shot, interpretable, and reliable alternative to end-to-end V-VLMs for post-hoc driving video understanding.

iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
Manyi Yao", Bingbing Zhuang', Sparsh Garg', Amit Boy-Chowdhury', Christian Shelton', Manmohan Chandraker Abhishek Aich#
NEC Laboratories, America, TUniversity of California, Riverside, *University of California, San Diego
Abstract
Grounding large language models (LLMs) in fomain-speciffe tasks like post-hoc dash-cam driving video analysis is challenging due to their general-purpose train- ing and lack of structured inductive biases. As vision is often the sole modal- ity available for such analysis (i.e, no LiDAR, GPS, etc.), existing rideo-based vision-language models (V-VLMss) struggle with spatial reasoning, causal infer- ence, and explainability of events in the input video. To this end, we introduce iFinder, structured semantic grounding framework that decouples perception from reasoning by translating dash-cam videos into a hierarchical, interpretable data structure for LLMs. iFinder operates that employs pretrained vision models to extract critical cues cues-object pose, lane positions, and object rrajectories-hiie are hierarchically organized into frame- and video-level structures. Combined with a three-block prompting strategy, it enables step-wise, grounded reasoning for the L.M to refine peer V-VI.M's outputs and provide accurate reasoning. Evaluations on four public dash-cam video benchmarks show that FFinder's proposed grounding with domain-specific cues-espeyially object orientation and global context- signixxiiign outperforms end-to-end V-VLMs on four zero-shot driving benchmarks, with up to 39% gains in accident reasoning accuracy. By grounding LLMs with driving domain-specific representations, iFinder offers a shot, interpretable, and reliable alternative to end-to-end V-VLMs for post-hoc driving video understanding.
modular, training-fre freee pipeline
1 Introduction
Grounding large language models (LLMs) to domain-specific requirements remains a significant challenge due to their general-purposs training and lack of inductive bias toward structured domain knowledge [1]. LLMS are often pretrained on broad internet data, which may lead imprecise outputs when applied to specialized domains such as driving video understanding [2]. Furthermore, the lack of interpretability and limited mechanisms for precise reasoning make it difficult to guar- antee reliability in high-stakes applications [3]. In this paper, we tackle the problem of generating causally and spatially grounded LLM responses to user queries, given front-view dash-cam video. The objective is to produce explanations that accurately reflect the underlying events in the scene.
The proliferation of advanced driver assistance systems has led an abundance of video data, often serving as the primary or sole modality for system analysis and validation. In many consumer-gran grade and fleet-level applications, supplementary sensors like LiDAR, GPS, or CAN bus data are either unavailable or impractical to collect due to cost and integration complexities [4, 5]. Consequently, extracting meaningful insights from visual spatio-temporal data becomes crucial for assessing sys- tem responses, diagnosing failures, and refining vehicle performance [6-11]. However, developing video analysis systems that generate grounded and reliable responses solely from camera input re- mains formidable challenge, without external supervision [12, 13).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VideoLLaMMA2: Them to so socdent in the vidoo. Horomer, tleee ure seceral tnstarocs caes steppiig rol Egltts and valling for then to turs gren before continiing on therr way.
VidkoLLaYA The
the axsideat
froat liee subllaii stups, collide with the foott of
VideoCluu2::
The case accidauti whitc car's stop
DfiveM
DriveMMM Tacre is no seciles the vidos. The italle is flowing secocialy with vetictes malte aiiinainin seio distance from each other.
Q: cuuse of the occiden?? Ground Truth: The car does give way normal driving vehicles when lurning or changing lanes.
FFiuderr A white is right aikeapped the lane, 13 moriig from 9.70 (Frane 26 to 6.1im (Franz 40), with rokar -21. indieaiig kehwaed
Cbyyet 1D
Figure I: Advantages of iFinder. Baselines VideoLLaMA2 [14], VideoLLaVA [15], and DriveMM [16] struggle with spatial reasoning, and fine-grained scene understanding, misinterpreting critical cues. iFinder's structured scene approach mitigates these errors for more accurate responses.
A seemingly straightforward solution to address the above problem is to develop end-to-end video- based Vision-Language Models (V-VLMs) [14-16]. However, although existing V-VLMs generate reasonable responses, they exhibit limitations in spatial reasoning, causal inference, and fine-graine g--airdded scene understanding [17-19] shown in Figure 1. This means misinterppetaatonn of critical visual cues can lead to incorrect conclusions about hazards, traffic signals, or object presence. Furthermore, incorporating new functionality without requiring extensive retraining or one-tuning is diffficult with VLMs [20]. To address these limitations, we build on the principle that perception should be decoupled from LLM reasoning. In particular, we propose FFideer, vision-language pipeline that extracts driving domain-relevant visual cues and passes them to the LLM via structured prompts, enabling post-hoc scenario understanding through symbolic, temporally grounded reasoning. This structured scene representations encode dynamic object pose, orientation, and semantic lane context in hierarchical format, enabling symbolic reasoning over frame-indexed data. iFinder leverages pre-trained vision models to extract domain relevantt cues and organizes them into hicrarchical data format. This structured input enables the LLM to reason accurately about driving scenarios, correcting or augmenting generic V-VLM outputs with grounded, verifiable evidence tailored to the driving video context. For example, in Figure we can observe that unlike baselines [14-16, 21] that provide incorrect or generic explanations, iFinder correctly identifies the white car's lane- cutting maneuver, aligning with the ground truth. It leverages the data structure of the input video and uses object tracking ('Object ID 13"), distance change ("9.78m - 6. 6m14m',, " and orientation ('rotyy -2.1 to indicate left turn') to draw conclusions. Our approach not only mitigates the limitations of end-to-end V-VLMs but also improves reliability and transparency.
The complete iFinder pipeline is as follows. The process begins with input video frames that are first undistorted to correct lens distortion. These frames are then processed by suite of pre- trained vision modules that extract critical driving cues: scene context, ego-vehicle motion, 2D/3D object detections, object tracking, lane assignments, object distances, and semantic attributes. This information is hierarchically structured into video-level (e.g., global context, ego state, peer VLM response) and frame-level (e.g., object properties per frame) representations. Simultaneously. peer V-VLM provides an initial answer to the user query, which may contain inaccuracies. The final reasoning is handled by the LLM, which is prompted using three components- instruction, step-by-step reasoning guidance, and key explanations of the structured inputs. Combining the peer's response with structured, domain-grounded visual cues enables the LLM to produce accurate and interpretable answers for complex driving event scenarios. Through rigorous analysis on four public benchmarks, provide three new insights to the community:
Improved Performance via Structured Grounding. By decoupled perception and LLM rea- soning, and grounding LLMs in hierarchical, interpretable video representations, iFinder beats both generalist and driving-specific VLMs, without any fine-tuning. iFinder, for example, in the accident reasoning dataset MM-AU [22], beats the best performing generalist V-VLM by 10.5% and atriving-specializee V-VLM by 39.17%.
Enhanced Explainability through Explicit Reasoning. Unlike end-to-end VLMS that rely on implicit cues, iFinde provides symbolic cues-such as object orientation, lane context, and dis- allows the LLM to generate transparent, verifiable explanations, as shown in Figure 5.

---

Object-orientation and Global Environmental Dominate. Surprisingly, our ablation studies in Table 5 reveal that object-orientation and global environmental context contribute more to reliable post-hoc reasoning than other factors like distance and lane location. For instance, removing object orientation information led a significant drop in reasoning accuracy (~4.5%), while removing distance or lane location had comparatively minor effect (~22.7%).
Related Works
Post-hoc Driving Video Analysis. Post-hoc driving video analysis for front-cam setting (rather than surveillance-cam setting, such as in traffic intersections [23]) has recently become strong research interest. Prior works on driving video analysis [22, 24] have been dominantly focused on accidents and their possible prevention analysis. Different from these, [25] provides detailed benchmark in which models can be tested on their understanding of the ego-vehicle's 's perspective based on their understanding of dynamic scenes, rather than merely their ability describe the video event. Recent work has explored zero-shot hazard identification and out-of-distribution challenges, with [26] introducing the COOOL benchmark for evaluating autonomous driving models on out-of- label objects. [27] demonstrating zero-shot hazard identification approaches on this benchmark, and [28] proposing multi-agent vision-language systems for detecting novel hazardous objects. In this work, unlike prior driving VLMs [16, 18], we develop a lasoun-anguuage pipeline that solely focuses on such offline driving video analysis.
Video Foundational Models for Driving Videos. The success of the image-bases VLSS has sparked a growing interest in the video-based VLMs within the research community. In the video- language domain, the prevailing approach now involves large-scale vidco-text pre-training followed by fine-tuning for specific tasks [15, 21, 29-48]. Within the driving video domain, multiple image- language models have been introduced [49-59] for various driving-related tasks. Although these methods perform well, their effectiveness is restricted to specific scenarios (Bird's-Eye L--Eye-View-tee View-based representations, multi-view representations, etc.) and driving-specific objectives that dominantly do not aid offline video analysis tasks. For video-domain driving specific foundational models, meth- ods like [16, 18, 60] have been introduced. However, they lack the capability to process different driving video with fine-grained analysis like ego-car attributes. Moreover, these VI.Ms are designed for tasks to represent in-moment vehicle driver, rather than video analysis.
3 Method
Motivation. We argue that grounding general-purpose LLMs using structured, interpretable scene representations offers more accurate alternative than V-VLMs. Further, decoupling perception from reasoning enables a more reliable analysis system. Hence, we introduce modular framework that grounds general-purpose purpose LLMs using structured, interpretable representations derived from pre- trained vision modules. These modules extract critical scene attributes- such as object pose, lane semantics, and motion cues cucs-which are hicrarchically organized into a -opeii- data struc- ture. This explicit grounding enables the LLM to reason with verifiable, context-aware inputs rather than relying solely on implicit visual cues. Before detailing each module, we summarize the full Finddr pipeline: raw dash-cam video frames are first undistorted and processed through pretrained vision models to extract scene-level and object-level cues. These are hierarchically organized into a structured data format and passed- along with peer V-VLM's output-intt a three-block prompt that guides a general-purpose LLM to generate post-hoc reasoning responses.
3.1 Proposed Framework
FFinder pipeline. Our pipeline is shown in Figure 2. Let front-view dash-cam video be repre-
sented as a sequence of T frames as V= (Ian, Taz, Iąr), where each frame Iat R l is the 1-th distorted or unrectified image of height H and width W. We define a unified function F that maps input video V to structured data DD for grounding the L.M..
F T + D
(I)
space of front-view dash-cam vidcos
3

---

Peer V-LLM
no our opposite direction.
D pose
any uppusile direction?
input video
sceme understanding
Ducaen Desdoo
A: Yés, there a thfiee stiam the the go lone.
ego-slate estimation
Undistortion
Dantionn Dear
LLM
object detection tracking
busi
Q
3D object
peer instruction
Video-Iave] -Infomation:
undistorted
attribute estimation
surrounding-foe Daern
frames
ego-vebicla difo veb-ccl---sff: Daota: Daan
steps instruction
uideo uideo-descriptionn descrsption Darm
estimation
Prums IIformation::
key explanation
frace, redes detected, objects:
(th Ru. den Ae Auga hoa
proposed prompt
lane detection
Au.i Auago
Figure 2: iFinder overview. The proposed pipeline transforms key scene properties such as object detection, lane detection, depth estimation, and ego-state estimation, into structured data, which, combined with peer-generated insights, enables the LLM to perform accurate and interpretable driv- ing scenario analysis.
Finally, the structured data 'D is passed to an LLM, denoted FiLMM. along with a prompt PLIM. generate the final response R the user's query. Next, we describe the function F that aims to extract D in an LLM-friendly manner. In practice, FF is decomposed into multiple specialized modules: 2D/3D object detection, lane localization, distance estimation. attribute estimation, ego- state estimation, and scene understanding. Each of the modules is instantiated with popular pre- trained open-souree models, requiring no fine-tuning when used within iFinder's pipeline.
3.1.1 Visual Information Extraction
Step 1: Undistorting frames. Front-view dash-cam videos often exhibit distortions due to the characteristics of front-facing cameras [61]. To address this and ensure optimal performance for perception models to follow, üFinder begins by correcting these distortions through the estimation of camera intrinsics K (focal length and principal point), and distortion coefficients, including radial distortion [ki, kng kg] and tangential distortion [P1, P2] coefficients. To correct for lens distortion, we compute a mapping function that transforms coordinates in the rectified (undistorted) image I to corresponding coordinates in the distorted image Ia. The rectified image is obtained by:
I [(z, y) Ia (x,
where R (x, y) computes the corresponding distorted coordinates for each undistorted pixel loca- tion (I, y) using the estimated camera intrinsic matrix and distortion coefficients.
Step Scene understanding. This step captures general environment details, such as weather con- ditions, road structure, and whether it is daytime or nighttime. We also extract a caption describing the events in the video.
Inuitiion. High-level scene understanding. including weather, traffic conditions, and time of day, along with video event descriptions, helps the LLM reason about vehicle movements, pedestrian actions, and traffic interactions.
Method. To extract this surrounding environment information D leverage image-based VLM Ji VLM to enable a precise and reliable interpretation of the video. Now, image-based VLMS lack the ability to capture the evolution of events time. To capture temporal dynamics, V-VLM FVVLLM is used to generate a detailed event description Dvideo-
FRVMM (V,Pr) D F (V,Pv) Dridso
(3)

---

Step 3: Ego-vehicle state estimation. This step captures the ego-vehicle's motion (mov- ing/stopped) and turn (left/righ//ttriight) action.
Intuition. Understanding the ego-vehicle's 's motion is crucial for reasoning in front-view dash-cam videos where the vehicle's perspective defines the driving scene. Given that we using front-view dash-cam videos, estimating the camera pose will directly provide the ego-vehicle's motion pattern in the input video. However, these numerical pose outputs are not inherently interpretable by an LLM. To bridge this gap, we transform the raw pose data into human-interpretable driving states by estimating the vehicle's 's turning behavior and motion status.
Method. We use a camera-pose estimation model caam-pose to map video frames V to sequence of translation vectors (Ti) where T. (X,,Y Yr, Ze) E R3 denotes the camera position at time t. (A) Vehicle turning estimation. With (X,, Vi E {1,... ,T), we estimate the heading angle of the camera A0; as
AQ, = tan (Z..1 2//X x.) - tan -1 (2. Ze 2e-1/Xi XI 1)
(4)
Next, Ad, is used to classify the vehicle's 's turn into the three categories (Fa as a threshold) as
D Duum "Straight" if A0,] Tax "Right Turn" if 20, Ta, else "Left Turn"
(5)
(B) Vehicle motion estimation. We compute the vehicle's motion over temporal window g using
= 112t,, Tell/a, V (1, ... T -g
(6)
where 8t denotes the approximate speed at time 6, used to classify the vehicle's motion state as
Dmotion "Stopped if s; Ts. else "Moving"
where Ta represents the speed threshold for detecting stopped vehicle. By incorporating both turning and motion status, we structure the ego-vehicle state as
Fego : (T),T + (Diotiion: Dum)
(8)
While video-level cues provide essential global understanding, many critical driving events, such as pedestrian crossings, vehicle interactions, and traffic signal changes, occur at the frame level. To fully comprehend the driving scenario, we extract frame-level information in Step 4-7, ensuring that the model can reason about both long-term motion trends and momentary scene dynamics.
Step 4: 2D Object detection and tracking. This step captures and tracks the objects in the video.
Intuition. To accurately analyze dynamic interactions of objects with the ego-vehicle in driving scene, necessary to not only detect objects in individual frames but also track them over time using unique identity. Furthermore, this step provides the necessary foundation for identifying attributes the objects, such as their lane location, distance, and attributes.
Method, For each video frame It, we apply a 2D object detection model F2D-der F2D-der: (4t)
( Here, be, = (Tmin: Vmin Tmax Mmaxx) Rdd is the bounding box and Cr is the class label for ith object. n4 is the number of objects detected in frame t. Then, we add a tracker on the detections in order to assign unique ID Yt to each detected object B, = (br,s, Ct,a) using (B,,K, + K,. Finally, for each video frame
multi-object tracking model FaD as F2D rack mackkckk
I, this step provides
des (I,) - {Te,s br,i, Ct. )
Step 5: Object lane location. This step assigns the lane location of the object in the scene.
Intuition. The ego-vehisle's driving decisions are influenced by the lane positions of surrounding objects. Therefore, it is crucial that the data structure DD encodes lane information for each detected object, particularly for vehicles and pedestrians. Similar to Step the numerical outputs of lane detection models are not inherently interpretable by an LLM. To bridge this gap, we transform the raw lane marking data into human- interpraaleee states by estimating the vehicle's lane location. Method. Once the objects are detected from Step 4. we perform the lane assignment as follows. We use lane detection model Frame and first obtain the predicted lane markings in each frame as Frane (T)) (I,) > (4. Here, 4.j represents the set of j-th lane marking coordinates, and me the total number of detected lane markings. Next, we divide the road into m, number of lane
sections formed by the lane markings. Each lane section is now defined as
8t.k = (F.,y) v) 4i,, S x S 34,511 y = [Umax, H])
(10)
where Fi,, is the c-corrdinate of the k-th lane marking, Ymax, = min, M. the highest point of all lane markings (assuming image coordinates have the origin at the tef-leff).
5

---

imput
output
input
output
Sclesst: Toart 484
9201,
bag
21 meleess
mean distance
éistance eslinatio
detector
tracker
Fq (main)
2D detector + tracker
segrenniation model
lane model
depth
vehidle rnfermnce
Figure 3: Lane location estimation. Detected objects are assigned lane by mapping the bot- tom midpoint of the corresponding bounding box (bottom middle point of image for ego) to sections identified by the lane detection model.
Figure 4: Distance estimation. Each object's distance is determined by averaging the depth values within its segmented region.
(A) Object lane estimation. For each ith object in frame t, we compute the midpoint Pr, of its bounding box bottom edge as Pt,i = (Zmin + Smaxx Umax). Then, its lane λe, is estimated as
Ae, = such that Pt.: E St.k
(11)
We estimate the ego-vchicle's lane Ax,ogo using the bottom-center pixel Pr.ego (W/2, H) as reference. The resulting lane data is then added to D for each frame.
Fiane (J,, K.) - :
(12)
Step 6: Object distance estimation. This step estimates the distance of the objects in the scene w.r.t. the eeo-vel from the video.
Intuition. Distance-aware awareness of each object will allow the LLM to analyze collisions, ego-vehicle navigation, and object interaction.
Method. Given input frame It, depth estimation model Faepth predicts metric depth map Faepth (I) Dr where, D, E H is the estimated depth map for frame t. For ith object's bounding box b = (Xmins Drain Tmax: Umaxx), ), the cropped depth region corresponding to the object Dr.i D2 [Tmin mmax: Vuui (umxx] Next, make the region of the object
precise and eliminate any background pixel, a segmentation model Frey predicts a binary mask M,,s for the object within b,, The final distance de, of the object from the eeo-veh is computed the mean distance of the masked region. de,i mean D., 3 M,,i). where @ denotes the element-wise multiplication. Finally, the distance information per object per frame is added D follows.
Faist (I,, K.) )
(13)
qualitative illustration of Step 5 and Step is shown in Figure and Figure 4, respectively.
Step 7:
Object attributes. This step generates object attributes like color to help the LLM distin- guish the objects in an interpretable manner.
Intuition. Object attributes enhance perception with human-like reasoning, useful for scene inter- pretation and understanding high-level secision-making.
Method. With by, (mmin: Vmin Tmaxx Fmaxx.. ) of vehicles, traffic light color, etc. using prompt Pa. See Supplementary Material for details on Pa-
Fi-YLM extract object attributes e.g., color
FIVLM
(Le[#min Tmaxs Mwin ymax], Pa) - 1
(14)
Step 8: 3D detection for object orientation. This step captures the object orientation from the 3D information of objects in the scene w.r.t. the ego-vehicle.
Intuition. From Step 4-8, all extracted information was from a 2D perspective. However, a critical aspect of the objects missing their orientation as per the ego-vehicle's 's view.
Method. Using 3D detection model FsD-det we predict p. 3D bounding boxes and extract the yaw
O,, E [-TT, π] for each object. as FaD-det (I:) + {Or, } Next, we project each 3D bounding box into 2D image space using the camera intrinsie matrix K from Step 1. We then apply the Hungarian algorithm [62] to match projected boxes with detected objects and transfer 0z, to the corresponding local object.

---

3.1.2 Proposed Data Structure and Prompt
Incorporating Peer V-VLM Reasoning, While structured visual information provides strong foundation for precise reasoning. we also incorporate a The peer V-VLM serves two complementary purposes: (I) it provides an initial, high-level response to the user query based on raw visual input, and (2) it exposes limitations in generic models leading to incorrect explanations. To this end, we query the peer V-VLM using the original video V and extract its response Dpeer This response is treated as a first-pass hypothesis, which is later refined by the LLM using structured evidence D. By comparing pear with structured scene information, our framework encourages the LLM to ground or correct its reasoning based on verifiable cucs. We now describe how the structured data D is used guide the final reasoning stage.
V-VLM as a peer module.
Hierarchical data structure. D is designed to organize information in hierarchical manner, distinguishing between video-level and frame-level details. This structure is particularly in- tuitive for an LLM because it aligns with how reasoning typically occurs over temporal quences. Note that the structured representation is provided to the LLM in JSON format.
D = [Vido-LLvvll-Inormaattt
response: Dpeer
surrounding-info: Dacene:
Lemme--Leel--Ifrraatee
{
ggo-can-information toofoomtaiin: Dmotion: Dum-
frame_ inder : t, detected objects:
description: Viddus
(T,, bt,i; Ct,i, de,5, Ae, Ot, i, At, At.e }}
Three-block prompt The prompt PILM is designed as three-block structure to optimize the
model's grounded reasoning. The first component, Key Explanation, provides a precise and explicit interpretation of the scene representation DD reducing ambiguity in the model's input. Prior work shows that I.LMs benefit from well-disambiguated inputs when handling symbolic data [63, 64]. The second component, Step Instructions, decomposes the reasoning task into explicit sub-goals. This design is motivated by findings in cognitive science and neural model alignment, where step- by-step prompting improves reasoning accuracy and consistency [65, 66]. The third component, Peer Instruction, informs the model that pger-generated answers may be unreliable and explicitly encourages independent reasoning. Together, these three components operationalize input ground- ing, procedural reasoning, and epistemic caution-thh necessary conditions for robust and gener- alizable performance in tasks involving multi-step inference from structured dynamic scenes. Our prompt has been provided in the Supplementary Material.
Note on efficiency. While iFinder involves multiple pretrained modules, each step is executed in- dependently and requires no retraining or gradient updates. Inference parallelizable across mod- ules. Our focus is not on real-time deployment, but enabling interpretable, post-hoc analysis pipelines-wher where accuracy is the primary objective.
Experiments
Experiment setup. iFinder leverages a combination of state-of-the-art models in each of its steps. In Step 1, we use GeoCalib [67] ] for estimating camera parameters and distortion coefficients. To correct the lens distortion,
use OpenCV's undistort [68] function for R. In Step 2, we use InternVL [69] for Fi-VLM and VideoLLaMA2 [14] for FAVLLM In Step 3, we use DROID-SLAM [70] for F Feam-pose. In Step for F2D-det, we use OWL-V2 [71] and ByteTracker [72] for FaD-urck- In Step 5, we use OMR [73] for Fume- In Step 6, Metric3D [74] is used for Fdepth and SAM [75] for Fug. In Step 7, we again use InternVL for FLVLM- In Step 8, we use CenterTrack [76] for FSD-da: For peer-informed reasoning, VideoLLaMA2 [14] serves as the default peer model unless otherwise specified. For final reasoning step, we use GPT-4o-mini [77] for FLLM- The full list of hyperparameters and prompts is in the Supplementary Material. Note that our goal is not to compare model variants, but to demonstrate the effectiveness of the combined linionnl pipeline. Each can be easily swapped for stronger versions for further improvements.
Experiment details. We choose datasets and baselines that require all the methods to analyze the complete video before answering the user query. To this end, we use four benchmarks: MM-AU (Multi-Modal Accident Video Understanding) [22], SUTD (Traffic Question Answering) [78], Lin- goQA [79], and Nexar [80] dataset. For baselines, we compare iFinder with V-VIMM that are trained to provide onen-ended responses to users' queries on input videos. In particular, we compare
7

---

Table 1: Multiple-choice VQA performance on MMAU and SUTD. pt prompt tun- ing. äFinder beats both all V-VLMs w/o fine-
Table 2: Result on SUTD categories. Basic Un- derstanding (U), Event Forecasting (F), Reverse
tuning, showing that fine-grat aeeeggaaddd details gen- erate more accurate choices.
Reasoning (R), Counterfactual Inference (C), In-
trospection (O), and Attribution (A).
Method
MM-AU
SUTD
Method
C
R
C
1
A
Generalist Models
Generalist Models
Vide [14]
50.95
47.31
VideoL [14]
49.2 39.0
48.5 53.5
35.8
45.2
LaM2 (w/ pt)
52.89
VideoChat2 [21]
42.5
38.1 43.8
49.2
30.4
42.8
VideoChat2 [2]]
49.56
42.17
Video-LLaVA [15]
39.7
37.2
35.8 40.5 31.1
36.4
Video-LLaVA [15]
43.63
38,35
swwning-pecializedd Methods
Methods
DriveMM [16]
47.6
38.6
40.1
43.2
38.5
37.7
DriveMM [T6]
24.22
43,90
iFinder (Ours)
52.2
43.5
50.2
56.8
39.2
49.6
iFinder (Ours)
63.39
50,93
Table 4: Accident Occurrence Prediction on Nexar dataset. Videol.LaMA2 and WiseAD ex- hibit a bias toward predicting accidents in all cases. We use VideoChat2 in peer-informed rea- soning, further enhancing its performance.
Table 3: Open-ended VQA result Lin- goQA dataset. iFinder outperforms others on the Lingo-Judge accuracy without fine-ture tmnettnnngg
Method
Lingo-J BLEU METEOR CIDBr
Generalist Models
Method
Acc (%) F1-Score Precision Recall
Ladc [14]
36.00
4.15
33.45
26.28
Generalist Models
VideoChat2 [21]
41.20
6.58
36.81
40.98
VideoChat2 [21]
58.0
0.62
0.57
0.68
Vidco-LLaVA [15]
21.00
4.26
26.99
31.23
Lad2 [14]
50.0
0.67
0.50
1.00
Driving-spccializze Methods
Driving-specilling Methods
WiseAD [18]
13.40
2.20
21.50
DriveMM [16]
49.0
0.32
0.48
0.24
iFinder (Ours)
44.20
6.07
35.80
42.01
WiscAD [18]
50.0
0.67
0.50
1.00
iFinder (Ours)
62.0
0.59
0.64
0.54
against recent state-of-the-art general V-VLMs VideoLLaMA2 [14], VideoChat2 [21], and Vide- oLLaVA [15], well as those proposed for driving video analysis DriveMM [16] and WiseAD [18]. The dataset and evaluation metric details are provided in the Supplementary Material. All the following evaluations are in a strictly zero-shot setting, with no fine-tuning.
Quantitative Results. We analyze the performance on accident-cause and traffic scene understand- ing on MM-AU and SUTD datasets under multi-choice VQA setup, shown in Table 1 and Table 2, and gain three insights on iFinder versatility and performance. One, iFFider outperforms both ceneral-purposs and highlighting its strong out-of-the- reasoning capabilities. Two, it achieves the best accuracy in all six SUTD categories, from basic understanding to attribution, showing robust generalization across diverse cognitive tasks. Three, the 39-point gap between iFinder and DriveMM on MM-AU under- scores the limitations of current domain-specific models and the strength of FFinder's architecture in complex, real-world driving scenarios.
models on MM-AU and SUTD without any fine-tuning,
Next, we analyzed all methods under open-ended VQA setup the LingoQA dataset in Table 3 and provided the following insights. One, iFinder achieves the best Lingo-Judge accuracy among all evaluated methods, outperforming Videol LaMA2 by 3%, VideoChat2 by 8%, and Video-LLL by 23.2%. Further, it outperforms driving-specialle method performs competitively on BLEU and METEOR metrics, it slightly lags behind VideoChat2. This suggests that VidcoChat2 generates responses that are more lexically similar to reference an- swers but do not necessarily reflect greater factual correctness. We also show this in Figure 5.
models, WiseAD by 30.8%. Two, while our
Finally, we analyze all methods for accident occurrence prediction on the Nexar dataset in Table 4. aFinder achieves the highest accuracy at 62.0%, outperforming both generalist models, as well as driving-specializeed models. Notably, while VideoL.aaMA and WiseAD achieve high F1 and perfect recall by always predicting accidents, this limits real-world reliability. In contrast, VideoChar2 balances precision and recall for better accuracy. Building on this, FFinde integrates within the peer V-VLM to enhance both accuracy and precision.
8

---

GTI You the pedlottins onfl singin the sail
GT: The mraiic figlt wil
hus uneiii dll cosorall
Hhe lig and spdiudan
Q: Can you spot any immeciiaee
Q- you comre slop? What for contriliute ohis decition?
Auzard on the road ahcad?
CTh Epocer cinie Fatt and hrik deranio
Valuo Thecon koodeaan
iaacee Eip willl sctemaber
Winnhll the
Q: What ir the couse of the accident?
Q: Did acridran occur video, and of wheee dlarr it star??
iiuda You
bicuie Laca (21-4)
Figure 5: Qualitative comparison on LingoQA (top), MM-AU (bottom, left), and Nexar (bottom, right) dataset. iFinder improves spatial reasoning and causal inference, and reduces peer- V-LM- errors. In the bottom-left example, iFinder corrects the peer V-VLM's inaccurate claim of a "de- celerated vehicle" by leveraging structured data that reveals the ego vehicle's 's rapid approach.
Table 5: Impact of each vision component and each prompt block on MM-AU dataset.
Table 6: Result breakdown MMAU
weather categories. iFinder achieves best per- formance across adversarial conditions, showing its adaptability to diverse environments.
Method
Accuracy (%)
VideoLLaMA2 (peer-VLL VLM)
52.89
Winder wio ego-state estimation
62.37
&Finder wlo lane detection
61.80
Method
Foggy
Rainy
Snowy
Sunny
Vision
aVinder wio distance estimation
60.62
aFinder w/o frame undistortion
60.47
Generalist Models
/Finder wlo object attributes
59.04
Lade (w/pt) [14]
66.67
44.83
40.25
54.56
Wiinder wio orientation estimation
58.83
VideoChat2 [21]
58.33
54.31
49.69
49.16
/Flnder wlo seene uaderstanding
$7.81
Video LLaVA [15]
58.33
41.38
42.77
43.76
/Finder wlo poor instruction
60.62
Driving specialized Methods
FFinder wlo steps instruction
60.06
DiveMM [16]
33.33
20.13
24.35
aFinder wio key explanation
58.73
iFinder (Ours)
75.00
65.52
$7.86
63.69
aFinder
63.39
Qualitative Results. We qualitatively analyze all methods in Figure 5 and observe the following. One. üFinder demonstrates superior perceptual grounding-for instance, in the first case, unlike VideoLLaMA2 and VideoChat2, iFinder correctly identifies the pedestrian as an immediate hazard, aligning with the ground truth. Two, iFinder shows fine fine-grained causal reasoning: in the second row, &Finde specifies that the ego vehicle reduced its distance from 24.75m to 6.33m, clearly linking this to the collision collision-sommthingg something missing in the generic response by VideoLLaMA2. Three, FFinder exhibits higher temporal precision, as seen in the final example where üFinder accurately detects the collision at frame 552 (18.4s), closely matching the GT (20.167s), while others offer vague or less aligned time windows. These examples underscore iFinder's robust visual grounding and decision-makee making fidelity.
Ablation studies. In Table 5, we assess the impact of each vision component, as well as the contri- bution of each prompt component in PLLM on the MM-AU dataset. Although each vision component contributes to overall performance, surprisingly, scene understanding and orientation estimation are of higher importance for semantic reasoning than even core perception modules like distance or lane estimation. For example, removing orientation estimation leads to the largest drop in accu- racy-from 63.39% to 58.83%. The results show that each block aid accuracy, with key explanations being especially crucial-withon without them, the LLM sometimes generates invalid responses, such as mis-formatted numerical outputs.
System analysis under adversarial conditions. Post-hoc driving video analysis also encompasses scenarios involving adversarial or long-tail conditions, such as poor weather or nighttime environ- ments. To evaluate the robustness of our method under such conditions, we break down the per- formance across different weather and lighting conditions using the MM-AU[22] benchmark in Table and Table 7. While generalist models (e.g., VideoL show steep drops in per- formance under rain (44.8%%) snow (40.25%). and night (50.23%). and driving-speciialzzed models like DriveMM perform poorly (e.g., 20.13% on snowy, 24. 14% on rainy), FFinder maintains con- sistently high accuracy achieving the best scores across all settings.

---

Table 7: Result breakdown on MMAU light categories. üFinder shows consistent improve- ments across different lighting conditions, indi- cating robustness to variations illumination.
Table Error propagation analysis of iFnddr on MM-AU. We evaluate iFinder un- der increasing confidence thresholds Tscore to simulate missed detections, showing strong silience to error propagation.
Method
Day
Night
Generalist Models
Method
Accuracy (%)
Objects Retained(())
Lide [w/
53.23
50.23
VidcoChat2[21]
50.29
43.84
iFinder
63,39
100.00
FFinder Tooee
Video-LLaVA[15]
43.25 46.58
iFinder w 0.5
61.09
32.63
spespiiialieed Methods
iFinder
0.6
58.73
7.96
iFinder "eome 0.7
58.42
3.16
DriveMM[16
23.41
30.59
Finder w 0.8
58.27
0.68
#Finder (Ours)
63.32
63.93
Impact of error propagation by object detection. With the object detector being the earliest and most important module to capture object semantics iFinder, we analyze the impact of error propagation missed detections occurring early in the system the overall performance in Table 8. It can be observed that even when fewer than % of objects are retained (0.68% at confidence score Tscore 0.8), the accuracy remains at 58.27%, better than the best baseline (VideoLLaMA2 at 52.89%). At Tscore = 0.5, retaining only 32.63% of objects still yields 61.09% accuracy, suggesting that iFinde''s reasoning remains stable under significant perceptual filtering. Overall, iFinder demonstrates graceful degradation and limited error propagation.
5 Conclusion
this paper, we argue that grounding general-purpose LLMs using structured, interpretable scene representations offers more accurate alternative than V-VLMs. Further, decoupling perception from reasoning enables a more reliable analysis system. Built on these principles, we introduce iFinder, modular, training-free free framework that advances post-hoc driving video analysis by struc- turally grounding purppss LLMs in domain-specific perception. Through explicit scene decomposition and hierarchical prompting, FFinde achieves superior spatial and causal reason- outperfrrnnin both generalist and driving specialized V-VLMs on accident reasoning bench- marks. Extensive evaluations confirm strong post-hoc video understanding performance under di- verse environmental conditions. This structured grounding method highlights promising direction for aligning LLMs with domain-specific reasoning requirements, where interpretability and reliabil- ity constitute the primary focus.
6 Acknowledgments and Disclosure of Funding
This work was supported by NSF grant CNS-2312395 and NEC Laboratories America, Inc.
References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07225 2021.
[2] Harsha Nori, Nicholas King, Scott M McKinney, Daniel Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint S.ti::2333.1377 2023. 1
[3] Marcel Binz, Eric Schulz, et al. Robustness and accuracy of large language models. arXiv preprint 106619,006.1169, 2023.
[4] Kevin Price. Aptiv discusses transition to video analysis in adas and ev validation. Automotive Testing Technology International, 2023. https://shortur/ http://horturl..t//.... at/V9XuD.
[5] P Andres. Data recording for adas development--scalable recording of sensor and ccu data. Elektronik Automot, pages 2-3, 2017.
[6] Alex Poms, Will Crichton, Pat Hanrahan, and Kayvon Fatahalian. Scanner: Efficient video analysis at scale. ACM Transactions Graphies (TOG), 37 2018.
10

---

[7] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, al. End to end learning for self-driving cars. arXiv preprint vr.iivII04..07.
2016.
[8] Long Chen, Yuchen Li, Chao Huang, Bai Li, Yang Xing, Daxin Tian, Li Li, Zhongxu Hu, Xiaoxiang Na, Zixuan Li, al. Milestones in autonomous driving and intelligent vehicles: Survey of surveys. IEEE Transactions on Intelligent Vehicles, 102:104461055, 2022.
[9] Schastian Baran and Przemyslaw Rola. Prediction of motor insurance claims occurrence as an imbalanced machine learning problem. arXiv preprint arXiv:2204.06109, 2022.
[10] Chenwei Lin, Hanjia Lyu, Jicbo Luo, and Xian Xu. Hamnessing 4v (ision) for insurance: A preliminary exploration. arXiv preprint 2Xiv:2444.09690 2024.
[11] Xingcheng Zhou and Alois C Knoll. Gpt-4v traffic assistant: in-depth look at vision language model on complex traffic events. arXiv preprint arXiv:2402.02205, 2024. 1
[12] Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking autonomous driving with large language models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 910-919, 2024.
[13] Nisaja Fernando, Abimani Kumarage, Vithyashagar Thiyaganathan, Radesh Hillary, and Lakmini Abey- wardhana. Automated vehicle insurance claims processing using computer vision, natural language pro- cessing. In 2022 22nd International Conference on Advances in ICT for Emerging Regions (ICTer), pages 124-129. IEEE, 2022.
[14] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Vidcollama 2: Advancing spatial-tempora modeling and audio understanding in vidco-llms. arXiv preprint MxXiv:2466.07477 2024. 2, 7, 8, 9, 10, 20
[15] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment hefore projection. arXiv preprint 2xiv:2311.10122, 2023. 2, 3, 8, 9, 10, 20
[16] Zhijian Huang, Chengjian Feng, Feng Yan, Baihui Xiao, Zequn Jic, Yujie Zhong. Xiaodan Liang, and Lin Ma. Drivemm: All-in-one large multimodal model for autonomous driving. arXiv preprint arXiv:2412.07689, 2024. 2, 2, 10, 20
[17] Licheng Wen, Xuemeng Yang. Daocheng Fu, Xiaofeng Wang. Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al. On the road with gpt-4v (ision): Early explorations of visual-language model autonomous driving. arXiv preprint arXh::2311.05332 2023. 2
[18] Songyan Zhang. Wenhui Huang, Zihui Gao, Hao Chen, and Chen Lv. Wisead: Knowledge augmented end-to-end autonomous driving with vision-language model. arXiv preprint arXiv:2412.09951, 2024. 3, 8, 20
[19] Pei Liu, Haipeng Liu, Haichao Liu, Xin Liu, Jinxin Ni, and Jun Ma. Vlm-e2e: Enhancing end-to-end autonomous driving with multimodal driver attention fusion, 2025. 2
[20] Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P. Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M. Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision, 2024. 2
[21] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 3, 8, 9, 10, 20
[22] Jianwu Fang, Lei-lei Li, Junfei Zhou, Junbin Xiao, Hongkai Yu, Chen Lv. Jianru Xue, and Tat-Seng Chua. Abductive ego-view accident video understanding for safe driving perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22000- 2024. 2, 3, 20, 23
[23] Yajun Xu, Huan Hu, Chuwen Huang, Yibing Nan, Yuyao Liu, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. Tad: A large-scale henchmark for traffic accidents detection from video surveillance. IEFE Access, 2024. 3
[24] Hoon Kim, Kangwook Lee, Gyeongjo Hwang, and Changho Suh. Crash to not crash: Learn to identify dangerous vehicles using a simulator. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 978-98,,100 2019. 3
11

---

[25] Gurkirt Singh, Stephen Akrigg. Manuele Di Maio, Valentina Fontana, Reza Javanmard Alitappeh. Salman Khan, Suman Saha, Kossar Jeddisaravi, Farzad Yousefi, Jacob Culley, al. Road: The road event aware- ness dataset for autonomous driving. IEEE transactions on pattern analysis and machine intelligence, ((11):003-10544 2022. 3
[26] Ali K AlShami, Ananya Kalita, Ryan Rabinowitz, Khang Lam, Rishabh Bezbarua. Terrance Boult, and Jugal Kalita. Coool: Challenge of outt-of-label novel benchmark for autonomous driving. arXiv preprint 1xXiv24422005462, 2024, 3
[27] Lukas Picek. Vojtech Cermak, and Marek Hanzl. Zero-shot hazard identification in autonomous driving: A case study on coool benchmark. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 654-663, 2025. 3
[28] Shashank Shriram, Srinivasa Perisetla, Aryan Keskar, Harsha Krishnaswamy, Tonko Emil Westerhof Bossen, Andreas Møgelmose, and Ross Greer. Towards a multi-agent vision-language system for zero- shot novel hazardous object detection for autonomous driving safety. arXiv preprint orXiv:2504.13399
2025. 3
[29] Antoine Miech, Jean- San-BBatistte Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020. 3
[30] Tianhao Li and Limin Wang. Learning spatiotemporal features via video and text pair discrimination. CoRR, abs/2001.055 .056910 2020.
[31] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu. Jae Sung Park, Jize Cao. Ali Farhadi. and Yejin Choi. Merlot: Multimodal neural script knowledge models. NeurlPS, 2021.
[32] Kunchang Li. Yali Wang. Yizhuo Li, Yi Wang, Yinan He, Limin Wang. and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. arXiv preprint arXiv:2303. 16058, 2023,
[33] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:221.009552, 2022.
[34] Hu Xu, Gargi Ghosh, Po- Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Meize, Luke Zettle- moyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre- pre-training for zero-shot video-text under- standing, arXiv preprint arXiv:2109, 14084, 2021.
[35] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu. Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, 2022.
[36] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang. Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, al. An empirical study of training end-to-end vinion-and- language transformers. In CVPR, 2022.
[37] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and languane tasks? arXiv preprint arXiv: 107.00..000 00333, 2021.
[38] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang. Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive iaguaage-mmage pre-training arXiv preprint arXiv:2 1eXiv:2/11.077) 1107783, 2021.
[39] Chen Sun, Austin Myers, Carl Vondrick. Kevin P. Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. ICCV, 2019.
[40]
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. CVPR, 2020.
[41] Wang, Kunchang Li, Yizhuo Li. Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03119/ 2022.
[42] Guo Chen. Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, et al. Internvideo-ego4d: A pack of champion solutions to ego4d challenges. arXiv preprint 22X::222110092222 2022.
[43] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint 05.4442306. 2023.
12

---

[44] Kunchang Li, Yali Wang, Yinan He. Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22195-22206, 2024.
[45] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llamt-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304110110, 2023.
[46] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024,
[47] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02855 028558 2023.
[48] Peng Jin, Ryuichi Takanobu, Wancai Zhang. Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700-13710, 2024. 3
[49] Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: knowledge-driven approach to autonomous driving with large language models. arXiv preprint arXiv:2309. 16292, 2023. 3
[50] Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint 2rXiv:331000111,, 2023.
[51] Zhenhua Xu, Yujia Zhang. Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
[52] Zhijian Huang. Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xiandan Liang. Making large language models better planners with reasoning-decision alignment. In European Conference on Computer Vision, pages 73-90. Springer, 2025.
[53] Hao Shao, Yuxuan Hu, Letian Wang. Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, pages 15120-155 15130, 2024.
[54] Wenhai Wang, Jiangwei Xie,
YhuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hamming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023.
[55] Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Rea- son2drive: Towards interpretable and chain-based reasoning for autonomous driving. In ECCV, pages 292-308. Springer. 2025.
[56] Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang. Wei Zhang, and Xiaomeng Li. Holistic tonomous driving understanding by bird's berd's-eye-view injected multi-modal large models. In CVPR, pages 13677, 2024.
[57] Shihao Wang. Zhiding Yu. Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li. and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.0555 01533, 2024.
[58] Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. arXiv preprint SBtv:2403.04593.. 2024.
[59] Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models self-driving corner cases. arXiv preprint arXiv:2404. 10555, 2024. 3
[60] Quang Minh Dinh, Minh Khoi Ho, Anh Quan Dang, and Hung Phong Tran. Trafficvim: A controllable visual language model for traffic video captioning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 7134-7143, 2024. 3
[61] Elena Giovannini, Arianna Giorgetti, Guido Pelletti, Alessio Giusti, Marco Garagnani, Jennifer Paola Pascali, Susi Pelotti, and Paolo Fais. Importance of dashboard camera (dash cam) analysis in fatal vehicle- pedestrian crash reconstruction. Forensic Science, Medicine and Pathology. 17(3):37--33 38777 2021. 4
13

---

[62] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955. 6
[63] Xuezhi Zhou, Simran Arora, Jingwei Huang, and et al. Least-to-most prompting enables complex rea- soning in large language models. In ICLR, 2023. 7
[64] Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In ACL, 2022. 7
[65] Jason Wei, Xuezhi Wang, Dale Schuurmans, and et al. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
[66] Max Nye, Nora Kassner, Michael Nye, and et al. Show your work: Scratchpads for intermediate compu- tation with language models. In NeurIPS, 2021.
[67] Alexander Veicht, Paul- -duuard Sarlin, Philipp Lindenberger, and Mare Pollefeys. GeoCalib: Single- image Calibration with Geometric Optimization. In ECCV, 2024.
[68] OpenCV Contributors. Image Processing cessed: 2025-03-07.7
Geometric Image Transformations: undistort(). 2024. Ac-
[69] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijic Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisic tasks. In CVPR, pages 24185-24198, 2024."
[70] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558-16569, 2021.
[71] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection, 2024.
[72] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zchuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 7
[73] Dongkwon and Chang-Su Kim. Omr: Occlusion-aware memory-based refinement for video lanc detection. In ECCV, 2024." 7
[74] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero- shot metric depth and surface normal estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024. 7
[75] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.00 02643, 2023.7
[76] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Tracking objects points. ECCV, 2020. 7.22
[77] OpenAL. Gpt-4o system card, 2024. 7, 20
[78] Li Xu, He Huang, and Jun Liu. SUTD- TrafficAA:::: A Question Answering Benchmark and an Efficient Network for Video Reasoning Over Traffic Events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9878-9888, June 2021. 7, 20,0,,2 23
[79] Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurahh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Video question swering for autonomous driving. In ECCV, 2024. 7, 20, 22, 23
[80] Daniel C. Moura, Shizhan Zhu, and Orly Zvitia. Nexar dashcam collision prediction dataset and chal- lenge, 2025. 7, 2020, 23
[81] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krish- nan, Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, pages 11621-11631, 2020. 22
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have provided quantitative justification of each claim in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a description of the limitations the Supplementary Material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the experimental details each component (vision mod- ules, LLM, and related prompts) and parameter in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Releasing the code is contingent upon receiving approval from the author's organization.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: potttss//nips.. cc/
cnsssiicyylly for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. cccccc/p..o cccppbllc///eccccccccccccccclcct wwwp///g///////eeee/eee// CCoeeS/mmmebbbii setttamiissiiiiiiiim ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The proposed method doesn't involve any training. For inference, we have provided the experimental details of each component (vision modules, LLM, and related prompts) and parameter in the Experiments section.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We provide an error analysis in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide a description of the compute resources and time of execution in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neurs cc wwwbblic/Etiisuutiee
Answer:
[Yes]
Justification:
We strictly followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide discussions on broader impacts of out method in the Supple- mentary Material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The licenses for existing assets used in this work are listed in Table 9,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

Table 9: Licenses and sources of datasets and models used in this work.
Asset
License
MM-AU[22]
CC-BY-NC-4.0
WTTD-TraffiQA [78]
Custom (research-only, non-commercial)
LingoQA [79]
https://github. com/wa///////o wayveai/LingooA (allowed for research)
Nexar [80]
nexar-open-datt-lieens
VideoLLaMA2 [14]
Apache-2.0
VideoChat2 [21]
MIT
VideoLLaVA [15]
Apache
DriveMM [16]
Apache-2.0
WiseAD [18]
MIT
GPT-4o-mini [77]
This code uses GPT-4o-mini via OpenAl's API. The model itself is proprietary and subject to OpenAl's Terms of Use.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, curated licenses for some datasets. Their licensing guide help determine the li- cense of dataset.
thcode com/ datasets has
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research
with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: disclose comprehensive information about the language model, including its official name and the specific prompts utilized.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Table 10: Wall-clock runtime of each pipeline module in iFinder.
Module
Runtime(s)
Frame Undistortion
66.7
3D Object Detection
14.8
Attribute Estimation
66.7
Distance Estimation
40.3
Lane Detection
21.5
A Limitations & Broader Impacts
Limitations. The current form of iFinder lacks mechanisms to incorporate reason about am- biguous, social, or normative aspects of driving scenes (e.g., intent behind a maneuver, yielding behavior, etc.). These elements are often critical in understanding traffic interactions but not eas- ily captured by purely spatial-temporal features or symbolic grounding. Future work should build this capability using hybrid reasoning mechanisms that combine structured perceptual data with commonsense knowledge bases.
Broader Impacts. On the positive side, by clearly separating how the system sees the world (per- ception) from how thinks about it (reasoning). &Finder supports a growing trend in Al that large language models (LLMs) have limits and need structured, trustworthy data to reason well. This makes the system's thinking more like human reasoning, which is especially important in high- stakes areas like self-driving cars. On the negative side, although iFinder improves clarity and explainability, it might unintentionally promote a narrow view knowledge where only LLM- readable information is seen valid. This could leave out important human factors that are harder to define, like driver's intent, ethical responsibility, or social rules of the road.
B Implementation choices
In Step 3, we sample the temporal points in order to reduce noise and make the estimation insensitive to small deviations. Further, we set Ta and T, as 30 and standard deviation of all speeds {s, )= For motion estimation, we set g as 2. In Step 4, since we use Owl-V2 V2, we set the 2D classes as ['mo- torcycle', 'police car', 'ambulance', 'bicycle', "traffic light', 'stop sign', 'road sign', 'construction worker', 'police officer', 'ambulance', 'fire truck', "construction vehicle', 'traffic cone', 'person', 'car", 'wheelchair', 'bus", 'truck'] with confidence threshold 0.25. In Step 5, we only estimate lane locations for vehicles and person categories. In Step 8, we use the default classes by Center- Track [76] for NuScenes dataset [81]. All the rest of the parameters
default model choices.
All the prompts are provided in Section E. Note that for peer V-VLM, we use the default prompt provided by the respective authors. All experiments were conducted on single NVIDIA A6000 GPU with 48 GB of memory. Table 10 reports the average wall-clock time required to execute each module in our pipeline per video the LingoQA dataset [79]. These timings reflect end-to-end processing, including loading, inference, and output serialization (for umoptimized python code).
C Dataset Details
Dataset Details. MMAU contains a test set of 1,953 ego-view accident videos, each associated with fixed question: "What is the cause of the accident?" along with five multiple-choice answer options. The with 6,075 multiple-choice questions designed to assess different aspects of scene understanding. The questions are categorized into six reasoning types: Basic Understanding, which involves direct perception of scene elements; Event Forecasting, which requires predicting future events; Reverse Reasoning, which focuses on deducing past events from the current scene; Counterfactual Inference, which evaluates hypothetical scenarios; Introspection, which involves providing preventive advice; and Attribution, which involve causal reasoning and responsibility assessment in driving scenarios. Performance on both datasets is measured using accuracy. For open-ended VQA, we evaluate on LingoQA [79], which consists of 100 videos with total of 500 questions in the evaluation set.
dataset includes a test set of 4,111 real-world driving videos, paired
22

---

Table 11: List of sampled videos from the Nexar dataset
01031, 00831, 00097, 02034, 01080, 01085, 01736, 00059, 02121, 01875,
01970, 01290, 00967, 01840, 00477, 01853, 00469, 00970, 01815, 02085,
00684, 00587, 01393, 02013, 00816, 01858, 01607, 00534, 02048, 00407,
01806, 01586, 00077, 01413, 00099, 01478, 00858, 00155, 01801, 01276,
02119, 01350, 01696, 00364, 01616, 01753, 00039, 01682, 00783, 01992,
01932, 01372, 01638, 01268, 01542, 00049, 01617, 00904, 02069, 00640,
00046, 00106, 00937, 01465, 00579, 00131, 01118, 00703, 00324, 00339,
00167, 01635, 00103, 01695, 00608, 00949, 00422, 01317, 00610, 00242,
00519, 00909, 01952, 01364, 01071, 00461, 01453, 01849, 01533, 00345,
00733, 00617, 00722, 00453, 01985, 00651, 00972, 01441, 00977, 00082
Table 12: Standard error of iFinder on MM-AU and SUTD datasets.
Method
MM-AU
SUTD
iFinde (ours)
63.391:.26
50.93:1.68
Unlike MM-AU [22] and SUTD-TrafficQA [78], which follow multiple-choice format, LingoQA [79] requires free-form natural language responses. accident occurrence prediction, we evaluate on the Nexar dataset [80]. Since the original test set does not include ground-truth labels, we ran- domly sample 100 videos from the training set to construct an evaluation set, maintaining balanced distribution of 50 accident and 50 non-accident videos, consistent with the ratio in the full training set. The list of sampled videos from the Nexar dataset's original training set, used for accident oc- currence prediction evaluation in this paper, is provided in Table 11. The list indicates the video names.
D Error Analysis
In Table 12, performance of FFnddr on MM-AU and SUTD datasets, reported as (mean accuracy + standard error) over five runs. We can observe that FFnder maintains its performance.
Although iFinder achieves strong results, it struggles in visually ambiguous collisions. For ex- ample, from the Nexar dataset, video "01031", a car cuts in front of the ego vehicle, but without clear cues such as steering correction or vibration, even humans cannot confirm contact; this re- veals iFinder 's reliance "00970", the ego car brakes sharply behind another vehicle, leaving only small gap. With no visi- ble impact signs like deformation or debris, it is unclear whether this was a collision or mesr-miss. Both cases show that subtle physical contact often leaves no visual trace, limiting vision-only ap- proaches. Future work should integrate other modalities (e.g., audio, IMU) and model uncertainty so the system can express lower confidence instead of making forced binary predictions.
observable patterns rather than imperceptible dynamics. In the video
E
Prompts
The prompts we use for the VLM are shown Listing 1, Listing 2 and Listing The system and user prompts we use for each of the tasks in the final reasoning are shown in Listing 4, Listing 5, Listing 6, Listing 7, and Listing 8.
Listing 1: Prompt for image-based VLM Pj in Step 2.
You are an expert autonomous driving, specializing in analyzing traffic scenes. the ego car.
You receive a series of traffic images from the perspective Your task is to describe the driving environment, focusing on weather, lighting road layout, surrounding environment, and any notable elements.
It essential that you strictly follow the rules instructions below. Any deviation from the specified structure or format will result in an invalid output.
23

---

STRICTLY follow Rules:
You must strictly follow the dictionary structure provided below.
Only use the specified terms for weather, light, road layout, environment.
not create your own terms.
No additional information or categories should be added.
You should strictly follow these instructions. If object element is not exist the scene, the value to 'None'. Ensure every field is filled with the appropriate value or 'None'.
visible or does
For the video description, base your analysia on the overall characteristice
observed throughout the video rather than a single frame.
Note any temporal changes that shifts, traffic light changes, road condition variations).
time in the video (e.g., traffic flow
Output the result in the following dictionary format:
"surrounding info": {
"weather": "[e.g., 'cloudy', 'sunny', 'rainy', 'fog', 'snowy']", "light": "[Choose 'day', 'night', 'dawn', 'dusk"]", "rood__ayout"; "[Choose from: 'straight road', 'curved road', 'intersection', -junction', 'ramp'l",
"environment": "[Choose from: 'city street', 'country road', 'highway',
residential area']",
"sun_viie www.iiiiiiityconniiiiee.
"[Choose from: 'clear', 'foggy', 'low visibility',
'hazy']"
"road__codiie condition" "[Choose from: 'wet', 'icy', 'normal', 'debris', 'potholes']",
"surface_ type:: [Choose from: 'asphalt', 'gravel', 'dirt', 'concrete']",
"traffic_flo=": [Choose from: 'light', 'moderate', 'heavy']",
"time_od _ay" "[Choose from: 'morning', 'afternoon', 'evening', 'night']", @rood_obstacleds [Choose from "debris visible", "no debris visible".] "road_ "road_demsity":: [Choose from "crowded", "normal", 'scarce'.
},
" "description": [Provide a concise yet informative summary the scene. Highlight notable objects, traffic conditions, movement patterns, mention any observable changes over time, and any potential driving hazarda.]"
Listing 2: Prompt for video-based VLM Py in Step 2.
Analyze the provided driving video and generate a detailed, sequential caption that accurately describes the vehicle's actions, road conditions, traffic dynamics, and surrounding environmental elements. Highlight key driving events, such as acceleration, braking, turning, interactions with other vehicles pedestrians and the presence of traffic signals, signs, notable landmarks.
Additionally, provide an in-depth analysis of the ego car's speed, discussing its impact on the scene and how it influences the behavior and dynamics of nearby objects and road users.
Listing 3: Prompt for video-based VLM Pa in Step 7.
are an expert autonomous driving, specializing in analyzing traffic scenes. You are driving the ego-vehicle and looking at the scene.
Your task is look at the red bounding box and output the response in the format below. If a person, say "person wearing black clothes", etc. it any other vehicle, say "black car", "black bus", "silver SUV", etc. Strictly follow the rules.
"color": "Chooose most dominant color of this object.]
24

---

Listing 4: System prompt in PLLM for multiple-choice VQA. You detailed traffic analyst, analyzing conclusions about vehicle behavior, lane positions, and potential hazards.
data draw fact-based
Listing 5: User prompt in PLEM for multiple-choice VQA.
You are analyzing JSON file representing a traffic scene from ego vehicle 's perspective. The video captures interactions with surrounding objects. Analyze only observable elements: bounding boxes ("bbox'), lane positions (* relative, Malative_lane_lccee aam__location'
object laae__ocationn ago_vehicle'), attributes (" rotations ('rot_y'), distances ('distance__fa fareeeerrrmeeoo___eeeeeeeee attributes"), additional environmental factors from "Video Level Information" such as weather, lighting, and road conditions.
peo_lane_locattae
USON Key Explanations
"bbox": Represents detected object's position in frame. Track changes size and location to determine motion and distance.
"distance_froo frrannee_rrommeeeee www.ftorroooooeeeeee ego_vehicle" Distance (in meters) from the ego vehicle.
geelative_lame_locttioo Description of how many lanes away an object is from the ego vehicle.
vobj_lame_locatior: Object's lane index relative to the road.
"ego_lame_location Ego vehicle's lame index relative to the road.
"attributes": Object features such as color.
"rot_y": Object's rotation angle, useful for detecting turns.
"loc": Object's position in space.
"abject_id". Unique identifier for objects each frame.
"surrounding. info": Describes the environment, including weather, lighting, road layout, surface type, traffic flow, and time of day. "motion_state"; Indicates the motion status (e.g., Moving, Stopped) of the ego vehicle.
"turn_action"; Describes the ego vehicle's turning behavior.
"description": Summary of the video. "response": Response from another model; it may be incorrect, but use it as basis for reasoning.
Instructions:
Follow the steps below to analyze the incident and formulate your response using JSON data and the description under "Video Level Information" " enhance
reasoning. Think step by step and use the exact format specified at end.
Step 1: Identify and Describe the Unusual Activity or Event
Step 1.1: Analyze the following data points to identify risky dangerous behaviors
Bounding box ('bbox'). Track object novements and changes in size or proximity.
Lane position booj__amm_llccttee Rotation ('rot_y"); Identify unusual rotation patterns suggesting erratic risky behavior.
Detect lane changes or encroachments.
Distance: Measure proximity of objects to the ego vehicle.
Describe any patterna or anomalies, such as:
Objects moving against traffic.
Lane cutting abrupt merging.
Unusual or sudden changes distance or rotation.
Sharp or erratic rotations ('rot_y"), e.g., sharp spinning of vehicle indicating slipping an icy wet road.
Use specific data points to explain behaviors:
If a vehicle shous sharp changes in rotation ('rot_y²) on icy or wet roads, classify it as "Vehicle slipping off-road due to wet/icy conditions."
If a vehicle moves across lanes unexpectedly into the ego vehicle's path, classify it "Lane cutting forceful merging incident." " If object (e.g., pedestrian, animal, or vehicle) suddenly enters the ego vehicle's path at close proximity, classify it accordingly (e.g., "Unexpected pedestrian crossing in front of ego vehicle"),
25

---

Step 1.2:
Based your amalysis, classify the incident using the following examples:
1. Lane cutting or forceful merging incident.
2. Close-proximity vehicle or pedestrian crossing in front of the ego vehicle.
3. Vehicle collision.
4. Vehicle slipping off-road due wet/icy conditions.
5. Traffic rule violation encounter.
Unexpected animal crossing front the ego vehicle.
Step 2: Provide Potential Reason for the Incident
Identify the possible reason why the incident occurred. The reason must based on specific observable data the JSON file. Use information such as:
Lane changes.
Rotation angles ('rot_y').
Object proximity to the ego vehicle.
Environmental indicators like weather conditions.
Other "video-level" information if included.
Step 3: Choose the Best Explanation from the Given Options
Based on JSON data, select the most appropriate option that best explains the cause the incident.
Key Requirements for Your Response:
1. Select only of the given multipl--hhiie options the final
3. not generate an open-ended response. The final answer must exactly one option from the provided list.
4. Base your answer strictly on observable data (bounding boxes, lane positions, rotations, distances, etc.).
5. the data incomplete, make an educated guess still select the most appropriate option.
6. Never state "not enough information" or "unable to determine". You must always pick the most reasonable answer
7. Format your final answer exactly specified-just the letter corresponding to your choice.
Ansver the question precisely and analytically based only on the observable data in the provided JSON file.
Response Format (Strictly Follow This Format): [Letter]
USON Data: (JSON data)
(Question)
Options: {0ptions}
Anawer with the option's letter from the given choices directly and only give the best option.
The best answer is:
Listing 6: System prompt in PLM for open-ended VQA and accident occurrence prediction.
You are detailed traffic analyst, analyzing scene data to draw fact-based conclusions about vehicle behavior, lane positions, and potential hazards. Focus on elements such as bounding boxes ("bbox'), lane changes, rotation (* rot_y"), and proximity to the ego vehicle. Use these attributes to form precise insights, noting any deviations from normal behavior, changes in object orientation, or risky maneuvers.
Listing 7: User prompt in PLLM for open-ended VQA.
You are analyzing JSON representing traffic video from the ego vehicle's perspective. The video captures interactions with surrounding objects. Analyze only observable elements: bounding boxes ("bbox'), lane positions (* object
relative lane__ocationn [ooj_lane_locatine 'ego_
26

---

attributes (* attributes"), and additional environmental factors from "Video Level Information" such as weather, lighting, and road conditions. Prioritize later frames for analysis.
rotations ('rot_y"), distances
Key Rules
Focus later frames for all interpretations.
Use common knouledge where applicable:
1. Traffic lighta can only ahow one color at time.
2. Ån object very far (e.g., 50+ meters) from the ego car is not considered in the ego lane.
For color-related questions:
1. Check the latest frames first.
2. If multiple colors exist, return only the most frequent color from later frames.
3. Return exactly ONE color. Never list multiple colors.
USON Key Explamations
"bbox": Represents the detected object'a position in the frame. Track changes in size and location to determine motion and distance. adistance_from.com vehicle"; "eo Distance (in meters) from the ego vehicle. trelative__ane_loctiiie ego vehicle.
Description how many lanes away an object is from the
"ootatiooooc Object's lane index relative to the road. "occ_lion__ocction Ego vehicle's lane index relative to the road. "attributes"; Object features such color.
"rot_y": Object's rotation angle, useful for detecting turns.
"loc": Object's position in 3D space.
"object_id"" Unique identifier for objects in each frame.
"surrounding_inte info": Describes the environment, including weather, lighting, road layout, surface type, traffic flow, and of day. "motion_state": Indicates the motion status (e.g., Moving, Stopped) the ego vehicle.
"turn_action": Describes the ego vehicle's turning behavior.
"description": Summary of the video. "respone": Response from another model; may incorrect, use it basis for reasoning.
Step-by-Step Analysis
Step Identify Key Event
Analyze movements using later frames first.
Categorize the event as lane change, pedestrian crossing, cyclist movement, turning vehicle, steady lane position, traffic sign, unexpected object, or other notable behavior.
Step 2:
Provide One Reason (10 Words) Provide one reason, exactly 10 words, using JSON data and the description under Video Level Information" to enhance reasoning.
Step 3: Answer as Driver
Do NOT mention JSON metadata (IDs, raw values). Answer naturally like driver.
Yes/Kio questions: Give direct, brief explanation.
Fact-based questions: Base reaponse visible elements.
Color-related questions: Return only ONE dominant color from later frames.
Key Constraints
1. Analyze later frames firat.
2. Use common knowledge (traffic light rules, far objects not in ego lane),
3. For color: Pick ONE most frequent color from later frames.
4. NEVER list multiple colors. Always return single color.
5. Make assumptions beyond the data.
USON Data:
(JSON data]
27

---

(Question)
Listing 8: User prompt in PLLM for accident occurrence prediction.
You analyzing JSON file representing traffic video from the ego vehicle's perspective. The video captures interactions with surrounding objects. Analyze only observable elements: bounding boxes ("bbox'), lane positions ( rotations ons ('rot_y"), distances ('distance_front attributes"), and additional environmental factors from "Video Level Information" such as weather, lighting, and road conditions. Your goal to determine whether an accident occurs and, if so, identify the frame index where it begins.
la_le_neeccaioon' object
attributes (f
USON Key Explanations
"bbox": Represents the detected object's position in the frame. Track changes in size and location to determine motion and distance.
"distance_from_ege" ". Distance (in meters) from the ego vehicle. Seattive_lame__lceee ego vehicle.
vehicle":
Description of how many lanes away object is from the
robj_lane_location Object's lane index relative to the road.
"ego_lame_loe lo._t_e_cociion Ego vehicle's lane index relative to the road.
"attributes": Object features such color.
"rot_y": Object's rotation angle, useful for detecting turns.
"loc": Object's position in space.
"abject_id" Unique identifier for objects in each frame.
"surrounding_intee info": Describes the environment, including weather, lighting, road layout, surface type, traffic flow, and time of day. "motion_state"; Indicates the motion status (e.g., Moving, Stopped) the ego vehicle.
"turn_action": Describes the ego vehicle's turning behavior.
"description": Summary of the video. "respone": Response from another model; may incorrect, it basis for reasoning.
Step-by-Step Analysis
Step 1: Identify Key Event
Categorize event as lane change, pedestrian crossing, cyclist movement, turning vehicle, steady lane position, traffic sign, unexpected object, or other notable behavior.
Check "motion_ster state" for abrupt stops and "bbox" overlaps for potential collisions
Step 2: Determine If an Accident Occurs
Always return within 10 words.
Start with "Yes" "No"
If accident detected, provide:
1. The frame number or timestamp of occurrence.
2. Example: "Yes, collision detected at frame 600.
If accident occurred: Example: "No."
If the data incomplete, make an educated guess.
1. Never state "not enough information" or "unable to determine"-yon must always pick one from "Yes" or "No"
Key Constraints
1. Do NOT mention raw numerical data from the JSON, except for the frame index.
2. Make no assumptions beyond the data.
USON Data:
(JSON data)
Did an accident occur in the video, and if so, when does it start (provide frame index)?
28

---

F Example of Extracted Data Structure
Figure presents an example of data extracted from the corresponding video (LingoQA dataset) using Finder.
G Additional Qualitative Results
We provide additional qualitative comparisons Figure and Figure 8 to further illustrate the advantages of FFinder over baseline methods. Figure 7 shows examples where iFinder corrects the peer V-VLM. Figure shows two examples where iFinder shows better grounded responses to users' questions compared to baselines.
29

---

frame index: 1": { mmttonnssaaee" "Stopped", "turn_action:; "Straight" "Video Level Information: {
ssurrounding.infoo info":
"weather": "sunny",
"light": "day",
"road_layout" "straight road",
"environment": "city street",
hoauu_/iiiiiilttyycooiiiitsssss "clear", corciidooo "normal",
"surface_type"; "asphalt",
"traffic_flo=" "light",
"time_of of of_a__of__"a "morning", "road "road_ : "no debris visible", droad_density" "normal"
ego-car-inffrmmaiion"
"frame_ sitrameiindxx: 0": { mmotion_state!s "Moving", "turn_action": "Straight"
"famm index: 2": { "motion_stater state": "Stopped", "turn_action" "Straight"
" "frame index: 3": { motion_state" "Stopped", "turn_action" "Straight"
"frame_ index: 4": { motion_state"" "Stopped", "turn_action", "Straight"
}.
"description": "The video depicts sunny day with clear visibility city street. The road is straight and devoid any debris. The traffic flow is light, with a red double-decker bus and pedestrian crossing the road. The surrounding environment includes buildings, trees, and bus stop. The scene is typical of morning commute with no significant hazards obstructions.",
"response"; "The current action is a car driving down a street. The justification is that the car is moving forward on the road."
}.
"Frame Level Information":
"frame_index": 0,
30

---

continued from previous page...
"detected objjctts:::
"class": "bus", "bbox": [677, 106, 1229, 875], "object_id": "distance_ errprce__rrooeee__eeeiiieeee "relative__am___occtiioo" "same ego vehicle", "attributes"; "Red color", "tracking.ire id":
"7.41 meters",
"class"; "person", "bbox": [180, 871], "object_id"; 11, "distance_ www._/eoo_eeeicclleeeee "relative__ane__ocata
"7.70 meters", "same lane ego vehicle", "attributes": "searing Black clothes", "loc": (-4.62, 1.33, 8.54], "rot_y": -1.64, :
{
"class"; "bicycle", "bbox": [1516, 650, 1540, 686], "object_id" 9, oettanco_frommeeo__ahiciio "41.14 meters"
"class": "bicycle",
"bbox": [1567, 1616, 731], "object_id"" 10, 75 meters"
H1.
"frame_index" 1, poatected_objectae
"class": "bus" "bbox": [811, 1087, 816], "object_id"; 2, "distance_ fwwm_frrmooooeeooeeeeoee fta_fforoomo_eebieee laan_ego_vehhiee ego_vehicle": "8.72 meters",
Dellaaiveelaae__e lame_location": "same lane ego vehicle", "attributes": "Red color", "tracking_id";
"class": "bicycle", "bbox": [1567, 670, 1616, 731], "'ojeet__aai"" id": "27.22 meters"
"class": "person",
"bbox": [180,54]] 541, 333, 871), "object_id"; 11, "12.62 meters", "same lane as ego vehicle", "attributes": "searing Black clothes", "loc"; [-3.8, 1.31, 8.0], "rot_y": -1.0, "tracking_id";
"distance_frome weatnnceefroomeeeeee wtt.f/oo_ooeeiccllllee Letiattive_laee_lccaee
31

---

continued from previous page...
"frame_index" 2, obejctt:d
"class": "bus" "bbox": [902,335, 1107, 768],
"object_id": 3, "distance_ from_erpppppr from_ego_vehicle". vehicle": "12.01 meters", Leeelative_lae_locaate
"same lane ego vehicle", "attributes": "Red color"
"loc": [0.23, 1.26, 19.88], "rot_y"; -1.43, "tracking.id"; id";
"class": "traffic light", "bbox": [1245, 1155,581, 1258, 608], "object_id". 5, "distance_froo
"79.30 meters",
"attributes": "Green light"
"class"; "bicycle", "bbox": (1512,647 1534, 678], "object_id"; 9, "distance_ from_ego__ehieeee vehicle"; "12.37 meters"
},
"frame "rrame_index". 3, obbjectedoojjects;
"class": "bus"
"bbox": [900,405, 1067, 734], "object_id"; 2, "16.71 meters",
"distance wtt.fffoooooooeeeeeeee ieeelative_lnee_ocatiio "same lane as ego vehicle", "attributes": "Red color",
"rot_y": -1.52, "loc": [-0.36, 1.46, 24.36], "tracking_id";
"class": "person",
"bbox": [959, 1048, 862], "object_id"" "distance www.__eoooeeeiieeeecoeeeeo trellative_lan__ocatiiee
"8.41 meters",
"same lane ego vehicle", "attributes": "wearing Black clothes", "loc": [-0.16, 1.47, 10.16], "rot_y": -0.31, "tracking_id"; id";
"class": "traffic light", "bbox": [814,514, 834, 834, 570], "object_ id": 19, "distance ceiistance_fro_eggo trom_ego_vehiceee "27.23 meters". "attributes"; "Green light"
},
32

---

continued from previous page... ...
"frame "rrame_iiddx"._. 4, obbjett"
"class": "bus",
"bbox": 1056, 713], "object_id": 2, "distance_ www.ffoooooeeeeeeeeeeeeeeoeoeeee trelative__ame_locatt
"22.00 meters",
"same lane as ego vehicle", "attributes": "Red color",
"loc": [-0.14, 1.46, 32.59], "rot_y": -1.53, "tracking.id":
"class"; "traffic light", "bbox": [805,518] 828, 569], "object_id"; 19, "distance_ fereeeefoooooeeeeeeeeeeeeee reg_ccei:""" "26.02 meters", "attributes": "Green light"
{
"class": "person", "bbox": [959, 680, 1013, 872], "object_id": 3, "distance from_errooooro egom_ego__o laaleeeo_veeeeeeee vehicle": "10.78 meters", "same lane as ego vehicle", "attributes"; "wearing Black clothes"
H
Figure 6: iFidee JSON data structure for LingoQA dataset video shown above
33

---

BAR CRASHES TIME
18+
Q: What is the cause of the accident??
GT: The car does not notice the coming vehicles when crossing the road.
AideoLLaMM2: The ahead decelerates stops suddenly.
iFinder: Car vered into ego-vehicle's 's lane, cansing potential collision incident.
CAR Cr CRien
Q: What the cause of the accident?
GT: The does not notice the coming vehicles when crossing the road.
VideoLLaMAA: The car's vision blocked and there is time to brake.
iFinder:
Black car (ID:6) abruptly merges into ego vchiclo's lane causing near-collision.
Figure 7: Qualitative visualization where iFinder corrects peer V- V-VLM MMAU dataset.
34

---

Q: there traffic in your lane? If so, what rypes of vehicles are present??
GT: Yes, cyclist.
VideoLLaMA2: Yes, there is traffic in the lane. The video shows red double-decker bus and other vehicles such as cars and trucks on the road.
VhaeoChat2: Yes, there is traffic in the lane. The vehicles present include a red double-decker bus, a black car, and blue van.
VideoLLaVA: Yes, there is traffic in the lane. The video shows cars and bus driving down the street.
iFinder: Yes, there is cyclist.
Q: Is there a traffic light in sight? If so, what color is exhibiting?
GT: Yes, green.
VideoLLaMA2: Yes, there traffic light visible in the video. It appears to be displaying red light.
Yes, there is traffic light in sight. It is exhibiting a red light.
VideoChat2:
VideoLLaVA: Yes, there is traffic light in sight, and it is displaying a red light.
iFinder: Yes, there is a traffic light in sight, and it is exhibiting a green light.
Figure 8: Qualitative comparison against baselines LingoQA dataset.
35


Reviews:
[{"review_id": "TgyDzeqqbw", "reviewer": "Reviewer_zz3N", "summary": "The paper introduces ı̇Finder, a training-free pipeline for post-hoc reasoning over dash-cam videos. It decouples perception from language reasoning by (i) extracting driving-specific cues—object pose, lane location, distances, ego motion, 3D orientation and scene context—using off-the-shelf vision models, (ii) packaging those cues into a hierarchical JSON structure, and (iii) feeding the structure, together with a peer VLM’s first-pass answer, to a GPT-4-class LLM via a three-block prompt (key explanations, step instructions, peer instruction). On four zero-shot driving benchmarks (MM-AU, SUTD-TrafficQA, LingoQA, Nexar), experimental results show that ı̇Finder outperforms both general-purpose Video-VLMs and driving-specialised models.", "questions": "1. What is the end-to-end runtime of ı̇Finder per 5s/10s/30s/60s video? A breakdown would clarify deployment feasibility.\n\n2. If stronger/weaker object trackers or depth estimators are substituted, do the gains persist? Ablation study can be conducted by using weaker object trackers and check their performance with iFinder.     \n\n3. Discuss two weaknesses.", "limitations": "They discussed it in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- It has solid engineering for all vision modules and connect them successfully with GPT-4o.   \n\n- The paper has rigorous zero-shot evaluation on 4 public ego-view driving benchmarks. Each of them contain problems that current VLMs can not resolve alone.    \n\n- The paper combines peer-LLM self-correction with structured visual grounding—a fresh twist on “LLM-agent” ideas for autonomous driving.     \n\n**Weaknesses**\n\n- Real-time latency/cost of running eight vision modules is not analysed. This might be a big problem if one or more module take a long time during inference. Besides, It relies exclusively on existing pretrained components. What if some module make mistakes? This is not discuss in the ablation study.   \n\n- It seems that gains come largely from adding extra model-derived priors (3D orientation, depth) rather than novel learning; may be less compelling once future VideoVLMs close the gap.", "was_revised": true, "final_justification": "Thanks for the author's reply. Although the latency is still a issue for the proposed model, I really appreciate comparison and detailed explanation of the answers. I will remain my positive score.", "rebuttal": "Thank you for the thoughtful comments and support for our paper. Below are our responses to your questions. Please do not hesitate to raise further questions.\n\n----\n- Response to  ***Real-time latency/cost of running eight vision modules is not analysed. This might be a big problem if one or more module take a long time during inference.***  \n    - We have analyzed the wall-clock run-time of each module in Table 1 in supplementary material. The full pipeline processes a LingoQA video in <4.5 minutes with unoptimized Python (Table 1 in supplementary material), suitable for offline post-hoc video analysis.\n\n----\n\n- Response to ***Besides, It relies exclusively on existing pretrained components. What if some module make mistakes? This is not discuss in the ablation study.*** \n    - We analyzed the impact of error propagation (or effect of compounding errors when the initial module, in our case the object detector, fails to detect objects in the scene) in Table 7 on MM-AU dataset. For example, to imitate this detection failure, we remove >97% of object detections (τ = 0.8) from the JSON files and observed that the overall accuracy still remains 58.27%, surpassing all baselines including VideoLLaMA2 (52.89%) (Table 7). This shows that our symbolic reasoning retains reliability under perceptual degradation. Further, Table 7 also shows that iFinder demonstrates graceful degradation and limited error propagation. Please see L326-333 for a detailed discussion. \n\n----\n\n- Response to ***It seems that gains come largely from adding extra model-derived priors (3D orientation, depth) rather than novel learning; may be less compelling once future VideoVLMs close the gap.*** \n    - We respectfully clarify that the gains in ı̇Finder result not from simply adding priors, but from a novel structured grounding framework that integrates visual cues into LLM reasoning via symbolic representations. Three quantitative findings support this:\n        - Removing 3D orientation reduces MM-AU accuracy from 63.39% → 58.83%, but this is still 5.94% above VideoLLaMA2 (52.89%), showing priors are necessary but not sufficient (Table 6). The largest drop occurs only when structured scene understanding is removed (→ 57.81%).\n        - End-to-end VLMs do not close the gap despite strong priors. VideoLLaMA2 and DriveMM have implicit access to visual cues but still lag by 10.5% and 39.17%, respectively, on MM-AU (Table 1). Structured reasoning over priors is critical, not their mere presence.\n        - Removing only “Key Explanation” in our prompt drops accuracy by 4.66%, more than removing any single visual input (Table 6). This suggests performance stems from reasoning over structure, not data quantity.\n\n    Our method is modular, zero-shot, and interpretable, offering complementary strengths to future end-to-end VLMs. We hope this clarifies that ı̇Finder is not a wrapper over priors, but a distinct, architectural contribution to structured LLM grounding.\n\n----\n\n - Response to ***What is the end-to-end runtime of ı̇Finder per 5s/10s/30s/60s video? A breakdown would clarify deployment feasibility.*** \n    - On a single A6000 GPU with 48 GB memory, the end-to-end runtime of ı̇Finder for video inputs of 5 seconds, 10 seconds, and 30 seconds (at 10 FPS with image size 640x360) is approximately 2.33 minutes, 3.85 minutes, and 18.75 minutes, respectively.\n\n----\n\n- Response to ***If stronger/weaker object trackers or depth estimators are substituted, do the gains persist? Ablation study can be conducted by using weaker object trackers and check their performance with iFinder.*** \n    - Thanks for the suggestion. We  replaced the tracker and distance estimation modules with their weaker counterparts and analyzed the performance of iFinder on MM-AU dataset. We observed that replacing the tracker with a weaker tracker (simple IoU based tracker vs ByteTracker) drops performance slightly to 61.70%, indicating that tracking quality contributes moderately to system accuracy. Similarly, using a weaker depth model (Base vs Large) causes a minor drop to 62.78%.\n\n----\n\n- Response to ***Discuss two weaknesses.*** \n    - We provide two observable weaknesses w.r.t. post-hoc analysis that future works can build upon.\n        - The goal of this work is to demonstrate how vision-based modules can effectively ground large language models (LLMs) for robust reasoning, without relying on external sensors like LiDAR or GPS. By focusing solely on visual input, iFinder highlights the potential of vision as a standalone modality for scene understanding and decision-making. While current implementation does not integrate LiDAR or GPS data, this design choice underscores the strength of vision-driven grounding and leaves open the possibility for future multi-modal extensions to further enhance performance in complex scenarios.\n        - Being entirely training-free, iFinder cannot self-correct based on prior mistakes or user feedback. Such adaptability can be built-in by adding in-context examples or LLM fine-tuning.\n  \n----"}, {"review_id": "7TfqRBrp2u", "reviewer": "Reviewer_XuKK", "summary": "The paper presents iFinder, a framework for zero-shot post-hoc reasoning on dash-cam videos. It decouples perception and reasoning by converting raw video into structured, hierarchical data (including object orientation, lane position, and ego motion), which is then passed to a general-purpose LLM. Such an approach enables interpretable, grounded reasoning and significantly improves performance over existing V-VLMs on 4 benchmarks. Despite its strong results, the possible use of the proposed framework is most likely limited \"just\" to offline analysis.", "questions": "* _Q1:_ Could the authors provide rough estimates of pipeline inference time and resource requirements? Even _coarse_ timing would be useful.\n* _Q2:_ Is there a plan to extend ı̇Finder toward semi-real-time scenarios (e.g., batch processing at the edge)? If not, what would be the major technical problem?\n* _Q3:_ How adaptable is the pipeline to other domains beyond driving? Could the same framework be applied to indoor navigation or surveillance video with minor changes?\n* _Q4:_ Would the authors consider evaluating ı̇Finder on more challenging or domain-shifted settings such as those proposed in [1]? These benchmarks focus on out-of-distribution hazard identification and symbolic reasoning, and would align well with ı̇Finder’s strengths.\n* _Q5:_ Since many LLMs are inherently non-deterministic, could the authors clarify whether multiple runs of the reasoning stage produce consistent outputs? Given the structured symbolic input and multi-step prompt design, some variability is expected, but it would be helpful to know if the system’s answers are stable in practice. Did the authors use any measures (e.g., fixed temperature, response filtering, or prompt anchoring) to manage output variability?", "limitations": "Mostly yes. The authors acknowledge the offline nature of iFinder and briefly mention its lack of real-time deployment support. However, several additional limitations deserve discussion. \n1. The system relies on many pre-trained models, each of which may introduce noise, yet, no formal analysis of this sensitivity is provided. \n2. The scene representation is handcrafted, which limits the method’s generalization. \n3. While the system claims interpretability and robustness, no formal evaluation or consistency analysis is conducted to support it.\n4. The results are most likely based on single-seed evaluation without statistical significance testing", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Note**: _Despite I'm a little bit \"negative\" in the review, I believe this is a strong and well-written paper. The design, structured reasoning approach, and strong zero-shot results make it an extremely valuable contribution. While there are areas for improvement, I view these more as opportunities for future work than fundamental flaws._\n\n### **Strengths**\n- **STRUCTURE and LANGUAGE**: The paper follows a standard structure, which makes it easier to follow and read. Overall, the paper is well-written. The level of English is great. Figures are of good quality and appropriately support the clarity. Formally, everything is on a high level.\n\n- **REPRODUCIBILITY**: Although the code is not provided, the paper and supplementary materials present outstanding implementation details, making it potentially \"easy\" to reproduce. Despite the large complexity and inherently non-deterministic nature of some LLMs, I believe there is no better way how to do it.\n\n- **METHOD DESIGN**: The core idea is well-designed and implemented. The framework is modular, does not require additional fine-tuning/training, and introduces a principled way to integrate domain knowledge into LLM reasoning via hierarchical scene representations and well-thought-out prompting.\n\n- **STRONG PERFORMANCE**: The method achieves consistent and significant improvements across four benchmarks, outperforming both general-purpose and domain-specific video-language models in zero-shot settings. Notably, the \"performance\" gains are much higher in tasks requiring spatial and causal reasoning.\n\n- **RELEVANCE**: The paper addresses a highly relevant problem in vision-language understanding for not just autonomous systems. Post-hoc reasoning from dash-cam videos is a challenging yet increasingly important task, especially given the limitations of real-time data collection in real-world driving environments.\n\n### **Weaknesses**\n\n- **SYSTEM COMPLEXITY**: The framework relies on many pretrained models (e.g., for object detection, depth estimation, 3D orientation, lane detection, ego-motion), which increases complexity and potential points of failure. Besides, the paper does not provide any system-level runtime or memory analysis, making it more challenging to assess practical deployment costs.\n\n- **RELIANCE ON RULE-BASED STRUCTURE**: The scene representation is handcrafted and domain-specific. While effective for driving, it’s unclear how easily this structure transfers to other domains (e.g., indoor navigation, robotics), or how it would perform under noisy or ambiguous visual inputs.\n\n- **UNCLEAR PEER CONTRIBUTION**: The peer V-VLM is treated as a noisy first-pass guess, but its actual contribution to the final performance is not deeply analyzed. A deeper ablation (e.g., replacing the peer) would clarify whether it meaningfully supports the reasoning or serves as a distractor.\n\n\n**[Additional Comments]:**\n- The Related Work section is rather short. While key methods are covered,  recent works on structured reasoning and OOD evaluation [1, 2, 3], particularly those addressing zero-shot hazard identification under domain shift, are missing, while it could be beneficial to include them. Evaluating iFinder on such a dataset/benchmark would significantly strengthen the paper’s narrative around robustness and generalization. Given the method’s zero-shot, modular nature, this would be a natural and valuable extension.\n- The qualitative examples are useful and illustrate the reasoning capabilities well. However, it would be valuable to also include some failure cases with a brief analysis, especially where the LLM misinterprets the structured input or the visual modules produce conflicting cues. \n- The authors apparently misunderstand what is expected regarding statistical significance. Reporting results from a single seed is insufficient, even if followed by a qualitative error analysis. Statistical significance typically requires multiple runs with different random seeds and corresponding variance or confidence intervals. The current justification (\"we provide an error analysis in the Supplementary Material\") does not address it. Although I understand the computation cost is rather heavy, a few repeated runs with 2–3 random seeds would make the results more trustworthy and support stronger conclusions.\n\n**[Grammar & Typos & Formating]**:\n- Line 240: The JSON example is overflowing the line width.\n- Table 1-7: The random verticall allignement does not look apealing. While minor, it feels like a huge ink stain on an otherwise good-looking and polished submission.\n\n___________________________________\n\n[1] @article{alshami2024coool,\n  title={Coool: Challenge of out-of-label a novel benchmark for autonomous driving},\n  author={AlShami, Ali K and Kalita, Ananya and Rabinowitz, Ryan and Lam, Khang and Bezbarua, Rishabh and Boult, Terrance and Kalita, Jugal},\n  journal={arXiv preprint arXiv:2412.05462},\n  year={2024}\n}\n\n[2] @inproceedings{picek2025zero,\n  title={Zero-shot hazard identification in Autonomous Driving: A Case Study on the COOOL Benchmark},\n  author={Picek, Lukas and Cermak, Vojtech and Hanzl, Marek},\n  booktitle={Proceedings of the Winter Conference on Applications of Computer Vision},\n  pages={654--663},\n  year={2025}\n}\n\n[3] @article{shriram2025towards,\n  title={Towards a multi-agent vision-language system for zero-shot novel hazardous object detection for autonomous driving safety},\n  author={Shriram, Shashank and Perisetla, Srinivasa and Keskar, Aryan and Krishnaswamy, Harsha and Bossen, Tonko Emil Westerhof and M{\\o}gelmose, Andreas and Greer, Ross},\n  journal={arXiv preprint arXiv:2504.13399},\n  year={2025}\n}", "was_revised": true, "final_justification": "The authors have addressed my main concerns with clear responses and additional results. If included in the camera-ready, it will largely improve the quality and transparency of the paper. \n\nReading the other reviewers’ comments did not change my mind about the contributions of the provided paper, and I will keep my rating.", "rebuttal": "Thank you for such a thorough and insightful review of our paper. We also appreciate the reviewer's strong support for the paper. Below are our responses to your questions. Please do not hesitate to raise further questions.\n\n----\n**Response to Weakness 1 \"SYSTEM COMPLEXITY\" and Q1:** We have analyzed the wall-clock run-time of each module in Table 1 in supplementary material. The full pipeline processes a LingoQA video in <4.5 minutes with unoptimized Python (Table 1 in supplementary material), suitable for offline post-hoc video analysis. Further (based on Reviewer zz3N's suggestion) we note the following. On a single A6000 GPU with 48 GB memory, the end-to-end runtime of ı̇Finder for video inputs of 5 seconds, 10 seconds, and 30 seconds (at 10 FPS with image size 640x360) is approximately 2.33 minutes, 3.85 minutes, and 18.75 minutes, respectively.\n\n----\n**Response to Weakness 2 \"RELIANCE ON RULE-BASED STRUCTURE\":** We agree that ı̇Finder targets the driving domain. However, we highlight the following that would enable generalization:\n   - Our representation is auto-generated via standard pretrained models (e.g., OWL-V2, SAM, CenterTrack) with no task-specific fine-tuning (Sec. 3.1). This decoupled design generalizes to any domain where object-level cues (position, orientation, interaction) are relevant.\n   - The robustness under visual noise is empirically evaluated on MM-AU. Even after filtering out >97% of object detections (τ = 0.8), accuracy remains 58.27%, surpassing all baselines including VideoLLaMA2 (52.89%) (Table 7). Our symbolic reasoning retains reliability under perceptual degradation.\n   - In Table 5, we also show that cross-condition generalization is strong. In adversarial settings (rain, snow, night), ı̇Finder outperforms the best V-VLM by 10–20% across all categories (Table 5), despite no domain adaptation. This suggests high resilience even under covariate shift.\n\nWhile domain transfer is not the paper’s focus, we believe ı̇Finder offers a template for symbolic grounding that can be instantiated in new domains via different pretrained perception modules. We will include this as a direction for future work.\n\n----\n**Response to Weakness 3 \"UNCLEAR PEER CONTRIBUTION\":** Note that iFinder improves the peer V-VLM by correcting its initial response with the proposed representational abstraction of vision inputs that enables symbolic reasoning over dynamic scenes using LLMs. We provided the impact of different peers within iFinder on LingoQA dataset in Table below. It can be observed that iFinder can improve enhance the overall performance irrespective of  the peer V-VLM involved.\n|                      | Acc(%) |\n|:--------------------:|:------:|\n| VideoLLama2          | 36.00  |\n| **Ours** (w/ VideoLLama2) | **41.20**  |\n| VideoChat2           | 41.20  |\n| **Ours** (w/ VideoChat2)  | **44.20**  |\n\n----\n**Response to [Additional Comments] 1 & Q4:** \n - We completely agree with the Reviewer that adding a discussion as well as evaluating iFinder on works like reviewer references [1, 2, 3] would demonstrate interesting insights and strengthen iFinder's position. We will add a discussion on these works in the related works section in the revised version of the manuscript.\n - In response to the reviewer’s suggestion to evaluate on the \"Coool\" benchmark [1], we explored the corresponding GitHub repository but found that ground truth annotations were not publicly available as of July 31, 2025. Despite this limitation, we conducted a qualitative analysis using an example to demonstrate the strengths of iFinder in Table below.\n| video name     | Github Baseline response (manually reformatted for better readability in the table)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |  iFinder response                                                                                                                                                                                                       | GT (visually observed)                                                               |\n|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|\n| video_0002.mp4 | Frame 2-109, 151-360: there is a white and black cat sitting on a table  focus and ( gaussian blur )  144p  may 1 0  white fog  extreme panoramic  zoom  see  zoom out  full width  teaser  dribbble 8k  8 k matte painting  may 6 8  high resolution image  untextured  without glasses  - signature Frame 110-150: there is a deer that is running down the street  discord profile picture  low quality footage  his legs spread apart  cryptid  captures emotion and movement  extremely close shot  half dog  loosely cropped  youtube thumbnail  moose  dynamic closeup | Animal in same lane as ego vehicle (frames 102-108, 116-119). Animal changing lanes unpredictably (frames 120-125). Car approaching ego vehicle (frames 151-152, 157-159). Debris on wet road (surrounding_info field). | An animal ran from the right side and crossed in front of the ego car to the left. |\n - Note that we evaluated iFinder under adversarial conditions (Table 5) in \"Foggy\", \"Rainy\", \"Snowy\" and \"Night\" conditions where it outperforms baselines (VideoLLama2 and VideoLLaVA) by (8.33%, 20.69%, 17.61%, 13.7%) and (16.67%, 24.14%, 15.09%, 17.35%), respectively. The improved accuracy and robustness validate the extremely significant value of leveraging vision-language grounding offered by iFinder. \n\n----\n **Response to [Additional Comments] 2:** We agree with the reviewer's suggestion. We will add the failure cases of iFinder in the revised manuscript. Please see below two examples from the Nexar dataset, marked by their respective video names (.mp4 files) where iFinder did not produce the expected response. We provide an analysis on possible reasons for these examples as well.\n   - “01031”: This dash cam video shows a vehicle cutting in from the left side of the ego car to the front. Since there is no obvious steering or vibration caused by a collision, it is even difficult for humans to determine whether a collision actually occurred.\n   - “00970”: This dash cam video shows a vehicle suddenly braking in front of the ego car. Although it is very close to the ego car, the video also shows the ego car braking, so it is not clear from the video whether an actual collision occurred.\n\n\n----\n **Response to [Additional Comments] 3 & Limitations 4 & Q5:** Based on reviewer's suggestion, we ran the iFinder module end-to-end with different random seeds on LingoQA dataset and provide the analysis below. Also note that we used fixed temperature for the LLM.\n\n|           | Acc(%) |\n|:------:|:------:|\n| Run 1 | 43.80  |\n| Run 2 | 44.20  |\n| Run 3 | 44.20  |\n\n----\n **Response to comments under [Grammar & Typos & Formating]:** Thank you for pointing this out. We agree with the reviewer and will fix this in the final version of the manuscript.\n\n----\n**Response to Q2:** Yes, we plan to extend iFinder toward semi-real-time scenarios (e.g. using quantized version of each module). The major (and exciting) challenge on this will be to maintain the performance of the adopted modules in their quantized form. \n\n----\n**Response to Q3 & Limitations 2:** Yes, iFinder’s pipeline is adaptable to domains beyond driving. This is because its schema—temporal object-centric data—is domain-structured, not domain-locked. Its modular design allows substitution of perception modules for other domains (e.g., indoor robotics), and the JSON format can accommodate arbitrary scene cues. Our ablations (Table 6) show performance is robust to component removal, indicating generalizable reasoning.\n\n----"}, {"review_id": "TdIBCAqrH2", "reviewer": "Reviewer_XWwS", "summary": "ı̇Finder is a zero-shot framework that grounds LLMs for driving video analysis by decoupling perception and reasoning. It converts dash-cam videos into structured, interpretable representations, capturing object motion, lane position, and orientation, etc. It also uses them to guide LLMs in accurate, causal reasoning, outperforming baselines on multiple benchmarks.", "questions": "1. What is the contributions of this paper?\n2. What is the inferencing time and memory usage during inference?\n3. What is the potential direction to improve the efficiency based on the current design?\n4. Why driving-specialized baselines performance pretty bad?", "limitations": "I think the largest limitation of the method is the inference latency, which is not mentioned in the limitation in the supplementary materials.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper is well-written and easy to follow.\n2. This method requires no training and has good interpretability. \n3. This method achieves competitive results among several benchmarks.\n\nWeakness\n1. My main concern is that the paper’s contribution is not clearly defined or significant. There is no explicit contribution statement, and the introduction only outlines three experimental findings without a clear methodological or conceptual innovation. This makes it difficult to assess the novelty and impact of the work.\n2. This paper introduces seven additional intermediate processing steps for dash-cam video reasoning, but nearly all of them rely on separate off-the-shelf models to extract specific cues. While the pipeline is training-free, the use of multiple models makes the system quite heavy. I am curious about the overall memory usage during inference, as excessive memory demands could make the approach impractical for real-time or resource-constrained driving scenarios.\n3. From Lines 252 to 255, the paper states that real-time performance is not the focus, emphasizing interpretability and accuracy instead. However, in driving scenarios, inference efficiency is just as critical as accuracy. The proposed method relies on seven intermediate modules and a peer V-VLM for final LLM processing, which raises concerns about latency and scalability. If the improved accuracy comes at the cost of significantly slower inference, the practical value of the approach becomes questionable. At least, the paper should report and compare inference time with existing baselines. Additionally, it would strengthen the work to propose a possible direction for improving efficiency based on the current design.\n4. In Tables 1–3, the driving-specialized baselines (DriveMM and WiseAD) perform significantly worse than generalist models. Notably, DriveMM achieves only 49% accuracy in Table 3, which is worse than random guessing (50%). This raises concerns about whether these methods are truly strong baselines or representative of the state-of-the-art driving-specialized  methods in driving video understanding.", "was_revised": true, "final_justification": "I acknowledge the contribution of this work in enabling symbolic reasoning over dynamic scenes using LLMs, which is an interesting and promising direction. The incorporation of more efficient sub-modules is also a practical design choice that helps alleviate the efficiency concerns to some extent.\n\nThat said, the technical novelty of the paper appears somewhat limited. The proposed approach primarily integrates existing components in a relatively straightforward manner, which diminishes the strength of its technical contribution.\n\nIn summary, I will raise my score to a 4, as most of my concerns have been addressed. However, I still believe that the technical depth of the paper is limited, despite the overall direction being compelling.", "rebuttal": "Thank you for the thoughtful and encouraging review of our paper. Below are our responses to your questions. Please do not hesitate to raise further questions.\n\n----\n\n**Response to Weakness 1 & Question 1.** The conceptual contribution of ı̇Finder is the idea of a representational abstraction of vision inputs that enables symbolic reasoning over dynamic scenes using LLMs. In other words, rather than passing raw pixel inputs or latent embeddings to the LLM, we transform each dash-cam video into a structured, temporally indexed JSON format. This representation bridges the gap between low-level perception and high-level reasoning, acting as a semantic interface between vision modules and language models.\n\n----\n**Response to Weakness 2 & Question 2.** We agree that ı̇Finder is not optimized for real-time deployment. As stated in the end of Sec. 3.1.2, we emphasize that ı̇Finder targets interpretability-critical scenarios, not embedded real-time inference. To clarify: \n - Note that modular design ≠ high memory usage. All modules run sequentially or in parallel without maintaining persistent GPU state. Peak memory remains below 48 GB on an A6000 GPU (see Appendix B). The largest modules—frame undistortion and object attribute extraction—are image-based and not memory-bound.\n - Wall-clock runtime is acceptable for post-hoc use. The full pipeline processes a LingoQA video in <4.5 minutes with unoptimized Python (Table 1 in supplementary material), suitable for offline analysis.\n - We can achieve scalability via decoupling. Each module can be independently swapped for faster or lower-memory alternatives (e.g., lighter depth or 3D estimators), offering scalability based on deployment needs.\n\nWe will clarify this use-case focus in the final version. \n\n\n----\n**Response to Weakness 3 & Question 3.** \n - As iFinder is built on top of correcting peer V-VLM (VLMs designed for high-level reasoning), it incurs higher computational cost relatively. However, this cost is justified by a notable increase in performance, particularly in tasks requiring nuanced understanding and disambiguation. For example in LingoQA dataset (Table 2), VideoLLama2 and VideoLLaVA incur 2.07 sec and 2.04 sec, respectively. However, iFinder (with 165 sec) outperforms them by 8.2% and 23.2%, respectively. Similarly under adversarial conditions (Table 5) in \"Foggy\", \"Rainy\", \"Snowy\" and \"Night\" conditions, iFinder outperforms them by (8.33%, 20.69%, 17.61%, 13.7%) and (16.67%, 24.14%, 15.09%, 17.35%), respectively. The improved accuracy and robustness validate the extremely significant value of leveraging vision-language grounding offered by iFinder.\n\n - There are two approaches to improve the framework’s efficiency:\n    - **Limited Compute (e.g., few GPUs):** Replace each model with a more efficient alternative to reduce latency.\n    - **Sufficient Compute (e.g., 8×A6000 GPUs):** Host API versions of the models on a GPU node. The iFinder vision pipeline can then query these APIs and retrieve JSON outputs. Additionally, enable parallel execution of independent modules, and aggregate their outputs into a final JSON.\n\n\n----\n**Response to Weakness 4 & Question 4.**  We understand the concern and clarify the following:\n - DriveMM [16] and WiseAD [18] are SOTA domain-specialized baselines for ego-centric driving scene understanding. We use the official implementations and pre-trained checkpoints. DriveMM (49%) and WiseAD (50%) on Nexar (Table 3) do not perform worse than chance; the class distribution for the experiment is 50/50 by design (see Appendix C in supplementary material). We investigated this performance further and observed high recall + low precision (e.g., 1.00 recall, 0.50 precision in WiseAD) which indicates bias towards predicting accidents (reported in Table 3).\n - Further note that, generalist models outperform driving-specialized models due to massive video pre-training. Models like VideoChat2 and VideoLLaMA2 benefit from scaling, not task specificity. That they outperform DriveMM highlights a gap in domain-adapted VLMs, a key motivation for ı̇Finder’s structured grounding.\n\n----"}, {"review_id": "3vOOj8Mr9t", "reviewer": "Reviewer_qGHL", "summary": "The paper introduces a structured semantic grounding framework, iFinder, to improve zero-shot vision-based reasoning of dash-cam driving videos by utilizing the reasoning capabilities of large language models. Instead of depending on end-to-end vision-language models (that are capable of video processing), this framework separates perception from reasoning. It uses pretrained vision models to extract driving-related information, such as object detection, lane position, and depth estimation. The framework then organizes the driving-related information and feeds this information to an LLM via structured prompts. This approach enables step-wise, grounded reasoning, leading to more accurate and interpretable driving scenario analyses than conventional VLMs, particularly in accident causation and hazard detection.", "questions": "Q1. How do you determine the prompts? I.e., how do you know the current prompts are \"optimal\"? Are there any ablation studies with varied prompts?\n\nQ2. How does the choice of the LLM (in Fig. 2) affect the performance?\n\nQ3. Can this framework be generalized to other domains? Or, will the framework still work if I remove a subset of modules (e.g., remove 2D object detection + tracking and keep the remaining modules)? What if I add more modules (e.g., global latitude & longitude)?\n\nQ4. A follow-up for Q3: Do you need to refine the prompts or retrain the LLM when adding or removing modules?\n\nQ5: Is this framework adaptable to multi-view cameras? E.g., self-driving cars have six cameras at different angles.", "limitations": "L1. The authors mention that real-time deployment is not a concern of this work. However, the authors could have a discussion on the applications of this framework beyond real-time deployment. I'm not clear where this framework can be applied except for real-time deployment.\n\nL2. While the paper includes some analysis on error propagation (showing graceful degradation when object detection confidence is low), it would be valuable to see a deeper discussion about how sensitive the entire pipeline is to failures or inaccuracies in upstream modules (e.g., tracking, depth estimation). How does ı̇Finder handle compounding errors, and are there mechanisms for uncertainty quantification or correction? Is there a way to provide probabilistic guarantees to the framework's outcomes?\n\nL3. The framework's performance heavily relies on the accuracy of multiple off-the-shelf vision models (e.g., for detection, tracking, lane estimation, depth, etc.). The overall reasoning can degrade if these modules perform poorly due to unusual conditions or domain shift.\n\nL4. The paper lacks a discussion of limitations and future work in the main body.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper provides structured, symbolic cues (such as object orientation, lane context, and object distance) to the LLM, enabling transparent reasoning, making the analysis process more interpretable and trustworthy for complex driving scenarios.\n- Sufficient empirical analysis shows that the framework significantly outperforms both general-purpose and driving-specific VLMs on multiple zero-shot driving benchmarks.\n- The framework uses a modular design based on pretrained vision models, which allows easy integration and parallelization of new perception modules without retraining.\n\n**Weaknesses**\n- The framework heavily relies on the quality of the existing deep learning models.\n- The framework cannot provide any guarantees for their outcomes.\n\n*Note to the authors*: More details of the weaknesses are in **Questions** and **Limitations**, so please respond to those sections ONLY during the rebuttal.", "was_revised": true, "final_justification": "The authors have addressed my concerns and added the necessary results. So, I decide to increase my score.", "rebuttal": "Thank you for the insightful review of our paper. Below are our responses to your questions. Please do not hesitate to raise further questions.\n\n----\n\n**Responses to Questions** \\\n**Response to Q1.** We agree that prompt design plays a crucial role in LLM-based reasoning and clarify the following:\n - Prompts are designed based on functional decomposition. As detailed in Sec. 3.1.2, our three-block prompt (Key Explanation, Step Instructions, Peer Response) is motivated by established findings in structured prompting and step-wise LLM reasoning [60–63]. It explicitly aligns with the hierarchical scene representation, guiding the LLM to reason over symbolic inputs.\n-  Yes, we perform prompt ablations in Table 6 and provide quantitative ablations over prompt components. Removing \"Key Explanation\" drops MM-AU accuracy by 4.66% (63.39 → 58.73), \"Step Instructions\" drops performance by 3.33%, and \"Peer Instruction\" drops performance by 2.77%.\n- Prompt design was not searched for optimality. Rather, it is guided by alignment between input structure and reasoning goals, and validated empirically via ablations (in Table 6). For example, if we drop a module from iFinder for ablation, we removed it's corresponding explanation from the main prompt \n\n\n----\n**Response to Q2.** Based on the reviewer's recommendation, we analyze the LLM choice on iFinder and present the analysis below (MM-AU dataset). \n| LLM                    | Acc (%) |\n|------------------------|---------|\n| GPT-4o-mini (in paper) | 63.39   |\n| GPT-4.1-mini           | 63.65   |\n| GPT-4.1-nano           | 52.53   |\n\n\n\n----\n**Response to Q3.**\n - **Generalization to other domains:** Yes, the iFinder framework can be generalized to other domains. This is because the iFinder pipeline is explicitly designed as a modular, training-free framework to facilitate flexibility in incorporating or removing specific vision components. The core idea remains the same: representational abstraction of vision information that enables symbolic reasoning over dynamic scenes in language via LLM. The schema—temporal object-centric data—is domain-structured, not domain-locked. Its modular design allows substitution of perception modules for other domains (e.g., indoor robotics), and the JSON format can accommodate arbitrary scene cues. Our ablations (Table 6) show performance is robust to component removal, indicating generalizable reasoning.\n - **Adding /Removing modules to iFinder:** As shown in our method (Section 3.1) and ablation results (Table 6), each module contributes uniquely to the final reasoning output, and modules can be swapped or extended without retraining. As aforementioned,   adding or removing modules would only affect the instructions sent to the LLM to understand the JSON file.\n\n\n\n----\n**Response to Q4.** We only need to change the prompt to include/remove information about added/removed modules for the LLM to understand the JSON file. No retraining of LLM is needed. \n\n----\n**Response to Q5.** Yes, ı̇Finder can be adapted to multi-view camera setups. Since our framework is modular and operates on per-frame visual inputs, each view can be processed independently through the same perception modules. The resulting structured cues (e.g., object pose, orientation, depth) from each camera can then be merged into a unified symbolic representation using view-specific metadata (e.g., extrinsics). While our current work focuses on monocular front-view videos, extending the pipeline to multi-view settings involves modifying the scene representation to accommodate cross-view spatial alignment—an area we identify as promising future work for enhancing coverage and reducing occlusion in complex scenarios.\n\n\n----\n----\n**Responses to Limitations** \\\n**Response to L1.** We thank the reviewer for raising this point. While we explicitly mention that real-time deployment is not a focus of our work, the primary goal of our framework is to enable post-hoc, structured analysis of driving scenes—a task that is fundamentally distinct from real-time inference. Our method is particularly suited for **offline** applications such as accident investigation, insurance assessment, driver behavior auditing, and fleet-level safety analytics, where *interpretability, causal attribution*, and *verifiable reasoning* are critical. As shown in our evaluations on MM-AU, SUTD, and Nexar datasets, our framework excels in post-event reasoning tasks, outperforming both generalist and driving-specific VLMs in accuracy and robustness. For example, On the MM-AU dataset—focused explicitly on accident reasoning—iFinder achieves 63.39% accuracy, significantly outperforming the best generalist V-VLM (VideoLLaMA2 with prompt tuning) at 52.89%, and the best driving-specialized model (DriveMM) at 24.22%.\n\n----\n**Response to L2.**\n\n- We analyzed the impact of error propagation (or effect of compounding errors when the initial module, in our case the object detector, fails to detect objects in the scene) in Table 7 on MM-AU dataset. For example, to imitate this detection failure, we remove >97% of object detections (τ = 0.8) from the JSON files and observed that the overall accuracy still remains 58.27%, surpassing all baselines including VideoLLaMA2 (52.89%) (Table 7). This shows that our symbolic reasoning retains reliability under perceptual degradation. Further, Table 7 also shows that iFinder demonstrates graceful degradation and limited error propagation. Please see L326-333 for a detailed discussion. \n - Also based on Reviewer zz3N ‘s suggestion, we performed a different error propagation analysis: replacing modules with their weaker counterparts for the object tracker and the depth estimation on MM-AU dataset. We observed that replacing the tracker with a weaker tracker drops performance slightly to 61.70%, indicating that tracking quality contributes moderately to system accuracy. Similarly, using a weaker depth model causes a minor drop to 62.78%.\n - Kindly note that while uncertainty quantification is not explicitly implemented in the current version of iFinder, the underlying LLM can support it. For instance, methods like Monte Carlo dropout, and token-level entropy estimation in the LLM’s outputs could be integrated to estimate uncertainty. This is a promising direction for future work. Further, discussing probabilistic guarantees for iFinder’s outcomes is currently beyond the scope of the paper.  \n\n\n----\n**Response to L3.** While iFinder integrates multiple vision modules, its robustness to degraded perception is quantitatively demonstrated. As shown in Table 5, iFinder outperforms all baselines under adverse conditions—achieving 65.52% in rain, 57.86% in snow, and 63.93% at night, while the best generalist model (VideoLLaMA2) drops to 44.83%, 40.25%, and 50.23%, respectively. These results underscore that iFinder’s structured grounding mitigates failure cascades and ensures stable, domain-shift-tolerant reasoning.\n\n\n----\n**Response to L4.** Thank you for highlighting this. Owing to space constraints, we included these discussions in the supplementary material. We will incorporate them into the main body of the manuscript in the final version.\n\n----"}]

Author rebuttal:

Meta-review: The paper presented iFinder, a novel, modular, training-free framework for dash-cam video reasoning by LLM. iFinder is a structured grounding framework that separates low-level perception from high-level reasoning to enhance LLM-based video understanding. This work designs a modular and training-free pipeline that combines multiple pretrained perception modules to generate interpretable inputs for LLMs. The experiments results show strong zero-shot performance on multiple driving datasets with thorough ablations on prompt structure, perception module substitution, and adversarial conditions. The reviewers also pointed out some weaknesses, including the limited novelty in model components, efficiency issue due to the multi-module pipeline, 
During the rebuttal phase, the authors effectively addressed the reviewers' concerns about latency, ablation coverage, and adaptability. The final ratings include a Strong Accept, and 3 Borderline Accepts, with 2 review ratings raised from initial Borderline Rejects). The authors' rebuttal improved the perception of the paper. The authors provided clarifications on latency, ablation studies, peer model contribution, and robustness to perception noise. The two initially most critical reviewers (XWwS and qGHL) explicitly noting that the authors' rebuttals addressed their concerns, justifying their final score adjustments.
In summary, this paper is technically sound, addresses an important problem in a principled manner, and shows promising experimental results on four benchmarking datasets. This paper provides excellent contributions to grounded vision-language reasoning. The AC agrees with the reviewers' suggestions and recommends accepting the paper.
Decision: Accept (poster)