Title: Scene-R1: Video-Grounded Large Language Models  for 3D Scene Reasoning without 3D Annotations
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Vision Grounding, Video-Grounded LLM, Reinforcement Learning, Without 3D Annotations

Abstract: Currently, utilizing large language models to understand the 3D world is becoming popular. Yet existing 3D‑aware LLMs act as black boxes: they output bounding boxes or textual answers without revealing how those decisions are made, and they still rely on pre‑trained 3D detectors to supply object proposals. We introduce Scene‑R1, a video‑grounded framework that learns to reason about 3D scenes without any point‑wise 3D instance supervision by pairing reinforcement‑learning‑driven reasoning with a two‑stage grounding pipeline.
In the temporal grounding stage, we explicitly reason about the video and select the video snippets most relevant to an open‑ended query. In the subsequent image grounding stage, we analyze the image and predict the 2D bounding box. This 2D prediction is then refined into a precise 3D localization by matching against high-fidelity candidates from a zero-shot segmentation module, which captures fine geometry while eliminating the need for detector-based proposals.
Scene-R1 can also adapt to the 3D visual question answering task to answer free-form questions directly from video. Our training pipeline only needs task-level 2D boxes or textual labels without dense 3D point-wise labels. Scene-R1 surpasses existing open-vocabulary baselines on multiple datasets, while delivering transparent, step‑by‑step rationales. These results show that reinforcement‑learning‑based reasoning combined with RGB‑D video alone offers a practical, annotation‑efficient route to trustworthy 3D scene understanding.