RODIN: Injecting 2D Foundational Features to 3D Vision Language Understanding

Authors: Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Alexander Sax, Franziska Meier, Katerina Fragkiadaki
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D vision-language understanding

Abstract:
We present RODIN (Referential ODIN), a novel model for 3D vision-language understanding that directly operates on posed RGB-D frames. Consuming posed RGB-D from sensors, such as those from an iPhone, simplifies and speeds up inference compared to existing models that train and test using pointclouds sampled from a reconstructed mesh provided by a dataset. We hypothesize that existing approaches consume pointclouds sampled from mesh instead of sensor RGB-D point clouds due to inaccurate camera poses in existing 3D grounding benchmarks, and show that using the "sensor" pointclouds indeed leads to a 5-10\% drop in performance on 3D referential grounding, for these methods. Yet sensor noise is unavoidable in real-world settings. RODIN instead addresses this with a scalable, end-to-end architecture for various 3D vision-language tasks. Specifically, RODIN combines powerful pretrained 2D weights trained on internet-scale data, adapts them to a 2D-3D encoder using the recently proposed ODIN, and combines that backbone with a proposed 3D mask-language decoder based on the Mask2Former used in SAM. RODIN achieves state-of-the-art performance on multiple 3D vision-language benchmarks, including referential grounding (SR3D, NR3D, ScanRefer), language-prompted object detection (ScanNet200 and Matterport3D), and question-answering (ScanQA and SQA3D). It outperforms previous methods for 3D vision-language tasks, despite consuming only sensor inputs. Because of its combination of effectively leveraging 2D pretrained architectures and finetuning end-to-end on sensor data, RODIN provides a scalable solution for embodied 3D perception.

Decision: reject
Reviewer scores: 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730572820626

thread_type:
official_review

<thread_object>
creator: reviewer_mefc

<content>
summary:
Building on prior work ODIN, RODIN introduces a language branch for 3D vision-language tasks, directly using posed RGB-D frames for 3D grounding, object detection, and 3D QA, thus bypassing mesh-based point clouds. It employs masked cross-attention modules, inspired by Mask2Former, to facilitate interaction between images and language. RODIN’s performance is validated across benchmarks such as SR3D, NR3D, ScanRefer, SQA, and ScanQA.

strengths:
1. The work shows promising performance across Referit3D and ScanRefer.
2. The first end-to-end method that leverages 2D features.

weaknesses:
1. The model primarily appears as a combination of ODIN with an added language branch and cross-modal interaction module, trained on large-scale 3D vision-language (VL) task heads. Compared to ODIN, which also uses RoBERTa as a language encoder, the technical contribution seems focused on engineering, incorporating Mask2Former’s masked cross-attention modules and an additional MLP language head, similar to PQ3D (lines 167-168).

2. The stated contribution, "The first end-to-end model leveraging pretrained 2D features and finetuning for 3D vision-language reasoning in object detection, referential grounding, and question answering," seems weak, as the visual backbone is ODIN-based (lines 195-201) and is not introduced in this paper.

3. The open-vocabulary claim lacks experiments to demonstrate zero-shot capability.

4. Despite incorporating more training data, the model underperforms ODIN on ScanNet-200 and Matterport3D. Moreover, ODIN already uses open-vocabulary detection prompts like GLIP.

5. The paper lacks citations to relevant 2D methods, such as MDETR [3] and GLIP [2].

6. Figure 2 lacks details on the visual encoder design, though it references ODIN, which uses a cross-view encoder.

[1] Jain, Ayush, et al. "ODIN: A Single Model for 2D and 3D Segmentation." CVPR 2024
[2] Li, Liunian Harold, et al. "Grounded language-image pre-training." CVPR 2022
[3] Kamath, Aishwarya, et al. "MDETR: Modulated Detection for End-to-End Multi-Modal Understanding." ICCV 2021

questions:
See the weakness.

1. Missing evaluation on additional ScanQA metrics such as CIDEr and BLEU, which would provide a more comprehensive assessment of performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604985949

thread_type:
official_review

<thread_object>
creator: reviewer_fl6t

<content>
summary:
This paper introduces a new model, RODIN, for 3D vision language understanding. RODIN leverages the powerful ODIN's backbone and extends it with a proposed 3D mask-language decoder based on Mask2Former or a text decoder to achieve state-of-the-art performance on multiple down-stream 3D tasks, including visual grounding, open-vocabulary object detection, and 3D QA. Unlike existing models, RODIN can process raw posed RGBD frames directly, increasing its scalability.

strengths:
1. This work presents a unique approach by combining an ODIN encoder and a 3D mask-language decoder or a text decoder to solve 3D vision-language understanding tasks directly on posed RGBD frames.

2. RODIN achieves significant improvements over previous methods on multiple downstream benchmarks.

3. The paper is well-structured, with a clear presentation.

weaknesses:
1. This paper's key idea or contribution is simply extending ODIN with existing mask or text decoder designs for 3D visual language understanding. The main components come from experiences of previous works, making it like an engineering combination, although figuring it out to work with SoTA performance is also appreciated. BTW, the benefit of this method, which can process raw RGB-D input to fit the practical setting, also can not be regarded as a contribution because it comes from ODIN. It is also encouraged to supplement some recent works related to this topic to make a more comprehensive discussion:

- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
- LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness

2. In the era of LLMs, I am curious about whether such an idea can be applied to MLLMs and how they perform, which I believe can bring more significant progress or attract more attention than solving these tasks in the previous specialist manner.

3. RODIN relies on datasets with segmentation annotations, however most large-scale datasets do not provide segmentation annotations, which may limit the model's scalability. The author discusses this point in Discussion, but it is not addressed.

4. This paper does not discuss the computational efficiency of RODIN, including hardware requirements and inference time. For real-world applications like robotics, efficiency is a critical factor.

questions:
1. The paper of ODIN mentions that "Inaccurate depth or camera poses cause a sharp decrease in performance." How do these affect the performance of the RODIN?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654108996

thread_type:
official_review

<thread_object>
creator: reviewer_hvvq

<content>
summary:
The authors proposed RODIN in this paper, which is a text-RGBD multimodal model for 3D vision language understanding. RODIN extends ODIN [1] to be able to reason with open-vocabulary texts. It uses the pretrained RGBD encoder in ODIN and a pretrained CLIP language encoder to encode RGBD images and texts. It follows Mask2Former’s mechanism to use a set of object queries and a transformer-based query refinement module. In this module, the authors propose to also update the features from RGB-D encoder with cross-attention and show performance gain in 3D referential grounding. The authors evaluate the proposed model in multiple datasets and benchmarks and show strong performance, especially when using RGB-D data directly from the sensors without mesh postprocessing.


[1] Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3564–3574, 2024.

strengths:
- The paper is overall well-written and easy to follow.
- The related work provides a comprehensive overview of 3D visual language understanding.
- The overall design of RODIN is sound and valid.
- The authors demonstrate that the model achieves state-of-the-art performance on several 3D language grounding datasets and benchmarks with a large margin.
- The ablation studies presented in Tables 4 and 5 are sound and well-designed, effectively demonstrating the effectiveness of each design component, particularly the visual token updating mechanism in the query refinement module.

weaknesses:
- The model designs are not new: it mostly extends the prior work ODIN to open vocabulary, and follows Mask2Former's decoding design. The architecture novelty is thus limited - most noticable change is the visual feature updating scheme in the query refinement module.
- The fairness in comparison: 
    - are the models shown in Table 1 trained on the same set of data? In L323, the authors mentioned that "This is similar in scale to datasets used by prior SOTA methods like PQ3D Zhu et al. (2024) and 3DVista Zhu et al. (2023b)." Could the authors elaborate more on this?
    - In terms of model size, are the models shown in Table 1 of similar model size?

questions:
- What do each of the losses terms in equation 7 refer to?
- On eq6, what if a none phrase consists of several words?
- typos:
    - L263: "do not update the visual features through during query refinement as we do" is not grammatically sound.
    - L272: naswers -> answers
    - eq 5 & 6: what does $V^T$ mean?
    - eq 6: missing parentheses around $V^T$
    - L286: $f\theta$ -> $f_\theta$
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696146629

thread_type:
official_review

<thread_object>
creator: reviewer_3mqz

<content>
summary:
This paper presents a novel 3D vision language model RODIN that directly operates on posted RGBD images rather than sensor point clouds and it argues that the sensor point cloud leads to 5~10% drop in 3D grounding tasks. RODIN extends ODIN model's architecture by adding text encoding/decoding capabilities for language understanding tasks. The whole model is end-to-end trainable. It achieves state-of-the-art performance on multiple 3D vision-language benchmarks including 3D grounding, segmentation, and vision question answering.

strengths:
1. The performance improvement is significant. RODIN outperforms previous state-of-the-art by more than 10% on SR3D, NR3D, and  ScanRefer.
2. The authors show comprehensive ablation studies validating the architectural designs.
3. This work directly work on the sensor data, without requirements for mesh reconstruction.
4. The whole model is end-to-end trainable and can solve multiple 3D vision-language tasks.
5. This architecture can fully leverage the pre-trained 2D vision foundation model features.

weaknesses:
1. The contribution might be limited as the main model architecture is based on ODIN, with the author incorporating text encoding and decoding to add text capabilities.
2. Since it depends on pre-trained 2D models, this approach might limit the model’s ability of generalization. Have authors fine-tuned these pre-trained 2D models, and would fine-tuning enhance the model’s performance?
3. How is camera pose inputted into the model? It’s unclear how the model handles noise in the camera pose. Why is this design performance better than using sensor point clouds, especially given that both designs need to address sensor noise?

questions:
1. Please provide more details about the “posed” RGB-D images. It’s unclear how the camera pose is encoded for each image.
2. How well does RODIN generalize to various camera configurations? If the test set includes different camera setups, will the model still perform effectively?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732216740786

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Respone

comment:
We appreciate the reviewers (3mqz, hvvq, fL6t, mEfc) for their valuable feedback. They recognize RODIN for its significant performance gains, surpassing previous state-of-the-art by over 10% on SR3D, NR3D, and ScanRefer (3mqz, hvvq, fL6t, mEfc), establishing it as the first end-to-end method utilizing 2D features that can solve multiple 3D Visual-Language tasks (fL6t, mEfc). The thorough ablation studies confirm the effectiveness of our architectural choices (3mqz, hvvq). Reviewers highlight the model's direct operation on sensor data without mesh reconstruction and its end-to-end training (3mqz). The paper is noted for its clarity and good structure (hvvq, fL6t), along with a comprehensive review of related work (hvvq).

\
**Reviewers 3mqz, hvvq, fL6t, mEfc are concerned that the proposed model has limited contribution over the past work of ODIN.** 

**ODIN is a model for instance segmentation** that, similar to our model, can process posed RGB-D sequences and uses pre-trained 2D foundational features. Our proposed model, **RODIN, is a model for 3D referential grounding and question answering. RODIN starts from ODIN’s backbone, and substantially innovates over its decoders.** Specifically, key architectural innovations of RODIN that are necessary for language grounding include: (1) updating visual features via attention to object query and language tokens in decoder (+24.1%) and (2) addition of a box supervision loss (+ 4.2%). **Using the original design of ODIN for referential grounding results in a 24.1% performance drop compared to RODIN (L477-481)**, which justifies our approach.

Apart from the technical contributions, **we further benchmark RODIN and prior SOTA baselines on more realistic settings for robotics/embodied AI** and demonstrate even larger performance gains in such real-world settings. RODIN is also the first end-to-end trainable 3D referential grounding method with SOTA performance, bypassing prior methods with multi-stage training.

Lastly, we have extended RODIN to use DINOv2 ViT backbones for image featurization, without 3D fusion inside the ViT 2D backbone (like the ODIN strategy), but rather, post 3D fusion after the 2D featurization across views. Using ViT 2D backbones improves performance over the initially reported RODIN results on all benchmarks by $4.2\%$, $1.9\%$, and $3.4\%$ @ 0.25 on SR3D, NR3D, and ScanRefer, respectively. Hence, the key message of RODIN is that using 2D features significantly boosts performance for 3D visual language tasks, and it is not restricted to the ODIN backbone. 



### **New Experiments since Submission**

Motivated by the reviewers’ comments, we have updated openreview with the revised submission (changes highlighted in magenta). In addition to responses to specific reviewers, below we summarize the major revisions and additional experiments we have added to the paper, guided by reviewers' comments:
- We have included CIDEr/BLEU Metrics on ScanQA in Table-8 of the paper (mEfc), details to architectural diagrams in Figure-5 (mEfc), and clarified details on the loss function and inference performance (hvvq). 
- We have measured robustness of RODIN and baselines against varying amounts of sensor noise  in Figure-6 (3mqz, hvvq)
- We reworded “open-vocabulary” object detection to “language-prompted” object detection (mEfc).
-  We tuned hyperparameters and managed to further close the performance gap between RODIN and ODIN in the object detection task (from ~4% mAP lower at the time of submission to only ~1% mAP lower currently), as pointed out by the reviewer mEfc. Currently, RODIN is the best performing model that can both detect objects in 3d and ground open vocabulary language. 
- We investigate the effects of fine-tuning 2D backbones on downstream performance as shown in Table-6 (3mQz).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents RODIN, a model that integrates 2D foundational features for 3D vision-language tasks, leveraging posed RGB-D data without mesh reconstruction. It builds on ODIN’s architecture but introduces modifications to enable referential grounding, question answering, and object detection. RODIN achieves SOTA performance on several benchmarks (e.g., SR3D, NR3D, ScanRefer). Reviewers raised several concerns about the difference from ODIN and some questions regarding details. Most concerns are addressed in the rebuttal process, however, the concern that the proposed model has limited contribution over ODIN still remains. Reviewer mEfc finds that "ODIN serves as a new baseline with good performance does not validate the significance or generality of RODIN’s modification. The modification still appears relatively minor, as it builds on ODIN, which already supports multi-modal prompts. If it is a generic 2D operation that enhances 3D features, additional experiments on 3D backbones, such as PointTransformer or Minkowski (as mentioned by the authors), would provide stronger evidence of its significance." AC carefully reads ODIN and RODIN, and agrees that the methodological difference from ODIN is not significant, while AC does acknowledge the engineering designs and their corresponding improvements. That said, AC believes the paper still does not meet the standard of ICLR due to the missing of adequately intellectual novelty. Therefore, AC recommends a weak rejection.