LUDVIG: Learning-free Uplifting of 2D Visual Features to Gaussian Splatting Scenes

Authors: Juliette Marrie, Romain Menegaux, Michael Arbel, Diane Larlus, Julien Mairal
Primary area: foundation or frontier models, including LLMs
Keywords: 3D vision, vision foundation models, graph diffusion

Abstract:
We address the problem of extending the capabilities of vision foundation models
such as DINO, SAM, and CLIP, to 3D tasks. Specifically, we introduce a novel
method to uplift 2D image features into 3D Gaussian Splatting scenes. Unlike traditional approaches that rely on minimizing a reconstruction
loss, our method employs a simpler and more efficient feature aggregation
technique, augmented by a graph diffusion mechanism. Graph diffusion
enriches features from a given model, such as CLIP, by leveraging pairwise
similarities that encode 3D geometry or similarities induced by another  embedding
like DINOv2.  Our approach achieves performance comparable to the state of
the art on multiple downstream tasks while delivering significant speed-ups.
Notably, we obtain competitive segmentation results using generic
DINOv2 features, despite DINOv2 not being trained on millions of
annotated segmentation masks like SAM.  When applied to CLIP
features, our method demonstrates strong performance in open-vocabulary,
language-based object detection tasks, highlighting the versatility of our 
approach.

Decision: reject
Reviewer scores: 3, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730611250373

thread_type:
official_review

<thread_object>
creator: reviewer_qrfz

<content>
summary:
This paper presents a method that lifts 2D features into the 3D representation space of 3DGS. With the lifted features, the method can perform segmentation of objects, and other applications. The paper tested features of SAM and DINO. For DINO, a graph diffusion step is performed to enhance the results.

strengths:
1. Previous approaches adopt optimizations, while this method is training-free and maybe fast.
2. The presentation is clear.

weaknesses:
1. The method needs additional post-processing steps such as graph diffusion.
2. The paper is not fully finished. There is no conclusion section.

questions:
1. The lifting algorithm is very simple, which is a weighted average of features collected from multiple views. This reviewer had an experience on object segmentation based on NeRF. I feel that the aggregated features may suffer from noises. For example, there may be errors in camera parameters, which can reduce the effectiveness of the aggregation. I suggest the authors to add more analysis on this problem (In fact, I do not know how to evaluate this, and expect the authors to provide more insights). 

2. The method is probably highly affected by the effectiveness of the feature extraction backbone. I am wondering whether the 2D segmentation masks obtained by the backbones applied to the images are similar to those of the lifted 3DGS projected to the 2D. Please provide this additional comparison.

3. I do not see connection between the proposed method and optimization-based approach in lines beginning from 205. Maybe this paragraph discusses the difference between the two kinds of approaches. I think a title of “Difference with optimization-based inverse rendering” may be more appropriate. 

4. The method needs a graph diffusion to further enhance its results. Why do you need this step? Please provide the results w/ and w/o this step. 

5. Please check the paper and make sure all sections are completed.

Overall, I think the accuracy of this method may be limited due to the noises in the features themselves caused by, e.g., misalignment between features of different views. The method is fast, but there is no argument supporting the necessity of this method at the cost of accuracy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615614738

thread_type:
official_review

<thread_object>
creator: reviewer_jxl3

<content>
summary:
The paper introduces LUDVIG, a learning-free method designed to uplift 2D visual features or semantic masks to 3D Gaussian Splatting scenes. LUDVIG avoids traditional, iterative optimization processes by using a simple aggregation technique, achieving comparable segmentation quality to state-of-the-art methods. The approach is applied effectively to features from both SAM and DINOv2, demonstrating high segmentation performance and computational efficiency across complex 3D scenarios.

strengths:
1. The learning-free feature uplifting method is both simple and effective, achieving strong results without training.
2. Experiments with SAM and DINOv2 demonstrate the method’s efficiency, yielding performance comparable to training-based approaches.
3. High Computational Efficiency: LUDVIG bypasses the costly and time-consuming optimization steps typical in 3D reconstruction methods, making it highly efficient.
4. Versatile Input Adaptability: The proposed method adapts seamlessly to various input types, such as features from SAM and DINOv2, showcasing robust performance across diverse feature sources.

weaknesses:
1. While the method is straightforward, it relies on hand-crafted processes, such as the segmentation score calculation and the graph diffusion process. These manual strategies may raise concerns about robustness, particularly in complex, real-world scenarios.
2. Certain sections, like Sec. 4.2, are challenging to follow. For example, the construction of 2D feature maps from DINOv2 is not clearly outlined. Including diagrams or visual aids could greatly enhance understanding and clarify complex steps.
3. Lack of qualitative comparison results with other baselines. 
4. The proposed method is primarily limited to foreground segmentation, whereas other feature lifting approaches support additional tasks, such as instance segmentation, 3D segmentation, and downstream applications like scene manipulation.

questions:
1. As in weakness 2, can you explain the process of feature map construction?
2. How is the time efficiency of the proposed method compared to other training-based methods? 
3. As in weakness 4, Can the proposed method performed on other tasks and how well is it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688614430

thread_type:
official_review

<thread_object>
creator: reviewer_n4zx

<content>
summary:
This submission proposes a simple yet effective scheme of uplifting visual features from 3D scenes represented by Gaussian splitting. In contrast to the previous approaches, the proposed method enables direct 3D segmentation without iterative optimization, while achieves comparable to the SotAs.

strengths:
+ The proposed scheme of connecting per-pixel 2D features and Gaussians are simple and intuitive. 

+ The segmentation can be directly done without iterative optimization on a trained Gaussian. 

+ The treatment on incorporating with DINOv2 feature into segmentation is nice, as it induces comparable results with the variant using the more tailored-for SAM.

weaknesses:
- There is neither limitation/failure nor future work discussion in the submission, what is the boarder impact of the work for the community? 

- The submission lacks report on running time.

questions:
The graph diffusion (Eq.(7)) is similar to power method for computing the dominant eigenvector of a matrix. While it is not guaranteed to converge to the dominant eigenvector, I guess there is a high probability. Is it possible to draw some theoretical connection to, say, spectral clustering methods, and therefore give some insight of the converging point of g_t? 

In figure 2, the rolling stair are separated from the ceiling in the multi-view case while merged to the latter in the single-view case, could the authors provide some explanation on such behavior?

In Sec. 4.2, the DINOv2 features seem to be of coarse scale (predicted in patches), would the recent works on feature super-resolution (e.g., Feature-up in ICLR 2024 by Fu et al.) help?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730839592756

thread_type:
official_review

<thread_object>
creator: reviewer_sblh

<content>
summary:
This paper is primarily concerned with generating 3D feature representations of a scene given multiple images of a stationary scene captured from different viewpoints. The main idea proposed here is that of extracting visual features from the multiple images (using a pretrained general purpose foundational model) and then fusing them with the scene representation computed using the 3D Gaussian splatting optimizer, i.e. fully calibrated camera poses and a set of 3D Gaussians with all the relevant parameters (position, covariance, size, opacity, spherical harmonics coefficients).

Specifically, feature vectors are computed and associated with the 3D centroids for each Gaussian in the 3DGS scene representation. The authors propose to compute these features by an aggregation scheme that is implemented as weighted averaging of the feature vectors extracted in multiple images. First 2D features are computed using existing visual feature extraction models. Features extracted for patches corresponding to the same Gaussian in multiple views are combined using a weighted averaging scheme presented in the paper. The 3D features associated with the 3DGS representation are then used in downstream tasks such as 3D segmentation (or multi-view segmentation) for which certain existing graph diffusion techniques are leveraged. 

The experimental results focus on segmentation performance on the Spin-NeRF and NVOS datasets where the method achieves comparable performance to existing techniques.

strengths:
The proposed approach to lift 2D features to 3D is efficient and avoids expensive iterative optimization schemes. The approach appears to be simple to implement and could be effective for downstream tasks such as 3D segmentation of stationary scenes where multiple views of the scene are available.

weaknesses:
My main concerns about this work revolves around:

(1) Low novelty:
- It was unclear to me to what extent the main idea of lifting features from images to 3DGS point couds was novel compared to existing approaches in the literature that have explored scene editing given a 3DGS reconstruction of a scene. The weighted averaging and aggregation scheme described here appears to be very similar to what was proposed in prior work such as Chen et al. 2024.

- The paper mostly focuses on the 3D segmentation application and shows the their proposed ideas work well for that application. They experiment with different features (DINOv2 vs SAM, etc.). However, the authors are not the first to segment or edit 3D Gaussian scenes. This was also explored in the prior work of Chen et al 2024 although they focused on binary segmentation, unlike in this work.

(2) More thorough comparison with existing approaches needed:
The main claimed contribution in the work is towards the technical approach for computing 3D features from 2D features using an efficient approach (referred to as uplifting in the paper). There are other approaches in the literature that have been explored to accomplish the same step. Those optimization based methods are cited and discussed [Kerr et al 2023, Zuo et al 2024] but there is no quantitative comparison between the proposed method and those related works. This makes it difficult to assess the contribution and impact of the novel ideas that are presented here. 

(3) Overall presentation has room for improvement:
The presentation and technical exposition could be improved for better clarity and presentation. I think the core objectives and the problem being tackled in the paper could have been explained better in the abstract and introduction. In the current version of the manuscript, it is difficult to understand what the authors precisely mean by “uplifting visual features to 3D scenes represented by Gaussian Splatting”. The abstract claims that “a simple yet effective aggregation technique yields excellent results.” – but the author have not explained what is being aggregated and for what purpose and what the success criteria is. The authors then talk about competitive segmentation results without clearly explaining the connection between the segmentation task and the 3D reconstruction or 3D scene representation task. Reading Section 1 (Introduction) doesn't give a concrete idea of the main objective, the key motivations and where the novelty lies.

questions:
Is the graph diffusion framework based on existing work? If so, the appropriate work in the literature should be cited for that. Otherwise, it would be best to clarify that the formulation is novel and developed by the authors themselves for the 3d segmentation task.

"Let I1, . . . , Im be a set of 2D frames from a 3D scene and d1, . . . , dm the corresponding viewing directions." -- the authors should clarify what they mean by viewing directions, and whether they are referring to camera poses, here? The phrase viewing direction is used throughout the paper and makes it difficult to understand the underlying mathematical operations that are being performed. 

The mathematical notation is difficult to follow. The expression \hat{F} = (\hat{F}_d, [) (lines 188) are not well explained. Similarly, the steps to construction the W and D matrices also need to clearly described.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732292902655

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Joint answer to the reviewers

comment:
### Joint answer

First, we would like to thank all reviewers for their careful reading and insightful comments. We have provided individual answers to each reviewer below. Those contain our current results on the ongoing experiments we are running to respond to the reviewers' questions and requests. The final results will be reported in the revised version of our paper that we plan to upload towards the end of the discussion period.
The additional experiments address comments from reviewers *sBLh* and *jxL3* on extending the experimental evaluation and running time comparaisons. 
We extended our evaluation to the task of open-vocabulary object detection. To this end, we uplifted CLIP features and refined them using graph diffusion with DINOv2 features.
Our approach achieves results comparable to the litterature while drastically reducing run times.

In the revised version of our paper we will:
- include these additional experiments on the task of open-vocabulary object detection, showing that our method does not only work for DINOv2 and SAM but also for CLIP (*answering sBLh and jxL3*)
- include comparative reports on running times (*answering jxL3 and N4ZX*)
- add qualitative comparisons with other baselines as well as visualizations of scene editing using 3D segmentation (*answering jxL3*) 
- discuss the theoretical inspiration of our graph diffusion process (*answering QRFz and N4ZX*)
- add a discussion on the scope and limitation of the method as concluding remarks based on the anwsers we provide to N4ZX and QRFz.
- clarify notations, add an illustration of the feature generation process (*answering sBLh and jxL3*), and revise the introduction and abstract so as to more clearly state the context, objectives and key contributions (*answering sBLh*). The new version of the abstract is available below.


### Abstract
The advent of powerful vision foundation models such as DINO, SAM, and CLIP has spurred interest in extending their capabilities to 3D tasks. These off-the-shelf models produce rich 2D features for tasks such as text embedding, semantic understanding, and segmentation at low computational cost.  Motivated by a recent breakthrough in 3D scene modeling, Gaussian Splatting, which enables real-time, high-resolution rendering of 3D features, we suggest a novel method to distill — or *uplift* — 2D image features into 3D.
While common approaches rely on minimizing a reconstruction loss, we instead propose a simple, efficient and parameter-free feature aggregation method that bypasses the need for costly optimization.  This method achieves comparable performance on multiple downstream tasks while offering significant speed-ups.

 When applied to semantic masks generated by the Segment Anything Model (SAM), our approach achieves segmentation quality comparable to state-of-the-art techniques. Remarkably, we also achieve competitive segmentation results using generic features from DINOv2, despite DINOv2 not being trained on millions of annotated segmentation masks like SAM. When applied to vision-language features from CLIP, our method achieves competitive results on open-vocabulary, language-based object detection. As a key contribution, we leverage the 3D scene geometry and the rich  similarities induced by DINO embeddings with a graph diffusion process, which refines the localization of coarse semantic features such as CLIP features or segmentation masks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553655858

thread_type:
official_comment

<thread_object>
creator: area_chair_samn

<content>
title:
Please read the rebuttal and response

comment:
Dear Reviewers,

Thanks again for serving for ICLR, the discussion period between authors and reviewers is approaching (November 27 at 11:59pm AoE), please read the rebuttal and ask questions if you have any. Your timely response is important and highly appreciated.

Thanks,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732789916200

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised manuscript

comment:
We thank again all the reviewers for their feedback. We have uploaded a revised version of our submission (with changes in blue) that includes:
- our additional experiments on the task of **open-vocabulary object detection** in Sections 4.2 and 5.4 (answering *sBLh* and *jxL3*)
- comparative reports on **running times** in Sections 5.4 and B.2 (answering *jxL3* and *N4ZX*)
- **qualitative comparisons** with other baselines in Sections C.2 and C.3 as well as visualizations of scene editing using 3D segmentation in Figure 7 (answering *jxL3*)
- a discussion on the theoretical inspiration of our graph diffusion process in Section 3.3 (answering *QRFz* and *N4ZX*)
- a discussion on the scope and limitation of the method in Section 6 (answering *N4ZX* and *QRFz*)
- clarified notations in Section 3, an illustration of the feature generation process in Figure 4 (answering *sBLh* and *jxL3*)
- a revised introduction and abstract (answering *sBLh*).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733145371030

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Consolidated results on LERF segmentation task

comment:
We thank the reviewers once again for engaging in the discussion during this rebuttal process. We have extended the experimental evaluation on open-vocabulary object localization by including additional results for the LERF Segmentation task [1], evaluated on the more challenging version introduced by LangSplat [2]. This task consists in predicting segmentation masks for different textual queries and complements our previous results on LERF Localization.

Our segmentation masks are predicted using SAM with point prompts extracted from CLIP relevancy maps. This aligns with FMGS's [3] approach (though FMGS does not evaluate this task) and adds only a small computational overhead during evaluation (1 to 4 seconds per test image). On this task, we outperform LangSplat by a significant margin.

These additional experimental results further confirm the effectiveness of our approach, reinforcing that its simplicity does not come at the cost of accuracy. We hope this addresses Rev. *QRFz*'s concerns and convinces Rev. *jxL3* and *sBLh* that our contribution has the potential to significantly impact the community. 

| LERF Segmentation (IoU)        | **LERF**  | **LangSplat**  | **Ours**             |
|--------------------|----------------|----------------|-----------------------|
| ramen              | 28.2           | **51.2**           |    50.2                   |
| figurines          | 38.6           | 44.7           |     **55.1**                  |
| teatime            | 45.0           | 65.1           |   **69.2**                    |
| waldo_kitchen      | 37.9           | 44.5           |  **59.0**                     |
| **overall**        | 37.4       | 51.4       |  **58.4**                     |
| **average time (minutes)** | 45 | 105       | **9**                |
-------------------------------------------------------------------

*[1] Kerr et al. (2023). “LERF: Language-Embedded Radiance Fields”*

*[2] Qin et al. (2024). “LangSplat: 3D Language Gaussian Splatting“*

*[3] Zuo et al. (2024). “FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding”*
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method that lift 2D features from foundational models (e.g., CLIP, SAM) to learned 3D Gaussians for a scene. The main idea is to apply weight aggregation on the features of each pixel, as well as a diffusion mechanism that diffuse features to nearby Gaussians. The propose approach is efficient compared to optimization-based methods. The weaknesses of the paper include low novelty (Reviewer sBLh), confusing presentation (Reviewer sBLh, QRFz), effectiveness and robustness of the method (Reviewer N4ZX, jxL3). The AC agrees with the reviewers that the paper has limited novelty and needs improvements on better differentiation with other methods, presentation and evidence to show the robustness of the proposed approach, thus the paper is recommended for rejection. The authors are encouraged to improve the paper given the comments of the reviewers.