LUDVIG: Learning-free Uplifting of 2D Visual Features to Gaussian Splatting Scenes

## Abstract

Abstract We address the task of uplifting visual features or semantic masks from 2D vision models to 3D scenes represented by Gaussian Splatting. Whereas common approaches rely on iterative optimization-based procedures, we show that a simple yet effective aggregation technique yields excellent results. Applied to semantic masks from Segment Anything (SAM), our uplifting approach leads to segmentation quality comparable to the state of the art. We then extend this method to generic DINOv2 features, integrating 3D scene geometry through graph diffusion, and achieve competitive segmentation results despite DINOv2 not being trained on millions of annotated masks like SAM.

## 1 Introduction

The field of image understanding has recently seen remarkable progress, driven by large pretrained models such as CLIP (Radford et al., ), DINO (Caron et al., ; Oquab et al., ), or SAM (Kirillov et al., ). A key factor behind their exceptional generalization capabilities lies in the vast size of their training datasets, often composed of millions or even billions of samples.

3D scene representation has also advanced with machine learning approaches like NeRF (Mildenhall et al., ) and model fitting techniques such as Gaussian Splatting (Kerbl et al., ). These methods typically rely on a few dozen views of the scene captured from different angles. While the resulting reconstructions effectively capture both appearance and geometrical information, they are not directly applicable to semantic tasks, which has led to further developments.

The complementarity of these two families of approaches has indeed recently been exploited by numerous methods that integrate geometry and semantics by uplifting image-level features extracted by large pretrained models into 3D NeRF or Gaussian Splatting representations. This has led to a surge in methods for tasks such as language-guided object retrieval (Kerr et al., ; Liu et al., ; Zuo et al., ), scene editing, (Kobayashi et al., ; Chen et al., ; Fan et al., ), or semantic segmentation (Cen et al., ; Ye et al., ; Ying et al., ).

The main limitation of most previous approaches lies in their reliance on optimization, which requires an iterative process to learn a scene-specific 3D representation by minimizing reprojection error across all training views. While this loss function is intuitive, a faster and more straightforward method for transferring 2D generic visual features to *already trained* Gaussian splatting 3D models would be preferable, which is the purpose of this work.

In this paper, we demonstrate that a simple, learning-free process is highly effective for uplifting 2D features or semantic masks into 3D Gaussian Splatting scenes. This process, which can be viewed as an ‘inverse rendering’ operation, is both computationally efficient and adaptable to any feature type. We showcase its efficiency by uplifting visual features from DINOv2 with registers (Oquab et al., ; Darcet et al., ), as well as semantic masks from SAM (Kirillov et al., ) and SAM2 (Ravi et al., ). When applied to SAM masks, our method achieves segmentation results comparable to state-of-the-art techniques while being significantly faster than previous approaches relying on gradient-based optimization. We also demonstrate the successful uplifting of generic DINOv2 features, which, when used for segmentation, perform surprisingly on par with prior work based on SAM. Our segmentation method with DINOv2 leverages simple dimensionality reduction and graph diffusion to incorporate 3D scene geometry.

To summarize, our contribution is threefold: (i) we introduce a simple, learning-free uplifting approach that can be directly integrated into the rendering process (Section 3), achieving state-of-the-art results when applied to SAM-generated semantic masks.
(ii) We demonstrate that uplifting DINOv2 features, combined with graph diffusion, produces strong 3D features capable of competing in segmentation tasks with recent SAM-based methods (Section 4), despite DINOv2 not being trained for segmentation. (iii) Additionally, our method can generate high-resolution generic feature maps for any rendered view as a useful byproduct.

## 2 Related work

##### Learning 3D semantic scene representations with NeRF.

NeRF (Mildenhall et al., ) provides an implicit 3D representation of a scene using a continuous function parametrized by a multilayer perceptron and predicts the volume density and radiance for any given 3D position and viewing direction.
Such representation can naturally be extended to not only predict radiance and density, but also semantic features.
The early works N3F (Tschernezki et al., ) and DFF (Kobayashi et al., ) distill DINO 2D (*i.e.*, image-level) features (Caron et al., ) in scene-specific NeRF representations. Kobayashi et al. () also distill LSeg (Li et al., ) a CLIP-inspired language-driven model for semantic segmentation. Shortly after, LERF (Kerr et al., ) and 3D-OVS (Liu et al., ) learned 3D CLIP (Radford et al., ) and DINO (Caron et al., ) features jointly for open-vocabulary segmentation. These works were extended to (combinations of) other large pretrained models such as latent diffusion models (Ye et al., ) or SAM (Kirillov et al., ) for semantic segmentation (Cen et al., ; Ying et al., ).

##### Learning 3D semantic scene representations with Gaussian splatting.

Subsequent work extended the approach to the more recent Gaussian splatting method (Kerbl et al., ), achieving high-quality novel-view synthesis while being orders of magnitude faster that NeRF-based models. The literature endowed 3D Gaussians with various semantic representations for tasks such as semantic segmentation using SAM (Cen et al., ; Ye et al., ; Kim et al., ), language-driven retrieval or editing using CLIP combined with DINO (Zuo et al., ) or SAM (Ye et al., ), or scene editing using diffusion models (Chen et al., ; Wang et al., ).
These works learn 3D semantic representations with gradient-based optimization of a reprojection loss.
As a single scene can be represented by over a million Gaussians, such gradient-based optimization has intrinsic limitations in terms of memory and compute.
To handle these, FMGS (Zuo et al., ) employs a multi-resolution hash embedding (MHE) of the scene for uplifting DINO and CLIP representations, while Feature 3DGS (Zhou et al., ) learns a $1\times 1$ convolutional upsampler of Gaussians’ features distilled from LSeg and SAM’s encoder. In contrast, our approach requires no learning, which significantly speeds up the uplifting process and reduces the memory requirements.

##### Leveraging 3D information to better segment in 2D.

Most prior works focusing on semantic segmentation leverage 2D models specialized for this task. The early work of Yen-Chen et al. () uplifts learned 2D image inpainters by optimizing view consistency over depth and appearance priors.
Later, subsequent works on novel-view semantic segmentation have mostly relied on uplifting either features from SAM’s encoder (Zhou et al., ), binary SAM masks (Cen et al., ; ), or masks automatically generated for all objects in the image (Ye et al., ; Ying et al., ; Kim et al., ).
In the latter case, mask predictions are inconsistent between views, and Gaussian Grouping (Ye et al., ) use a universal 2D temporal propagation model to match predictions, while GARField (Kim et al., ) directly resolves ambiguities in 3D by grouping objects with similar size together. Concurrently, Ying et al. () uplift inconsistent 2D segmentation masks produced by SAM into a 3D feature field using a hierarchical contrastive learning approach. However, as mentioned by the authors, their learning approach does not guarantee that two instances of the same object will be assigned the same semantic label. In our work, we uplift DINOv2 features into 3D which yields a smooth feature field with semantic consistency across the whole scene. Our simple aggregation is effective enough to yield results comparable to prior work leveraging SAM without suffering from semantic inconsistencies or resorting to complex learning schemes. Compared to prior work uplifting DINO features (Tschernezki et al., ; Kobayashi et al., ; Kerr et al., ; Liu et al., ; Ye et al., ; Zuo et al., ), we quantitatively show that DINOv2 features can be used on their own for semantic segmentation and rival SAM-based models through a simple graph diffusion process that incorporates the 3D geometry of the scene.

## 3 Uplifting 2D visual representations into 3D

In this section, we present a simple yet effective method for lifting 2D visual features or semantic masks into 3D using Gaussian splatting and discuss its relation with more expensive gradient-based optimization techniques.

### 3.1 Background on Gaussian splatting

##### Scene representation.

The Gaussian splatting method consists in modeling a 3D scene as a set of $n$ Gaussians densities ${\mathcal{N}}_{i}$, each defined by a mean $\mu_{i}$ in $\mathbb{R}^{3}$, a covariance $\Sigma_{i}$ in $\mathbb{R}^{3\times 3}$, an opacity $\sigma_{i}$ in $(0,1)$, and a color function $c_{i}({d})$ that depends on the viewing direction $d$.

A 2D frame at a given view is an image rendered by projecting the 3D Gaussians onto a 2D plane, parametrized by the viewing direction $d$. This projection accounts for the opacity of the Gaussians and the order in which rays associated with each pixel pass through the densities.
More precisely, a pixel $p$ for a view $d$ is associated to an ordered set ${\mathcal{S}}_{{d},p}$ of Gaussians and its value is obtained by their weighted contributions:

$$ $\hat{I}_{{d}}(p)=\sum_{i\in{\mathcal{S}}_{d,p}}c_{i}(d)w_{i}(d,p).$ (1) $$

The above weights are obtained by $\alpha$-blending, i.e. $w_{i}(d,p)=\alpha_{i}(d,p)\prod_{j\in\mathcal{S}_{d,p},j<i}\left(1-\alpha_{j}(
d,p)\right)$,
where the Gaussian contributions $\alpha_{i}(d,p)$ are computed by multiplying the opacity $\sigma_{i}$ by the Gaussian density ${\mathcal{N}}_{i}$ projected onto the 2D plane at pixel position $p$.

##### Scene optimization.

Let $I_{1},\dots,I_{m}$ be a set of 2D frames from a 3D scene and $d_{1},\dots,d_{m}$ the corresponding viewing directions. Gaussian Splatting optimizes the parameters involved in the scene rendering
function described in the previous section. This includes the means and covariances of the Gaussian densities, their opacities, and the color function parametrized by spherical harmonics. Denoting by $\theta$ these parameters, the following reconstruction loss is used

$$ $\min_{\theta}\frac{1}{m}\sum_{k=1}^{m}\mathcal{L}(I_{k},\hat{I}_{d_{k},\theta}),$ (2) $$

where $\hat{I}_{d_{k},\theta}$ is the rendered frame of the scene in the direction $d_{k}$, as in Eq. (1), by using the parameters $\theta$, and $\mathcal{L}$ is a combination of $\ell_{1}$ and SSIM loss functions (Kerbl et al., ).

### 3.2 Uplifting of 2D feature maps into 3D

Given a set of $m$ 2D training frames and corresponding 3D scene obtained by the Gaussian Splatting method, our goal is to compute generic features $f_{i}$ in $\mathbb{R}^{c}$ for each Gaussian $i$, which would be effective for solving future downstream tasks, *e.g.*, high-resolution semantic segmentation for new frames of the scene, or robot navigation. In other words, $f_{i}$ can be seen as an extension of the color function $c_{i}$, even though, for simplicity, we do not consider view-dependent features in this work.

A natural approach is to consider a pre-trained vision model that provides 2D feature maps for each of the $m$ frames used in Gaussian splatting, and then devise a technique to *uplift* these 2D feature maps into 3D. This uplifting principle can also be directly applied to semantic masks instead of generic features, as demonstrated in Section 5.
Interestingly, once the features $f_{i}$ are computed for each Gaussian $i$, it is possible to *render* two-dimensional feature maps for any new view, at a resolution that can be much higher than the feature maps computed for the $m$ training frames, thus performing effective feature super-resolution.

##### Uplifting with simple aggregation.

We construct uplifted features for each 3D Gaussian of the 3D Gaussian Splatting scene as a weighted average of 2D features from all frames.
Each 2D feature $F_{d,p}$ from a frame at a given viewing direction $d$ and pixel $p$ contributes to the feature $f_{i}$ by a factor proportional to the rendering weight $w_{i}(d,p)$, if the Gaussian $i$ belongs to the ordered set $\mathcal{S}_{d,p}$ associated to the view/pixel pair $(d,p)$. The resulting features are then normalized to maintain the same order of magnitude as the original 2D features, thus resulting in the following simple equation:

$$ $f_{i}=\sum_{d=1}^{m}\sum_{p}\bar{w}_{i}(d,p)F_{d,p}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \text{with} \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode \nobreak\ \bar{w}_{i}(d,p)=\frac{\mathds{1}_{i\in\mathcal{S}_{d,p}}w_{i}(d,p)} {\sum_{d=1}^{m}\sum_{p}\mathds{1}_{i\in\mathcal{S}_{d,p}}w_{i}(d,p)},$ (3) $$

where $\mathds{1}_{i\in\mathcal{S}_{d,p}}$ is equal to $1$ if the Gaussian $i$ belongs to $\mathcal{S}_{d,p}$ and 0 0 otherwise.
We can interpret this equation as a normalized version of the transposed rendering operation over the $m$ viewing direction.
More precisely, the rendering of any view-independent collection of features ${\bf{f}}=(f_{i})$ attached to the $n$ Gaussians into the $m$ training frames can be represented as a linear operator $W$ acting of the collection $\bf{f}$ and returning a collection of 2D feature maps ${\bf{\hat{F}}}=(\hat{F}_{d,p})$, see (4) below. Here, the matrix $W$ consists of all rendering weights $\mathds{1}_{i\in\mathcal{S}_{d,p}}w_{i}(d,p)$ at row $(d,p)$ and column $i$.
Similarly, the uplifting expression introduced in Eq. (3) can be expressed in terms of transpose of $W$ and a diagonal matrix $D$ of size $m$ representing the normalization factor and whose diagonal elements are obtained by summing over the rows of $W$ as in Eq. (5) below:

$$ Rendering to $m$ frames $$

$$ ${\bf\hat{F}}=W{\bf{f}},$ (4) $$

$$ Uplifting from $m$ frames $$

$$ ${\bf f}=D^{-1}W^{\top}{\bf{F}}.$ (5) $$

The procedure in (5), illustrated in Figure 1, bears similarity with the one from Chen et al. () for uplifting 2D binary masks to a 3D Gaussian splatting scene.
In their method, uplifted masks are thresholded to create 3D binary masks that can be rendered into different 2D frames. Such a thresholding operation would not be appropriate for uplifting generic features such as those from DINOv2.
Moreover, unlike in Eq. (3) and (5), Chen et al. () propose to normalize their uplifted masks based on the total count of view/pixel pairs $(d,p)$ contributing to the mask of a Gaussian $i$, i.e. $\sum_{d=1}^{m}\sum_{p}\mathds{1}_{i\in\mathcal{S}_{d,p}}$, without taking the rendering weight $w_{i}(d,p)$ into account. Consequently, the uplifted features tend to have larger values for large, opaque Gaussians, making the rendering of these features more likely to ignore details provided by smaller and more transparent Gaussians.

Figure: Figure 1: Illustration of the inverse and forward rendering. In the inverse rendering (or uplifting) phase, features ${\bf{f}}$ are created for each 3D Gaussian by aggregating coarse 2D features ${\bf F}$ over all viewing directions. For forward rendering, the 3D features ${\bf{f}}$ are projected on any given viewing direction as in regular Gaussian splatting. The rendering weight $\bar{w}_{i}(d,p)$ represents the relative influences of the Gaussian $i$ on pixel $p$, defined in Eq. (3).

##### Connection with optimization-based inverse rendering.

An alternative approach to uplifting 2D features ${\bf F}$ is to minimize a reconstruction objective $\mathcal{L}({\bf f})$, where the goal is to find uplifted features ${\bf f}$ whose rendering closely matches the original 2D features ${\bf F}$ (Tschernezki et al., ; Kerr et al., ; Zuo et al., ). A natural choice is to minimize the mean squared error between the 2D features ${\bf F}$ and the rendered ones ${\bf\hat{F}}$ as defined by Eq. (4):

$$ $\displaystyle\min_{{\bf f}}\mathcal{L}({\bf f}):=\frac{1}{2}\|{\bf F}-W{\bf f} \|^{2}.$ (6) $$

Such an approach requires using an optimization procedure which would be costly compared to the proposed uplifting method. Nevertheless, it is possible to interpret the proposed uplifting scheme in Eq. (5) as a single pre-conditioned gradient descent step on the reconstruction objective, starting from a ${\bf 0}$ feature, *i.e.*, ${\bf f}=-D^{-1}\nabla\mathcal{L}({\bf 0})$. In practice, we found that performing more iterations on the objective $\mathcal{L}({\bf f})$ did not result in particular improvement of the quality of the features, thus suggesting that the cheaper scheme in Eq. (5) is already an effective approach to uplifting.

##### Gaussian filtering

The normalization $\beta_{i}=\sum_{d=1}^{m}\sum_{p}\mathds{1}_{i\in\mathcal{S}_{d,p}}w_{i}(d,p)$ serves as an estimator of the relative importance of each Gaussian in the scene. Therefore, it can be used as a criterion to prune the set of Gaussians for memory efficiency. In our experiments, we filter out half of the Gaussians based on $\beta_{i}$ and observe no qualitative nor quantitative degradation of the results. This approach is inspired by prior work on efficient Gaussian Splatting representation such as proposed by Fan et al. () that also prunes Gaussians based on their contribution to each pixel in the training frames.

## 4 From 3D Uplifting to Segmentation

The previous section describes how to uplift features in 3D. Although it is difficult to assess the intrinsic quality of 3D features directly, one approach is to evaluate their performance in the task of multiple-view segmentation. We describe two approaches based on uplifting features from either DINOv2 or SAM.
These models are either completely agnostic to segmentation tasks (DINOv2), or specifically designed for such tasks (SAM).

As in Sec. 3, we are given a set of 2D frames $I_{1},\dots,I_{m}$, with viewing direction $d_{1},\dots,d_{m}$ and corresponding 3D scene obtained by the Gaussian Splatting method, which can be used to uplift 2D features from the $m$ frames to 3D.
Additionally, a foreground mask of the object to be segmented is provided on a *reference frame* taken to be the first frame $I_{1}$.
We consider two types of foreground masks: either *scribbles* or a whole *reference mask* of the object, both of which define a set of foreground pixels $\mathcal{P}$.
In the following subsections, we present the proposed approaches for segmentation using SAM and DINOv2 features, based on both types of foreground masks.

### 4.1 Multiple-view Segmentation with SAM

SAM (Kirillov et al., ; Ravi et al., ) is a powerful image segmentation model, that can generate object segmentation masks from point prompts, on a single 2D image.
Aggregating SAM 2D segmentation masks in 3D allows for cross-view consistency and improves single-view segmentation results.
In order to leverage SAM, we propose a simple mechanism for generating SAM 2D features for each frame from a foreground mask in the reference frame.

##### Construction of 2D feature maps.

The key point is to generate point prompts on each training frame from the foreground mask provided on the reference frame. To this end, we perform an uplifting of the foreground mask (Eq. (3)) and re-project it on all frames (Eq. (4)).
This results in 2D scalar maps of real numbers that we further normalize by their average value and for which higher values indicates the presence of the target object to segment. For each frame at viewing direction $d$, we retain a subset of pixels $\mathcal{P}_{d}$ with values higher than a threshold fixed for all scenes and select point prompts for SAM from this subset.
Finally, we compute 2D segmentation masks for each frame using SAM by randomly selecting $3$ points prompts from $\mathcal{P}_{d}$, repeating the operation $10$ times and averaging the resulting masks for each view to obtain the final 2D SAM feature maps.

##### Segmentation with uplifted SAM masks.

The 2D segmentation masks generated by SAM are uplifted using the aggregation scheme in Sec. 3.2. Our final prediction is obtained by rendering the uplifted feature maps into the target frame.

### 4.2 Multiple-view segmentation with DINOv2

DINOv2 (Oquab et al., ) is a self-supervised vision model recognized for its generalization capabilities.
In this work, we aggregate the patch-level representations produced by DINOv2 with registers (Darcet et al., ) into a high resolution and fine-grained 3D semantic representation.

##### Construction of 2D feature maps.

We construct the 2D feature maps using a combination of a sliding windows mechanism and dimensionality reduction of the original DINOv2 features.
Specifically, we compute DINOv2 features across multiple overlapping crops of a whole image corresponding to a given frame. This results in several features maps, one for each crop, of size $\frac{L}{p}\times\frac{W}{p}$ where $L\times W$ is the size of the crop and $p\times p$ represents the image patches processed by DINOv2.
Dimensionality reduction is then performed on extracted features from all training frames and all crops pooled together, resulting in compact representations. Finally, the features of reduced-dimension are up-sampled and aggregated to obtain feature maps at the original image size. This approach enhances the granularity of spatial representations by aggregating patch-level features from several overlapping crops to compute the pixel-level features, therefore resulting in a finer feature map at the original image size while still using features of a relatively small dimension ($c$=40).

##### Segmentation with uplifted DINOv2 features.

The 2D feature maps from the $m$ training views are uplifted using Eq. (3) and the resulting 3D features are then re-projected into any viewing direction $d$ using Eq. (4) to compute rendered 2D features $(\hat{F}_{d,p})$.
To obtain segmentation masks, we construct a score $P(\hat{F}_{d,p})$ for a 2D pixel $p$ to belong to the foreground, based on its corresponding rendered feature. The predictor $P$ is constructed by exploiting the foreground mask. More precisely, $P$ relies on the rendered foreground features $\mathcal{F}_{ref}:=(\hat{F}_{d_{1},p})_{p\in\mathcal{P}}$ corresponding to the foreground mask computed on the reference frame $I_{1}$.
We propose two approaches for constructing $P$.
The first one is a simple approach that sets $P(\hat{F}_{d,p})=\mathcal{S}_{F}(\hat{F}_{d,p},\bar{F})$ where $\bar{F}$ is the average over foreground features $\mathcal{F}_{ref}$, and $\mathcal{S}_{F}$ is defined based on the cosine similarity.
The second approach is more discriminative and first trains a logistic regression model $P$ on all rendered 2D features of the reference frame, so that the foreground features $\mathcal{F}_{ref}$ are assigned a positive label. Then $P(\hat{F}_{d,p})$ gives the probability that a pixel $p$ belongs to the foreground. The final mask is then obtained by thresholding.

Experimentally, the second approach is extremely efficient when the set of foreground pixels $\mathcal{P}$ covers the whole object to segment, so that $P$ captures all relevant features. This is the case when a whole reference mask of the object is provided. When the foreground pixels $\mathcal{P}$ does not cover the whole object, as with scribbles, $P$ can be discriminative to parts of the object that are not covered by $\mathcal{P}$. Therefore, we rely on the second approach for tasks where a reference mask is provided, and use the simpler first approach when only scribbles serve as reference.

### 4.3 Enhancing segmentation with DINOv2 using 3D graph diffusion

DINOv2 provides generic visual features that do not explicitly include information for segmentation, unlike models such as SAM that were specifically trained for such a task.
Consequently, using the 2D projections of uplifted DINOv2 features, as proposed in Sec. 4.2, might fail to separate different objects that happen to have similar features while still being distinct entities.
This challenge can be mitigated by incorporating 3D spatial information in which the objects are more likely to be well-separated.
To this end, we propose an approach for segmentation based on 3D graph diffusion for leveraging the 3D spatial information. The approach constructs a graph whose nodes are given by the 3D Gaussians and whose edges, represented by a matrix $A$ of size $n\times n$, encode both the 3D Euclidean geometry between the nodes and the similarity between their associated DINOv2’s uplifted features.
Starting from a suitably initialized vector of weights $g_{0}\in\mathbb{R}^{n}$, representing a coarse estimation of the contribution of each Gaussian to the segmentation mask, we perform $m$ diffusion steps to construct a sequence of $T$ diffused vectors $(g_{t})_{1\leq t\leq T}$ defined as follows:

$$ $g_{t}=Ag_{t-1}.$ (7) $$

We retain the last weight vector $g_{T}$ and render it into 2D for segmentation (Eq. (4)).
Below, we describe the initialization of the weight vector $g_{0}$ and the construction of the adjacency matrix $A$.

##### Initialization of the weight vector.

The initial weight vector $g_{0}$ is computed by uplifting the 2D foreground mask (either scribbles or a reference mask) from the reference frame into 3D using Eq. (3), normalizing the 3D mask by its mean value over all nodes and setting to zero all values below a fixed threshold. The nodes for which $g_{0}$ has a positive value define a set of anchor nodes $\mathcal{M}$ that are more likely to contribute to the foreground. The resulting weight vector is a coarse estimation of how much each Gaussian contributes to a rendered 2D segmentation mask.

##### Construction of the graph edges.

The graph is constructed so that each node $i$ has edges only with its $k$ nearest neighbors $\mathcal{N}(i)$ as measured by the Euclidean distance between the positions of the 3D Gaussians.
The edges are then weighted by a factor that accounts for the contribution of a local feature similarity $S_{f}(f_{i},f_{j})$ between uplifted features of neighboring nodes $i$ and $j$, and a global unary regularization term $P(f_{i})$ on each node $i$ that encourages similarity between the uplifted node feature $f_{i}$ and those belonging to the foreground. The resulting adjacency matrix is obtained by exponentiation of the sum of both terms weighted by bandwidth parameters $b_{f},b_{p}$:

$$ $A_{ij}=\mathds{1}_{j\in\mathcal{N}(i)}\exp\left(\frac{S_{f}(f_{i},f_{j})}{b_{f }}+\frac{P(f_{i})}{b_{p}}\right).$ (8) $$

We define $S_{f}$ based on the cosine similarity between features and define the unary term $P$ using a similar approach as in Sec. 4.2: either as a similarity $P(f_{i})=\mathcal{S}_{f}(f_{i},\bar{f})$ with the averaged feature $\bar{f}$ over the anchor nodes $\mathcal{M}$ (in the case when scribbles are provided), or as a logistic regression model trained on the uplifted features, so that anchor nodes’ features are assigned a positive label (in the case when a full foreground mask is available).
The local term $S_{f}$, typically a cosine similarity, allows diffusion to neighbors that have similar features while the unary term prevents leakage to background nodes during diffusion by encouraging closeness to the foreground features and allows using an arbitrary number of diffusion steps.

## 5 Experiments

### 5.1 Experiment details

##### 3D scene training and pruning.

All scenes are trained using the original Gaussian Spatting implementation (Kerbl et al., ) with default hyperparameters. For memory efficiency, half of the Gaussians are filtered out based on their importance, as described in Sec. 3.2.

##### 2D vision models.

Our experiments are conducted using DINOv2’s ViT-g with registers (Darcet et al., ), SAM (Kirillov et al., ) and SAM 2 (Ravi et al., ).

##### Datasets.

We consider two segmentation tasks: i) Neural Volumetric Object Selection (NVOS,  Ren et al. ), which is derived from the LLFF dataset (Mildenhall et al., ), and ii) SPIn-NeRF, which contains a subsets of scenes from NeRF-related datasets (Knapitsch et al., ; Mildenhall et al., ; ; Yen-Chen et al., ; Fridovich-Keil et al., ).
The NVOS dataset consists of forward-facing sequences in which one frame is labeled with a segmentation mask and another one is labeled with scribbles to be used as reference. SPIn-NeRF contains both forward-facing and 360-degree scenes, in which all frames are labeled with segmentation masks, and the standard evaluation protocol uses the segmentation mask from the first frame as reference to label the subsequent frames.

##### Evaluation and hyperparameter tuning.

Our segmentation results are averaged over 3 independent runs.
Segmentation with 3D SAM masks requires setting a threshold for foreground/background pixel assignment, and optionally choosing one of the three masks proposed by SAM (representing different possible segmentations of the object of interest).
Segmentation with DINOv2 uses three hyperparameters: the two bandwidths for graph diffusion and the threshold for foreground/background pixel assignment.

For SPIn-NeRF, all hyperparameters are chosen based on the IoU for the available reference mask. For NVOS, only reference scribbles are provided, hence i) for SAM/SAM2, only one mask is generated, and the threshold for segmentation is fixed for all scenes for SAM and automatically chosen using Li iterative Minimum Cross Entropy method (Li & Lee, ) for SAM 2, ii) for DINOv2 we predict a SAM mask based on the scribbles of the reference frame, and choose the hyperparameters maximizing the IoU with this SAM mask. This is consistent with a scenario where the user, here SAM, would choose hyperparameters based on visual inspection on one of the frames.

### 5.2 Qualitative results

##### DINOv2 feature uplifting.

First, we illustrate the effectiveness of our simple uplifting approach. Figure 2 shows the first three PCA components (one channel per component) over DINOv2’s patch embeddings. The coarse patch-level representations from every view (middle) are aggregated using Eq. 5 to form a highly detailed 3D semantic representation, and reprojected into 2D (right) using Eq. 4. The aggregation is very fast, as it is directly implemented in the Gaussian Splatting CUDA-based rendering process. The procedure takes about 1.5ms per view and can be parallelized across the feature dimension. The first principal component (encoded in the red channel) mostly captures the foreground object, and the subsequent ones allow refining the foreground representations and delivering a detailed background.

Figure: (a) RGB image
Refer to caption: extracted/5937544/figures/pcas/trex.jpg

##### Graph diffusion.

Figure 3 illustrates the effectiveness of the diffusion process. In the Fern scene, diffusion progressively spreads through the branches to their extremities and the regularization (red background) prevents it from leaking beyond the trunk.
As illustrated with the case of Horns, diffusion filters out unwanted objects that are similar to the object of interest (here the two skulls on the side).
The graph nodes are initialized with the reference scribbles and the diffusion spreads through the object of interest and stop at its borders. The regularization sets a constraint that prevents leakage, even after a large number of iterations. This is also illustrated in Appendix Figure 5 for the Flower and Trex scenes: diffusion rapidly spreads, with near-full coverage after only 5 steps, before reaching all the much smaller Gaussians on the border, allowing for a refined segmentation.

Figure: (a) RGB image
Refer to caption: extracted/5937544/figures/diffusion/fern_rgb_pos.jpg

### 5.3 Segmentation results

In this section, we quantitatively evaluate the segmentation task on NVOS (Ren et al., ) and SPin-NeRF (Mirzaei et al., ).
We evaluate segmentation based on SAM and SAM2 mask uplifting, and on DINOv2 feature uplifting combined with graph diffusion. We compare our segmentation results to the current state of the art: SA3D (Cen et al., ), SA3D-GS (Cen et al., ), SAGA (Cen et al., ), OmniSeg3D (Ying et al., ). All these methods are specifically designed for uplifting the 2D segmentation masks produced by SAM into 3D using gradient-based optimization of a projection loss. We also report results from NVOS (Ren et al., ) and MVSeg (Yen-Chen et al., ), who respectively introduced the NVOS and SPIn-NeRF datasets.

##### Segmentation on SPIn-NeRF.

We report our segmentation results for the SPin-NeRF dataset (Mirzaei et al., ) in Table 1.
Our results are comparable to the state of the art while not relying on optimization-based approaches. Surprisingly, our segmentation with DINOv2 using graph diffusion also gives results on par with models leveraging SAM masks.
Our lower segmentation results compared to OmniSeg’s can be partly attributed to poor Gaussian Splatting reconstruction of highly specular scenes such as the Fork, in which semi-transparent Gaussians floating over the object try to represent reflections or surface effects that are difficult to capture with standard rasterization techniques (Jiang et al., ).

**Table 1: Segmentation (IoU) on SPIn-NeRF (Mirzaei et al., ) with DINOv2, SAM and SAM2.**
|  | MVSeg | SA3D-GS | SAGA | OmniSeg3D | LUDVIG (Ours) |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 3D representation: | NeRF | GS | GS | NeRF | GS |  |  |
| Uplifting: |  | SAM | SAM | SAM | DINOv2 | SAM | SAM2 |
| Orchids | 92.7 | 84.7 | - | 92.3 | 92.6 | 91.9 | 90.7 |
| Leaves | 94.9 | 97.2 | - | 96.0 | 93.9 | 96.4 | 96.4 |
| Fern | 94.3 | 96.7 | - | 97.5 | 95.6 | 96.8 | 96.7 |
| Room | 95.6 | 93.7 | - | 97.9 | 94.7 | 96.5 | 96.6 |
| Horns | 92.8 | 95.3 | - | 91.5 | 94.4 | 92.3 | 94.9 |
| Fortress | 97.7 | 98.1 | - | 97.9 | 97.6 | 98.3 | 98.3 |
| Fork | 87.9 | 87.9 | - | 90.4 | 81.6 | 87.1 | 86.8 |
| Pinecone | 93.4 | 91.6 | - | 92.1 | 90.1 | 90.8 | 90.8 |
| Truck | 85.2 | 94.8 | - | 96.1 | 94.8 | 94.3 | 92.6 |
| Lego | 74.9 | 92.0 | - | 90.8 | 93.2 | 92.8 | 92.9 |
| Average | 90.9 | 93.2 | 93.4 | 94.3 | 92.8 | 93.7 | 93.7 |

##### Segmentation on NVOS.

We report our segmentation results for the NVOS dataset (Ren et al., ) in Table 2.
Our results are comparable to those obtained by prior work. Again, DINOv2 performs surprisingly well while not having been trained on billions of labeled images like SAM. Compared to SAM, DINOv2 better captures complex objects, but sometimes also capture some background noise. This can be seen in Appendix Figure 4 with the example of Trex: while SAM misses out the end of the tail as well as the end of the ribs, DINOv2 captures the whole Trex, but also captures part of the stairs behind. Visualisations of Orchids in Appendix Figure 4 also explain the lower performance of SAM on this scene: the two orchids SAM is missing are not covered by the positive scribbles, which makes the task ambiguous.

**Table 2: Segmentation (IoU) on NVOS (Ren et al., ) with DINOv2, SAM and SAM2.**
|  | Fern | Flower | Fortress | HornsC | HornsL | Leaves | Orchids | Trex | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NVOS | - | - | - | - | - | - | - | - | 70.1 |
| SA3D | 82.9 | 94.6 | 98.3 | 96.2 | 90.2 | 93.2 | 85.5 | 82.0 | 90.3 |
| OmniSeg3D | 82.7 | 95.3 | 98.5 | 97.7 | 95.6 | 92.7 | 84.0 | 87.4 | 91.7 |
| SA3D-GS | - | - | - | - | - | - | - | - | 92.2 |
| SAGA | - | - | - | - | - | - | - | - | 92.6 |
| Ours-DINOv2 | 84.4 | 96.3 | 95.3 | 95.4 | 93.4 | 95.9 | 92.1 | 86.4 | 92.4 |
| Ours-SAM | 85.5 | 97.6 | 98.1 | 97.9 | 94.1 | 96.4 | 73.1 | 88.0 | 91.3 |
| Ours-SAM2 | 84.8 | 97.2 | 98.3 | 97.7 | 92.4 | 96.9 | 73.0 | 89.0 | 91.2 |

### 5.4 Ablation studies

**Table 3: Segmentation (IoU) on SPIn-NeRF (Mirzaei et al., ). We compare purely geometrical reference mask uplifting and reprojection and single-view prediction, feature/mask uplifting or graph diffusion leveraging DINOv2 or SAM2.**
|  | Geometry only | Single view | Uplifting | Graph diffusion |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model: | Reference ask | DINOv2 | SAM2 | DINOv2 | SAM2 | DINOv2 |
| Orchids | 80.9 | 91.4 | 79.2 | 91.7 | 90.7 | 92.6 |
| Leaves | 94.8 | 89.3 | 96.6 | 94.3 | 96.4 | 93.9 |
| Fern | 95.5 | 94.4 | 96.7 | 96.7 | 96.7 | 95.6 |
| Room | 85.7 | 94.5 | 96.3 | 97.1 | 96.6 | 94.7 |
| Horns | 90.4 | 90.7 | 92.7 | 93.1 | 94.9 | 94.4 |
| Fortress | 95.4 | 96.8 | 97.8 | 98.7 | 98.3 | 97.6 |
| Fork | 66.3 | 85.6 | 77.2 | 88.4 | 86.8 | 81.6 |
| Pinecone | 58.8 | 92.9 | 90.3 | 86.7 | 90.8 | 90.1 |
| Truck | 60.0 | 86.2 | 89.3 | 88.8 | 92.6 | 94.8 |
| Lego | 77.2 | 63.5 | 89.1 | 72.4 | 92.9 | 93.2 |
| Average | 80.4 | 88.3 | 90.5 | 90.8 | 93.7 | 92.8 |

In Table 3, we compare our segmentation protocol using DINOv2 and SAM2 to multiple simpler variants. More precisely, we evaluate i) a purely geometrical variant that does not use SAM2 or DINOv2, ii) single-view segmentation in 2D based on SAM2 or DINOv2 2D predictions, iii) uplifting DINOv2 features or SAM2 masks into 3D then rendering them for segmentation, as described in Sec. 4.1 and 4.2, and iv) segmenting using graph diffusion over DINOv2 3D feature similarities.

The purely geometrical approach works well on the forward-facing LLFF scenes (Orchids to Fortress). In these scenes, the reference mask is accurately uplifted and reprojected as the viewing direction changes only a little between each frame. However, it fails on the 360-degree scenes (Fork, Pinecone, Truck, Lego). This points to a suboptimal 3D reconstruction of the scene, likely due to overfitting on the limited numbers of available training views (Chung et al., ).

The single-view variants use a similar process for constructing the features and using them for segmentation as in Sec. 4.1 and 4.2 but without uplifting and rendering. It improves from a purely geometrical approach and performs reasonably well on average, the foreground being well isolated from the rest of the scene. However, as illustrated in Figure 2, the semantic features are at a much lower resolution than those resulting from 3D uplifting, leading to a coarser segmentation.

3D uplifting considerably boosts results compared to single-view approaches. However, performing segmentation in 2D based on the uplifted DINOv2 features does not benefit from the 3D spatial information and typically fails on the 360-degree scenes (Pinecone, Truck and Lego) which have higher variability between frames from different views. Introducing 3D spatial information through 3D graph diffusion results in a boosted performance on these scenes.

#### Acknowledgments

This project was supported by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003) and by ERC grant number 101087696 (APHELEIA project). This work was granted access to the HPC resources of IDRIS under the allocation [AD011013343R2] made by GENCI.

## Appendix A Additional visualisations

### A.1 Segmentation on NVOS

Figure 4 shows our segmentation masks from SAM and DINOv2 for the three most challenging scenes of the NVOS dataset: Fern, Orchids and Trex.

Figure: (a) Reference image
Refer to caption: extracted/5937544/figures/nvos/fern_reference.jpg

### A.2 Diffusion process

Figure 5 illustrates different steps of the diffusion process for Fern, Leaves, Flower and Trex from the NVOS (Ren et al., ) dataset. Starting from the reference scribbles, the diffusion rapidly spreads through the large neighboring Gaussians. Covering the entire object takes more time for complex structures such as Fern, or for masks with disconnected components such as Orchids. As illustrated in the case of Flower, the last diffusion steps allow spreading to the smaller Gaussians on the flowers’ edges, yielding a refined segmentation mask. For Trex, the parts being reached the latest are the head and tail. Their features are further away from the reference features (defined as the average feature over 3D reference scribbles), and therefore the regularization for diffusion is stronger in these regions. Overall once the object has been fully covered, the regularization is very effective at preventing leakage, which allows diffusion to run for an arbitrary number of steps.

Figure: Figure 5: Illustration of the graph diffusion process. 2D projections of i) first three PCA components of DINOv2 3D features, ii) unary regularization term (red), iii) weight vector $g_{t}$ at timesteps $t\in\{0,3,5,10,100\}$, iv) RGB segmentation obtained using a mask based on the 2D projection of $g_{100}$.
Refer to caption: extracted/5937544/figures/diffusion_appendix/fern_pca.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Is the graph diffusion framework based on existing work? If so, the appropriate work in the literature should be cited for that. Otherwise, it would be best to clarify that the formulation is novel and developed by the authors themselves for the 3d segmentation task.

"Let I1, . . . , Im be a set of 2D frames from a 3D scene and d1, . . . , dm the corresponding viewing directions." -- the authors should clarify what they mean by viewing directions, and whether they are referring to camera poses, here? The phrase viewing direction is used throughout the paper and makes it difficult to understand the underlying mathematical operations that are being performed. 

The mathematical notation is difficult to follow. The expression \hat{F} = (\hat{F}_d, [) (lines 188) are not well explained. Similarly, the steps to construction the W and D matrices also need to clearly described.
soundness: 2
strengths: The proposed approach to lift 2D features to 3D is efficient and avoids expensive iterative optimization schemes. The approach appears to be simple to implement and could be effective for downstream tasks such as 3D segmentation of stationary scenes where multiple views of the scene are available.
summary: This paper is primarily concerned with generating 3D feature representations of a scene given multiple images of a stationary scene captured from different viewpoints. The main idea proposed here is that of extracting visual features from the multiple images (using a pretrained general purpose foundational model) and then fusing them with the scene representation computed using the 3D Gaussian splatting optimizer, i.e. fully calibrated camera poses and a set of 3D Gaussians with all the relevant parameters (position, covariance, size, opacity, spherical harmonics coefficients).

Specifically, feature vectors are computed and associated with the 3D centroids for each Gaussian in the 3DGS scene representation. The authors propose to compute these features by an aggregation scheme that is implemented as weighted averaging of the feature vectors extracted in multiple images. First 2D features are computed using existing visual feature extraction models. Features extracted for patches corresponding to the same Gaussian in multiple views are combined using a weighted averaging scheme presented in the paper. The 3D features associated with the 3DGS representation are then used in downstream tasks such as 3D segmentation (or multi-view segmentation) for which certain existing graph diffusion techniques are leveraged. 

The experimental results focus on segmentation performance on the Spin-NeRF and NVOS datasets where the method achieves comparable performance to existing techniques.
weaknesses: My main concerns about this work revolves around:

(1) Low novelty:
- It was unclear to me to what extent the main idea of lifting features from images to 3DGS point couds was novel compared to existing approaches in the literature that have explored scene editing given a 3DGS reconstruction of a scene. The weighted averaging and aggregation scheme described here appears to be very similar to what was proposed in prior work such as Chen et al. 2024.

- The paper mostly focuses on the 3D segmentation application and shows the their proposed ideas work well for that application. They experiment with different features (DINOv2 vs SAM, etc.). However, the authors are not the first to segment or edit 3D Gaussian scenes. This was also explored in the prior work of Chen et al 2024 although they focused on binary segmentation, unlike in this work.

(2) More thorough comparison with existing approaches needed:
The main claimed contribution in the work is towards the technical approach for computing 3D features from 2D features using an efficient approach (referred to as uplifting in the paper). There are other approaches in the literature that have been explored to accomplish the same step. Those optimization based methods are cited and discussed [Kerr et al 2023, Zuo et al 2024] but there is no quantitative comparison between the proposed method and those related works. This makes it difficult to assess the contribution and impact of the novel ideas that are presented here. 

(3) Overall presentation has room for improvement:
The presentation and technical exposition could be improved for better clarity and presentation. I think the core objectives and the problem being tackled in the paper could have been explained better in the abstract and introduction. In the current version of the manuscript, it is difficult to understand what the authors precisely mean by “uplifting visual features to 3D scenes represented by Gaussian Splatting”. The abstract claims that “a simple yet effective aggregation technique yields excellent results.” – but the author have not explained what is being aggregated and for what purpose and what the success criteria is. The authors then talk about competitive segmentation results without clearly explaining the connection between the segmentation task and the 3D reconstruction or 3D scene representation task. Reading Section 1 (Introduction) doesn't give a concrete idea of the main objective, the key motivations and where the novelty lies.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The graph diffusion (Eq.(7)) is similar to power method for computing the dominant eigenvector of a matrix. While it is not guaranteed to converge to the dominant eigenvector, I guess there is a high probability. Is it possible to draw some theoretical connection to, say, spectral clustering methods, and therefore give some insight of the converging point of g_t? 

In figure 2, the rolling stair are separated from the ceiling in the multi-view case while merged to the latter in the single-view case, could the authors provide some explanation on such behavior?

In Sec. 4.2, the DINOv2 features seem to be of coarse scale (predicted in patches), would the recent works on feature super-resolution (e.g., Feature-up in ICLR 2024 by Fu et al.) help?
soundness: 3
strengths: + The proposed scheme of connecting per-pixel 2D features and Gaussians are simple and intuitive. 

+ The segmentation can be directly done without iterative optimization on a trained Gaussian. 

+ The treatment on incorporating with DINOv2 feature into segmentation is nice, as it induces comparable results with the variant using the more tailored-for SAM.
summary: This submission proposes a simple yet effective scheme of uplifting visual features from 3D scenes represented by Gaussian splitting. In contrast to the previous approaches, the proposed method enables direct 3D segmentation without iterative optimization, while achieves comparable to the SotAs.
weaknesses: - There is neither limitation/failure nor future work discussion in the submission, what is the boarder impact of the work for the community? 

- The submission lacks report on running time.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. As in weakness 2, can you explain the process of feature map construction?
2. How is the time efficiency of the proposed method compared to other training-based methods? 
3. As in weakness 4, Can the proposed method performed on other tasks and how well is it?
soundness: 3
strengths: 1. The learning-free feature uplifting method is both simple and effective, achieving strong results without training.
2. Experiments with SAM and DINOv2 demonstrate the method’s efficiency, yielding performance comparable to training-based approaches.
3. High Computational Efficiency: LUDVIG bypasses the costly and time-consuming optimization steps typical in 3D reconstruction methods, making it highly efficient.
4. Versatile Input Adaptability: The proposed method adapts seamlessly to various input types, such as features from SAM and DINOv2, showcasing robust performance across diverse feature sources.
summary: The paper introduces LUDVIG, a learning-free method designed to uplift 2D visual features or semantic masks to 3D Gaussian Splatting scenes. LUDVIG avoids traditional, iterative optimization processes by using a simple aggregation technique, achieving comparable segmentation quality to state-of-the-art methods. The approach is applied effectively to features from both SAM and DINOv2, demonstrating high segmentation performance and computational efficiency across complex 3D scenarios.
weaknesses: 1. While the method is straightforward, it relies on hand-crafted processes, such as the segmentation score calculation and the graph diffusion process. These manual strategies may raise concerns about robustness, particularly in complex, real-world scenarios.
2. Certain sections, like Sec. 4.2, are challenging to follow. For example, the construction of 2D feature maps from DINOv2 is not clearly outlined. Including diagrams or visual aids could greatly enhance understanding and clarify complex steps.
3. Lack of qualitative comparison results with other baselines. 
4. The proposed method is primarily limited to foreground segmentation, whereas other feature lifting approaches support additional tasks, such as instance segmentation, 3D segmentation, and downstream applications like scene manipulation.

---

confidence: 2
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. The lifting algorithm is very simple, which is a weighted average of features collected from multiple views. This reviewer had an experience on object segmentation based on NeRF. I feel that the aggregated features may suffer from noises. For example, there may be errors in camera parameters, which can reduce the effectiveness of the aggregation. I suggest the authors to add more analysis on this problem (In fact, I do not know how to evaluate this, and expect the authors to provide more insights). 

2. The method is probably highly affected by the effectiveness of the feature extraction backbone. I am wondering whether the 2D segmentation masks obtained by the backbones applied to the images are similar to those of the lifted 3DGS projected to the 2D. Please provide this additional comparison.

3. I do not see connection between the proposed method and optimization-based approach in lines beginning from 205. Maybe this paragraph discusses the difference between the two kinds of approaches. I think a title of “Difference with optimization-based inverse rendering” may be more appropriate. 

4. The method needs a graph diffusion to further enhance its results. Why do you need this step? Please provide the results w/ and w/o this step. 

5. Please check the paper and make sure all sections are completed.

Overall, I think the accuracy of this method may be limited due to the noises in the features themselves caused by, e.g., misalignment between features of different views. The method is fast, but there is no argument supporting the necessity of this method at the cost of accuracy.
soundness: 2
strengths: 1. Previous approaches adopt optimizations, while this method is training-free and maybe fast.
2. The presentation is clear.
summary: This paper presents a method that lifts 2D features into the 3D representation space of 3DGS. With the lifted features, the method can perform segmentation of objects, and other applications. The paper tested features of SAM and DINO. For DINO, a graph diffusion step is performed to enhance the results.
weaknesses: 1. The method needs additional post-processing steps such as graph diffusion.
2. The paper is not fully finished. There is no conclusion section.

Meta-review: This paper proposes a method that lift 2D features from foundational models (e.g., CLIP, SAM) to learned 3D Gaussians for a scene. The main idea is to apply weight aggregation on the features of each pixel, as well as a diffusion mechanism that diffuse features to nearby Gaussians. The propose approach is efficient compared to optimization-based methods. The weaknesses of the paper include low novelty (Reviewer sBLh), confusing presentation (Reviewer sBLh, QRFz), effectiveness and robustness of the method (Reviewer N4ZX, jxL3). The AC agrees with the reviewers that the paper has limited novelty and needs improvements on better differentiation with other methods, presentation and evidence to show the robustness of the proposed approach, thus the paper is recommended for rejection. The authors are encouraged to improve the paper given the comments of the reviewers.

Decision: Reject