# RegSegField: Mask-Regularization and Hierarchical Segmentation for Novel View Synthesis from Sparse Inputs

Kai Gu, Thomas Maugey, Sebastian Knorr, Christine Guillemot

► To cite this version:

Kai Gu, Thomas Maugey, Sebastian Knorr, Christine Guillemot. RegSegField: Mask-Regularization and Hierarchical Segmentation for Novel View Synthesis from Sparse Inputs. CVMP 2024 - 21st ACM SIGGRAPH European Conference on Visual Media Production, Nov 2024, London, United Kingdom. pp.1-10, 10.1145/3697294.3697299 . hal-04725951

HAL Id: hal-04725951

https://inria.hal.science/hal-04725951v1

Submitted on 8 Oct 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# RegSegField: Mask-Regularization and Hierarchical Segmentation for Novel View Synthesis from Sparse Inputs

Kai Gu

Inria centre at Rennes University
France

Technische Universität Berlin
Germany

kai.gu@inria.fr

Knorr Sebastian

Hochschule für Technik und Wirtschaft Berlin
Germany

Sebastian.Knorr@HTW-Berlin.de

Thomas Maugey

Inria centre at Rennes University
France

thomas.maugey@inria.fr

Christine Guillemot

Inria centre at Rennes University
France

christine.guillemot@inria.fr

<figure><img src="image_2.png" /><figcaption>Figure 1: The RegSegField regularizes randomly initialized 3D-GS by matching 2D masks and bounding 3D object regions.</figcaption></figure>

## Abstract

Radiance Field (RF) representations and their latest variant, 3D-Gaussian Splatting (3D-GS), have revolutionized the field of 3D vision. Novel View Synthesis (NVS) from RF typically requires dense inputs, and for 3D-GS in particular, a high-quality point cloud from a multi-view stereo model is usually necessary. Sparse input RFs are commonly regularized by various priors, such as smoothness, depth, and appearance. Meanwhile, 3D scene segmentation has also achieved significant results with the aid of RFs, and combining the field with different semantic and physical attributes has become a trend. To further tackle NVS and 3D segmentation problems under sparse-input conditions, we introduce RegSegField, a novel pipeline to utilize 2D segmentations to aid the reconstruction of objects and parts. This method introduces a novel mask-visibility loss by matching 2D segments across different views, thus defining the 3D regions for different objects. To further optimize the correspondence of 2D segments, we introduce a hierarchical feature field supervised by a contrastive learning method, allowing iterative

updates of matched mask areas. To resolve the inconsistent segmentation across different views and refine the mask matching with the help of RF geometry, we also employed a multi-level hierarchy loss. With the help of the hierarchy loss, our method facilitates scene segmentation at discrete granularity levels, whereas other methods require sampling at different scales or determining similarity thresholds. Our experiments show that our regularization approach outperforms various depth-guided NeRF methods and even enables sparse reconstruction of 3D-GS with random initialization.

## CCS Concepts

* Computing methodologies → 3D imaging.

## Keywords

View synthesis, light field, deep learning, (neural) radiance field, Gaussian splatting, regularization.

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution International 4.0 License.

CVMP '24, November 18–19, 2024, London, United Kingdom

© 2024 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-1281-4/24/11

https://doi.org/10.1145/3697294.3697299

## ACM Reference Format:

Kai Gu, Thomas Maugey, Knorr Sebastian, and Christine Guillemot. 2024. RegSegField: Mask-Regularization and Hierarchical Segmentation for Novel View Synthesis from Sparse Inputs. In *European Conference on Visual Media Production (CVMP '24), November 18–19, 2024, London, United Kingdom*. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3697294.3697299
# 1 Introduction

The radiance field (RF) is the most up-to-date solution for the novel view synthesis (NVS) problem. RF reconstruction from sparse input is a rising research area within the fields of computer vision and graphics. This domain has seen a variety of approaches leveraging different forms of prior knowledge, such as monocular depth cues and ray-density distributions, to enhance reconstruction quality. Furthermore, various constraints are employed to ensure local smoothness, including controlling depth total variation and limiting high-frequency components in features [Niemeyer et al. 2022; Yang et al. 2023]. Recent advancements have integrated generative models with RFs, such as diffusion denoising probability models (DDPM [Ho et al. 2020]), latent diffusion models [Blattmann et al. 2022; Rombach et al. 2021a] enhancing image detail and eliminating artifacts, enabling high-quality scene generation from single inputs [Liu et al. 2023b] and new viewpoints [Jain et al. 2021a; Wu et al. 2023; Wynn and Turmukhambetov 2023], despite potential hallucinations and time-consuming training.

Beyond these solutions, visibility constraints have proven effective in eliminating the artifacts and reducing the solution space [Wynn and Turmukhambetov 2023]. In 3D reconstruction, points are typically considered visible only if they are observable by at least two cameras. Reducing the likelihood of sampling points outside the viewing frustums helps remove incorrect reconstructions. For object-centered scenes, employing foreground-background masks limits the sampling space and prevents unwanted background reconstruction. However, this constraint becomes less applicable in 360-degree scenes, captured either with 360 cameras or employing conventional cameras from multiple angles, comprehensively covering a 360-degree view, where most of the 3D points are visible from more than two cameras due to wider spatial coverage. Even for the object-centered scenes, limiting the sampling within the multiple view frustums only helps to remove some near-camera floaters.

As the demand for incorporating scene understanding and other feature information into RFs grows, 2D segmentation has become a necessary preprocessing step for downstream 3D tasks. With high-quality 2D image masks from the segment-anything model (SAM) [Kirillov et al. 2023] and advancements in RF, the 3D segmentation problem is addressed by supervising 3D field segmentation with 2D masks [Kim et al. 2024; Ying et al. 2023] or embedding semantic features into the geometry [Kerr et al. 2023].

Inspired by the frustum visibility constraints and combined with the help of 2D segmentation models, we address the sparse input NVS problem by introducing a novel mask-matching and mask-visibility regularization process with hierarchical segmentation (RegSegField), aiming to limit the solution space of the geometry, i.e., allowing RF to come up with the correct reconstruction without monocular depth or image priors. RegSegField is a novel pipeline that takes posed sparse input views and their SAM masks to improve the reconstruction of different RF representations (e.g., Neural Radiance Field (NeRF) [Mildenhall et al. 2020] and 3D-GS [Kerbl et al. 2023]). The approach employs mask-visibility regularization by finding corresponding image segments in different views and thus bounds the 3D region of different objects or parts by the intersection of mask-view-frustums projected from different views.

Fig. 1 demonstrates the regularization process by visualizing the intersections of different object masks, while the naive 3D-GS fails to produce reasonable geometry for a sparse input of 3 views.

To apply mask-visibility regularization, correspondences of masks among different views need to be found. In principle, the matching of masks can be done by different metrics of similarity. Here, we reuse the sparse keypoint matching from Structure-from-Motion (SfM) obtained by Colmap [Schönberger and Frahm 2016; Schönberger et al. 2016]. The sparse keypoint matching can be noisy and incomplete due to limited overlapping among views, therefore we introduce a mask embedding module that maps the mask indices to learnable mask descriptors representing their correspondence. The multi-level descriptor, which carries the hierarchical information of local patches, is initialized by the matched keypoints and optimized jointly with the RF.

The initialized descriptors offer a sub-optimal regularization for the RF optimization at the beginning. To refine the matching, we assign the descriptors to the geometry of the RF and optimize the feature field by contrastive learning. Thus, mask matching between views is implicitly achieved with the help of the RF. To deal with potential inconsistent segmentation among different views, we apply a smoothed hierarchy loss that distributes the supervision to neighboring hierarchical levels. Finally, we update the mask-embedding module and refine visibility regularization iteratively.

RegSegField offers a novel way to regularize few-shot RFs by fully utilizing spatial information from 2D segmentations. This approach improves NVS quality and enables 3D segmentation from sparse input. Our contributions are summarized as follows:

* We regularize different RF representations (NeRF and 3D-GS) using sparse input views by applying mask visibility constraints and demonstrate the method's superior performance compared to state-of-the-art NeRF and 3D-GS methods.

* We designed a novel mask-embedding module to extract the correspondences and hierarchy of 2D segments by applying mask matching.

* We introduce a hierarchical feature field supervised by a smoothed hierarchy loss, allowing the refinement of mask matching and providing discrete-level hierarchical 3D segmentation at different granularity.

# 2 Related work

We present a new approach for mask regularization and 3D segmentation in novel view synthesis (NVS) from sparse inputs. We first review related works in NVS/NeRF, 3D segmentation, and few-shot radiance fields.

## 2.1 Novel View Synthesis and Neural Radiance Fields

The Neural Radiance Fields (NeRF) [Mildenhall et al. 2020] have revolutionized view synthesis. It originally employs a simple multi-layer perceptron (MLP) to render 2D views from 3D points sampled along rays. NeRF relies on MLP queries for determining colors and opacities, which present a significant efficiency bottleneck in training and rendering. Innovations like instantNGP [Müller et al. 2022], TensoRF [Chen et al. 2022], and Plenoxels [Fridovich-Keil et al. 2022] introduce more efficient methods to represent RF either
explicitly or semi-explicitly. Further advancements such as 3D-Gaussian splatting (3D-GS) [Kerbl et al. 2023] utilizes anisotropic 3D-Gaussian kernels for differentiable radiance field composition, enabling high-quality real-time rendering and seamless integration into modern rendering frameworks.

Although these methods have solved the efficiency issues of NeRF, the faster training speeds often mean that these methods end up in local minima during the training for sparse inputs. Models like the hash-encoding-based method InstantNGP and Nerfacto [Tancik et al. 2023] as well as randomly initialized 3D-GS are more vulnerable than the pure-MLP-based NeRF and can easily overfit to individual training views, forming erroneous structures like reliefs that appear in front of cameras [Niemeyer et al. 2022; Wynn and Turmukhambetov 2023].

## 2.2 3D Image Segmentation

Alongside advancements in RF, 3D image segmentation has seen substantial enhancement. Leveraging the spatial consistency of RF, 3D image segmentation problems can be transformed into RF segmentation tasks. The segment-anything model (SAM) [Kirillov et al. 2023] and related technologies [Liu et al. 2023c; Ren et al. 2024] have simplified the process of obtaining high-quality automated 2D image segmentation. SANeRF-HQ [Liu et al. 2023a] assigns features from the SAM to a feature field, which, through a decoder, produces multi-view object masks for training an object field. However, these methods still require user input for prompts and necessitate redecoding and optimizing features for each query to generate the object field. Recent studies have exploited the segment-anything model's capability to autonomously generate masks, achieving automatic segmentation of 3D scenes. Gaussian Grouping [Ye et al. 2024] introduces a compact identity encoding for the grouping of 3D Gaussians. Panoptic NeRF [Fu et al. 2022] transfers 2D semantic labels to NeRF and also improves the geometry with the consistency of semantic masks. Innovations like Garfield [Kim et al. 2024] introduce a scale-conditioned affinity field, employing contrastive learning [Hadsell et al. 2006a] to achieve spatially consistent field segmentation across varying granularities. Similarly, OmniSeg3D [Ying et al. 2023] utilizes hierarchical clustering [Li et al. 2021], trained through contrastive learning, to segment a feature field into object segments at different similarity levels. However, these methods' granularity-dependent segmentation results rely on multiple samplings across continuous scales or clustering thresholds, necessitating additional post-processing computations.

We have found that SAM masks inherently exhibit containing and overlapping relationships, which reveal the local hierarchical structure. Similarly, we developed a feature field that shares the RF geometry but incorporates an explicit multi-level design. This design allows users to select the desired granularity directly.

## 2.3 Few-Shot Radiance Fields

Despite the significant success of techniques like 3D-GS in NVS, they typically require numerous input images from various angles and positions, often sourced from one or several high-quality videos or camera arrays, which reduces practicality and increases storage capacity. To tackle this challenge, numerous studies have explored various methods to enhance the reconstruction quality of

few-shot RF. Depth information, whether from monocular depth estimation models [Guangcong et al. 2023; Xiong et al. 2023; Zhu et al. 2023], sparse data from multi-view stereo [Deng et al. 2022], or depth sensor data [Guangcong et al. 2023], has proven effective in improving geometric accuracy under limited viewpoint conditions. Recent sparse reconstruction efforts on 3D-GS platforms also leverage depth priors, such as in FSGS [Zhu et al. 2023] and SparseGS [Xiong et al. 2023]. RegNeRF [Niemeyer et al. 2022] enhances reconstruction consistency by imposing depth smoothness constraints, while DietNeRF [Jain et al. 2021b] utilizes semantic consistency across multiple views for regularization. The use of pre-trained diffusion models [Ho et al. 2020; Rombach et al. 2021a] as generative image priors has become a hot topic in sparse or even single-input NVS. DiffusioNerf [Wynn and Turmukhambetov 2023], for instance, trains an RGB-D prior to regularize patch appearance and geometry, whereas SparseGS [Xiong et al. 2023] employs pre-trained stable diffusion model [Rombach et al. 2021b] directly for enhancing image quality and partial generation from new viewpoints. These methods typically require additional pre-trained depth or RGB models as priors. While they can produce high-quality renderings, they occasionally introduce the risk of hallucination [Wu et al. 2023].

RegSegField effectively utilizes 3D segmentation as prior information to regularize the reconstruction of different radiance field representations with sparse input and allows 3D-GS to function on sparse inputs without the need for point-cloud initialization.

## 3 Method

The entire pipeline of our approach is depicted in Fig. 2. RegSegField requires standard radiance fields (RF) inputs, i.e., the images with poses estimated from Structure from Motion (SfM) models like Colmap [Schönberger and Frahm 2016; Schönberger et al. 2016]. In addition, the model also takes the intermediate outcome of Colmap, the keypoint pairs from 2D image pairs, and an optional input of sparse 3D points for the 3D-GS [Kerbl et al. 2023] implementation of RegSegField. Moreover, the approach requires 2D segmentation masks at different semantic granularity generated from the Semantic Segment Anything Model (Semantic SAM) [Li et al. 2023].

The pipeline involves three stages: first, the mask-matching stage assigns masks of the same object in different views with similar descriptors with the help of matched key points. Second, in the regularized RF training stage, the RF is optimized by limiting the reconstruction of the objects within the 3D regions offered by their corresponding 2D masks; and third, in the feature field training and refinement of the regularization stage, a feature field is optimized to enhance the 3D consistency of mask descriptors from stage 1, and further improves the regularization step in stage 2. The individual processing steps of the pipeline are detailed in the following subsections.

### 3.1 Preliminary

We first give an overview of the base representations that our approach builds upon, the Neural Radiance Field (NeRF) [Mildenhall et al. 2020] and 3D Gaussian splatting (3D-GS).

The NeRF uses a neural network to model the volumetric scene by predicting the color (c) and density (σ) at any 3D point (x) and
the viewing direction (d). The pixel color C can be calculated by volume rendering, which samples N 3D points discretely at an interval $\delta_i$ along a camera ray:

$$C = \sum_{i=1}^{N} T_i(1 - \exp(-\sigma_i \delta_i))c_i, \quad \text{with} \quad T_i = \exp(-\sum_{j=1}^{i-1} \sigma_j \delta_j) \quad (1)$$

$T_i$ represents the accumulated transmittance. The NeRF can be optimized by minimizing the photometric loss between the ground truth image and the rendered one.

The 3D-GS method instead represents the scene as a collection of anisotropic 3D Gaussians with covariance matrix $\Sigma \in \mathbb{R}^{3 \times 3}$ centered at point $\mu \in \mathbb{R}^3$. Each Gaussian has an impact to any 3D point $\mathbf{x}$ following the Gaussian distribution $G(\mathbf{x}) = \exp(\mathbf{x} - \mu^T \Sigma^{-1} \mathbf{x} - \mu)$.

An opacity value $\alpha \in \mathbb{R}$ and D-degree spherical harmonics (SH) coefficients $c^{SH} \in \mathbb{R}^{D^2}$ are associated to each Gaussian. The pixel value is calculated by summing all the Gaussians that contribute to it as:

$$C = \sum_{i=1} c_i \alpha_i T_i \quad (2)$$

where the $c_i \in \mathbb{R}^3$ is calculated from the $c^{SH}$ of i-th Gaussian given the view direction d. $\alpha_i$ is computed by evaluating the projected 2D covariance matrix $\Sigma'$ multiplied by $\alpha$. $T_i = \prod_{j=1}^{i-1} (1 - \alpha_j)$ is the accumulated transmittance. For a detailed derivation of $\alpha_i$ and $\Sigma'$, readers are encouraged to refer to [Yifan et al. 2019; Zwicker et al. 2001].

During the optimization process, Gaussians with low opacity are removed, while those with high view-space position gradients are densified. 3D-GS optimization is effective when initialized with ample 3D points from Structure from Motion (SfM) model or in densely observed scenes. However, with sparse inputs, the point cloud may be too sparse, causing Gaussians to overfit to training views, and random point initialization is infeasible.

To clarify and simplify the description, we consolidate the depiction of both rendering techniques as the sum of the colors $c_x$ and blending weights $w_x$ at 3D points $\mathbf{x} \in \mathcal{X}_p$ that contribute to pixel $p$.

$$C_p = \sum_{\mathbf{x} \in \mathcal{X}_p} c_x w_x \quad (3)$$

## 3.2 2D Mask Generation

Our RegSegField approach first extracts 2D masks from all input images. SAM [Kirillov et al. 2023] offers high-quality masks automatically. The mask generator processes a grid of points, generating multiple candidate masks per query and subsequently post-processing these to eliminate duplicates and low-confidence entries, resulting in overlapping masks. Due to SAM's limited control over output granularity, we utilize Semantic SAM [Li et al. 2023] for mask extraction in our approach. Semantic SAM accepts semantic granularity, from instance to different part levels, as input prompts alongside the standard SAM input, enabling the production of masks at varying hierarchical levels.

## 3.3 Mask Matching

To enable mask-visibility regularization, we need to find the corresponding masks across different views and thus, identify if a mask

is visible in those views. A typical way is to use a union-find data structure [Kozen 1992] to group known matches and end up with a collection of disjoint sets. However, such a flat data structure destroys the overlapping and containing relations between masks, and cannot deal with incorrect matching and conflicting segments. To avoid this, we train a per-scene mask embedding module showing global matching information between views while maintaining the local hierarchical structures.

For an image $I_i$, a set of binary masks $M_i = \{M_1, M_2, \dots, M_{S_i}\}$ is collected and sorted by the area of the masks as shown in Fig. ?? (a). Here $S_i$ is the number of masks from $I_i$. Stacking these binary masks allows us to construct a multi-hot mask vector $\mathbf{m}_p^i \in \{0, 1\}^{S_i}$ for each pixel $p = (u, v)$ of image $I_i$, given by $\mathbf{m}_p^i = [M_1(p), M_2(p), \dots, M_{S_i}(p)]^T$. We consider pixels with the same mask values as patch $P$, and the image segments are represented as a collection of patches as depicted in Fig. 3 (b).

Given that the semantic segmentation presents masks of varying granularity (semantic, instance, parts), meaning that an individual pixel may belong to multiple masks reflecting a multi-level hierarchy, we propose a hierarchical global descriptor to encapsulate the intra-image object mask hierarchy and inter-view global matching.

We define the mask descriptor of a pixel $p$ as $\mathbf{\gamma}_p \in \mathbb{R}^{L \times C}$, where $L$ is the maximum number of hierarchy levels and $C$ is the dimensionality of features per hierarchy level. We aim for the descriptors at level $l$ of pixels from different views to be as similar as possible when projected from the same group of 3D points at hierarchy level $l$, while ensuring that descriptors from different groups are as distinct as possible. We then propose a mask embedding module to obtain the mask descriptor that meets our requirements by assuming that a mapping $\Gamma(\mathbf{m}_p, i) \to \mathbf{\gamma}_p$ exists (When discussing mappings from the same image, we omit the image index $i$). Hence, the problem of matching 2D masks can be converted to finding the correct mapping.

Given the limited observation of different objects and inconsistent masks across different views, mask matching is an ill-posed problem, making it infeasible to achieve such mapping with a simple lookup table. Instead, we approximate $\Gamma_\Psi$ using a multi-layer perceptron (MLP) with parameters $\Psi$. The MLP has $N$ input heads that fit the different number of the masks for $N$ input views, while they share the same output head. We optimize the network using a contrastive learning approach by minimizing both a **match loss** and a **hierarchical loss**, which ensures global consistency and maintains local hierarchy.

**Match Loss:** Consider two pixels, each taken from images $I_i$ and $I_j$ respectively. Their descriptors are considered a positive pair when they belong to masks of the same object or the same part of an object. Such pairs can be identified by various measures of similarity between pixels or segments. In this case, we use pre-obtained Colmap keypoint pairs as matches for the pixels. For the image pair $(I_i, I_j)$ with $K_{ij}$ keypoint-pairs, their descriptors are pulled together by minimizing the L2 distance across all levels $l$:

$$\mathcal{L}_{\text{match}}^{ij} = \frac{1}{L} \sum_{l=1}^{L} \sum_{k=1}^{K_{ij}} \frac{\|\Gamma(\mathbf{m}_{p_k}^i, i) - \Gamma(\mathbf{m}_{p_k}^j, j)\|_2}{K_{ij}}. \quad (4)$$
<figure><img src="image_4.png" /><figcaption>Figure 2: The pipeline starts with mask matching, supervised by sparse matching from Colmap. Matched masks serve as global descriptors to regularize mask visibility during RF optimization, alongside optimizing a hierarchical feature field under 2D image supervision. The feature field then updates the mask descriptor, enhancing the regularization.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 3: Illustration of the hierarchical supervision of the mask descriptor.</figcaption></figure>

Here, the $m_{p_k}^i, m_{p_k}^j$ are the mask vectors of k-th keypoint on image i and j.

**Hierarchy Loss:** As defined by DrLIM [Hadsell et al. 2006b], the margin-based contrastive loss function has two main components: one component draws features within the same group closer together, while the other pushes features from different groups apart. Consider two pixels $p_a, p_b$ in two different segment patches of the same image, each with their multi-hot mask represented as $m_a, m_b$. Their hierarchical relationship can be expressed as a degree of overlapping $t_{ab} = m_a \cdot m_b + 1$, all the mask combinations will form a correlation matrix $T \in \mathbb{R}^{S \times S}$ as depicted in Fig. 3 (c). We define the naive hierarchy loss in two parts: the descriptor-pairs are pulled together from levels 1 to $t_{ab}$ and pushed apart at the remaining levels:

$$ \mathcal{L}_{\text{pull}}^{ab} = \|\Gamma^{1:t_{ab}}(\mathbf{m}_a) - \Gamma^{1:t_{ab}}(\mathbf{m}_b)\|_2 \quad (5) $$

$$ \mathcal{L}_{\text{push}}^{ab} = \min(0, \eta - \|\Gamma^{t_{ab}+1:L}(\mathbf{m}_a) - \Gamma^{t_{ab}+1:L}(\mathbf{m}_b)\|_2). $$

$\eta$ represents the margin, a minimum distance allowed for negative pairs. The sum of the pull and push losses forms the hierarchy loss

$\mathcal{L}_{\text{hierarchy}}:$

$$ \mathcal{L}_{\text{hierarchy}} = \frac{2}{n(n-1)} \sum_{1 \le a < b \le S} \mathcal{L}_{\text{pull}}^{ab} + \mathcal{L}_{\text{push}}^{ab} \quad (6) $$

For example in Fig. 3, patch 4 and patch 5, the descriptors from levels 1 to 3 will be pulled together, and the descriptor will be pushed part at level 4, which means these 2 patches will be grouped together until the hierarchy level 4.

This formulation captures both the hierarchical structure within individual images and the correspondence across images. We train the parameters of $\Gamma_{\Psi}$ by minimizing the match loss (inter-image step) and the hierarchy loss (intra-image step). The inter-image and intra-image steps are carried out alternatively.

## 3.4 Mask Visibility Regularization

Due to excessive ambiguity in sparse input scenarios, we want to limit the sampling region as much as possible. Hence, our goal is to ensure that each object/part is reconstructed within the 3D region bounded by their 2D masks in different views as illustrated in Fig. 1, where the mask-frustums of different objects are visualized in different colors. Having pre-trained mask descriptors by the mask-matching process, we obtained the correspondence of segments across different views, allowing us to achieve the goal by applying mask-visibility constraints. During the optimization of a RF, a pixel is considered visible if the 3D point x, which contributes to the final rendering, falls in the regions of corresponding segments in n-other views. Here, x refers to one 3D point in the NeRF or the center of one 3D Gaussian in the 3D-GS model, respectively. Similar to the frustum loss proposed in DiffusioNeRF [Wynn and Turmukhambetov 2023], the mask visibility loss is defined as:

$$ \mathcal{L}_{\text{mask}} = \sum_{\{x | n_x \le 1\}} w_x \quad (7) $$
<figure><img src="image_6.png" /><figcaption>Figure 4: Given 3D points in a patch from view (a), reprojected to target views (b) and (c), the color indicates the descriptor distance of patches between source and target views.</figcaption></figure>

where $w_x$ is the blending weight at 3D points $x$, and $n_x$ represents the number of matched masks in which the points are contained:

$$n_x = \sum_{\substack{j=1 \\ j \neq i}}^{N} \mathbf{1}(\|\Gamma(\mathbf{m}_{p_x}^i, i) - \Gamma(\mathbf{m}_{p_x'}^j, j)\|_2 < \epsilon), \quad (8)$$

where $\Gamma_i(\mathbf{m}_{p_x}^i)$ is the descriptor of the corresponding pixel in the source view $i$, and the $\Gamma_j(\mathbf{m}_{p_x'}^j)$ is the descriptor of reprojected $x$ onto the target view $j$. $\epsilon$ is a threshold determining the maximum distance allowed for a match. Weights of 3D points lying in fewer than two matched masks will be penalized. Fig. 4 illustrates the matching process in 2D with the 3D points in the source view and target views, the colors of the points represent the dissimilarity between corresponding descriptors.

Minimizing the $L_{mask}$ loss jointly with the RF photometric objectives penalizes the density of points outside the region of the objects given by the 2D masks. As shown in Fig. 1, the object will be reconstructed effectively in the overlapping regions of corresponding masks.

## 3.5 Hierarchical Feature Field

Due to the sparse matching and inconsistent masks, the correspondence of 2D masks from the mask embedding module is suboptimal as illustrated in Fig. 6. Applying a visibility loss from the wrong matching will prevent the reconstruction of some object that is actually visible. To solve this, we train an L-level hierarchical feature field ($H(x) \to \mathbb{R}^{L \times C}$), which enables hierarchical 3D segmentation and fine-tuning of the mask descriptors in the refinement stage of visibility regularization. The features can be rendered with a weighted average using the same RF density-based rendering weights to obtain a feature value $\hat{\Gamma}(p)$ on a per-ray basis as:

$$\hat{\Gamma}(p) = \sum_{x \in \mathcal{X}_p} H(x) w_x. \quad (9)$$

To ensure the consistency between mask descriptors and 3D features, we initialize the feature field with the 2D mask descriptor by minimizing the rendering loss:

$$L_{SegRender} = \|\hat{\Gamma}(p) - \Gamma(p)\|_2. \quad (10)$$

With the training process of the RF, mask-matching is implicitly carried out during the optimization, allowing us to further refine the hierarchical feature field. After the initialization is done, we further apply the contrastive supervision to the rendered feature. It is worth mentioning that the naive hierarchy loss in Eq. (5) does not

<figure><img src="image_7.png" /><figcaption>Figure 5: Example for showing the conflict-resolving process for the multi-level hierarchy supervision.</figcaption></figure>

resolve the inconsistency from different native hierarchy trees from individual 2D segments. The illustration in Fig. 5 demonstrates the challenge. We address the problem by applying a smoothed loss to the multi-level supervision. Given 2 3D points $(x_a, x_b)$ on the surface of objects from the scene, their corresponding pixels on views $i$ and $j$ are $(p_a^i \to p_a^j)$, $(p_b^i \to p_b^j)$, respectively, assuming that there is an underlying ground truth hierarchy relation as shown in Fig. 5. For the hierarchy level $t=2$, the supervision from view $i$ pulls $\Gamma_a, \Gamma_b$ together, while in view $j$ they should be pushed apart. Therefore, we assign a decreasing weight $\omega_t$ to the $L_{hierarchy}$ of all the levels, where $\omega_t$ is a Gaussian distributed weight centered at $t$ with standard deviation $\sigma$:

$$L_{sh} = \frac{1}{\Omega} \sum_{t=1}^{L} \sum_{l=1}^{L} \omega_t(l) L_{hierarchy}^l, \quad \text{where } \omega_t(l) = \exp\left(-\frac{(l-t)^2}{2\sigma^2}\right), \quad (11)$$

$\Omega$ is a normalization term that ensures the sum of weighting equals 1. Considering the level of hierarchy from SAM model of different views to be a noisy Gaussian distribution centered at the underlying ground truth $t$, applying soft supervision across all the hierarchy levels, we effectively resolved the conflict hierarchy and inconsistent segmentation among different views.

## 3.6 Visibility Refinement

We notice that wrong matches at the beginning will also lead to floaters in incorrect intersection regions. Applying visibility loss with higher weight after the optimized matching will further improve the result. As we update the hierarchical feature field, we can further improve the mask embedding module $\Gamma_{\Psi}$ for further regularization. In this case, we apply the same render loss in Eq. (10) to the predicted mask descriptor $\Gamma(p)$, while we detach the gradient of $\hat{\Gamma}(p)$. Fig. 6 shows how the mask descriptors are updated after the refinement of the feature field.

## 3.7 Auxiliary loss

To further improve the geometry accuracy, the depth smoothness evaluated by total variation loss has proven to be effective by RegNeRF [Niemeyer et al. 2022]. The high-quality image segmentation can serve not only our main goal for visibility regularization, the image segments also provide additional information that can be used to improve the auxiliary loss of depth. We render depth $\hat{d}$ of pixel $p$ by $\sum_{x \in \mathcal{X}_p} d_x w_x$, $d_x$ is the distance of $x$ to the camera.

Penalizing TV of the rendered depth also helps to remove some artifacts like floaters. The TV loss penalizes sudden depth changes while preserving the edges. However, we find that there can be a
depth discontinuity along the boundary between objects, which
can also be over-smoothed. As we have obtained high-quality im-
age segmentation, we apply a more dedicated smoothness loss by
excluding the pixels between different segment patches. For pixel
p at coordinate (u,v), we apply the TV loss to all the pixels with the
same mask values as their vertical and horizontal neighbors as:

$$
\{(u, v) | \mathcal{M}(p_{uv}) = \mathcal{M}(p_{(u+1)v}) = \mathcal{M}(p_{u(v+1)})\}
$$

$$
\mathcal{L}_{\text{maskTV}} = \sum_{u,v} \left[ (\hat{d}(p_{uv}) - \hat{d}(p_{(u+1)v}))^2 + (\hat{d}(p_{uv}) - \hat{d}(p_{u(v+1)}))^2 \right]. \quad (12)
$$

In summary, our whole pipeline matches 2D image masks with the introduced mask embedding module and outputs mask descriptors for regularization. Optimization of the RF is achieved by minimizing a combined loss comprising photometric, mask visibility, and mask-aided smoothness losses. Following the initial warm-up steps, further refinement of mask matching and 3D segmentation is accomplished by minimizing smooth-hierarchy and feature render losses. The implementation details of the model and the training process can be found in the supplementary materials and our upcoming code release.

4 Experiment

4.1 Experimental settings

We integrate our regularization method in the Nerfacto and Splafacto
implementations of the Nerfstudio [Tancik et al. 2023] environ-
ment, a modular implementation of state-of-the-art RF models that
is user-friendly and computationally more efficient. We use three
datasets (DTU [Jensen et al. 2014], MIP-NeRF 360 [Barron et al.
2022], LERF [Kerr et al. 2023]), respectively, to validate the effec-
tiveness of our method for NeRF-like and 3D-GS-like methods to
address the view synthesis problems from sparse inputs.

The **DTU Dataset** involves scenes with objects on a table, where we followed the 3-view setup from RegNeRF [Niemeyer et al. 2022] and selected 10 representative scans that consist of multiple objects and we focus only on foreground reconstruction. **MIP-NeRF 360 Dataset** consists of 9 scenes with a central object and a complex background, captured by cameras rotating 360°. For this dataset, we used 5 scenes with clear object boundaries, selecting 12 views for training and holding out every 8th image as a test set, following community standards [Mildenhall et al. 2020]. **LERF Dataset** contains complex object compositions, typically used for segmentation tasks, from which we used 8 views as model input.

Our quantitative analysis compares our methods against depth-
guided baselines, D-Nerfacto and D-Splafacto, using local depth
ranking loss[Guangcong et al. 2023; Zhu et al. 2023]. We also com-
pared with recent few-shot 3D-GS methods, SparseGS [Xiong et al.
2023] and FSGS [Zhu et al. 2023], measuring PSNR, SSIM [Wang
et al. 2004], and LPIPS [Zhang et al. 2018] on the test dataset. Ad-
ditionally, we conducted a qualitative comparison with GARField
[Kim et al. 2024] on segmentation tasks.

4.2 Few-shot radiance field

As RegSegField focuses on the regularization of few-shot radiance
fields, the method is mostly effective in dealing with the random ray-
sampling phase. The 3D-GS methods usually rely on 3D SfM points

<table><caption>Table 1: Quantitative results on DTU scenes with 3 input and Mip-NeRF 360 scenes with 12 input views.</caption>
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th colspan="3">DTU</th>
      <th colspan="3">Mip-NeRF 360</th>
    </tr>
    <tr>
      <th>PSNR ↑</th>
      <th>SSIM ↑</th>
      <th>LPIPS ↓</th>
      <th>PSNR ↑</th>
      <th>SSIM ↑</th>
      <th>LPIPS ↓</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>D-Nerfacto</td>
      <td>13.53</td>
      <td>0.756</td>
      <td>0.260</td>
      <td>15.49</td>
      <td>0.355</td>
      <td>0.542</td>
    </tr>
    <tr>
      <td>Ours-NF</td>
      <td><strong>18.04</strong></td>
      <td><strong>0.772</strong></td>
      <td><strong>0.161</strong></td>
      <td><strong>18.61</strong></td>
      <td><strong>0.551</strong></td>
      <td><strong>0.290</strong></td>
    </tr>
    <tr>
      <td>D-Splafacto</td>
      <td>18.67</td>
      <td>0.836</td>
      <td>0.122</td>
      <td>18.84</td>
      <td>0.588</td>
      <td>0.260</td>
    </tr>
    <tr>
      <td>FSGS</td>
      <td>16.58</td>
      <td>0.805</td>
      <td>0.115</td>
      <td><strong>19.44</strong></td>
      <td><strong>0.623</strong></td>
      <td><strong>0.216</strong></td>
    </tr>
    <tr>
      <td>SparseGS</td>
      <td>17.648</td>
      <td>0.810</td>
      <td><strong>0.113</strong></td>
      <td>18.68</td>
      <td>0.599</td>
      <td>0.222</td>
    </tr>
    <tr>
      <td>Ours-GS</td>
      <td><strong>18.97</strong></td>
      <td><strong>0.833</strong></td>
      <td>0.117</td>
      <td>19.10</td>
      <td>0.595</td>
      <td>0.248</td>
    </tr>
    <tr>
      <td>D-Splafacto-R</td>
      <td>10.02</td>
      <td>0.593</td>
      <td>0.747</td>
      <td>13.18</td>
      <td>0.254</td>
      <td>0.564</td>
    </tr>
    <tr>
      <td>Ours-GS-R</td>
      <td><strong>17.92</strong></td>
      <td><strong>0.783</strong></td>
      <td><strong>0.159</strong></td>
      <td><strong>16.54</strong></td>
      <td><strong>0.462</strong></td>
      <td><strong>0.346</strong></td>
    </tr>
  </tbody>
</table>

as initialization and progressively grow and densify the Gaussian
from the existing ones, which makes our methods less effective
than in the NeRF methods. Here, we also report the results of 3D-GS
with random points as initialization, which makes it hardly possible
for 3D-GS to reconstruct the scene from sparse input.

Tab. 1 shows a quantitative comparison of our method against
depth-guided baseline methods and SOTA sparse 3D-GS methods.
Our methods outperform all other methods in the 3-view setup
of the DTU scenes by 0.3 to 7.9 dB. Fig. 9 shows that our method
can remove the artifacts around the objects and make the floating
Gaussians attach to the surface of the objects correctly. For the ex-
periment using 3D-GS method with random initialization and NeRF
method, the result is degraded heavily. Even with monocular-depth
guidance, the rendering in Fig. 10 appears to overfit to training
views, while our methods reconstruct objects correctly. Please note
that some methods use different training splits in their reported
results. To ensure a fair comparison, we retrained all models on our
selected scenes using the same training configuration (following
community practice). Therefore, the results may differ from those
in the original reports.

In Tab. 1 and Fig. 9, we observe that even in complex 360-degree
scenes (Mip-NeRF dataset), our method still improves the results
of various baseline approaches. This improvement is particularly
evident when using the Nerfacto and randomly initialized 3D-GS
methods as can be seen in Fig. 10 (more than 3dB in PSNR). However,
we found that when 3D-GS is initialized with point clouds, our
method becomes less effective. Regularization with mask-visibility
does not always enhance the final result due to the challenges in
initial mask-matching. The FSGS achieves the best performance
with the help of dense multi-view stereo (MVS) matching and depth
guidance. It is worth mentioning that our method penalizes the
unseen parts of the scene and eliminates the potential "outpainting"
from the border of the visible part, causing a black background to
show through as depicted in the red box in Fig. 7, this leads to a
significant bias in quality metrics, although our method presents
better results as clearly visible in the green box.
<figure><img src="image_8.png" /><figcaption>Figure 6: Visualization of the initial/optimized descriptors of 2 input views (mapped to RGB with PCA). The optimized descriptors are well-matched after the optimization process.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 7: Illustration of the outpainting bias. The top right corner is the region not well covered by the training views.</figcaption></figure>

<table><caption>Table 2: Ablation study on one DTU scene with 3 input views: results improve as more dedicated constraints are applied.</caption><thead><tr><th></th><th>PSNR ↑</th><th>SSIM ↑</th><th>LPIPS ↓</th></tr></thead><tbody><tr><td>+ frustum</td><td>11.72</td><td>0.784</td><td>0.230</td></tr><tr><td>+ fore-background</td><td>17.30</td><td>0.839</td><td>0.122</td></tr><tr><td>+ mask-visibility</td><td>21.41</td><td>0.890</td><td>0.064</td></tr><tr><td>+ refined mask-visibility</td><td>21.80</td><td>0.893</td><td>0.065</td></tr><tr><td>+ naive TV loss</td><td>21.84</td><td>0.898</td><td>0.058</td></tr><tr><td>full-model</td><td><strong>23.08</strong></td><td><strong>0.905</strong></td><td><strong>0.049</strong></td></tr></tbody></table>

## 4.3 Hierarchical segmentation

As an additional contribution of RegSegField, we trained a field with hierarchical segmentation features, demonstrating scene segmentation at different levels of granularity. We directly provide discrete levels for selection, whereas Garfield requires adjusting the scale values to compute the segmentation granularity from the feature field at different scales. We visualize the result of one scene from MIP-NeRF-360 and one scene from the LERF (Fig. 11).

## 4.4 Ablation study

RegSegField incorporates 3 key regularization components for few-shot RF reconstruction: mask-visibility constraints, mask-TV loss, and mask descriptor refinement. To highlight their necessity and effectiveness, we conducted an ablation study. We trained a Splafacto model with random Gaussian initialization, starting with frustum visibility regularization. Then, we substituted fore-background visibility, followed by mask-visibility, and finally refined mask-visibility. To assess the mask-aided TV loss, we compared the result with naive TV loss using our full model. Results in Tab. 2 demonstrate

the effectiveness of visibility regularization and the improvement introduced by the mask-descriptor optimization. Furthermore, the mask-aided TV loss will also further improve the result largely.

## 5 Limitation

While the mask-matching step using keypoints can be replaced by other similarity metrics without affecting the core concept of mask visibility regularization, we have not tested alternative matching methods, such as semantic labels, or deep feature correlation. Currently, Colmap is required for sparse matching and accurate camera pose estimation. In highly complex scenes, keypoint-based sparse matching often suffers from insufficient keypoints or inconsistent masks, leading to noisy or incorrect matches. This degrades the regularization quality, potentially removing objects and creating holes in the final rendering (Fig. 8). Although the regularization occurs only at the start of training, it loops over all reference views in one step, making training iterations slower. The process is 3× slower than Splafacto and 1.2× slower than Nerfacto, with training time increasing linearly as more input views are added.

## 6 Conclusion

Our proposed RegSegField approach successfully utilizes the spatial consistency information provided by 2D masks to constrain the reconstruction of the radiance field from sparse inputs without the need for estimated monocular depth. Thanks to the descriptor-based mask-matching approach and visibility regularization, we not only enhanced the reconstruction quality of state-of-the-art NeRF methods for sparse setups but also enabled high-quality reconstructions through 3D Gaussian splatting (3D-GS) without the need for SfM point clouds. The mask-visibility regularization enables 3D-GS to sample the entire space more uniformly, similar to traditional NeRF methods, without relying on 3D point clouds.

Additionally, to optimize the regularization process, we proposed the use of discrete-level hierarchical feature fields. Through contrastive learning, we obtained segmentation features with spatial consistency. Benefiting from discrete supervision, our method avoids the need to resample different scales or compute similarity-based clustering to determine thresholds, allowing users to directly switch between different levels to obtain corresponding segmentation results. Meanwhile, as the feature field is optimized, we allow iterative updates of the mask-descriptor to achieve more consistent mask-matching, thereby enhancing subsequent regularization.

Finally, our regularization approach offers valid 3D regions and 2D visibility information for object-wise reconstruction, enabling potential downstream applications like object-based 3D generation, scene editing, and more dedicated inpainting/outpainting tasks for less-visible regions. These promising results and findings open the floor for further research on better 2D segmentation methods and alternative mask-matching approaches to enhance the performance of our proposed regularization steps.

## Acknowledgments

The work was supported by the European Joint Doctoral Programme on Plenoptic Imaging (PLENOPTIMA) through the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Grant Agreement No. 956770.
<figure><img src="image_10.png" /><figcaption>Figure 8: Bad reconstruction results from the "Stump" scene, the regularization degrades the final rendering.</figcaption></figure>

## References

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR). 5470–5479.

Andreas Blattmann, Robin Rombach, Kaan Oktay, and Björn Ommer. 2022. Retrieval-Augmented Diffusion Models.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022. TensoRF: Tensorial Radiance Fields. In the European Conf. on Comput. Vis. (ECCV).

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. 2022. Depth-Supervised NeRF: Fewer Views and Faster Training for Free. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR). 12882–12891.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance Fields Without Neural Networks. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR). 5501–5510.

Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. 2022. Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation. In the Int. Conf. on 3D Vis. (3DV).

Guangcong, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. 2023. SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis. the Int. Conf. on Comput. Vis. (ICCV) (2023).

Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006a. Dimensionality Reduction by Learning an Invariant Mapping. 2 (2006), 1735–1742.

R. Hadsell, S. Chopra, and Y. LeCun. 2006b. Dimensionality Reduction by Learning an Invariant Mapping. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR), Vol. 2. 1735–1742.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Advances in Neur. Inf. Process. Syst., H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 6840–6851.

Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. 2021a. Zero-Shot Text-Guided Object Generation with Dream Fields. arXiv (2021).

Ajay Jain, Matthew Tancik, and Pieter Abbeel. 2021b. Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis. In the Int. Conf. on Comput. Vis. (ICCV). 5885–5894.

Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. 2014. Large scale multi-view stereopsis evaluation. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR). IEEE, 406–413.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. on Graph. 42, 4 (July 2023).

Justin\* Kerr, Chung Min\* Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. 2023. LERF: Language Embedded Radiance Fields. In Int. Conf. on Comput. Vis. (ICCV).

Chung Min\* Kim, Mingxuan\* Wu, Justin\* Kerr, Matthew Tancik, Ken Goldberg, and Angjoo Kanazawa. 2024. GARField: Group Anything with Radiance Fields. In arXiv.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 (2023).

Dexter C. Kozen. 1992. Union-Find. Springer New York, New York, NY, 48–51.

Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. 2023. Semantic-SAM: Segment and Recognize Anything at Any Granularity. arXiv preprint arXiv:2307.04767 (2023).

Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. 2021. Prototypical Contrastive Learning of Unsupervised Representations. In the Int. Conf. on Learn. Representations (ICLR).

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 2023b. Zero-1-to-3: Zero-shot One Image to 3D Object. arXiv:2303.11328 [cs.CV].

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023c. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023).

Yichen Liu, Benran Hu, Chi-Keung Tang, and Yu-Wing Tai. 2023a. SANeRF-HQ: Segment Anything for NeRF in High Quality. arXiv preprint arXiv:2312.01531 (2023).

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Lecture Notes in Comput. Science (including subseries Lecture Notes in Artif. Intelligence and Lecture Notes in Bioinformatics) 12346 LNCS (2020), 405–421.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. on Graph. (TOG) 41, 4, Article 102 (2022), 15 pages.

Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. 2022. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR).

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. arXiv:2401.14159 [cs.CV].

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021a. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs.CV].

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021b. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs.CV].

Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conf. on Comput. Vis. and Patt. Recog. (CVPR).

Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In the European Conf. on Comput. Vis. (ECCV).

Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. 2023. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH (SIGGRAPH '23).

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. Trans. on Image Process. (TIP) 13, 4 (4, 2004), 600–612.

Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. 2023. ReconFusion: 3D Reconstruction with Diffusion Priors. arXiv (2023).

Jamie Wynn and Daniyar Turmukhambetov. 2023. DiffusioNeRF: Regularizing Neural Radiance Fields With Denoising Diffusion Models. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR). 4180–4189.

Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. 2023. SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting.

Jiawei Yang, Marco Pavone, and Yue Wang. 2023. FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR).

Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. 2024. Gaussian Grouping: Segment and Edit Anything in 3D Scenes. In the European Conf. on Comput. Vis. (ECCV).

Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019. Differentiable Surface Splatting for Point-based Geometry Processing. ACM Trans. on Graph. (Proc. of ACM SIGGRAPH ASIA) 38, 6 (2019).

Haiyang Ying, Yixuan Yin, Jinzhi Zhang, Fan Wang, Tao Yu, Ruqi Huang, and Lu Fang. 2023. OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning. arXiv preprint arXiv:2311.11666 (2023).

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In the Conf. on Comput. Vis. and Patt. Recog. (CVPR). 586–595.

Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. 2023. FSGS: Real-Time Few-Shot View Synthesis using Gaussian Splatting. arXiv:2312.00451 [cs.CV].

Matthias Zwicker, Hanspeter Pfister, Jeroen Baar, and Markus Gross. 2001. EWA volume splatting. IEEE Visualization (01 2001).
<figure><img src="image_11.png" /><figcaption>Figure 9: Visualization of results on the DTU dataset and Mip-NeRF 360 dataset from different state-of-the-art 3D-GS methods.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 10: Visualization of results on the DTU dataset and Mip-NeRF 360 dataset compared to all depth-guided baseline methods.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 11: Visualization of the hierarchical segmentation by showing the feature map of different hierarchy levels from coarse to fine. The colors are rendered by applying PCA to the higher-dimensional features. Please note that since Garfield cannot reconstruct the scene or achieve meaningful segmentation in sparse scenarios, we used 60 input views to train Garfield. (Top: ours (LERF-Ramen), Middle: ours (MIP-NeRF-360-Counter), Bottom: Garfield (MIP-NeRF-360-Counter))</figcaption></figure>