3DGS-Det: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for 3D Object Detection

## Abstract

Abstract Neural Radiance Fields (NeRF) are widely used for novel-view synthesis and have been adapted for 3D Object Detection (3DOD), offering a promising approach to 3D object detection through view-synthesis representation. However, NeRF faces inherent limitations: (i) It has limited representational capacity for 3DOD due to its implicit nature, and (ii) it suffers from slow rendering speeds. Recently, 3D Gaussian Splatting (3DGS) has emerged as an explicit 3D representation that addresses these limitations with faster rendering capabilities. Inspired by these advantages, this paper introduces 3DGS into 3DOD for the first time, identifying two main challenges:
(i) Ambiguous spatial distribution of Gaussian blobs – 3DGS primarily relies on 2D pixel-level supervision, resulting in unclear 3D spatial distribution of Gaussian blobs and poor differentiation between objects and background, which hinders 3DOD; (ii) Excessive background blobs – 2D images often include numerous background pixels, leading to densely reconstructed 3DGS with many noisy Gaussian blobs representing the background, negatively affecting detection.
To tackle the challenge (i), we leverage the fact that 3DGS reconstruction is derived from 2D images, and propose an elegant and efficient solution by incorporating 2D Boundary Guidance to significantly enhance the spatial distribution of Gaussian blobs,
resulting in clearer differentiation between objects and their background (see Fig. 1 ).
To address the challenge (ii), we propose a Box-Focused Sampling strategy using 2D boxes to generate object probability distribution in 3D spaces, allowing effective probabilistic sampling in 3D to retain more object blobs and reduce noisy background blobs. Benefiting from the proposed Boundary Guidance and Box-Focused Sampling, our final method, 3DGS-DET , achieves significant improvements ( +5.6 on mAP@0.25, +3.7 on mAP@0.5) over our basic pipeline version,
without introducing any additional learnable parameters.
Furthermore, 3DGS-DET significantly outperforms the state-of-the-art NeRF-based method, NeRF-Det, achieving improvements of +6.6 on mAP@0.25 and +8.1 on mAP@0.5 for the ScanNet dataset, and impressive +31.5 on mAP@0.25 for the ARKITScenes dataset.
Codes and models are publicly available at: https://github.com/yangcaoai/3DGS-DET .

## 1 Introduction

3D Object Detection (3DOD) (Qi et al., ; ) is a fundamental task in computer vision, providing foundations for wide realistic application scenarios such as autonomous driving, robotics, and industrial production, as accurate localization and classification of objects in 3D space are critical for these applications.
Most existing 3DOD methods (Rukhovich et al., ; ) explored using non-view-synthesis representations, including point clouds, RGBD, and multi-view images, to perform 3D object detection. However, these approaches mainly focus on the perception perspective and lack the capability for novel view synthesis.

Neural Radiance Fields (NeRF) (Mildenhall et al., ) provide an effective manner for novel view synthesis and have been adapted for 3D Object Detection (3DOD) through view-synthesis representations (Xu et al., ; Hu et al., ).
However, as a view-synthesis representation for 3D object detection, NeRF has inherent limitations: 1) Its implicit nature restricts its representational capacity for 3DOD, and 2) it suffers from slow rendering speeds.
Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., ) has emerged as an explicit 3D representation that offers faster rendering, effectively addressing these limitations.
Inspired by these
strengths,
our work is *the first* to introduce 3DGS into 3DOD. In this exploration, we identify two primary challenges:
(i) Ambiguous spatial distribution of Gaussian blobs – 3DGS primarily relies on 2D pixel-level supervision, resulting in unclear 3D spatial distribution of Gaussian blobs and insufficient differentiation between objects and background, which hinders effective 3DOD;
(ii) Excessive background blobs – 2D images often contain numerous background pixels, leading to densely populated 3DGS with many noisy Gaussian blobs representing the background, negatively impacting the detection of foreground 3D objects.

To address the above-discussed challenges, we further empower 3DGS with two novel strategies for 3D object detection (i) *2D Boundary Guidance Strategy*: Given the fact that 3DGS reconstruction is optimized from 2D images, we introduce a novel strategy by incorporating 2D Boundary Guidance to achieve a more suitable 3D spatial distribution of Gaussian blobs for detection. Specifically, we first perform object boundary detection on posed images, then overlay the boundaries onto the images, and finally train the 3DGS model. This proposed strategy can facilitate the learning of a spatial Gaussian blob distribution that is more differentiable for the foreground objects and the background (see Fig. 1).
(ii) *Box-Focused Sampling Strategy*: This strategy further leverages 2D boxes to establish 3D object probability spaces, enabling an object probabilistic sampling of Gaussian blobs to effectively preserve object blobs and prune background blobs. Specifically, we project the 2D boxes that cover objects in images into 3D spaces to form frustums. The 3D Gaussian blobs within the frustum have a higher probability of being object blobs compared to those outside. Based on this strategy, we construct 3D object probability spaces and sample Gaussian blobs accordingly, finally preserving more object blobs and reducing noisy background blobs.

In summary, the contributions of this work are fourfold:

- •
To the best of our knowledge, we are the first to integrate 3D Gaussian Splatting (3DGS) into 3D Object Detection (3DOD), representing a novel contribution to the field. We propose 3DGS-DET, which empowers 3DGS with Boundary Guidance and Box-Focused Sampling for 3DOD.
- •
We design *Boundary Guidance* to optimize 3DGS with the guidance of object boundaries, which achieves a significantly better spatial distribution of Gaussian blobs and clearer differentiation between objects and the background, thereby effectively enhancing 3D object detection.
- •
We propose *Box-Focused Sampling*, which establishes 3D object probability spaces, enabling a higher sampling probability to be assigned to object-related 3D Gaussian blobs. This probabilistic sampling strategy preserves more object blobs and suppresses noisy background blobs, therefore producing further improved detection performance.
- •
With zero additional learnable parameters, Boundary Guidance and Box-Focused Sampling improve detection by 5.6 points on mAP@0.25 and 3.7 points on mAP@0.5 as demonstrated in our ablation study. Furthermore, our final approach, 3DGS-DET, significantly outperforms the state-of-the-art NeRF-based method, NeRF-Det, on both ScanNet (+6.6 on mAP@0.25, +8.1 on mAP@0.5) and ARKITScenes (+31.5 on mAP@0.25).

## 2 Related Works

3D Gaussian Splatting (3DGS) is an effective explicit representation that models 3D scenes or objects using Gaussian blobs – small, continuous Gaussian functions distributed across 3D space, enabling faster rendering. Recent works (Shen et al., ; Liu et al., ; Lee et al., ) have shown that 3DGS is highly suitable for dynamic scene modeling. Additionally, some studies (Lin et al., ; Zhang et al., ; Xiong et al., ; Wang & Xu, ; Liu et al., ; Feng et al., ) also demonstrate its efficiency in processing large-scale 3D scene data. A key focus of recent 3DGS research is integrating semantic understanding to enhance perception capabilities. Researchers (Zhou et al., ; Qin et al., ; Shi et al., ; Zuo et al., ) leverage advanced 2D foundational models, such as SAM (Kirillov et al., ) and CLIP (Radford et al., ), along with feature extraction methods like DINO (Zhang et al., ), to boost perception effectiveness.
Unlike previous methods that often overlook specific challenges of 3D Object Detection (3DOD), our approach uniquely introduces Boundary Guidance and Box-Focused Sampling, marking the first exploration of 3DGS as a representation for the 3D object detection task.

Non-View-Synthesis Representation-Based 3D Object Detection. Traditional 3D detection tasks primarily utilize the following representations: (i) Point cloud-based methods (Yang et al., ; Ali et al., ; Shi et al., ; Qi et al., ; ; Wang et al., ; Peng et al., ; Wang et al., ; Rukhovich et al., ; Cao et al., ; ) directly process unstructured 3D points captured by sensors like LiDAR or depth cameras. Techniques such as VoteNet (Qi et al., ) and CAGroup3D (Wang et al., ) efficiently handle point clouds, capturing detailed geometries while facing challenges in computational efficiency due to their irregular structure. Some researches (Zhou & Tuzel, ; Ye et al., ; Deng et al., ; Mao et al., ; Noh et al., ; Chen et al., ; Mahmoud et al., ) divide 3D space into uniform volumetric units, enabling 3D convolutional neural networks to process the data, although they encounter trade-offs between resolution and memory usage. (ii) Multi-view image-based methods (Wang et al., ; Xiong et al., ; Wang et al., ; Chen et al., ; Feng et al., ; Tu et al., ; Shen et al., ) leverage 2D images from multiple perspectives to reconstruct 3D structures.
(iii) RGB-D based methods (Qi et al., ; ; Luo et al., ) enhance 3D object detection by combining 2D images cues, with 3D data to improve accuracy. However, these representations predominantly focus on perception and lack the capability for novel view synthesis.

View-Synthesis Representation-Based 3D Object Detection.
Neural Radiance Fields (NeRF) (Mildenhall et al., ) have become popular for novel-view-synthesis and have been adapted for 3D Object Detection (3DOD) (Hu et al., ; Xu et al., ). These adaptations present promising solutions for detecting 3D objects using view-synthesis representations. For instance, NeRF-RPN (Hu et al., ) employs voxel representations, integrating multi-scale 3D neural volumetric features to perform category-agnostic box localization rather than category-specific object detection. NeRF-Det (Xu et al., ) incorporates multi-view geometric constraints from the NeRF component into 3D detection. Notably, NeRF-RPN focuses on class-agnostic box detection, while NeRF-Det targets class-specific object detection. Our work follows the class-specific setting of NeRF-Det. However, NeRF faces significant challenges: its implicit nature limits its representational capacity for 3D object detection, and it suffers from slow rendering speeds.
3D Gaussian Splatting (3DGS) (Kerbl et al., ) has emerged as an explicit 3D representation, offering faster rendering and effectively addressing these limitations. Motivated by these advantages, our work introduces 3DGS into 3DOD for the first time, and presents novel designs to adapt 3DGS for detection, making significant differences from NeRF-based methods (Hu et al., ; Xu et al., ).

Figure: Figure 2: Pipeline overview (zooming in for a clearer view). The top row illustrates our basic pipeline detailed in Sec. 3.2. The bottom row shows our 3DGS-DET pipeline with both Boundary Guidance (Sec. 3.3) and Box-Focused Sampling (Sec. 3.4) embedded. The Boundary Guidance can significantly improve the 3D spatial distribution of Gaussian blobs, and thus produce clearer differentiation between objects and the background. The Box-Focused Sampling effectively preserves more object-related blobs while suppressing noisy background blobs, compared to random sampling. These two proposed strategies together largely advance the 3D detection performance.

## 3 Methodology

The pipeline of our 3DGS-DET is illustrated in the bottom row of Fig. 2. Initially, we train the 3D Gaussian Splatting (3DGS) on the input scenes using the proposed Boundary Guidance, which significantly enhances the spatial distribution of Gaussian blobs, resulting in clearer differentiation between objects and the background. Subsequently, we apply the proposed Box-Focused Sampling, which effectively preserves object-related blobs while suppressing noisy background blobs. The sampled 3DGS is then fed into the detection framework for training.
In this section, we detail our method step by step. First, we introduce the preliminary concept of 3D Gaussian Splatting (3DGS) in Sec. 3.1. As the first to introduce 3DGS in 3D object detection, we establish the basic pipeline in Sec. 3.2, utilizing 3DGS for input and output detection predictions. We then present Boundary Guidance in Sec. 3.3. Finally, we describe the Box-Focused Sampling Strategy in Sec. 3.4.

### 3.1 Preliminary: 3D Gaussian Splatting

In our proposed method, 3DGS-DET, the input scene is represented using 3D Gaussian Splatting (3DGS) (Kerbl et al., ), formulated as follows:

$$ $G=\left\{(\bm{\mu}_{i},\bm{S}_{i},\bm{R}_{i},\bm{c}_{i},\bm{\alpha}_{i})\right \}_{i=1}^{N},$ (1) $$

where $N$ denotes the number of Gaussian blobs. Each blob is characterized by its 3D coordinate $\bm{\mu}_{i}$, scaling matrix $\bm{S}_{i}$, rotation matrix $\bm{R}_{i}$, color features $\bm{c}_{i}$, and opacity $\bm{\alpha}_{i}$. These attributes define the Gaussian through a covariance matrix $\Sigma=\bm{R}\bm{S}\bm{S}^{T}\bm{R}^{T}$, centered at $\bm{\mu}$:

$$ $G(\bm{x})=\exp^{\left(-\frac{1}{2}(\bm{x}-\bm{\mu})^{T}\Sigma^{-1}(\bm{x}-\bm{ \mu})\right)}.$ (2) $$

During rendering, opacity modulates the Gaussian. By projecting the covariance onto a 2D plane (Zwicker et al., ), we derive the projected Gaussian, and utilize volume rendering (Max, ) to compute the image pixel colors:

$$ $C=\sum_{k=1}^{K}\alpha_{k}c_{k}\prod_{j=1}^{k-1}(1-\alpha_{j}),$ (3) $$

where $K$ is the number of sampling points along the ray. $\alpha_{i}$ is determined by evaluating a 2D Gaussian with covariance $\Sigma$, multiplied by the learned opacity (Yifan et al., ). The initial 3D coordinates of each Gaussian are based on Structure from Motion (SfM) points (Schonberger & Frahm, ). Gaussian attributes are refined to minimize the image reconstruction loss:

$$ $L_{\text{render}}=(1-\lambda)L_{1}(I,\hat{I})+\lambda L_{\text{D-SSIM}}(I,\hat {I}),$ (4) $$

where $\hat{I}$ represents the ground truth images. Additional details can be found in Kerbl et al. ().

Figure: Figure 3: Illustration of the proposed Boundary Guidance and Box-Focused Sampling strategies. In the top row, Boundary Guidance is constructed by three steps, *i.e.,* detecting boundaries on posed images, overlaying them to images, and training a 3DGS model to achieve a more distinct spatial distribution of Gaussian blobs for objects and the background. In the bottom row, Box-Focused Sampling is achieved by conducting object detection on posed images. The predicted 2D boxes are projected into the 3D domain to establish object probability spaces, allowing probabilistic sampling of Gaussians to preserve more object blobs and suppress noisy background blobs.

### 3.2 Proposed Basic Pipeline of 3DGS for 3D Object Detection

In this section, we build our basic pipeline by directly utilizing the original 3D Gaussian Splatting (3DGS) for 3D Object Detection (3DOD) without any further improvement. As depicted in the top row of Fig. 2, we train the 3DGS representation of the input scene using posed images, denoted as $G=\left\{(\bm{\mu}_{i},\bm{S}_{i},\bm{R}_{i},\bm{c}_{i},\bm{\alpha}_{i})\right
\}_{i=1}^{N}$.
Given that the number of Gaussian blobs $N$ is too large for them to be input into the detector, we perform random sampling to select a subset of Gaussian blobs, denoted as $\hat{G}=\left\{(\bm{\mu}_{i},\bm{S}_{i},\bm{R}_{i},\bm{c}_{i},\bm{\alpha}_{i})
\right\}_{i=1}^{M}$, where $M<N$.
We then concatenate the attributes of the Gaussian blobs along the channel dimension as follows:

$$ $\hat{G}_{\text{input}}=\text{Concat}(\bm{\mu}_{i},\bm{S}_{i},\bm{R}_{i},\bm{c} _{i},\bm{\alpha}_{i})\quad\forall i\in\{1,\ldots,M\}.$ (5) $$

This concatenated representation $\hat{G}_{\text{input}}$ is then fed into the subsequent detection tool. Note that since 3DGS is an explicit 3D representation, $\hat{G}_{\text{input}}$ can be utilized with any point-cloud-based detector by retraining the detector model on 3DGS representation. In our study, the research focus is on enhancing 3DGS for 3DOD in general, rather than designing a specific detector. Therefore, we utilize the existing work (Rukhovich et al., ) as the detection tool.
The final detection predictions are obtained as follows:

$$ $P=\mathrm{F}(\hat{G}_{\text{input}})=\left(\bm{p},\bm{z},\bm{b}\right),$ (6) $$

where $\mathrm{F}$ denotes the detector tool and $P$ represents the predictions, including classification probabilities $\bm{p}$, centerness $\bm{z}$, and bounding box regression parameters $\bm{b}$.

The training loss (Rukhovich et al., ) is defined as:

$$ $\begin{split}L_{\text{det}}=\frac{1}{N_{\text{pos}}}\sum_{\hat{x},\hat{y},\hat {z}}\left(\mathds{1}_{\{p(\hat{x},\hat{y},\hat{z})\neq 0\}}L_{\text{reg}}(\hat {\bm{b}},\bm{b})+\mathds{1}_{\{p(\hat{x},\hat{y},\hat{z})\neq 0\}}L_{\text{ cntr}}(\hat{\bm{z}},\bm{z})+L_{\text{cls}}(\hat{\bm{p}},\bm{p})\right),\end{split}$ (7) $$

where the number of matched positions $N_{\text{pos}}$ is given by $\sum_{\hat{x},\hat{y},\hat{z}}\mathds{1}_{\{p(\hat{x},\hat{y},\hat{z})\neq 0\}}$. Ground truth labels are indicated with a hat symbol. The regression loss $L_{\text{reg}}$ is based on Intersection over Union (IoU), the centerness loss $L_{\text{cntr}}$ uses binary cross-entropy, and the classification loss $L_{\text{cls}}$ employs focal loss. Further details on the detection tool can be found in Rukhovich et al. ().

Building upon this basic pipeline, we develop our method, 3DGS-DET, by introducing two novel designs to improve the 3DGS representation, as illustrated in the bottom row of Fig. 2. These designs are detailed in the following sections  Sec. 3.3 and Sec. 3.4.

### 3.3 Boundary Guidance

Given the fact that 3DGS reconstruction
is derived from 2D images, we design the novel Boundary Guidance strategy by incorporating 2D Boundary Guidance
to achieve a more suitable 3D spatial distribution of Gaussian blobs for detection. In this section, we present our Boundary Guidance strategy in detail.
As illustrated in the top row of Fig. 3,
to provide the guidance priors for 3DGS reconstruction, we first generate category-specific boundaries for posed images:

$$ $B_{\text{bd}}=\mathrm{H}_{\text{bd}}(I)=\{b_{\text{bd}}^{c}\}\quad c\in C,$ (8) $$

where $\mathrm{H}_{\text{bd}}$ is the boundary generator, and $b_{\text{bd}}^{c}$ represents the binary boundary map for category $c$. If $b_{\text{bd}}^{c}(x,y)=1$, the pixel at $(x,y)$ belongs to the boundary for objects of category $c$. The set $C$ includes all categories.
In practice, the operations of $\mathrm{H}_{\text{bd}}$ are as follows: we use Grounded SAM (Ren et al., ) to generate category-specific masks. Then, the Suzuki-Abe algorithm (Suzuki et al., ) is employed to extract the boundaries of these masks, along with category information. The category-specific boundaries are then overlaid on the posed images in different colors:

$$ $I_{\text{bd}}(x,y)=I(x,y)\cdot\left(1-\sum_{c\in C}b_{\text{bd}}^{c}(x,y) \right)+\sum_{c\in C}b_{\text{bd}}^{c}(x,y)\cdot\text{color}(c),$ (9) $$

where $I_{\text{bd}}(x,y)$ is the pixel at position $(x,y)$ of the final image with overlaid boundaries. $I(x,y)$ is from the original image, $b_{\text{bd}}^{c}(x,y)$ is the boundary map for category $c$, and $\text{color}(c)$ is the color associated with category $c$.
These $I_{\text{bd}}$ images are used as ground truth to train the 3DGS representation $G_{\text{bd}}$
by the following loss:

$$ $L_{\text{render}}=(1-\lambda)L_{1}(I,I_{\text{bd}})+\lambda L_{\text{D-SSIM}}( I,I_{\text{bd}}).$ (10) $$

To effectively reduce $L_{\text{render}}$ during training, it is crucial to ensure the rendering quality of boundaries and the multi-view stability of boundaries. In this way,
the Boundary Guidance lead 3DGS to incorporate boundary prior information into the 3D space.
As shown in Fig. 2 (better viewed when zoomed in), 3DGS trained with Boundary Guidance demonstrates improved spatial distribution of Gaussian blobs compared to those trained without it,
without introducing additional learnable parameters.

### 3.4 Box-Focused Sampling

Considering that 2D images often
include numerous background pixels, leading to densely reconstructed 3DGS with
many noisy Gaussian blobs representing the background, negatively affecting detection.
To reduce the excessive background blobs,
in this section, we propose the Box-Focused Sampling strategy in detail. As depicted in the bottom row of Fig. 3,
to provide priors for the following sampling,
we utilize a 2D object detector to identify object bounding boxes:

$$ $B_{\text{bb}}=\mathrm{H_{\text{bb}}}(I)=\{(b_{\text{bb}},p^{C})\},$ (11) $$

where $\mathrm{H}_{\text{bb}}$ is the box detector, and we select Grounding DINO (Liu et al., ) as the detector in our experiments. Here, $b_{\text{bb}}$ denotes the bounding box positions, and $p^{C}$ is the probability vector for the box belonging to each category in $C$. We define $p_{\max}=\max_{c\in C}p^{c}$ as the highest category probability for a given bounding box, which helps to establish object probability spaces in later step. Then,
we project the 2D boxes into 3D space:

$$ $F_{\text{ft}}=\{K^{-1}\begin{bmatrix}x_{i}\\ y_{i}\\ z\end{bmatrix}\mid(x_{i},y_{i})\in b_{\text{bb}},z\in\{z_{\min},z_{\max}\}\},$ (12) $$

where $F_{\text{ft}}$ is the projected 3D frustum from $b_{\text{bb}}$, and $K^{-1}$ is the inverse camera matrix used to map 2D bounding box corners $(x_{i},y_{i})$ and depth values $z_{\min}$ and $z_{\max}$ into 3D space.
Next, we establish object probability spaces using $F_{\text{ft}}$ and $p_{\max}$. Specifically, for each bounding box, the maximum probability $p_{\max}$ models the likelihood of each Gaussian blob within the corresponding frustum being an object blob:

$$ $p_{\text{obj}}(g_{i}\mid g_{i}\in F_{\text{ft}})=p_{\max},$ (13) $$

where $p_{\text{obj}}(g_{i}\mid g_{i}\in F_{\text{ft}})$ indicates the probability of each Gaussian blob $g_{i}$ within frustum $F_{\text{ft}}$ being an object blob. To integrate priors from different view frustums, we select the maximum probability as the aggregated probability:

$$ $p_{\text{agr}}(g_{i})=\max_{v\in V}p_{\text{obj}}(g_{i}\mid g_{i}\in F_{\text{ ft}}^{v}),$ (14) $$

where $p_{\text{agr}}(g_{i})$ is the aggregated probability for Gaussian blob $g_{i}$, and $V$ represents the set of all views. Gaussian blobs not belonging to any frustum are assigned a small probability $p_{\text{bg}}$, set to 0.01 in practice.
In this way,
we obtain the object probability spaces $P_{\text{obj}}$, where each Gaussian blob has an associated probability of being an object. We then perform independent probabilistic sampling based on $P_{\text{obj}}$ to achieve Box-Focused Sampling, resulting in the sampled Gaussian set $\hat{G}_{\text{bd}}^{\text{bs}}$ as:

$$ $\hat{G}_{\text{bd}}^{\text{bs}}=\{g\mid g\sim P_{\text{obj}}(g)\}.$ (15) $$

In this way, it allows object blobs to be better preserved due to their higher probabilities, while most background points, having lower probabilities, are effectively reduced.
Then, based on $\hat{G}_{\text{bd}}^{\text{bs}}$, we proceed with the training of the detector, as formulated by Equ. 5-Equ. 7 as described in Sec. 3.2.
As shown in Fig. 2, 3DGS sampled via Box-Focused Sampling retains more object blobs and reduces background noise.

## 4 Experiments

**Table 1: Comparison of mAP@0.25 across different methods on ScanNet. The first block includes methods using non-view-synthesis representations, such as point cloud, RGB-D, and multi-view images. The second block includes methods utilizing view-synthesis representations (NeRF-based and our 3DGS-based method). Our 3DGS-DET significantly outperforms the NeRF-based method NeRF-Det by 6.6 points. For other representations, 3DGS-DET surpasses all methods except for the point-cloud-based methods, FCAF3D and CAGroup3D, which have inherent advantages by directly using sensor-captured 3D data, specifically point clouds, as input.**
| Methods | cab | bed | chair | sofa | tabl | door | wind | bkshf | pic | cntr |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seg-Cluster (Wang et al., ) | 11.8 | 13.5 | 18.9 | 14.6 | 13.8 | 11.1 | 11.5 | 11.7 | 0.0 | 13.7 |
| Mask R-CNN (He et al., ) | 15.7 | 15.4 | 16.4 | 16.2 | 14.9 | 12.5 | 11.6 | 11.8 | 19.5 | 13.7 |
| SGPN (Wang et al., ) | 20.7 | 31.5 | 31.6 | 40.6 | 31.9 | 16.6 | 15.3 | 13.6 | 0.0 | 17.4 |
| 3D-SIS (Hou et al., ) | 12.8 | 63.1 | 66.0 | 46.3 | 26.9 | 8.0 | 2.8 | 2.3 | 0.0 | 6.9 |
| 3D-SIS (w/ RGB) (Hou et al., ) | 19.8 | 69.7 | 66.2 | 71.8 | 36.1 | 30.6 | 10.9 | 27.3 | 0.0 | 10.0 |
| VoteNet (Qi et al., ) | 36.3 | 87.9 | 88.7 | 89.6 | 58.8 | 47.3 | 38.1 | 44.6 | 7.8 | 56.1 |
| FCAF3D (Rukhovich et al., ) | 57.2 | 87.0 | 95.0 | 92.3 | 70.3 | 61.1 | 60.2 | 64.5 | 29.9 | 64.3 |
| CAGroup3D (Wang et al., ) | 60.4 | 93.0 | 95.3 | 92.3 | 69.9 | 67.9 | 63.6 | 67.3 | 40.7 | 77.0 |
| ImGeoNet (Tu et al., ) | 40.6 | 84.1 | 74.8 | 75.6 | 59.9 | 40.4 | 24.7 | 60.1 | 4.2 | 41.2 |
| CN-RMA (Shen et al., ) | 42.3 | 80.0 | 79.4 | 83.1 | 55.2 | 44.0 | 30.6 | 53.6 | 8.8 | 65.0 |
| ImVoxelNet (Rukhovich et al., ) | 30.9 | 84.0 | 77.5 | 73.3 | 56.7 | 35.1 | 18.6 | 47.5 | 0.0 | 44.4 |
| NeRF-Det (Xu et al., ) | 37.6 | 84.9 | 76.2 | 76.7 | 57.5 | 36.4 | 17.8 | 47.0 | 2.5 | 49.2 |
| 3DGS-DET (Our basic pipeline) | 39.6 | 82.5 | 75.8 | 78.0 | 53.6 | 36.1 | 26.9 | 41.8 | 11.9 | 56.0 |
| 3DGS-DET (Our basic pipeline+BG) | 38.9 | 83.5 | 81.7 | 82.6 | 54.4 | 36.2 | 26.0 | 39.6 | 13.5 | 52.8 |
| 3DGS-DET (Our basic pipeline+BG+BS) | 44.1 | 82.7 | 81.7 | 79.6 | 56.0 | 35.4 | 27.6 | 45.2 | 17.3 | 61.9 |
| Methods | desk | curt | fridg | showr | toil | sink | bath | ofurn | mAP@0.25 |  |
| Seg-Cluster (Wang et al., ) | 12.2 | 12.4 | 11.2 | 18.0 | 19.5 | 18.9 | 16.4 | 12.2 | 13.4 |  |
| Mask R-CNN (He et al., ) | 14.4 | 14.7 | 21.6 | 18.5 | 25.0 | 24.5 | 24.5 | 16.9 | 17.1 |  |
| SGPN (Wang et al., ) | 14.1 | 22.2 | 0.0 | 0.0 | 72.9 | 52.4 | 0.0 | 18.6 | 22.2 |  |
| 3D-SIS (Hou et al., ) | 33.3 | 2.5 | 10.4 | 12.2 | 74.5 | 22.9 | 58.7 | 7.1 | 25.4 |  |
| 3D-SIS (w/ RGB) (Hou et al., ) | 46.9 | 14.1 | 53.8 | 36.0 | 87.6 | 43.0 | 84.3 | 16.2 | 40.2 |  |
| VoteNet (Qi et al., ) | 71.7 | 47.2 | 45.4 | 57.1 | 94.9 | 54.7 | 92.1 | 37.2 | 58.7 |  |
| FCAF3D (Rukhovich et al., ) | 71.5 | 60.1 | 52.4 | 83.9 | 99.9 | 84.7 | 86.6 | 65.4 | 71.5 |  |
| CAGroup3D (Wang et al., ) | 83.9 | 69.4 | 65.7 | 73.0 | 100.0 | 79.7 | 87.0 | 66.1 | 75.12 |  |
| ImGeoNet (Tu et al., ) | 70.9 | 33.7 | 54.4 | 47.5 | 95.2 | 57.5 | 81.5 | 36.1 | 54.6 |  |
| CN-RMA (Shen et al., ) | 70.0 | 44.9 | 44.0 | 55.2 | 95.4 | 68.1 | 86.1 | 49.7 | 58.6 |  |
| ImVoxelNet (Rukhovich et al., ) | 65.5 | 19.6 | 58.2 | 32.8 | 92.3 | 40.1 | 77.6 | 28.0 | 49.0 |  |
| NeRF-Det (Xu et al., ) | 52.0 | 29.2 | 68.2 | 49.3 | 97.1 | 57.6 | 83.6 | 35.9 | 53.3 |  |
| 3DGS-DET (Our basic pipeline) | 69.8 | 36.7 | 38.3 | 55.3 | 93.5 | 64.0 | 80.8 | 37.5 | 54.3 |  |
| 3DGS-DET (Our basic pipeline+BG) | 68.6 | 45.2 | 52.7 | 45.0 | 98.3 | 69.6 | 84.3 | 48.0 | 56.7 |  |
| 3DGS-DET (Our basic pipeline+BG+BS) | 72.8 | 40.7 | 56.6 | 71.9 | 98.5 | 72.2 | 88.3 | 46.7 | 59.9 (+6.6) |  |

### 4.1 Experimental setup

Dataset: To thoroughly evaluate the performance of our proposed method in 3D detection tasks, we selected two representative datasets: ScanNet (Dai et al., ) and ARKitScene (Baruch et al., ). ScanNet is a large-scale indoor scene dataset containing over 1,500 real-world 3D scanned scenes, encompassing various complex indoor environments such as residential spaces, offices, and classrooms. The ARKitScene dataset is constructed from RGB-D image sequences, offering detailed geometric information and precise object annotations. For each scene, a maximum of 600 posed images are extracted. The category settings follow the standard 18 categories for ScanNet and 17 categories for ARKitScene.

Metrics: We use mAP@0.25 and mAP@0.5 as the primary evaluation metrics. Mean Average Precision (mAP) is calculated at different IoU thresholds, providing a comprehensive measure of the detection model’s performance across various categories.

Implementation Details:
For training 3DGS, we follow Kerbl et al. () to initialize the 3D coordinates of Gaussian blobs using Structure from Motion (SfM) points. The training hyperparameters are the same as those in Kerbl et al. (). We employ
pretrained GroundedSAM (Ren et al., ) and the Suzuki-Abe algorithm (Suzuki et al., ) as the boundary detector in Boundary Guidance. The pretrained GroundingDINO (Liu et al., ) is used as the box detector in the Box-Focused Sampling strategy. For the detection tool, we utilize the FCAF3D (Rukhovich et al., ) architecture implemented in MMDetection3D (Contributors, ). The training hyperparameters are the same as those in FCAF3D. In our ablation study, to ensure a fair comparison, all model versions are trained with the same hyperparameters, such as the same number of epochs, specifically 12 epochs. All the ablation experiments (Sec. 4.3) are conducted on ScanNet.

### 4.2 Main Results

Quantitative Results.
For the ScanNet dataset, we present the mAP@0.25 and mAP@0.5 performances of various methods in Tab. 1 of the main paper and Tab. 6 of the Appendix, respectively. Note that some methods did not report mAP@0.5 in previous studies, resulting in blank entries for these methods in Tab. 6 of the Appendix.

In both Tab. 1 and Tab. 6 of the Appendix, the methods listed in the first block (Wang et al., ; He et al., ; Hou et al., ; Qi et al., ; Rukhovich et al., ; Wang et al., ; Tu et al., ; Shen et al., ; Rukhovich et al., ) are non-view-synthesis representation-based 3D detection methods. These methods utilize point clouds, RGB-D data, or multi-view images for 3D object detection. The second block consists of view-synthesis representation-based 3DOD methods, including NeRF-Det (Hu et al., ) and our proposed 3DGS-DET. NeRF-Det is the closest work to ours, leveraging Neural Radiance Fields (NeRF).
Our approach variants are detailed as follows: ‘3DGS-DET (Our basic pipeline)’ represents the basic pipeline method established in Sec. 3.2. ‘3DGS-DET (Our basic pipeline+BG)’ incorporates the proposed Boundary Guidance as detailed in Sec. 3.3. ‘3DGS-DET (Our basic pipeline+BG+BS)’ is our full method, utilizing both Boundary Guidance and Box-Focused Sampling as described in Sec. 3.4.
As illustrated in Tab. 1 and Tab. 6 of the Appendix, all versions of our methods significantly outperform NeRF-Det. Notably, our full method (’Our basic pipeline+BG+BS’) surpasses the state-of-the-art NeRF-based method, NeRF-Det, by +6.6 on mAP@0.25 and +8.1 on mAP@0.5, showcasing the superiority of our approach.

**Table 2: Comparison of the ‘whole-scene’ performance on the ARKITScenes validation set. Our 3DGS-DET significantly outperforms NeRF-Det by 31.5 points. Note that we follow the setup described in the NeRF-Det (Xu et al., ) supplementary materials: ‘In our experiments, we utilize the subset of the dataset with low-resolution images’, considering it is the closest work to ours. Other methods that do not use the same setting are not listed in this table.**
| Methods | cab | fridg | shlf | stove | bed | sink | wshr | tolt | bthtb |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ImVoxelNet (Rukhovich et al., ) | 32.2 | 34.3 | 4.2 | 0.0 | 64.7 | 20.5 | 15.8 | 68.9 | 80.4 |
| NeRF-Det (Xu et al., ) | 36.1 | 40.7 | 4.9 | 0.0 | 69.3 | 24.4 | 17.3 | 75.1 | 84.6 |
| 3DGS-DET (Ours) | 45.2 | 84.4 | 33.3 | 41.4 | 87.3 | 75.5 | 67.6 | 87.2 | 90.8 |
| Methods | oven | dshwshr | frplce | stool | chr | tble | TV | sofa | mAP@.25 |
| ImVoxelNet (Rukhovich et al., ) | 9.9 | 4.1 | 10.2 | 0.4 | 5.2 | 11.6 | 3.1 | 35.6 | 23.6 |
| NeRF-Det (Xu et al., ) | 14.0 | 7.4 | 10.9 | 0.2 | 4.0 | 14.2 | 5.3 | 44.0 | 26.7 |
| 3DGS-DET (Ours) | 74.3 | 6.0 | 56.4 | 26.3 | 70.3 | 60.6 | 0.7 | 81.8 | 58.2  (+31.5) |

Regarding the ARKitScene dataset, considering NeRF-Det is the closest work to ours, we follow the same setup described in the NeRF-Det (Xu et al., ) supplementary materials: ‘In our experiments, we utilize the subset of the dataset with low-resolution images.’ Similarly, we adopt the same subset of the ARKitScenes dataset. Other methods that report performance on ARKitScene use the full dataset, so our 3DGS-DET is only compared with ImVoxelNet and NeRF-Det under the same conditions as described in NeRF-Det. The results in Tab. 2 demonstrate that 3DGS-DET performs better across most categories, achieving an mAP@0.25 of 58.2, which significantly outperforms NeRF-Det by +31.5, highlighting the superiority of our method.

Qualitative results.
We provide a qualitative comparison with NeRF-Det in Fig. 4. As shown, our methods detect more objects in the scene with greater positional accuracy compared to NeRF-Det (Xu et al., ), demonstrating the superiority of our approach. More qualitative comparisons can be found in Fig. 6 and Fig. 7 in the Appendix.

Figure: Figure 4: Qualitative comparison. Our methods identify more 3D objects in the scene with better positional precision, highlighting the advantages of our approach over NeRF-Det (Xu et al., ). In this figure, the scene is represented using mesh to clearly show the boxes.

### 4.3 Ablation Study

#### 4.3.1 Analysis on the Effect of Proposed Designs

In this section, we demonstrate the effectiveness of our contributions by first presenting the performance of our proposed basic 3DGS detection pipeline and then incrementally incorporating our additional designs to analyze the resulting performance improvements.

Our Proposed Basic 3DGS Detection Pipeline.
As shown in Tab. 1, ’3DGS-DET (Our basic pipeline)’ represents our proposed detection pipeline utilizing 3DGS, as described in Section 3.2. Benefiting from the advantages of 3DGS as an explicit scene representation, our basic pipeline surpasses NeRF-Det by 1 point (54.3 vs. 53.3), underscoring the significance of introducing 3DGS into 3DOD for the first time.

**Table 3: Ablation study on guidance from different priors.**
| Different Priors | mAP@0.25 | mAP@0.5 |
| --- | --- | --- |
| 2D Center Point | 54.4 | 33.9 |
| 2D Mask | 54.9 | 34.2 |
| 2D Boundary (ours) | 56.7 | 36.9 |

Boundary Guidance. ‘3DGS-DET (Our basic pipeline+BG)’ incorporates the proposed Boundary Guidance as detailed in Sec. 3.3. Introducing Boundary Guidance into the basic pipeline results in a significant improvement of 2.4 points (56.7 vs. 54.3), demonstrating the effectiveness of the proposed Boundary Guidance.
To further explore the impact of Boundary Guidance on 3DGS representations,
we present a visual comparison of the spatial distribution of trained Gaussian blobs in Fig. 8 in the Appendix.
As we can see, Gaussian blobs trained with Boundary Guidance demonstrate clearer spatial distribution and more distinct differentiation between objects and the background.
We also present rendered images from different views by 3DGS trained with Boundary Guidance in Fig. 9 and Fig. 10 in the Appendix.
As can be observed, the category-specific boundaries are clearly rendered and show multi-view stability, indicating that the 3D representation has effectively embedded the priors from Boundary Guidance. All these results clearly verify the effectiveness of the proposed Boundary Guidance for 3D detection with 3DGS.

Box-Focused Sampling. Furthermore, we introduce Box-Focused Sampling detailed in Sec. 3.4, represented by ‘3DGS-DET (Our basic pipeline+BG+BS)’ in Tab. 1. This addition leads to a further performance boost of 3.2 points (59.9 vs. 56.7), proving the effectiveness of Box-Focused Sampling.
The visual comparison of sampled Gaussian blobs is shown in Fig. 11 in the Appendix.
We can observe that the proposed Box-Focused Sampling significantly
retains more object blobs and suppresses noisy background blobs.

Figure: Figure 5: Analysis of guidance from different priors: (a) Center Point Guidance, (b) Mask Guidance, and (c) Boundary Guidance. In (a) and (b), the spatial distribution of Gaussian blobs for objects like the chair, trash bin and sink is incomplete and ambiguous. Gaussian blobs trained with Boundary Guidance exhibit a clearer spatial distribution. The reason behind this phenomenon is that the center point provides only positional guidance, lacking richer information like shape or size. The mask highlights shape and size but hides the object’s surface, reducing texture and geometric information. Boundary Guidance offers positional cues and richer information, such as shape and size, while preserving texture and geometric details on the object’s surface, leading to the best performance.

#### 4.3.2 Ablation Study on Guidance from Different Priors

In this section, we analyze the impact of guidance from various priors. As described in Sec. 3.3, we utilize the object’s boundary as the guidance prior. Here, we perform an ablation study considering the object’s center point and mask as alternative priors. To obtain the center point, we detect the object’s bounding box using GroundingDINO (Liu et al., ) and compute its center coordinates. The mask is generated with GroundedSAM (Ren et al., ). Note that all priors are category-specific, with each class associated with a fixed color. These priors are overlaid on the posed images, as shown in Fig. 5, and then used to train the 3DGS for detection. Tab. 4.3.1 presents the detection performance for 3DGS trained with the different priors.
As reported in Tab. 4.3.1, the 3DGS-DET method using boundary guidance achieved 56.7% in mAP@0.25 and 36.9% in mAP@0.5, demonstrating significant superiority over the center point and mask priors.

Let’s explore the visualizations for further insights. In (a) and (c) of Fig. 5, we observe that the spatial distribution of Gaussian blobs with Point Guidance is less distinct compared to Boundary Guidance. This is because the center point provides only positional guidance, lacking richer information like shape or size, making it less effective compared to the boundary prior.
For the mask prior, as shown in (b) and (c) of Fig. 5, the Gaussian blobs’ spatial distribution with Mask Guidance is more ambiguous than with the Boundary Guidance. Although the mask highlights shape and size information, it hides the object’s surface, reducing texture and geometric information, thus being less effective than the boundary prior.
Overall, Boundary Guidance offers positional cues and richer information such as shape and size while preserving texture and geometric details on the object’s surface, leading to the best performance.

#### 4.3.3 Analysis on Different Sampling Methods

In this section, we compare two additional sampling methods with our Box-Focused Sampling: 1) Random Sampling and 2) Farthest Point Sampling (Qi et al., ). The latter iteratively selects points farthest from those already chosen, ensuring even distribution for better scene coverage, focusing on global distribution rather than specific geometric features of objects.
The results in Tab. 4.3.1 demonstrate that our Box-Focused Sampling achieves the highest performance, with mAP@0.25 and mAP@0.5 reaching 59.9% and 37.8%, respectively. This is because 3DGS often contain excessive background blobs. Our Box-Focused Sampling is specifically designed to preserve more object-related blobs while suppressing noisy background blobs. In contrast, other sampling methods primarily focus on global scenes without differentiation between objects and background blobs.

## 5 Conclusion

In this work, we introduce 3D Gaussian Splatting (3DGS) into 3D Object Detection (3DOD) for the first time. We propose 3DGS-DET, a novel approach that leverages Boundary Guidance and Box-Focused Sampling to enhance 3DGS for 3DOD. Our method effectively addresses the inherent challenges of 3DGS in 3D object detection by improving spatial distribution and reducing background noise. By incorporating 2D Boundary Guidance, we achieve clearer differentiation between objects and background, while Box-Focused Sampling retains more object points and minimizes background noise. Our method demonstrates significant improvements, with gains of +5.6 on mAP@0.25 and +3.7 on mAP@0.5 over the basic pipeline. It also outperforms state-of-the-art NeRF-based methods, achieving +6.6 on mAP@0.25 and +8.1 on mAP@0.5 on the ScanNet dataset, and an impressive +31.5 on mAP@0.25 on the ARKITScenes dataset. These results underscore the effectiveness and superiority of our designs.

## Appendix A Appendix

### A.1 Performance on NeRF-RPN Setting

**Table 5: Performance on the NeRF-RPN setting, which targets class-agnostic box detection. Our method significantly outperforms NeRF-RPN in this setting.**
| Methods | mAP@0.25 | mAP@0.5 |
| --- | --- | --- |
| NeRF-RPN (Hu et al., ) | 55.5 | 18.4 |
| 3DGS-DET (ours) | 75.6  (+20.1) | 52.3  (+33.9) |

In this section, we adapt our 3DGS-DET to the NeRF-RPN Setting (Hu et al., ), which targets class-agnostic box detection. To achieve this, we labeled all the ground-truth boxes with a single ‘object’ category and trained 3DGS-DET accordingly.
Additionally, NeRF-RPN uses a different train/validation split compared to the official ScanNet dataset, with its validation set overlapping the official ScanNet training set. To address this, we excluded the overlapping parts between the NeRF-RPN test set and the ScanNet official training set from our training data. We then used the remaining scenes for training, and tested on the same validation set provided by NeRF-RPN.
As shown in Tab. 5, 3DGS-DET achieved an mAP@0.25 of 75.6% and an mAP@0.5 of 52.3%, significantly outperforming NeRF-RPN (Hu et al., )’s 55.5% and 18.4%. This demonstrates the significant superiority of our method in the class-agnostic setting.

### A.2 Future Work

As the first work to introduce 3DGS into 3DOD, our paper mainly focuses on the primary stage of this pipeline: empowering 3DGS for 3DOD.
Diverse experiments demonstrate that our designs can lead to significant improvements. Beyond empowering the 3DGS representation, a subsequent detector specifically designed for 3DGS could hold promise in the future. Besides, exploring joint training of 3DGS and the detector is also an interesting direction. We hope our exploration knowledge, open-source codes and data will inspire further research.

**Table 6: Comparison of mAP@0.5 across different methods on ScanNet. The first block presents methods that employ non-view-synthesis representations, including point clouds, RGB-D, and multi-view images. The second block lists methods using view-synthesis representations, such as NeRF-based and our 3DGS-based techniques. Our 3DGS-DET significantly surpasses the NeRF-based NeRF-Det by 8.1 points. Among other representations, 3DGS-DET outperforms all except the point-cloud-based methods FCAF3D and CAGroup3D, which benefit from directly using sensor-captured 3D data, specifically point clouds, as input. Note that some methods did not report mAP@0.5 in previous works, resulting in blank entries for these methods.**
| Methods | cab | bed | chair | sofa | tabl | door | wind | bkshf | pic | cntr |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seg-Cluster (Wang et al., ) | - | - | - | - | - | - | - | - | - | - |
| Mask R-CNN (He et al., ) | - | - | - | - | - | - | - | - | - | - |
| SGPN (Wang et al., ) | - | - | - | - | - | - | - | - | - | - |
| 3D-SIS (Hou et al., ) | 5.1 | 42.2 | 50.1 | 31.8 | 15.1 | 1.4 | 0.0 | 1.4 | 0.0 | 0.0 |
| 3D-SIS (w/ RGB) (Hou et al., ) | 5.7 | 50.3 | 52.6 | 55.4 | 22.0 | 10.9 | 0.0 | 13.2 | 0.0 | 0.0 |
| VoteNet (Qi et al., ) | 8.1 | 76.1 | 67.2 | 68.8 | 42.4 | 15.3 | 6.4 | 28.0 | 1.3 | 9.5 |
| FCAF3D (Rukhovich et al., ) | 35.8 | 81.5 | 89.8 | 85.0 | 62.0 | 44.1 | 30.7 | 58.4 | 17.9 | 31.3 |
| CAGroup3D (Wang et al., ) | 41.4 | 82.8 | 90.8 | 85.6 | 64.9 | 54.3 | 37.3 | 64.1 | 31.4 | 41.1 |
| ImGeoNet (Tu et al., ) | 15.8 | 74.8 | 46.5 | 45.7 | 39.9 | 8.0 | 2.9 | 32.9 | 0.3 | 7.9 |
| CN-RMA (Shen et al., ) | 21.3 | 69.2 | 52.4 | 63.5 | 42.9 | 11.1 | 6.5 | 40.0 | 1.2 | 24.9 |
| ImVoxelNet (Rukhovich et al., ) | 8.9 | 67.1 | 35.0 | 33.1 | 30.5 | 4.9 | 1.3 | 7.0 | 0.1 | 0.9 |
| NeRF-Det (Xu et al., ) | 12.0 | 68.4 | 47.8 | 58.3 | 42.8 | 7.1 | 3.0 | 31.3 | 1.6 | 11.6 |
| 3DGS-DET (Our basic pipeline) | 18.5 | 73.5 | 44.6 | 61.9 | 42.2 | 9.3 | 5.6 | 28.7 | 2.3 | 2.0 |
| 3DGS-DET (Our basic pipeline+BG) | 16.1 | 77.0 | 51.6 | 62.4 | 44.7 | 11.7 | 11.3 | 24.4 | 1.7 | 19.0 |
| 3DGS-DET (Our basic pipeline+BG+BS) | 19.2 | 73.8 | 52.7 | 65.2 | 46.2 | 9.6 | 8.2 | 31.8 | 4.2 | 20.9 |
| Methods | desk | curt | fridg | showr | toil | sink | bath | ofurn | mAP@0.5 |  |
| Seg-Cluster (Wang et al., ) | - | - | - | - | - | - | - | - | - |  |
| Mask R-CNN (He et al., ) | - | - | - | - | - | - | - | - | - |  |
| SGPN (Wang et al., ) | - | - | - | - | - | - | - | - | - |  |
| 3D-SIS (Hou et al., ) | 13.7 | 0.0 | 2.7 | 3.0 | 56.8 | 8.7 | 28.5 | 2.6 | 14.6 |  |
| 3D-SIS (w/ RGB) (Hou et al., ) | 23.6 | 2.6 | 24.5 | 0.8 | 71.8 | 8.9 | 56.4 | 6.9 | 22.5 |  |
| VoteNet (Qi et al., ) | 37.5 | 11.6 | 27.8 | 10.0 | 86.5 | 16.8 | 78.9 | 11.7 | 33.5 |  |
| FCAF3D (Rukhovich et al., ) | 53.4 | 44.2 | 46.8 | 64.2 | 91.6 | 52.6 | 84.5 | 57.1 | 57.3 |  |
| CAGroup3D (Wang et al., ) | 63.6 | 44.4 | 57.0 | 49.3 | 98.2 | 55.4 | 82.4 | 58.8 | 61.3 |  |
| ImGeoNet (Tu et al., ) | 43.9 | 4.3 | 24.0 | 2.0 | 68.8 | 24.5 | 61.7 | 17.4 | 28.9 |  |
| CN-RMA (Shen et al., ) | 51.4 | 19.6 | 33.0 | 6.6 | 73.3 | 36.1 | 76.4 | 31.5 | 36.8 |  |
| ImVoxelNet (Rukhovich et al., ) | 35.5 | 0.6 | 22.1 | 4.5 | 67.7 | 18.9 | 60.2 | 10.1 | 22.7 |  |
| NeRF-Det (Xu et al., ) | 46.0 | 5.8 | 26.0 | 1.6 | 69.0 | 25.5 | 55.8 | 21.1 | 29.7 |  |
| 3DGS-DET (Our basic pipeline) | 53.5 | 18.1 | 30.7 | 3.4 | 77.0 | 29.0 | 68.3 | 24.2 | 34.1 |  |
| 3DGS-DET (Our basic pipeline+BG) | 47.4 | 27.2 | 30.4 | 8.3 | 87.0 | 36.3 | 78.3 | 28.8 | 36.9 |  |
| 3DGS-DET (Our basic pipeline+BG+BS) | 52.4 | 22.2 | 36.9 | 15.7 | 82.6 | 35.1 | 74.0 | 28.9 | 37.8 (+8.1) |  |

Figure: Figure 6: More qualitative comparison. Our methods identify more objects in the scene with better positional precision, highlighting the advantages of our approach over NeRF-Det (Xu et al., ). In this figure, the scene is represented using mesh to clearly display the boxes. Note that, Black and white boxes indicate predictions with incorrect categories, while boxes of other colors represent predictions with the correct category.

Figure: Figure 7: More qualitative comparison. Our methods identify more objects in the scene with better positional precision, highlighting the advantages of our approach over NeRF-Det (Xu et al., ). In this figure, the scene is represented using mesh to clearly display the boxes. Note that, Black and white boxes indicate predictions with incorrect categories, while boxes of other colors represent predictions with the correct category.

Figure: Figure 8: Analysis on the effect of Boundary Guidance. Gaussian blobs trained with Boundary Guidance exhibit clearer spatial distribution and more distinct differentiation between objects and background. Note that we visualize only the positions of the Gaussian blobs to highlight their spatial distribution, omitting other attributes.

Figure: Figure 9: Rendered images from different views by 3DGS trained with Boundary Guidance. The category-specific boundaries are well rendered and exhibit multi-view stability, demonstrating that the 3D representation has successfully embedded the priors provided by Boundary Guidance.

Figure: Figure 10: Rendered images from different views by 3DGS trained with Boundary Guidance. The category-specific boundaries are well rendered and exhibit multi-view stability, demonstrating that the 3D representation has successfully embedded the priors provided by Boundary Guidance.

Figure: Figure 11: Analysis on the effect of Box-Focused Sampling. Box-Focused Sampling significantly retains more object blobs and reduces noisy background blobs. Note that we visualize only the positions of the Gaussian blobs to highlight their spatial distribution, omitting other attributes.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses
soundness: 3
strengths: + To my best knowledge, this is the first paper that got 3D object detection working using the 3D Gaussian Splatting representation
+ The performance of the method is good. The proposed method outperforms SotA method such as NeRF-Det by a reasonable margin
+ The presentation of the paper is good. The paper is relatively easy to follow.
summary: The paper proposes to use 3D Gaussian Splatting (3DGS) as the representation to do 3D object detection. To make 3DGS works well for the 3D object detection task, the authors tackle two obstacles: (1) due to the natural of Gaussian, the object boundaries are ambiguous and hard to distinguish, and (2) excessive amount Gaussian blobs in the background. To solve (1), the authors rely on 2D image boundary guidance, and to resolve (2), the authors propose "box-focused" sampling. Experimental results show that the proposed method outperform the current SotA (NeRF-Det) by a reasonable margin on ScanNet and ARKit datasets.
weaknesses: - The proposed method seems to be relying on some heuristics and some hyper-parameters seem to be set to work well for the particular test sets. For example, "Gaussian blobs not belonging to any frustum are assigned a small probability pbg, set to 0.01 in practice." lines (315-316)
- The paper does not discuss anything related to latency. The proposed "boundary guidance" and "box-focused sampling" could be taking more computational time than the baseline NeRF-Det
- The upper-bound performance of the proposed method is bounded by various methods used to provide "boundary guidance" and "box-focused sampling", such as Grounded SAM, Suzuki-Abe algorithm etc. If NeRF-Det is aided by these segmentation method, it could potentially perform better as well

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: NA
soundness: 2
strengths: - The guidance of the GS reconstruction via boundary coloring and object frustum weighted resampling is clever and effective as can be observed qualitatively in the figures in the paper. These also quantitatively lead to improved 3D object detection as shown via ablation studies. These guidances in effect make condition the GS reconstruction on available 2D object evidence which is a very interesting paradigm. In the extreme one would aim to only reconstruct the objects?
- The improvement of 3D detection performance on ARKitScenes relative to NeRF-Det is impressive. I am quite curious to see some qualitative examples showing the improvements. For example the chair class jumped from 4% to 70.3%. I would love to see a qualitative comparison on a scene with many chairs. 
- The paper is well written and has good figures to support the claims (effects of the two guidances on GS centers). As well as good qualitative figures showing the 3D bounding box detections (although I would really like to see some on ARKitScenes - see weaknesses).
summary: The manuscript introduces an approach to 3D object detection based on Gaussian Splats (GS) reconstructions. The method uses two mechanisms to guide the 3D GS reconstruction towards reconstructing the objects in the scene with high boundary fidelity before using a standard sparse 3D convolution-based 3D object detector on the GS parameters. The GS reconstructions are guided by (1) coloring object boundaries in the images and (2) resampling Gaussians with higher probability if they fall within an object frustum. The object boundary coloring leads to strong edges in the images which via the GS optimization translate into more Gaussians on object boundaries. The frustum resampling down-weighs background surfaces and focuses Gaussians to reconstruct primarily the objects in the scene. As a result the 3D detector has an easier job and performs well.
weaknesses: - The ablation studies are effective but could be improved by adding lower bounds (no guidance mAP to Table 3) and upper bounds (sampling GS according to GT OBBs to table 4). The center-point guidance seems unnecessary - clearly a pixel-level optimization algorithm like GS will not be able to leverage the guidance since it wont be multi-view consistent. 
- To some degree that the 2D boundary guidance works is surprising since those 2D boundaries often stem from occlusion boundaries where multiview consistency across larger baselines is not given. I would love to see some examples where the 2D boundary is a occlusion boundary and the camera observes it from different angles - does this guidance still work?
- It is unclear why the box-focused sampling cannot re-use the segmentation masks? All that is needed is to assign object probabilities to Gaussian splats which does not need frustra to be unprojected. The GS centers can simply be projected into all images, to assign mask confidence values from Grounded SAM. This would make for a more simple story and system. 
- The main results Table 1 incorrectly identifies only NeRF-Det as a baseline for the proposed approach. Other methods like ImGeoNet, CN-RMA, ImVoxelNet also only rely on posed images and are valid baselines for the proposed method. At a minimum each related method needs to have called out the input modalities. As is the table is not useful in providing comparisons to the right kind of related work using also multiview posed images.
- Feature-metric GS/NERF reconstructions such as in LangSplat, EgoLifter, LERF could in principle be prompted for 3D segmentations that of course can be used to extract 3D bounding boxes. Any one of those would be a great additional point of comparison.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors can find the most important questions under the Weaknesses section.
soundness: 3
strengths: - The paper proposes the first method that directly considers 3D Gaussians for 3D object detection.
- The method achieves good accuracy on two public datasets.
summary: The paper proposes a guidance approach for 3DGS that facilitates the separation of 3D Gaussians representing distinct 3D object instances. Additionally, it introduces guided downsampling of the generated Gaussians, ensuring that most are associated with object instances rather than the background. The method shows improvements in two frequently used datasets.
weaknesses: I have mixed feelings about the paper. On the one hand, the contributions appear questionable or minor. On the other, the method achieves competitive accuracy compared to baselines, nearly matching methods that do not rely on Gaussians. My primary concerns are as follows:

- The boundary guidance lacks justification. In Eq. 9, the authors incorporate a region constraint by linearly mixing the RGB color and a region-specific color at each pixel. However, they do not clarify how this region color is chosen — a critical detail, as the choice of color heavily impacts the optimization. For example, if the chosen region color is similar to nearby content (e.g., a white region color on a brown table near a white wall), the boundary constraint would fail. Additionally, the region's color likely interferes with both the object and surrounding colors, ultimately altering the Gaussians' colors, which may no longer represent the original scene. Maybe I misunderstood, but this choice seems odd; an alternative could have been to increase the number of channels per pixel or use the approach of LangSplat [a] to get the guidance. 
- Section 3.4 describes Gaussian subsampling to retain the most representative objects. However, it is unclear how this impacts final accuracy. This approach seemingly does not enhance object detection but merely reduces the Gaussian cloud density. While this could affect the metrics, the practical value remains uncertain. Retaining a single well-localized Gaussian per object might optimize segmentation accuracy but would likely impair novel view synthesis and geometric accuracy. This raises the following point:
- It would be essential to assess novel view synthesis and geometric accuracy. Does this approach compromise other objectives of 3DGS, or do they remain intact?

Minor points and typos:
- Section 3.4: The process here is somewhat unclear. To my understanding, Gaussian splatting with boundary guidance runs first, followed by Gaussian subsampling to retain those likely corresponding to objects. This appears to involve guided sampling based on unspecified probabilities. The precise calculation of these "probability" scores is not explained. Regardless of my interpretation, the authors should clarify this part, as much of it is speculative.
- L080/L097: "empower" → "improve"
- L085: "distribution that is more differentiable" > I don't understand what the authors want to say here. 
- L087: "to establish 3D object probability spaces" > "object probability spaces" is misleading. This term refers to a 3D subspace where objects are located, and it is unrelated to probabilities. The authors use "probability" inconsistently, such as in Eq. 14, to justify heuristics. Heuristics are acceptable, but mislabeling them as probabilities is not.
- L155: "significantly enhances the spatial distribution"—the opposite is true; it restricts distribution to independent object representations, losing background information.
- L316: What is "independent probabilistic sampling"?
- Fig. 2 does not effectively illustrate the pipeline. It should be improved.

[a] Qin, M., Li, W., Zhou, J., Wang, H. and Pfister, H., 2024. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20051-20060).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The paper mentions that background Gaussians affect detection, but there are backgrounds and target objects in point clouds as well. Do the point cloud-based 3DOD face the same issue? If not, why not directly perform point cloud object detection and rendering tasks separately?

2. From my understanding, this paper enhances the accuracy of 3D object detection by unprojecting 2D priors, i.e. edges and 2D detection information, into 3D space based on 3DGS. This enhancement seems reasonable. However, if this process only improves 3D object detection while not enhancing or even degrading neural rendering, it can be seen as a strategy solely for improving 3D object detection. The use of 3D Gaussians might not be necessary. For instance, could converting 3D Gaussians into point clouds for 3D object detection after training achieves similar improvements?
soundness: 2
strengths: 1. This paper solves 3DOD problem on 3DGS at the first time.
2. The improvement over NeRF-based detection is significant.
summary: This paper pioneers using 3DGS for 3DOD, tackling two main challenges: (i) unclear spatial distribution of Gaussians, which affects object-background separation, and (ii) excessive background noise. They propose the 2D boundary guidance to improve spatial clarity and a Box-Focused sampling strategy for efficient object-focused sampling. The experiments show the improvement of object detection compared to their baseline and nerf-based method.
weaknesses: 1. Combining neural rendering and 3D object detection is a relatively new direction. In the introduction, I think the authors should discuss more about the significance or application of performing 3D object detection with novel view synthesis, especially for the paper which first uses 3D Gaussians for object detection. In NeRF-based methods(NeRF-Det), their neural rendering and object detection mutually enhance each other; accurate geometry improves object detection, while the object detection task promotes geometric learning, ultimately enhancing the quality of both object detection and neural rendering. However, in this paper, the experiments only analyze 3D object detection, without evaluating the quality of rendering. Therefore, it is not possible to thoroughly investigate the impact between the rendering of 3D Gaussians and the object detection tasks under this method. In conclusion, why not independently perform the tasks of rendering and 3D object detection, if both yield better results when performed independently? Besides, for NeRF-based 3DOD, they train a feed-forward network for generalizable neural rendering, which could be benificial for perception. But this paper still need per-scene optimization (to my understanding).

2. The paper mentions that performing 3D object detection on 3DGS has higher rendering speed, but it does not compare the differences with NeRF-based methods in terms of training time, rendering time, and detection time.