Gaussian-Det: Learning Closed-Surface Gaussians for 3D Object Detection

## Abstract

Abstract Skins wrapping around our bodies, leathers covering over the sofa, sheet metal coating the car – it suggests that objects are enclosed by a series of continuous surfaces, which provides us with informative geometry prior for objectness deduction.
In this paper, we propose Gaussian-Det which leverages Gaussian Splatting as surface representation for multi-view based 3D object detection.
Unlike existing monocular or NeRF-based methods which depict the objects via discrete positional data,
Gaussian-Det models the objects in a continuous manner by formulating the input Gaussians as feature descriptors on a mass of partial surfaces.
Furthermore, to address the numerous outliers inherently introduced by Gaussian splatting, we accordingly devise a Closure Inferring Module (CIM) for the comprehensive surface-based objectness deduction.
CIM firstly estimates the probabilistic feature residuals for partial surfaces given the underdetermined nature of Gaussian Splatting, which are then coalesced into a holistic representation on the overall surface closure of the object proposal.
In this way, the surface information Gaussian-Det exploits serves as the prior on the quality and reliability of objectness and the information basis of proposal refinement.
Experiments on both synthetic and real-world datasets demonstrate that Gaussian-Det outperforms various existing approaches, in terms of both average precision and recall.

## 1 Introduction

3D object detection in indoor scenes [Dai et al., , Hu et al., , Rukhovich et al., ] has been a popular topic in industry and academia, fueling various applications such as robotic navigation and augmented reality.
While many works [Qi et al., , Liu et al., , Cheng et al., , Rukhovich et al., , Ran et al., ] have demonstrated the effectiveness on point cloud data captured by expensive sensors such as lidars and depth cameras, RGB camera systems offer a cost-effective alternative.
However, the absence of depth information poses major challenges for inferring the three-dimensional geometries of the target objects from two-dimensional visual clues.

To overcome the aforementioned ill-posedness in image-based 3D object detection, pure multi-view based approaches [Rukhovich et al., , Tu et al., ] projects the features on the image plane into a 3D volume representation.
However, such projective geometry is still dominated by 2D features which may lead to ambiguous detection results.
On the other hand, recent advances in Neural Radiance Field (NeRF) [Mildenhall et al., ] enjoy multi-view consensus and consistency via learning a plenoptic mapping function.
Leveraging NeRF as the 3D scene representation, NeRF-RPN [Hu et al., ] consumes volumetric information extracted from NeRF and directly outputs 3D bounding boxes through a general region proposal network (RPN).
However, due to the implicit and continuous representation of NeRF, these methods are computationally expensive to optimize.
More importantly, similar to monocular-based methods [Rukhovich et al., , Tu et al., ], they depend on uniform and discrete sampling in 3D space which may lead to sub-optimal performance by failing to capture the true objectness.
In contrast, as indicated by the bottom (a-b) part of Figure 1, real-world 3D objects are usually enclosed by a series of continuous surfaces. Such surface information provides informative visual clues for objectness deduction [Ren and Sudderth, ].

Figure: Figure 1: Top Left: Gaussians with color and coordinate. Top Right: Gaussians with color, coordinate and surface information, but in a much fewer amount. Bottom (a-b): Object composed of Gaussians exhibits a textured surface as opposed to the top left illustration. Bottom (c-d): Gaussian splatting inherently introduces numerous outliers around the object.
Refer to caption: x1.png

In this paper, we propose Gaussian-Det which exploits continuous surface representation for multi-view based 3D object detection.
Gaussian-Det leverages the recently-proposed Gaussian Splatting [Kerbl et al., ] and formulates the input Gaussians into a mass of descriptors on partial surfaces.
As shown in the top part of Figure 1, even with less than a third of the number, Gaussians utilizing surface information better demonstrate the texture information and showcase the smoothness and continuity.
While it is feasible to directly employ point cloud based detectors, they can be distracted by numerous outliers inherently introduced by Gaussian Splatting process shown in Bottom (c-d) part of Figure 1.
Moreover, the point-based detectors regard the input Gaussians as infinitesimal positional representations and discard the explicit depiction on continuous surface, which may lead to sub-optimal detection proposals.
To address this, we design a Closure Inferring Module (CIM) for the comprehensive surface-based objectness deduction in Gaussian-Det.
Specifically, taking the underdetermined nature of Gaussian Splatting into account, CIM firstly conducts the partial surface inference via estimating a variational term as a probabilistic feature residual to the original representation of partial surfaces.
Under the assumption that the partial surfaces corresponding to an object can form a closed surface, CIM then coalesces them into a holistic representation which measures the closure of the overall proposal, serving as the prior on the quality and reliability of objectness prediction and the information basis of proposal refinement.
To validate the effectiveness of the proposed Gaussian-Det, we have conducted experiments on both synthetic 3D-FRONT [Fu et al., ] and real-world ScanNet [Dai et al., ] datasets.
The experimental results demonstrate that Gaussian-Det outperforms various existing approaches in terms of both average precision and recall.

## 2 Related Work

### 2.1 Neural Scene Representation

In recent years, the methods of neural scene representation have presented astonishing performances on visual computing and geometry learning.
They usually take a coordinate as input and output corresponding scene properties, such as signed distance [Park et al., , Chabra et al., ], occupancy [Mescheder et al., ], color and density.
Notably, in the scenario of multi-view geometry reconstruction, the Neural Radiance Field (NeRF) [Mildenhall et al., ] learns a plenoptic mapping from the querying coordinate and viewing direction into the point-wise radiation and volume density.
The NeRF-based scene modeling has inspired numerous extended works spanning from low-level geometry reconstruction [Barron et al., , Fridovich-Keil et al., , Wang et al., ] to high-level visual understanding [Zhi et al., , Jeong et al., , Hu et al., , Xu et al., , Tian et al., , Irshad et al., ].
Since NeRF requires a massive amount of queries during both training and inference phases, some improved representations are introduced as proxies such as hash grids [Müller et al., ], tri-planes [Chen et al., ] and points [Xu et al., ].
Among these advancements, the 3D Gaussian Splatting (3D-GS) [Kerbl et al., ]
stands out for its balance between quality and efficiency.
3D-GS employs a set of anisotropic Gaussians initialized by Structure from Motion (SfM), whose amount and properties are iteratively updated during training.
Apart from the positional information as in point cloud based or NeRF-based representation, a 3D Gaussian naturally depicts a continuous surface through its mean, covariance parameters.
Based on 3D-GS, several works have explored on 3D content generation [Tang et al., ], surface reconstruction [Guédon and Lepetit, ], semantic segmentation [Shi et al., ] and editing [Chen et al., ].

### 2.2 Indoor 3D Object Detection

Indoor 3D object detection has drawn extensive attention owing to its broad applications in robotic vision, augmented reality, etc.
The techniques can be categorized according to the input modalities, such as point cloud, gridded voxels and 2D image(s).
For example, Song and Xiao [] uses a volumetric CNN to create 3D Region Proposal Network (RPN) on a voxelized 3D scene.
VoteNet [Qi et al., ] directly operates on point cloud by using a PointNet++ [Qi et al., ] backbone and center voting strategy.
The follow-up point-based approaches include the VoteNet-style  [Cheng et al., , Liu et al., ] and the sparse convolution based methods [Rukhovich et al., , Wang et al., ].
However, the reliance on expensive lidars or depth sensors limits their applicability.
In contrast, the RGB camera systems offer a cost-effective alternative.
To overcome the inherent ill-posedness in image-based 3D object detection, ImVoxelNet [Rukhovich et al., ] lifts the 2D features into a 3D volume representation followed by a FCOS [Tian et al., ]-style detection head.
ImGeoNet [Tu et al., ] introduces an image-induced geometry-aware voxel representation for better preservation of geometry.
However, the projective geometry extracted by Rukhovich et al. [], Tu et al. [] still originates from 2D features and may thus result in ambiguous detections.
On the other hand, several advances [Hu et al., , Xu et al., , Irshad et al., ] seek to leverage the multi-view consensus and consistency of NeRF representation [Mildenhall et al., ] that embeds the geometry into the voxel grids.
NeRF-RPN consumes the volumetric features sampled from NeRF and then directly regresses the bounding boxes via a generic RPN.
NeRF-Det learns a two-branch framework that simultaneously conducts geometry reconstruction and bounding box regression.
These two approaches rely on uniform and discrete sampling which may fail to capture the true objectness in 3D scenes.
In contrast, our proposed Gaussian-Det strives for representing the targets in a continuous manner by leveraging the descriptors on partial surfaces and coalescing them into a holistic measurement which serves as a comprehensive prior on the reliability of the deducted objectness.

## 3 Approach

In this section, we elaborated the proposed Gaussian-Det by dividing the framework into the construction of surface-based Gaussian representation, object proposal initialization, partial surface feature inference and holistic surface closure coalescence.
The overall framework is illustrated in Figure 2.

Figure: Figure 2: Framework of our Gaussian-Det. Gaussian-Det takes as input the Gaussians reconstructed from scanned multi-view images of an indoor scene. We formulate surface-based Gaussian representation which is firstly used to predict the initial object proposal. Then the closure inferring module, which contains partial surface feature inference and holistic surface closure coalescence, learns a reliability measurement of the proposal and estimates the refined detection result, whose absolute closure score is low similar to the ground truth.
Refer to caption: x2.png

### 3.1 Preliminary: 3D Gaussian Splatting

The input scene of our proposed Gaussian-Det is represented by 3D Gaussian Splatting (3D-GS) [Kerbl et al., ] that are reconstructed from posed images, comprising a set of 3D Gaussian primitives $\mathbb{G}=\{\mathbf{g}_{p}=(\mathbf{\mu}_{p},\mathbf{\Sigma}_{p},\alpha_{p},
\mathbf{c}_{p})\}^{P}_{p}$.
Each 3D Gaussian $\mathbf{g}_{p}$ is parameterized by a mean vector $\mathbf{\mu}_{p}\in\mathbb{R}^{3}$ indicating its center, an anisotropic covariance matrix $\mathbf{\Sigma}_{p}\in\mathbb{R}^{3\times 3}$, the opacity value $\alpha_{p}\in\mathbb{R}$ and color $\mathbf{c}_{p}\in\mathbb{R}^{3}$.
$\mathbf{\Sigma}_{p}$ is a positive semidefinite matrix and can be factorized as $\mathbf{\Sigma}_{p}=\mathbf{R}_{p}\mathbf{S}_{p}\mathbf{S}_{p}^{T}\mathbf{R}_{
p}^{T}$.
$\mathbf{R}_{p}\in\mathbb{R}^{3\times 3}$ and $\mathbf{S}_{p}\in\mathbb{R}^{3\times 3}$ refer to a diagonal size matrix $diag(s_{p}^{1},s_{p}^{2},s_{p}^{3})$ and a rotation matrix converted from a unit quaternion respectively.
Subsequently, the 3D Gaussians are projected onto the image plane using known camera matrices.
$\alpha$-blending is executed to aggregate the overlapping Gaussians on each pixel and compute the final color:

$$ $C=\sum\limits^{n}_{i=1}\mathbf{c}_{i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j})$ (1) $$

where $n$ is the number of overlapping Gaussians.
The attributes of $\mathbf{g}_{p}$ are optimized through backward propagation of the gradient flow.
Particularly, the density (i.e., $P$) of 3D Gaussians is also updated during training via cloning and splitting for the sake of rendering quality.

### 3.2 Surface-based Gaussian Representation and Proposal Initialization

Compared with NeRF-based [Mildenhall et al., ] representation, 3D Gaussian Splatting enjoys the real-time rendering speed owing to a specially designed rasterizer.
More importantly, a 3D Gaussian $\mathbf{g}_{p}$ approximates the shape of a bell curve with depiction on a continuous surface (especially for a flat Gaussian [Guédon and Lepetit, ]), as apposed to the discrete representation of infinitesimal positional data such as point cloud [Rukhovich et al., ] and uniformly sampled volumes  [Rukhovich et al., , Xu et al., , Hu et al., ].
To this end, we concentrate the Gaussian representation onto surface-related information by formulating:

$$ $\mathbb{G}^{surf}\coloneqq\{\mathbf{g}_{p}^{surf}=(\mathbf{x}_{p},\mathbf{n}_{ p},A_{p})\}_{p}^{P}\in\mathbb{R}^{P\times 7}$ (2) $$

where $\mathbf{x}_{p}=\mathbf{\mu}_{p}$, $\mathbf{n}_{p}$ and $A_{p}$ specify the information about location, perpendicular orientation and area respectively.
The normal vector $\mathbf{n}_{p}$ is determined by solving the eigenvalue problem $\mathbf{\Sigma}_{p}\mathbf{n}_{p}=\lambda\mathbf{n}_{p}$. Specifically, $\mathbf{n}_{p}$ is the eigenvector corresponding to the smallest eigenvalue. The in-ward or out-ward orientation is determined by setting the orientation of $\mathbf{n}_{p}$ to align with the orientation from the center of the object proposal to the Gaussian mean. Please refer to the Appendix pages for the derivative process.
$A_{p}$ is defined as the largest cross-section area of 3D Gaussian Ellipsoid:

$$ $A_{p}=\pi\frac{s_{p}^{1}\cdot s_{p}^{2}\cdot s_{p}^{3}}{\min(s_{p}^{1},s_{p}^{ 2},s_{p}^{3})}$ (3) $$

The reformulated Gaussian representation $\mathbb{G}^{surf}$ initializes the depiction on multiple partial surfaces.
Given the huge amount of Gaussians in $\mathbb{G}^{surf}$, we use an off-the-shelf backbone $\mathcal{B}(\cdot)$ for trimming:

$$ $\mathbb{G}^{cand},\mathcal{F}^{cand}=\mathcal{B}(\mathbb{G}^{surf})$ (4) $$

where $\mathbb{G}^{cand}=\{\mathbf{g}_{1},\mathbf{g}_{2},\dots,\mathbf{g}_{M}\}\in
\mathbb{R}^{M\times 7}$ is a subset of $\mathbb{G}^{surf}$ and $\mathcal{F}^{cand}=\{\mathbf{f_{1}},\mathbf{f_{2}},\dots,\mathbf{f_{M}}\}\in
\mathbb{R}^{M\times C}$ denotes the corresponding feature in the latent space.
From $\mathbb{G}^{surf}$, we then accordingly initialize the partial-aware object proposals $\mathbf{P}^{part}=\{p_{1},p_{2},\dots,p_{K}\}$ by employing a VoteNet [Qi et al., ]-style head [Cheng et al., ] which groups $\mathbb{G}^{cand}$ in Euclidean space and predicts the bounding boxes with a MLP-based architecture [Qi et al., ].
Meanwhile, the partial-aware features of each proposal is extracted:

$$ $\mathbf{f}^{part}_{k}=\mathcal{M}(\mathcal{F}^{cand},p_{k})$ (5) $$

where $\mathcal{M}$ is a querying-and-pooling function [Cheng et al., ] that operates around the location of each proposal.

As mentioned above, it is feasible to directly employ point cloud based detectors [Qi et al., , Rukhovich et al., , Liu et al., ] and regard $\mathbb{G}^{surf}$ as a mass of point cloud concatenated with the corresponding feature.
However, as shown in Figure 1, there still severely exists Gaussian outliers around the object surfaces that may distract the point-based detectors.
We also demonstrate in Experiment section that this leads to incorrect objectness deduction and inferior detection performances evaluated on $\mathbf{P}^{part}$.

To address this, we devise a Closure Inference Module (CIM) in Gaussian-Det responsible for comprehensive surface-based objectness deduction in Gaussian-Det, which takes the holistic closure measurement into account to refine the object proposals.

### 3.3 Closure Inference Module for Proposal Refinement

As illustrated in Figure 2, the Closure Inference Module (CIM) consists of two stages: partial surface feature inference and holistic surface closure coalescence.

Partial Surface Feature Inference:
CIM firstly conducts the partial surface modeling via probabilistic feature construction.
In 3D Gaussian Splatting, inferring local surface features for spatial points is underdetermined due to the supervision using 2D signals, leading to the unstructured outliers [Guédon and Lepetit, ] with unreliable local surface descriptors.
To this end, we introduce a variational term $\mathcal{F}^{V}$ into the candidate features $\mathcal{F}^{cand}$, establishing a probabilistic feature residual:

$$ $\mathcal{F}^{cand}\leftarrow\mathcal{F}^{cand}+\alpha\mathcal{F}^{V}$ (6) $$

where $\alpha$ is a weight hyper-parameter (in practice we set $\alpha=0.1$).
The randomness is modeled by assuming $\mathcal{F}^{V}$ can be sampled from $\mathbf{p}_{\theta}(\mathcal{F}^{V})=N(\mathcal{F}^{V};\mathbf{0},\mathbf{I})$.
To fit the posterior $\log q_{\phi}(\mathcal{F}^{V}|\mathcal{F}^{cand})$ and likelihood $p_{\theta}(\mathcal{F}^{cand}|\mathcal{F}^{V})$ parameterized by $\phi$ and $\theta$ respectively, the following evidence lower bound (ELBO) is optimized for the residual estimation:

$$ $\mathcal{L}_{res}=\mathcal{L}_{\bm{\mu}_{\mathbf{g}},\mathbf{\sigma}_{\mathbf{ g}}}(\mathcal{F}^{cand})=\mathbb{E}_{q(\mathcal{F}^{V}|\mathcal{F}^{cand})}[ \log p(\mathcal{F}^{cand}|\mathcal{F}^{V})]-\mathrm{KL}(q(\mathcal{F}^{V}| \mathcal{F}^{cand})||p(\mathcal{F}^{V}))$ (7) $$

By modeling the variational term, CIM explicitly represents this uncertainty, thereby better describing the distribution of local surface features.

Figure: Figure 3: Example of utilizing closure as a prior for determining the quality of objectness prediction.
Refer to caption: extracted/5895448/fig/teaser_v2.png

Holistic Surface Closure Coalescence:
The partial surfaces are then coalesced into a holistic representation by CIM, which provides informative clues on surface closure for objectness deduction.
As illustrated in Figure 3, we assume that the partial surfaces corresponding to an object can form a closed surface, which serves as a prior for determining the quality of objectness predictions.
The prior can be mathematically expressed by Theorem 1.

###### Theorem 1 .

Given a constant vector field $\mathbf{T}$ and a closed surface $\mathbf{S}$, the flux $\mathbf{\Phi}$ of the vector field $\mathbf{T}$ through the closed surface $\mathbf{S}$, also expressed as the surface integral of the $\mathbf{T}$ over $\mathbf{S}$ is zero:

$$ $\mathbf{\Phi}=\oiint\limits_{S}\mathbf{T}\cdot\mathbf{n}\mathrm{d}S=0$ (8) $$

###### Proof.

It is a special case of Gauss’ Flux Theorem by setting $\mathbf{T}$ as any constant vector field:

$$ $\oiint\limits_{S}\mathbf{T}\cdot\mathbf{n}\mathrm{d}S=\iiint\limits_{V}\nabla \cdot\mathbf{T}\mathrm{d}V\stackrel{{\scriptstyle\nabla\cdot\mathbf{T}=0}}{{=} }0\qed$ (9) $$

Numerically we implement the integrated flux in Eqn. (9) for the $k$-th proposal as quadrature:

$$ $\hat{\bm{\Phi}}_{k}=\sum_{\mathbf{g}_{i}\in\mathbb{G}^{cand}_{k}}\mathbf{T} \cdot\mathbf{n}_{i}A_{i},\;\text{where}\;\mathbb{G}^{cand}_{k}=\{\mathbf{g}_{i }\mid i\in\mathcal{I}_{k}\}\in\mathbb{R}^{N_{k}\times 7}$ (10) $$

where $\mathcal{I}_{k}$ is a set containing the indices of the candidate Gaussians within the proposal $p_{k}$. $N_{k}$ is the number of candidate Gaussians within $p_{k}$.
Without loss of generality, we set $\mathbf{T}$ as $(\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}})$.

With the estimated flux as the measurement of surface closure, we formulate the geometrically coalesced representation as:

$$ $\mathbf{f}^{hol}_{k}=S_{k}\cdot\mathcal{M}(\mathcal{F}^{cand},p_{k}),\;\text{ where}\;S_{k}=\exp(-\gamma_{k}|\hat{\mathbf{\Phi}}_{k}|)$ (11) $$

where $\mathcal{F}^{cand}$ is the inferred features with the probabilistic residual in Eqn.(6).
$S_{k}$ serves as the closure weight parameter.
$\gamma_{k}$ is a flexible coefficient mapped from $\mathcal{F}^{V}$ through a MLP and sigmoid activation function.
In this way, it controls the preservation (when $\hat{\mathbf{\Phi}}_{k}$ is close to zero) or suppression (when the absolute value of $\hat{\mathbf{\Phi}}_{k}$ is very large) of $\mathbf{f}^{hol}_{k}$.

We finally formulate the refined proposal feature which comprehensively combines the information on partial-aware surfaces and prior on holistic closure-based measurement on objectness as:

$$ $\mathbf{f}^{final}_{k}\coloneqq[\mathbf{f}_{k}^{part}||\mathbf{f}_{k}^{hol}]$ (12) $$

where $[\cdot||\cdot]$ denotes the concatenation operation along the feature dimension.
$\mathbf{f}^{final}_{k}$ is used for proposal refinement by estimating a parameter offset to the original proposals through a stacked MLP:

$$ $p_{k}^{refined}=p_{k}+MLP(\mathbf{f}^{final}_{k})$ (13) $$

Given the final refined object proposals $p_{k}^{refined}$, the overall loss function is computed as:

$$ $\displaystyle\mathcal{L}(\{p^{gt}_{k}\},\{p_{k}\},\{p_{k}^{refined}\})=\lambda _{res}\mathcal{L}_{res}$ $\displaystyle+\lambda_{pred}\mathcal{L}_{2}(\{p^{gt}_{k}\},\{p_{k}\})$ (14) $\displaystyle+\lambda_{refine}\mathcal{L}_{2}(\{p^{gt}_{k}\},\{p_{k}^{refined}\})$ $$

where $\{p^{gt}_{k}\}$ and $\mathcal{L}_{2}$ denote the ground truth bounding boxes and L2 norm loss function.
$\lambda_{res}$, $\lambda_{pred}$ and $\lambda_{refine}$ are pre-set weight hyper-parameter.

###### Theorem 1 .

###### Proof.

## 4 Experiments

We validate the proposed Gaussian-Det by conducting benchmark experiments on both synthetic and real-world datasets on 3D object detection.
We then validate the design choices via ablation studies.

### 4.1 Experimental Settings and Implementation Details

We experimented Gaussian-Det on the officially-released benchmark by Hu et al. [] which contain both the synthetic 3D-FRONT [Fu et al., ] and the real-world ScanNet [Dai et al., ] datasets.
We faithfully followed Hu et al. [] to take the posed images as input.
The camera poses in the datasets released by Hu et al. [] were estimated via Structure from Motion (SfM) which used the groundtruth point cloud as initialization.

The synthetic 3D-FRONT [Fu et al., ] dataset is featured with its large-scale room layouts and textured furniture models.
The complex scenes are manually downsized into 159 usable rooms after data cleansing by Hu et al. [].
The indoor scenes are densely annotated with oriented bounding boxes.
We follow Hu et al. [] to remove the annotations of construction objects such as ceilings and floors, and merge the components that belong to the same object instance into a whole.

The real-world ScanNet [Dai et al., ] dataset consists of complex and challenging scenes captured by Structure Sensor in an ego-centric manner.
We followed Hu et al. [] to uniformly divide the frames into 100 bins and choose the sharpest one in each bin measured by Laplacian variance.
Experiments were done on the 90 scenes that are randomly selected by Hu et al. [].

For the input Gaussians $\mathbb{G}$, we used the official implementation of 3D-GS [Kerbl et al., ] equipped with the SuGaR regularization [Guédon and Lepetit, ], which takes 30,000 training iterations.
We excluded the Gaussians outside the room cuboid defined by Hu et al. [] and filtered out Gaussians with opacity value lower than 0.3.
Note that such a combined implementation [Kerbl et al., , Guédon and Lepetit, ] aims at decent 2D rendering results.
The issue of outlier Gaussians around the 3D objects still severely exists as shown in Figure 1.
We employed PointNet++ [Qi et al., ] as the backbone $\mathcal{B}$.
We used a single NVIDIA RTX A6000 48GB GPU for both training and evaluation.
The total training epochs of detection is 50 for both 3D-FRONT and ScanNet.
The evaluation metrics are mainly average recall (AR) and average precision (AP), thresholded by IoU value at both 0.25 and 0.5.
Please refer to the Appendix pages for the details on model hyper-parameters.

In Figure 4, we have illustrated the statistics on the surface closure of two datasets measured by $|\Phi|$.
The bar charts show that the majority of the flux values are clustered near zero, which indicates that most objects in both datasets contain a higher degree of surface closure and our assumption is applicable to the employed datasets.

Figure: Figure 4: Statistics on surface closure of two datasets [Fu et al., , Dai et al., ]. Most objects in both datasets contain a higher degree of surface closure. The statistics are collected from our reconstructed 3D Gaussians.
Refer to caption: x3.png

### 4.2 Main Results

**Table 1: Quantitative results on the 3D-FRONT and ScanNet datasets. $\dagger$: we report the results of NeRF-MAE which involves the pre-training on 3D-FRONT and two other exterior databases, including HM3D Ramakrishnan et al. [] and Hypersim Roberts et al. [].**
| Methods | 3D-FRONT | ScanNet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AR25 | AR50 | AP25 | AP50 | AR25 | AR50 | AP25 | AP50 |  |
| ImVoxelNetRukhovich et al. [] | 88.3 | 71.5 | 86.1 | 66.4 | 51.7 | 20.2 | 37.3 | 9.8 |
| NeRF-RPN Hu et al. [] | 96.3 | 69.9 | 85.2 | 59.9 | 89.2 | 42.9 | 55.5 | 18.4 |
| NeRF-MAE$\dagger$ Irshad et al. [] | 97.2 | 74.5 | 85.3 | 63.0 | 92.0 | 39.5 | 57.1 | 17.0 |
| G-VoteNet Qi et al. [] | 81.5 | 61.6 | 73.0 | 49.6 | 78.5 | 34.2 | 66.8 | 18.2 |
| G-GroupFree Liu et al. [] | 84.9 | 63.7 | 72.1 | 45.1 | 75.2 | 37.6 | 60.1 | 20.4 |
| G-FCAF3D Rukhovich et al. [] | 89.1 | 56.9 | 73.1 | 35.2 | 90.2 | 42.4 | 63.7 | 18.5 |
| G-BRNet Xu et al. [] | 89.7 | 75.3 | 88.2 | 71.0 | 71.1 | 32.2 | 63.1 | 19.3 |
| Gaussian-Det (Ours) | 97.9 | 82.3 | 96.7 | 77.7 | 87.3 | 43.0 | 71.7 | 24.5 |

Compared Methods:
We have summarized the quantitative results on 3D-FRONT and ScanNet in Table 1.
For the comparison with the pure multi-view based methods, we follow Irshad et al. [] to include ImVoxelNet [Rukhovich et al., ], NeRF-RPN [Hu et al., ] and NeRF-MAE [Irshad et al., ] in Table 1.
ImVoxelNet directly extracts 3D volumetric features projected from 2D image planes.
The NeRF-based methods [Hu et al., , Irshad et al., ] apply 3D convolutions on the uniformly sampled points in the radiance fields.
Note that NeRF-MAE includes the pre-training on extra datasets.
Moreover, to demonstrate the effectiveness of Gaussian-Det more intuitively, we experimented on the point cloud based detectors [Qi et al., , Liu et al., , Rukhovich et al., , Cheng et al., ] (prefixed by “G-”) which merely contains the backbone and detection heads in Section 3.2.
The upper part of Table 1 are cited from Irshad et al. [], for which we have provided a more in-depth comparison in the Appendix pages.

Analysis on Quantitative Results:
The observation from Table 1 is two-fold.
Firstly, the point cloud based methods are able to achieve comparable performances compared with previous multi-view [Rukhovich et al., ] or state-of-the-art NeRF-based [Irshad et al., , Hu et al., ] approaches.
This suggests the potential of Gaussian Splatting as an explicit input modality for 3D object detection.
Secondly, except for AR thresholded by 0.25 on ScanNet, we can see that Gaussian-Det precedes all the compared methods on all evaluation metrics, especially on AP which is more widely-used in 3D object detection.
For example, on 3D-FRONT evaluated by AP25, Gaussian-Det outperforms the second-best G-BRNet [Cheng et al., ] by a significant margin of $+8.5\%$.
Although ScanNet is a challenging dataset for its lower image quality and diverse layouts, our Gaussian-Det still achieves the state-of-the-art average precisions (+4.9% on AP25 and +4.1% on AP50).
This indicates that Gaussian-Det better leverages the surface-based information inherently condensed in Gaussians.
By coalescing partial surfaces into object-wise and closure-aware holistic representations, Gaussian-Det better perceives the quality and reliability of the object proposals, thereby providing favourable information basis for proposal refinement.

Figure: Figure 5: Failure case on occluded objects.
Refer to caption: x4.png

Failure Case:
As for illustrative comparison, we firstly present the failure case by Gaussian-Det as shown in Figure 5.
Despite focusing on more informative surface representations, Gaussian-Det may still struggle at handling detecting the object with severe occlusion, which is an inherent challenge for current multi-view based 3D object detectors.
Moreover, since the surfaces of the objectness that contains jointed objects can also be regarded as closed, Gaussian-Det may mistake them as a whole.
Please refer to the Appendix pages for more failure cases.

Qualitative Results:
We then display the qualitative comparisons in Figure 6, where we set the 2D rendered results of the input Gaussians as visual reference.
We can see that compared with ground truth bounding boxes, Gaussian-Det is capable of preventing redundant object predictions that are potentially caused by outliers, leading to higher detection accuracy.
Figure 7 provides a more in-depth visualization on the discriminative objectness deduction of Gaussian-Det.
For each detected bounding box, we have marked the corresponding flux value computed in Eqn.(10) that measures the surface closure.
Predictions whose absolute flux values are larger or less than one are marked with blue or red respectively.
We can see that in the presented examples, a favourable detection well corresponds to a relatively higher degree of closure (lower absolute value of flux).
With the holistic surface coalescence by CIM, Gaussian-Det is aware of the reliability of proposals and able to refine them for higher accuracy.
Please refer to the Appendix pages for more qualitative results.

Figure: Figure 6: Qualitative Results on 3D-FRONT (top two rows) and ScanNet (bottom two rows).
Refer to caption: x5.png

Figure: Figure 7: Illustration on the detected objects and the surface closure calculated from the 3D Gaussians within each estimated bounding boxes, measured by the absolute flux values (in square decimetre).
Refer to caption: x6.png

### 4.3 Ablation Studies and Analysis

To evaluate the effectiveness of the design choices and the components in our framework, we conducted five groups of ablational experiments on the 3D-FRONT dataset.
The performances evaluated by average precision (AP) are summarized in Table 2.

**Table 2: Quantitative results of ablation studies on 3D-FRONT.**
|  | AP25 | AP50 |
| --- | --- | --- |
| (1) Discard the Holistic Surface Coalescence | 92.8 | 73.9 |
| (2) Naive Holistic Surface Coalescence | 93.5 | 75.0 |
| (3) Deterministic Partial Surface Representation | 95.7 | 74.7 |
| (4) No Flexibility in Closure Measurement | 96.4 | 76.2 |
| (5) Alternative Surface Orientation | 96.2 | 75.7 |
| (6) The Full Framework | 96.7 | 77.7 |

(1)-(2) Discarded or Naive Holistic Surface Coalescence:
The trial (1) aims at experimenting with information on partial surfaces only.
This is achieved by concatenating two copies of $\mathbf{f}^{part}_{k}$ in Eqn. (12).
In trial (2), the partial surfaces are coalesced without any consideration on the closure properties of objects.
This is achieved by setting the closure score $S_{k}=1$ for all object proposals.
By comparing (1), (2) and (6), we can see that the coalescence into holistic surface representation is necessary (+0.7% on AP25 and +1.1% on AP50) despite its naive implementation in (2).
The closure-based measurement augments the proposal representation with prior on the reliability information, which furtherly enhances the detection accuracy sizably and is reflected by the performance gains (+3.2% on AP25 and +2.7% on AP50).

(3) Deterministic Partial Surface Representation:
In Gaussian-Det, the feature representation of the trimmed Gaussian candidates are supplemented with a probabilistic residual in Eqn. (6).
Here we ignored the variational inference of $\mathcal{F}^{cand}$ by setting $\lambda_{res}=0$.
We can see that the deterministic partial surface representation in (3) is hampered by the under-determined nature of Gaussian Splatting, which leads to inferior performances (-1.0% on AP25 and -3.0% on AP50).

(4) No Flexibility in Closure Measurement:
The trial cancels the allowance for the potential truncation in indoor scenes.
We empirically set $\gamma_{k}=0.5$ for all object proposals.
It can be seen that despite the failure case in Figure 5, a learnable $\gamma_{k}$ endows the closure measurement with the consideration on flexible cases such as truncations and occlusions.
Improvements on both AP25 (+0.3%) and AP50 (+1.5%) can be observed.

(5) Alternative Surface Orientation:
By default we set the $\mathbf{n}_{p}$ of a partial surface by solving the eigenvalue problem and determine its in-ward or out-ward by setting the orientation of $\mathbf{n}_{p}$ to align with the orientation from the center of the object proposal to the Gaussian mean. An alternative way is to determine orientation following [Ran et al., ].
We experimented on two different implementation of closure scores.
As it influences the surface-based Gaussian representation as well as the closure computation, a sizable drops on AP50 (-2.0%) can be observed in Table 2.

Please refer to the Appendix pages for additional ablational experiments, including the choices of hyper-parameters and the comparison on learning curves before and after incorporating the probabilistic feature residual.

Model Efficiency:
As shown in Table 4, Gaussian-Det exhibits remarkable inference speed at 33.3 FPS, which is an order of magnitude faster than both NeRF-RPN and NeRF-MAE. Additionally, our model size is substantially smaller at 24.0MB, less than 8% of NeRF-RPN size and 4% of NeRF-MAE. These results indicate that our Gaussian-Det demonstrates significant improvements in detection accuracy while maintaining real-time processing speed and favourable model compactness.

Comparison with Point-based Partial Surface Modeling:
It is worth mentioning that RepSurf [Ran et al., ], a point cloud based convolutional operator, also considers the surface structure within local regions.
We accordingly replaced the backbone of G-BRNet [Cheng et al., ] with the umbrella variant of RepSurf.
The results on ScanNet are shown in Table 4.
The observation is two-fold.
Firstly, the incorporation of RepSurf does augment the point features with informative depiction on local surfaces (+0.4% on AP50).
However, RepSurf is prone to be severely distracted by the outlier Gaussians since it only considers the very local surfaces, which harms the detection quality drastically (-15.34% on AP25).
Secondly, merely focusing on the very local (umbrella-shaped in this case) surface while ignoring the holistic surface modeling leads to sub-optimal results (-4.8% on AP50).
This indicates that the comprehensive surface modeling strategy enables Gaussian-Det to better capture the intricate geometry of objects, thereby contributing to a sizably improved 3D detection accuracy.
Such awareness of holistic surface representation is essential to prevent the distraction by outliers.

**Table 3: Comparison on model efficiency.**
| Method | FPS | Model Size |
| --- | --- | --- |
| NeRF-RPN | 1.51 | 312.7MB |
| NeRF-MAE | 1.11 | 604.2MB |
| Gaussian-Det (Ours) | 33.3 | 24.0MB |

## 5 Conclusion

In this paper, we present Gaussian-Det for multi-view 3D object detection. Gaussian-Det leverages Gaussian Splatting and exploits the continuous surface representation.
To overcome the numerous outliers inherently introduced by the data modality, we devise a Closure Inferring Module (CIM) for comprehensive surface-based objectness deduction.
CIM learns the probabilistic feature residual for partial surfaces and coalesces them into holistic representation on closure measurement, thereby serving as a prior on proposal reliability.
Experimental results on both synthetic and real-world datasets validate the superiority of our Gaussian-Det over various existing mult-view based approaches.

Broader Impact:
The proposed method utilizes continuous surface representations to enhance the applications like augmented reality and robotic navigation. While it offers cost-effective solutions via the use of RGB cameras instead of expensive lidar systems, it is largely affected by the quality of input data and may struggle at handling occluded objects.
It also raises privacy concerns due to the demand for surveillance devices. To mitigate these issues, the adoption of ethical guidelines is crucial to ensure responsible use and maintain the reliability of the technology.

Limitations:
Gaussian-Det may be struggling to handle occlusion scenes (as shown in Figure 5), which is an inherent challenge in the setting of multi-view 3D object detection. Specifically, the detectors are unaware of the appearances and detailed geometries of the occluded part, thereby further worsening the inherent ill-posedness of 3D perception given 2D images.

Future Works:
Since the degree of surface closure of the objectness that contains joint objects is also relatively high, such scenarios might be challenging for Gaussian-Det. Future work can incorporate the color information to discriminate the objects and calculate the flux values separately. In addition, the overall closed surfaces can potentially be an informative clue to rule out the outlier 3D Gaussians, thereby in turn improving the quality of 3D reconstruction.

## Appendix A More Technical Details

### A.1 Details on Calculating Surface Normal

We provide details of calculating surface normal, which is determined by solving the eigenvalue problem:

$$ $\mathbf{\Sigma_{p}}\mathbf{n_{p}}=\lambda\mathbf{n_{p}}$ (15) $$

where $\mathbf{\Sigma}_{p}=\mathbf{R}_{p}\mathbf{S}_{p}\mathbf{S}_{p}^{T}\mathbf{R}_{
p}^{T}$. Then we made identical deformation to both sides of the equation.

$$ $\displaystyle\mathbf{R}_{p}\mathbf{S}_{p}\mathbf{S}_{p}^{T}\mathbf{R}_{p}^{T} \mathbf{n}_{p}$ $\displaystyle=\lambda\mathbf{n}_{p}$ (16) $\displaystyle\mathbf{S}_{p}^{T}\mathbf{R}_{p}^{T}\mathbf{R}_{p}\mathbf{S}_{p} \mathbf{S}_{p}^{T}\mathbf{R}_{p}^{T}\mathbf{n}_{p}$ $\displaystyle=\lambda\mathbf{S}_{p}^{T}\mathbf{R}_{p}^{T}\mathbf{n}_{p}$ $$

We set $\mathbf{S}_{p}^{T}\mathbf{R}_{p}^{T}\mathbf{n}=\mathbf{x}$. As $\mathbf{R}_{p}$ is a rotation matrix, we get $\mathbf{R}^{T}_{p}\mathbf{R}_{p}=1$. Then:

$$ $\mathbf{S}_{p}^{T}\mathbf{S}_{p}\mathbf{x}=\lambda\mathbf{x}$ (17) $$

The solution is $\mathbf{x}_{0}=[1,0,0]^{T},\mathbf{x}_{1}=[0,1,0]^{T},\mathbf{x}_{2}=[0,0,1]^{T}$. Therefore:

$$ $\mathbf{n}_{p,i}=\mathbf{R}_{p}\mathbf{S}_{p}^{-1}\mathbf{x}_{i},i=0,1,2$ (18) $$

We define eigenvector with smallest eigenvalue as the Surface Normal $\mathbf{n}_{p}$. The defination is consistent with [Guédon and Lepetit, ]. However, $\mathbf{n}_{p}$ is unoriented, meaning it can point either inside or outside. Therefore, we determine the surface normal direction by considering the object center (in practice, we use proposal center). To elaborate, the orientation of surface normal will align with the orientation pointing from the object center to the Gaussian center. An alternative strategy is to follow [Ran et al., ]
to determine the orientation. Specifically, we ensure that the first element of the normal vector is non-negative. Then we determine the orientation using an instance-level random inversion with $p=50\%$. Ablation study in Table 2 shows that the result of strategy of Ran et al. [] is lower than the aforementioned strategy.

### A.2 More Details on Training

In training stage, we optimize the residue loss by maximizing the evidence lower bound.

$$ $\mathcal{L}_{res}=\mathcal{L}_{\bm{\mu}_{\mathbf{g}},\mathbf{\sigma}_{\mathbf{ g}}}(\mathcal{F}^{cand})=\mathbb{E}_{q(\mathcal{F}^{V}|\mathcal{F}^{cand})}[ \log p(\mathcal{F}^{cand}|\mathcal{F}^{V})]-\mathrm{KL}(q(\mathcal{F}^{V}| \mathcal{F}^{cand})||p(\mathcal{F}^{V}))$ (19) $$

After applying the reparameterization trick [Bengio et al., ], the total variational loss can be reduced to:

$$ $\displaystyle\mathcal{L}_{res}=$ $\displaystyle\approx\frac{1}{M}\sum\limits^{M}_{i=1}\log p_{\theta}(\mathcal{F }^{cand}|\mathcal{F}^{V})$ (20) $\displaystyle+\frac{1}{2M}\sum\limits^{M}_{i=1}\sum\limits^{C}_{j=1}\Bigl{(}1$ $\displaystyle+\log\bigl{(}(\bm{\sigma}^{(j)}_{\mathbf{g}_{i}})^{2}\bigr{)}-( \bm{\mu}^{(j)}_{\mathbf{g}_{i}})^{2}-(\bm{\sigma}^{(j)}_{\mathbf{g}_{i}})^{2} \Bigr{)}$ $\displaystyle\triangleq\mathcal{L}_{rec}+\mathcal{L}_{appr}$ $$

where $C$ refers to channel and $M$ refers to the number of candidate Gaussians. $\mathcal{L}_{rec}$ refers to reconstruction loss and $\mathcal{L}_{appr}$ refers to Gaussian approximation loss. During training we implement square difference for simplicity.

## Appendix B More Ablation Experiments and Analysis

### B.1 Hyper-parameter Setup

**Table 5: The hyper-parameter setup in Gaussian-Det.**
| Name | Description | Value |
| --- | --- | --- |
| $\lambda_{res}$ | Weight of $\mathcal{L}_{res}$ | 1 |
| $\lambda_{pred}$ | Weight of $\mathcal{L}_{pred}$ | 1 |
| $\lambda_{refine}$ | Weight of $\mathcal{L}_{refine}$ | 1 |
| $\alpha$ | Residue weight | 0.1 |
| $\delta$ | Threshold of grouping the Gaussians | 0.2 |
| $P$ | The number of input Gaussians | 40000 |
| $M$ | The number of candidate Gaussians $\mathbb{G}^{cand}$ | 1024 |
| $C$ | Channel of candidate feature $\mathcal{F}^{cand}$ | 256 |

We summarize the hyper-parameters used in Table 5. These include hyper-parameters for loss function, grouping threshold, input and network setup. We have also tried different choices of $\lambda_{res}$. We choose $\lambda_{res}=1$ as the default setting.

**Table 6: Experimental results of parameter settings on Gaussian-Det. $\lambda_{res}=0$ denotes deterministic partial surface representation in ablation study. Performances of detection on 3D-FRONT are reported.**
| $\lambda_{res}$ | Is Default | AP25 | AP50 |
| --- | --- | --- | --- |
| 0 |  | 95.7 | 74.7 |
| 0.1 |  | 95.5 | 76.1 |
| 0.5 |  | 96.2 | 76.6 |
| 1 | $\checkmark$ | 96.7 | 77.7 |
| 2 |  | 96.2 | 75.7 |

### B.2 Impacts of F V subscript 𝐹 𝑉 F_{V} italic_F start_POSTSUBSCRIPT italic_V end_POSTSUBSCRIPT on the Training Process

Figure: Figure 8: Training curves before and after removing $F_{V}$.
Refer to caption: x7.png

We have also collected the statistics before and after removing $F_{V}$ and illustrated the learning curve in Figure 8.
We can observe that the probabilistic feature residual term contributed to a slightly faster initial convergence and less fluctuation throughout the training process.
This suggests that Gaussian-Det with $F_{V}$ is more stable during training.

## Appendix C More Visulization Results

### C.1 More Qualitative Results under Challenging Scenarios

Apart from the failure case in Figure 5, we illustrate another case in Example 1, Figure 9 where Gaussian-Det successfully detects the occluded object. In this example, Gaussian-Det benefits from the prior that the objectness contains a high degree of surface closure ($|\Phi|=0.074$), thereby tackling the occlusion caused by the sofa ($|\Phi|=0.025$).

On the other hand, for those challenging objects with higher that take up a small proportion, we have illustrated similar cases of ScanNet in Example 2&3 of Figure 9.
In Example 2, a bookshelf ($|\Phi|=0.833$) is close to a suitcase and two walls.
In Example 3, a desk ($|\Phi|=0.380$) and two garbage bins ($|\Phi|=0.556,0.218$) are tightly positioned to each other and are against a wall.
The absolute flux values ($|\Phi$) of these exemplified objects are relatively higher due to less exposure to camera shooting. While our assumption is weaker in these examples, our Gaussian-Det is still able to detect the target objects.

Figure: Figure 9: Detection results (upper) under challenging scenarios and corresponding ground-truths (bottom).
Refer to caption: x8.png

### C.2 More Failure Cases

Figure: Figure 10: Additional failure cases.
Refer to caption: x9.png

Apart from the example provided, we have further presented more failure cases in Figure 10. Since the surfaces of the objectness that contains jointed objects can also be regarded as closed, Gaussian-Det may mistake them as a whole. Future researches can focus on incorporating the color information to discriminate joint objects.

## Appendix D More Detailed Quantitative Comparison

### D.1 Detailed Comparison with NeRF-RPN and NeRF-MAE

**Table 7: Detailed quantitative comparison on 3D-FRONT and ScanNet.**
| Reference | Methods | Pretrained | 3D-FRONT | ScanNet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AR25 | AR50 | AP25 | AP50 | AR25 | AR50 | AP25 | AP50 |  |  |  |
| Hu et al. [] | NeRF-RPN (AB, VGG19) |  | 97.8 | 76.5 | 65.9 | 43.2 | 88.7 | 42.4 | 40.7 | 14.4 |
| NeRF-RPN (AB, R50) |  | 96.3 | 70.6 | 65.7 | 45.1 | 86.2 | 32.0 | 34.4 | 9.0 |  |
| NeRF-RPN (AB, Swin-S) |  | 98.5 | 63.2 | 51.8 | 26.2 | 93.6 | 44.3 | 38.7 | 12.9 |  |
| NeRF-RPN (AF, VGG19) |  | 96.3 | 69.9 | 85.2 | 59.9 | 89.2 | 42.9 | 55.5 | 18.4 |  |
| NeRF-RPN (AF, R50) |  | 95.6 | 67.7 | 83.9 | 55.6 | 91.6 | 35.5 | 55.7 | 16.1 |  |
| NeRF-RPN (AF, Swin-S) |  | 96.3 | 62.5 | 78.7 | 41.0 | 90.6 | 39.9 | 57.5 | 20.5 |  |
| Irshad et al. [] | NeRF-RPN |  | 96.3 | 69.9 | 85.2 | 59.9 | 89.2 | 42.9 | 55.5 | 18.4 |
| NeRF-MAE (F3D) | ✓ | 96.2 | 67.5 | 78.0 | 54.3 | 89.7 | 36.1 | 51.0 | 14.5 |  |
| NeRF-MAE (F3D, Aug) | ✓ | 96.3 | 74.3 | 83.0 | 59.1 | 90.5 | 39.1 | 54.3 | 15.5 |  |
| NeRF-MAE (F3D&HM3D&HS, Aug) | ✓ | 97.2 | 74.5 | 85.3 | 63.0 | 92.0 | 39.5 | 57.1 | 17.0 |  |
|  | Gaussian-Det (Ours) |  | 97.9 | 82.3 | 96.7 | 77.7 | 87.3 | 43.0 | 71.7 | 24.5 |

In Table 7, we report all the available officially-reported results of NeRF-RPN and NeRF-MAE for a thorough comparison with Gaussian-Det.
The checkmark indicates that NeRF-MAE employs self-supervised pre-training before finetuning on the downstream detection task.
F3D, HM3D and HS refer to the 3D-FRONT, HM3D and Hypersim datasets for self-supervised pre-training.
Aug. indicates augmentations done during pre-training.
Note that without pre-training on exterior databases, Gaussian-Det still outperforms variants of both NeRF-RPN and NeRF-MAE on most evaluation metrics.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: See the questions in the weakness section.
soundness: 2
strengths: The proposed method takes advantage of the Gaussian splatting representation for 3D object detection. Gaussian splatting is a more compact representation than the point clouds.
summary: This paper proposed a 3D object detection method based on the Gaussian splatting. With the Gaussian splatting as input, the proposed deep model extracts the 3D object bounding boxes. Experiments show the proposed methods give higher recall and precision than the competing methods.
weaknesses: - The paper combines several methods: (1) Official implementation of Gaussian splatting and (2) PointNet++. There is no new network structure
developed. The novelty is low.

- The overall network is not explicitly defined in the paper. please add this, otherwise it is impossible to check the details.

- The writing of the paper is quite unclear, especially the section 3.3. Theorem 1 is directly from the existing theorem. No need to include here. It is also confusing where this theorem is used anywhere in the paper. The equations are sloppy. Notations are a mess. The features
used two notations F and f. F^{cand} is not defined. f^{part} is also never defined anywhere in the paper. The equations from (6), (7), (11) need motivation and clarification.

- The experimental setting is not clearly defined. The authors used a dataset originally for 3D object detection in Nerf. It is not clear how the point cloud based methods, which need a point cloud input, are used in the experiment comparison.

- The qualitative examples of the proposed method show the quality of the 3D boxes is sometimes a lot worse than other competing methods. The boxes are often bigger and quite off. The competing methods also show many overlapping bounding boxes. No non-max suppression This may cause the lower numbers from the competing methods.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: To be honest, I do not have many issues with the current submission. 

In figure 6, maybe the authors could use a different color per object. 

How could a noisy estimate of the cameras affect overall performance?
soundness: 3
strengths: The paper is addressing a complex and important problem. 

The authors include most of the details to understand their proposal, being the technical section clear enough and well written. 

Gaussian-Det is simple and effective. To this end, the overall framework is divided into the construction of surfaces based on Gaussian representations, an object proposal initialization, as well as a partial surface feature inference and holistic surface closure coalescence. 

Both synthetic and real experiments are provided in the paper. The results are competitive with respect to several competing approaches.

The ablation study helps the reader.
summary: In this paper is presented a novel method for multi-view 3D object detection, exploiting Gaussian splatting to obtain a continuous scene representation. Then, to handle outliers in the previous representation, a closure inferring module is included that learns a probabilistic feature residual for partial surfaces and coalesces them into holistic representation on closure measurement. Experimental evaluation is reported on both synthetic and real datasets, including an ablation study and some comparisons with respect to competing approaches.
weaknesses: - Camera poses exploit a type of ground truth (point cloud) to be inferred. In my opinion, this is clearly a strong limitation of the paper. 

- Lack of challenging cases. In my opinion, some qualitative instances could be included in the paper, showing the effectiveness of the method in complex cases, especially, in comparison with other approaches.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: Improving readability and providing a visual comparison of the ablations would better present this work.
soundness: 3
strengths: - The proposed method show SOTA performance compared with previous NeRF based methods.
- Use 3DGS is faster than NeRF.
- The CIM module effectively handles outliers in Gaussian Splatting.
summary: This paper introduces a novel 3D object detection method, Gaussian-Det, which uses 3DGS to represent objects as continuous surfaces rather than NeRF discrete points. The method uses Gaussian as a surface descriptor and introduces the Closure Inference Module (CIM), which solves the outlier problem inherent in the 3DGS method by considering surface closure as a prior for objectness. The CIM operates in two phases: firstly, the probabilistic feature residuals are estimated for some of the surfaces, and then they are merged to form a holistic representation that measures the overall surface closure. The method is evaluated on synthetic (3D-FRONT) and real-world (ScanNet) datasets, and the results show a significant improvement over previous SOTA methods. The main innovations of the method are its surface-based representation and the use of surface closure as a geometric prior to improve the quality of detection.
weaknesses: - There is a lack of qualitative comparisons between different components, particularly regarding the use of Holistic Surface Coalescence and Residual Estimation. These comparisons are important for readers to better understand the effectiveness of each component.
- It would be interesting to analyze how the underlying Gaussian representation affects the results. For example, what is the impact of using the original 3DGS or a 2DGS representation?
- The overall writing is unclear and difficult to follow.
	- L.208 should be "From G^{cand}" 
	- There is multiple M in FIg. 2, are they the same?
	- Figure 3 is confusing. It is unclear what “open surfaces” and “closed surfaces” refer to. In Figure 3 (a) and (b), the primary visual difference appears to be the incompleteness of the Gaussians in (b). Please clarify the relationship between open and closed surfaces.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: See Weaknesses section.
soundness: 3
strengths: - Interesting solution using Gaussian-Splatting technologies for 3D object detection.
- Proposing Closure Inferring Module (CIM) to deal with the noisy nature of Gaussian-Splatting representations.
summary: The paper presents Gaussian-Det, a method to detect 3D objects via Gaussian Splatting. The proposed method leverages Gaussian Splatting as surface representation for multi-view based 3D object detection. Gaussian-Det proposes Closure Inferring Module (CIM) to deal with the outliers present using Gaussian-Splatting technologies. The paper presents experiments on both synthetic and real 3D-object detection datasets where Gaussian-Det mostly outperforms previous methods.
weaknesses: - Clarity of the paper is not good and needs improvement (my most concerning weakness). First, overall the high-level idea is not clearly explained. Although, Fig. 2 tries to do this, it falls short. Figure 2 lacks a clear illustration of the goal using the Backbone, and most importantly, fails to explain clearly how the CIM module (in my opinion the main contribution). Second, given the lack of clarity, it is hard to judge the novelty and quality of the solution. For example, I don't find the description of the Theorem if the high-level idea is not explained clearly. In my opinion it is just using space that could've been used to add Figures to explain CIM in a better way.

- Lack of details in the experiments. First, in line 320-321, the narrative says that the scenes are manually downsized into 159 usable rooms. However, it is not clear if the data was verified to ensure high quality. This is important since this data is used to measure quality.
 
- Insufficient experiments. In my opinion, showing results only on 2 datasets is not that convincing. Given that 3D object detection has been around for some time, it would've been more informative to know the performance of this proposed method on more real-scenes benchmarks/datasets (e.g., Waymo Open datasets, Objectron, etc.). I think that these datasets could've revealed the practical benefit of the proposed approach.

Meta-review: The proposed method, Gaussian-Det, uses Gaussian Splatting (3DGS) to represent the surfaces for multiview 3D object detection. Gaussian-Det formulates the input Gaussians as feature descriptors on partial surfaces. Outliers derived from 3DGS are reduced by the Closure Inferring Module, which estimates the probabilistic feature residuals for partial surfaces and combines them to produce 3D object proposals. 

The paper received mixed reviews, with scores of 3, 5, 6, and 6. After the rebuttal, one reviewer increased the rating, resulting in final scores of 3, 6, 6, and 6. It should be noted that the reviewer who gave the score of 3 did not provide any further responses during the rebuttal and reviewer discussion phases even though the authors adequately addressed the reviewer. Additionally, some comments from this reviewer contained apparent errors, which suggests a potentially low-quality review. Consequently, the area chair decided to disregard this review when making the final decision. Given that the other three reviewers provided positive feedback and consider the paper to be above the acceptance threshold, the area chair recommends accepting the paper.

Decision: Accept (Poster)