Gaussian-Det: Learning Closed-Surface Gaussians for 3D Object Detection
Authors: Hongru Yan
Yu Zheng
Yueqi Duan
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Gaussian Splatting
3D Object Detection
Surface Closure
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Skins wrapping around our bodies, leathers covering over the sofa, sheet metal coating the car – it suggests that objects are enclosed by a series of continuous surfaces, which provides us with informative geometry prior for objectness deduction. In this paper, we propose Gaussian-Det which leverages Gaussian Splatting as surface representation for multi-view based 3D object detection. Unlike existing monocular or NeRF-based methods which depict the objects via discrete positional data, Gaussian-Det models the objects in a continuous manner by formulating the input Gaussians as feature descriptors on a mass of partial surfaces. Furthermore, to address the numerous outliers inherently introduced by Gaussian splatting, we accordingly devise a Closure Inferring Module (CIM) for the comprehensive surface-based objectness deduction. CIM firstly estimates the probabilistic feature residuals for partial surfaces given the underdetermined nature of Gaussian Splatting, which are then coalesced into a holistic representation on the overall surface closure of the object proposal. In this way, the surface information Gaussian-Det exploits serves as the prior on the quality and reliability of objectness and the information basis of proposal refinement. Experiments on both synthetic and real-world datasets demonstrate that Gaussian-Det outperforms various existing approaches, in terms of both average precision and recall.

Published as a conference paper at ICLR 2025
GAUSSIAN-DET: LEARNING CLOSED-SURFACE GAUS- SIANS FOR 3D OBJECT DETECTION
Hongru Yan,, Yu Zheng", Yueqi Duan Tsinghua University yanhr21 @ma ils.iss.a s.. tss masunguua... edu. cn,
yu-zhenge ts inghuug edu.ch. Semnyueqietsingiiee edu.co
Gaussians (w./o. Surface, >150K)
Caussians (w/ Surface, <0K)
(a)
(b)
(c)
We leverage Gaussian Splatting as surface representation for multi-view based 3D object detection. Top Left: Gaussians with color and coordinate. Top Right: Gaussians with color, coordinate and surface information, but in a much fewer amount. Bottom (a-b): Object composed of Gaussians exhibits a textured surface as opposed to the top left illustration. Bottom (c-d): However, Gaussian splatting inherently introduces numerous outliers around the object.
Figure I:
ABSTRACT
Skins wrapping around our bodies, leathers covering over the sofa, sheet metal coating the car it suggests that objects are enclosed by series of continuous surfaces, which provides us with informative geometry prior for objectness deduc- tion. In this paper, we propose Gaussian-Der which leverages Gaussian Splatting surface representation for multi-view based 3D object detection. Unlike existing monocular or NeRF-based methods which depict the objects via discrete positional data, Gaussian- models the objects in continuous manner by formulating the input Gaussians as feature descriptors on a mass of partial surfaces. Further- more, address the numerous outliers inherently introduced by Gaussian splatting, we accordingly devise a Closure Inferring Module (CIM) for the comprehensive burecce-bsed objectness deduction. CIM firstly estimates the probabilistic feature residuals for partial surfaces given the underdetermined nature of Gaussian Splat- ting, which are then coalesced into holistic representation on the overall surface closure of the object proposal. In this way, the surface information Gaussian-Der exploits serves as the prior on the quality and reliability of objectness mation basis of proposal refinement. Experiments both synthetic and real-world datasets demonstrate that Gaussian-Ded outperforms various existing approaches, in terms of both average precision and recall.
the infor-
INTRODUCTION
3D object detection in indoor scenes (Dai et al. 2017: Hu et al. 2023 Rukhovich et al. 2022b
has been a popular topic in industry and academia, fueling various applications such robotic navigation and augmented reality. While many works (Qi et al. 2019 Liu et al. 2021 Cheng et al. 2022) have demonstrated the effectiveness on point cloud
2021: Rukhovich al. 2022a Ran et
"gqual Contributions.
Corresponding Author.

---

Published as a conference paper at ICLR 2025
Lower U
Hiihhr (
Tuge Oljen
Bouxday
Graziass Enclond Blooddng
More Closed Surfaces by Gaussaans
Openet Surfaces Formed Gauasians
Predkted Boanding Bocos
Accurato Detoction
Inaocuraee Detection
Figure 2: Illustration of the surface closure prior Gaussian-Det. For 3D Gaussians within bounding box, those forming relatively closed surfaces (lower ($|) indicate an accurate detection, and vice versa. Note that 3D Gaussians outside the bounding box are not shown for clarity.
data captured by expensive sensors such as lidars and depth cameras, RGB camera systems offer a cost-effective alternative. However, the absence of depth information poses major challenges for inferring the liceesiimenion geometries of the target objects from two-dimensional visual clues.
To overcome the aforementioned ill- po-posdddees in image-based 3D object detection, pure multi-view based approaches (Rukhovich et al. 20226 Tu al. 2023 projects the features the image plane into a 3D volume representation. However, such projective geometry is still dominated by 2D features which may lead to ambiguous detection results. On the other hand, recent advances Neural Radiance Field (NeRF) (Mildenhall et al., (020) enjoy multi-view consensus and consistency via learning plenoptic mapping function. Leveraging NeRF as the 3D scene representation, NeRF-RPN (Hu et al.) 2023) consumes volumetric information extracted from NeRF and directly outputs 3D bounding boxes through a general region proposal network (RPN). However, due to the implicit and continuous representation of NeRF, these methods are computationally expensive to optimize. More importantly, similar to monocular-based methods (Rukhovich et al., 2022b Tu et al., 2023). they depend on uniform and discrete sampling in 3D space which may lead to sub- sub-optimal performance by failing to capture the true objectness. In contrast, as indicated by the bottom (a-b) part of Figure [ 3D objects usually enclosed by series of continuous surfaces. Such surface information provides informative visual clues for objectness deduction (Ren and Sudderth 2018).
real- real-world
In this paper, we propose Gaussian-Det which exploits continuous surface representation for multi- view based 3D object detection. Gaussian-Det leverages the recent Gaussian Splatting (Kerbl et al., 2023) and formulates the input Gaussians into mass of descriptors on partial surfaces. As shown by top of Figure [ even with less than a third of the number, Gaussians utilizzng surface information better demonstrate the texture information, smoothness and continuity. While is feasible to directly employ point cloud based detectors, they can be distracted by numerous outliers inherently introduced by Gaussian Splatting process shown Bottom (c-d) part of Figure [ Moreover, the point-based detectors regard the input Gaussians as infinitesimal positional representations and discard the explicit depiction on continuous surface, which may lead to sub-optimal detection proposals. To address this, we design Closure Inferring Module (CIM) for the comprehensive surface-based objectness deduction in Gaussian- Det. As displayed in Figure 2 CIM is built upon the assumptive prior that, the partial surfaces in an accurately detected bounding box can form relatively closed surface measured by lower flux value. Specifically, taking the underdetermined nature of Gaussian Splatting into account, CIM firstly conducts the partial surface inference via estimating variational term probabilistic feature residual to the original representation of partial surfaces. CIM then coalesces them into a holistic representation and uses the flux value measures the closure of the overall proposal, serving as the prior on the quality and reliability of objectness prediction and the information basis of proposal refinement. To validate the effectiveness of the proposed Gaussian- experiments on both synthetic 3D-FRONT (Fu et al.] (021) and real-world ScanNet (Dai et al.) 2017) datasets. The experimental results demonstrate that Gaussian- outperforms various existing approaches in terms of both average precision and recall.
we have conducted
2 RELATED WORK
Neural Scene Representation: In recent years, neural scene representation have presented aston- ishing performances on visual computing and geometry learning. They usually take a coordinate as input and output corresponding scene properties, such signed distance (Park et al.) 2019: Chabra eet al. 2020). occupancy (Mescheder et al. 2019), color and density. Notably, in the scenario of multi-view geometry reconstruction, the Neural Radiance Field (NeRF) (Mildenhall et al. 2020
2

---

Published as a conference paper at ICLR 2025
learns a plenoptic mapping from the querying coordinate and viewing direction into the point-wise radiation and volume density. The NeRF-based scene modeling has inspired numerous extended
works spanning from low-level geometry reconstruction (Barron al. [222]] Fridovich-Kell et al.
2022
Wang et al., (021) to high-level visual understanding (Zhi et al., 2021 Jeong et al. 2022 Hu
al.
2023:
Xu et al.. 2023 Liu et al. 2023 TTian et al. 2024 Irshad al., 2024). Since NeRF
requires a massive amount of queries during both training and inference phases, some improved
representations are introduced as proxies such as hash grids (Müller et al. 2022), tri-planes (Chen
et al. 2022) and points (Xu et al. 2022a). Among these advancements, the 3D Gaussian Splatting
GS) (Kerbl et al. (023) stands out for its balance between quality and efficiency. 3D-GS employs set of anisotropic Gaussians initialized by Structure from Motion (SfM), whose amount and proper- ties are iteratively updated during training. Apart from the positional information as point cloud based or NeRF-based representation, a 3D Gaussian naturally depicts a continuous surface through its mean, covariance parameters. Based on 3D-GS, several works have explored on novel view syn- thesis Charatan et al. (2024); Chen et al. (2024b); Zhang et al. (2024). 3D content generation (Tang 2024), surface reconstruction (Guédon and Lepetit 2024), semantic segmentation (Shi et al. and editing (Chen et al. (024a)).
et al.
2024
Indoor 3D Object Detection: Indoor 3D object detection has drawn extensive attention owing to its broad applications in robotic vision, augmented reality, etc. The techniques can be categorized according to the input modalities, such as point cloud, gridded voxels and 2D image(s). For example, (Song and Xiao 2016) uses volumetric CNN to create 3D Region Proposal Network (RPN) on a voxelized 3D scene. VoteNet (Qi et al. 2019) directly operates on point cloud by using a PointNet++ (Qi et al. 2017b) backbone and center voting strategy. The follow-up point-based approaches include the VoteNet-style (Cheng et al. 202T; Liu et al. 2021 Zheng et al. 2024) and the sparse convolution based methods (Rukhovich et al. [2022a] Wang et al. (002). However, the reliance on expensive lidars or depth sensors limits their applicability. In contrast, the RGB camera systems offer a cost-effective alternative. To overcome the inherent ill-posedness in image-based 3D object detection, Im VoxeINet (Rukhovich et al. 2022b) lifts the 2D features into 3D volume representation followed by a FCOS (Tian et al.) 2019) styll detection head. ImGeoNet [Tu et al. (002) introduces an image-induced geometry-aware voxel representation for better preservation of geometry. However, the projective geometry extracted by (Rukhovich et al. 20226) Tu et al. 2023) still originates from 2D features and may thus result in ambiguous detections. On the other hand, several advances (Hu)
let al,, [023] Xu et al.] 2023; Irshad et al., (024) seek to leverage the multi-view consensus and consistency of NeRF representation (Mildenhall et al.) 2020) that embeds the geometry into the voxel grids. NeRF-RPN consumes the volumetric features sampled from NeRF and then directly regresses the bounding boxes via a generic RPN. NeRF-Det learns a two-branch framework that simultancously conducts geometry reconstruction and bounding box regression. These two approaches rely on uniform and discrete sampling which may fail capture the true objectness in 3D scenes. In contrast, our proposed Gaussian-Da Det strives for representing the targets in continuous manner by leveraging the descriptors on partial surfaces and coalescing them into a holistic measurement which serves as a comprehensive prior on the reliability of the deducted objectness.
APPROACH
In this section, we elaborated the proposed Gaussian-Det by dividing the framework into the construc- tion of surface-based Gaussian representation, object proposal initialization, partial surface feature inference and holistic surface closure coalescence. The overall framework is illustrated in Figure]
3.1 PRELIMINARY: 3D GAUSSIAN SPLATTING
The input scene of our proposed Gaussian-Det is represented by 3D Gaussian Splatting (3D- GS) (Kerbl et al. 2023) that are reconstructed from posed images, comprising set of 3D Gaussian primitives G = gg, (Mpp Ep, ap,, Hp Each 3D Gaussian gp is parameterized by mean vector Up E R indicating its center, an anisotropic covariance matrix Σ, E 1R333, the opacity value ap E IR and color Cp E R23 Σp is positive semidefinite matrix and can be factorized as Σp R.S.S.ST S.R.R. Rp E max3 and Sp R3x3 refer to a diagonal size matrix diag(s), s2 s²) and a rotation matrix converted from a unit quatermion respectively. Subsequently, the 3D Gaussians are projected onto the image plane using known camera matrices. a-blending is executed to aggregate the overlapping
3

---

Published as a conference paper at ICLR 2025
Surfoox-boseed Goussion Repressatotion
Preponol Iniholigetion
Mults-viww Imsages
(3), 4p)
Condidare Gamlass
Taitial
Gassdan Sylinti
Sarface-onnd
Backbans
Fanmatation
Gi> 1008)
Garl
Goml Frnny
Proposals
Offie
Residual Estimation
Protaistiatic Roiddal
Xy Rg
Ground Tiuuth Reference
Hicher 19y7
CEW: Portial Surface Inference
CLA Holisthc Surfoce Cleaure Crolescence
Figure 3: From the original Gaussians G, we formulate surface-based Gaussian representation Gsurf, which is firstly used predict the initial object proposal with opener surfaces. Then the closure inferring module, which contains partial surface feature inference and holistic surface closure coalescence. CIM learns informative prior on the quality of the predicted objectness, thus controlling the support or suppression (8) of the holistic object representation. The partial and holistic representations are combined estimate the refined detection result, whose degree of surface closure is similar to that of ground truth measured by the absolute flux value [xx]-
Gaussians each pixel and compute the final color:
i-1 C Σ caa (1-)
(I)
j=1
where is the number of overlapping Gaussians. The attributes of g are optimized through backward propagation of the gradient flow. Particularly, the density (i.e., P) of 3D Gaussians is also updated during training via cloning and splitting for the sake of rendering quality.
3.2 SURFACE-BASED GAUSSIAN REPRESENTATION AND PROPOSAL INITIALIZATION
Compared with NeRF- based (Mildenhall al. 2020) representation, 3D Gaussian Splatting enjoys the real-time rendering speed owing to a specially designed rasterizer. More importantly, 3D Gaussian g, approximates the shape of bell curve with depiction on continuous surface (especially for flat Gaussian (Guédon and Lepetit 2024)), ), as apposed to the discrete representation of infinitesimal positional data such as point cloud (Rukhovich et al. 2022a) and uniformly sampled volumes (Rukhovich et al. 2022b, Xu et al. 2023 Hu et al.) 2023). To this end, we concentrate the Gaussian representation onto surface-related information by formulating:
Gourf = -gg:// = (xp, Dp, Ap)} E
where Xp Mp, np and Ap specify the information about location, perpendicular orientation and area respectively. The normal vector n, is determined by solving the eigenvalue problem Σ, Anp- Specifically, n, is the eigenvector corresponding the smallest eigenvalue. The in-ward or out-ward orientation is determined by setting the orientation of n, to align with the orientation from the center of the object proposal to the Gaussian mean. Please refer to the Appendix pages for the derivative process. Ap is defined as the largest cross-section area of 3D Gaussian Ellipsoid:
Ap min( {s), $2 8
(3)
The reformulated Gaussian representation Gaaff initializes the depiction on multiple partial sur- faces. Given the huge amount of Gaussians in Gsurf, we use an off-the-shelf backbone B(-) )) for simultaneously trimming them into candidate subset and extracting their corresponding features:
Gond Frand = B(G
where Grand [gi. 18..gg...... .., gM E Mx7 a subset of G Gaurf and Frand (f, f2 fe...., fmt e M denotes the corresponding feature in the latent space. From Geond, then accordingly
4

---

Published as a conference paper at ICLR 2025
initialize the partial-aware object proposals Pnant
{p1 P2..... Px by employing VoteNet (Qi
et al. -tylee head (Cheng et al., 2021) which groups Grand in Euclidean space and predicts the bounding boxes with MLP-basce architecture
Qi et al. 2017a). Meanwhile, the partial-aware
features of each proposal is extracted:
fgart MMpart (Feand pp)
(5)
where Mpprt is querying-anc caiiing-ndd-oooolin function (Cheng al. 2021 that operates around the location of each proposal.
As mentioned above, it is feasible to directly employ point cloud based detectors (Qi et al. 2019 Rukhovich et al. 2022a; Liu et al. 2021) and regard Geurs as mass of point cloud concatenated with the corresponding feature. However, as shown in Figure T there still severely exists Gaussian outliers around the object surfaces that may distract the point-based detectors. We also demon- strate Experiment section that this leads to incorrect objectness deduction and inferior detection performances evaluated on Pport.
To address this, we devise a Closure Inference Module (CIM) in Gaussian-Det responsible for comprehensive surface-based objectness deduction in Gaussian- which takes the holistic closure measurement into account to refine the object proposals.
3.3 CL.OSURE INFERENCE MODULE FOR PROPOSAL. REFINEMENT
As illustrated in the bottom part of Figure 3] the Closure Inference Module (CIM) consists of two stages: Given the underdetermined nature 3D Gaussian Splatting. CIM firstly enhances the representation of partial surfaces via variationally estimating the probabilistic feature residuals. Subsequently, under the assumption that the higher degree of surface closure indicates a more accurate detection (and vice versa), CIM seeks holistic surface closure coalescence, serving an informative prior on the quality of the predicted objectness.
Partial Surface Inference: In Gaussian Splatting, inferring local surface features for spatial points is underdetermined due to the supervision from 2D signals, leading to unstructured outliers (Guédon) and Lepetit 2024) with unreliable local surface descriptors. To this end, we model such randomness in CIM by incorporating a variational term FV into the candidate features Frand establishing probabilistic feature residual:
Foand & Frand + taFV
(6)
where a is weight hyper-parameter (in practice we set 0.1). The randomness is modeled by assuming FV can be sampled from (F(F") = N (FV, 0,11). To fit the posterior log (JV [Feandd and likelihood pa [peand|]F" parameterized by ø and respectively, the following evidence lower bound (ELBO) is optimized for the residual DiiimationKKingnnn and Welling 2014):
Cres F (Feand) = [ plogea/olopppppppepppppp p( Feaand FV ]] (L(g(FV [Fvand) (Ip(F()) (7)
By modeling the variational term, CIM explicitly represents this uncertainty, thereby better describing the distribution of local surface features.
The partial surfaces are then coalesced into a holistic representation by CIM. Specifically, illustrated in Figure 2 CIM leverages key geometric property that the partial surfaces corresponding to an object can approximate a closed surface. As formulated by Theorem (see Appendix for proof), this property can be mathematically expressed that the net flux & of a constant vector field through a closed surface is zero.
Holistic Surface Closure Coalescence:
Theorem 1. Given a constant vector field T and a closed surface S, the flux & of the vector field T through the closed surface S, also expressed as the surface integral of the T over S is zero:
+ = # T nds =
(
In CIM, we utilize this theorem by computing the flux of a pre-set constant vector field over the partial surfaces enclosed by an object proposal. The closer the total flux is to zero, the better the surfaces form a closed object, suggesting higher-quality detection. To practically implement this concept for the k-thh object proposal, CIM computes the estimated flux & numerically using quadrature:
5

---

Published as a conference paper at ICLR 2025
- = Σ T-n,A, where G
{g: Ix} E RNXT
(9)
where Iş is set containing the indices of the candidate Gaussians within the proposal Pk. N₂ is the number of candidate Gaussians within ps- Without loss of generality, we set as V3' v3 v3).
With the estimated flux as the quantitative measurement of surface closure, CIM then formulate the geometrically coalesced representation as:
Fhol Sk .Mhal (Feand p&), where S exp(-7x (x)
(10)
where Frandd is the enhanced feature with the probabilistic residual in Eqn. 66) Yh is flexibility coefficient mapped from FV through a MLP and sigmoid activation function. Mhhal is another querying-and and-anddooooo moo-looinng function sharing the same architecture with M Sy serves as the closure
weight parameter which controls the support @ k|| 4, Sk 1) or suppression (|0 I t. Ss 4) rhot.
We finally formulate the refined proposal feature which comprehensively combines the information on partial-aware surfaces and holistic closure-based measurement on objectness as:
& final
Ippart 1|
(11)
where [-I|-] denotes the concatenation along the feature dimension. & ffnaal used for proposal refinement by estimating a parameter offset the original proposals through stacked MLP:
Pk
refened = p + MMP
(12)
refined the overall loss function is computed as: Given the final refined object proposals Ps:
[(ipk
gt }. (pkt. (pk
refined)) AresCres + X2 {pph)
(13)
+ >re C2 (pt {pk (prefined))
where (e%') and Le denote the ground truth bounding boxes and L. norm loss function. Aress Apread and Arefine are pre-set weight hyper-parameter.
EXPERIMENTS
We validate the proposed Gaussian-Det by conducting benchmark experiments on both synthetic and real-world datasets on 3D object detection. We then validate the design choices via ablation studies.
4.1 EXPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
We experimented Gaussian-Dee on the officially benchmark by (Hu al. 2023 which contain both the synthetic 3D-FRONT (Fu et al. 2021) and the real-world ScanNet (Dai et al. 2017)
datasets. We faithfully followed (Hu et al. (202)) to take the posed images input. The camera poses in the datasets released by (Hu et al. 2023) were estimated via Structure from Motion (SfM) which used the groundtruth point cloud as initialization.
The synthetic 3D-FRONT (Fu et al. 2021) dataset is featured with its large-scale room layouts and textured furniture models. The dataset contains 159 usable rooms that are manually selected, cleaned and rendered by (Hu et al. 2023). The indoor scenes are densely annotated with oriented bounding boxes. We follow (Hu al. (223) remove the annotations of construction objects such as ceilings and floors, and merge the components that belong to the same object instance into whole. Please refer to the Appendix pages for illustrative examples of this dataset.
The real-world ScanNet (Dai et al. 2017) dataset consists of complex and challenging scenes captured by Structure Sensor in an ego-centric manner. We followed (Hu et al. 2023) to uniformly divide the frames into 100 bins and choose the sharpest one in each bin measured by Laplacian variance. Experiments were done on the 90 scenes that are randomly selected by (Hu et al. 2023).
For the input Gaussians G, used the official implementation of 3D-GS (Kerbl al. 2023) equipped with the SuGaR regularization (Guédon and Lepetit| 2024).. which takes 30,000 training iterations. We excluded the Gaussians outside the room cuboid defined by (Hu et al. 2023) and filtered out

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative results on the 3D-FRONT and ScanNet datasets. t: we report the results of NeRF-MAE which involves the pre-training 3D-FRONT and two other exterior databases, including HM3D (Ramakrishnan et al. 2021) and Hypersim (Roberts et al. 2021).
3D-FRONT
ScanNet
Methods
AR25
ARgo
AP
AP
AR
ARgo
AP23
AP
Im VoxelNet Rukhovich et al
2022b
88.3
71.5
86.1
66.4
51.7
20.2
37.3
9.8
NeRF-RPN Hu et al. 2023
96.3
69.9
85.2
59,9
89.2
42,9
55.5
18.4
DeRF-MAE) IIshad et al. 2024
97.2
74.5
85.3
63.0
92.0
39.5
57.1
17.0
G- VoteNet (Qi
al. 2019
81.5
61.6
73.0
49.6
78.5
34.2
66.8
18.2
G- GroupFrre (Lau al. 2021
84.9
63.7
72.1
45.1
75.2
37.6
60.1
20.4
G- G-FCAF3D (uuhhvvihh et 2022a
89.1
56.9
73.1
35,2
90.2
42.4
63.7
18.5
G-BRNet (u et 20226
89.7
75.3
88.2
71.0
71.1
32.2
63.1
19.3
Gaussian- Det (Ours)
97.9
82.3
96.7
77.7
87.3
43.0
71.7
24.5
G-FCAF3D Figure 4: Qualitative Results on 3D-FRONT (top two rows) and ScanNet (bottom two rows). We visualize each bounding box with a unique color for clear illustration.
2D Reference
Ground Truth
Ours (GaussianDeet)
NeRF-RPN
Gaussians with opacity value lower than 0.3. Note that such combined implementation (Kerbl et al., 2023 Guédon and Lepetit 2024) aims at decent 2D rendering results. The issue of outlier Gaussians around the 3D objects still severely exists as shown in Figure [ et al., 2017b) the backbone B. We used single NVIDIA RTX A6000 48GB GPU for both training and evaluation. The total training epochs detection is 50 for both 3D-FRONT and ScanNet. The evaluation metrics are mainly average recall (AR) and average precision (AP), thresholded by loU value at both 0.25 and 0.5. Please refer to the Appendix pages for the details on model hyper-parameters.
employed PointNet++ (Qi
From the statistical results (see Appendix) of our reconstructed 3D Gaussians two datasets, we observe that the majority of flux values are clustered near zero. This indicates that most objects contain higher degree of surface closure and our assumption is applicable to the employed datasets.
4.2
MAIN RESULTS
We have summarized the quantitative results on 3D-FRONT and ScanNet in T For the comparison with the pure multi view based methods, we follow (Irshad al. 2024 to include Im VoxelNet (Rukhovich et al. 2022b). NeRF-RPN (Hu et al. 2023) and NeRF-MAE (Irshad]
Compared Methods:
ct al. 2024 in Table Im VoxelNet directly extracts 3D volumetric features projected from 2D image planes. The NeRF-based methods (Hu et al.) 2023 Irshad et al.) 2024 apply 3D convolutions
the uniformly sampled points in the radiance fields. Note that NeRF-MAE includes the pre-training extra datasets. Moreover, to demonstrate the effectiveness of Gaussian Det more intuitively, experimented on the point cloud based detectors (Qi et al., 2019 Liu et al. 2021 Rukhovich et al.) 2022al Cheng et al. 2021) (prefixed by "G-") which merely contains the backbone and detection
7

---

Published as a conference paper at ICLR 2025
OuEs (Gaussian- Det)
G-FCAR3D
Figure 6: Illustration on the detected objects and the surface closure calculated from the 3D Gaussians within each estimated bounding boxes, measured by the absolute flux values (in square decimetre).
The upper part of Table 1 are cited from (Irshad al. 2024), for which we have provided a more in-depth comparison in the Appendix pages.
heads in Section 3.2
Analysis on Quantitative Results:
The observation from Table D is two-fold. Firstly, the point cloud based methods are able to achieve comparable performances compared with previous multi-
view (Rukhovich et al., 2022b) or state-of-the-at NeRF-based (Irshad et al.) 2024 Hu et al. (023) approaches. This suggests the potential of Gaussian Splatting 3D object detection. Secondly, except for AR thresholded by 0.25 on ScanNet, we can see that Gaussian- precedes all the compared methods on all evaluation metrics, especially on AP which is more widely-used in 3D object detection. For example, on 3D-FRONT evaluated by AP25, Gaussian- Det outperforms the second-best G-BRNet (Cheng et al. 2021) by a significant margin of +8.5%. Although ScanNet is challenging dataset for its lower image quality and diverse layouts, our Gaussian-Der still achieves the state-of-the-art average precisions (+4.9% on AP25 and +4. 1% on AP This indicates that Gaussian-Det better leverages the surface-based information inherently condensed in Gaussians. By coalescing partial surfaces into object-wise and closure-aware holistic representations, Caussian-Det better perceives the quality and reliability of the object proposals, thereby providing favourable information basis for proposal refinement.
explicit input modality for
Qualitative Results: As for illustrative comparison, we firstly display the qualitative comparisons in Figure 4 where we set the 2D rendered results of the input Gaussians visual reference. We can see that compared with ground truth bounding boxes, Gaussian- Det is capable of preventing redundant object predictions that are potentially caused by outliers, leading higher detection accuracy. Figure 6 provides a more in-depth visualization on the discriminative objectness deduction of Gaussian-Des For each detected bounding box, we have marked the corresponding flux value computed in Eqn. (9) that measures the surface closure. Predictions whose absolute flux values are larger or less than one are marked with blue or red respectively. We can see that in the presented examples, favourable detection well corresponds to relatively higher degree of closure (lower absolute value of flux). With the holistic surface coalescence by CIM, Gaussian reliability of proposals and able to refine them for higher accuracy. Please refer to the Appendix pages for more qualitative results under challenging scenarios.
aware of the
We then present the failure case by Gaussian-Det in Figure [] Despite focus- ing on more informative surface representations, Gaussian-Dee Det may still struggle at handling de- tecting the object with severe occlusion, which is an inherent challenge for current multi-view based 3D object detectors. Moreover, since the surfaces of the objectness that contains jointed objects can also be regarded as closed, Gaussian-
Failure Case:
Ous (Gaussian Del) Figure 5: Failure case on occluded objects. Det may mistake them whole. Please refer to the Appendix for more failure cases.
Ground Truth
4.3 ABLATION STUDIES AND ANALYSIS
To evaluate the effectiveness of the design choices and the components in our framework, we conducted five groups of ablational experiments on the 3D-FRONT dataset. The performances evaluated by average precision (AP) are summarized in Table
(1)-(2) Discarded or Naive Holistic Surface Coalescence: The trial (1) aims at experimenting with information on partial surfaces only. This is achieved by concatenating two copies of epart in Eqn. (1T). In trial (2), the partial surfaces are coalesced without any consideration the closure
8

---

Published as a conference paper at ICLR 2025
properties of objects. This is achieved by setting the closure score S₃ =1 for all object proposals. By comparing (1), (2) and (6), we can see that the coalescence into holistic surface representation is necessary (+0.7% AP25 and +1.1% % 50 despite its naive implementation in (2). The closure- based measurement augments the proposal representation with prior on the reliability information, which furtherly enhances the detection accuracy sizably and is reflected by the performance gains (+3.2% on AP25 and +2.7% on AP5o).
(3) Deterministic Partial Sur- face Representation: Gaussian-Det, the feature repre-
Table 2: Quantitative results of ablation studies on 3D-FRONT.
In
AP25
APso
sentation of the trimmed Gaus-
(1) Discard the Holistic Surface Coalescence
92.8
73.9
sian candidates are supplemented with probabilistic residual in
(2) Naive Holistic Surface Coalescence
93.5
75.0
(3) Deterministic Partial Surface Reprexentation
95.7
74.7
Eqn. (6). Here we ignored the
(4) No Flexihility in Closure Measurement
96.4
76.2
variational inference of Feand by
(5) Alternative Surface Orientation
96.2
75.7
setting Aves O. We can see that
(6) The Full Framework
96.7
77.7
the deterministic partial surface representation in (3) hampered by the under-determined nature of
Gaussian Splatting, which leads to inferior performances (-1.0% on AP25 and -3.0% on 5P
(4) No Flexibility in Closure Measurement: The trial cancels the allowance for the potential truncation in indoor scenes. We empirically set ÿ = 0.5 for all object proposals. It can be seen that despite the failure case in Figure 5 learnable 7 endows the closure measurement with the consideration on flexible cases such as truncations and occlusions. Improvements on both AP25 (+0.3%) and AP50 (+1.5%) can be observed.
(5) Alternative Surface Orientation: By default we set the np of a partial surface by solving the eigenvalue problem and determine its in-ward or out-ward by setting the orientation of n, to align with the orientation from the center of the object proposal to the Gaussian mean. An alternative way is to determine orientation following (Ran et al.) 2022). We experimented on two different implementation of closure scores. As it influences the serface-based Gaussian representation well as the closure computation, a sizable drops on AP (-2.0%) can be observed in Table 2
Please refer to the Appendix pages for additional ablational experiments, including the choices of hyper-parameters, Gaussian Splatting representation and the comparison on learning curves before and after incorporating the probabilistic feature residual.
Model Efficiency: As shown in Table Gaussian- DDta exhibits remarkable inference speed at 33.3 FPS, which is an order of magnitude faster than both NeRF-RPN and NeRF-MAE. Additionally, our model size substantially smaller 24.0MB, less than 8% of NeRF-RPN size and 4% of NeRF-MAE. These results indicate that our Gaussian-Det demonstrates significant improvements in detection accuracy while maintaining real-time processing speed and favourable model compactness.
Detecting with Noised Poses: To furtherly validate on inferring the objectness from low-quality data, we experimented on adding normally distributed perturbations with the mean of 0 and variance of 0.07 to camera extrinsics when reconstructing them. The corresponding quantitative results are shown in Table 5 We can see that with noised poses, Gaussian- DDet showcases less precision decline on both AP and AR, which verifies the effectiveness of Gaussian-Des against either inherent outlier 3D Gaussians low-quality input poses. Please refer to the Appendix for qualitative comparisons.
Efficacy on 3D Instance Segmentation: We have further validated the generality of our proposed method on the task of open-vocabulle
3D instance segmentation. On the Gaussian- Grrupinng baseline YYe et al., 2024). we incorporated the estimation of surface closure as prior the reliability the object instance, thus supporting or suppressing the segmented results. From the results in Table [6] we can that leveraging the surface closure provides informative prior objectness deduction and largely enhances the performances of instance segmentation in 3D scenes. The rendering results of qualitative examples containing multiple object instances are shown in Figure 7 It can be seen that the incorporation of the prior of surface closure contributes to fewer noisy predictions off the table. Moreover, it substantially speeds up the training period and saves the GPU memory footprint (see Appendix) compared with the baseline method.
Comparison with Point-based Partial Surface Modeling: We note that RepSurf (Ran et al., 2022). point cloud based convolutional operator, also considers the surface structure within local regions.

---

Published as a conference paper at ICLR 2025
Table Comparison on model efficiency.
Table Comparison with RepSurf.
Method
FPS
Model Size
Method
AP25
APso
NeRF-RPN (Hu et al. 2023
1.51
312.7MB
G-BRNet (Xu et al. 2022b
63.1
19.3
NcRF-MAE (Irhhad et a.. 2024
1.11
604.2MB
-ReeSurf-U U (an el al 2022
47.7
19.7
Gaussian-Det (Ours)
33.3
24.0MB
Gaussian Det (Ours)
71.7
24.5
Table 5: Comparison under the setting of adding noises to camera poses.
Table 6: Comparison on 3D instance seg- mentation.
Methouss
Notsod Poses
AR2s
APas
Methol
nBBLUU
G-BRNer
Cheng al.
2021
89.7
88.2 67.6 (-20.6)
DEVA
LERE
Chere of RUAS
33.5
46.2
45.1
30.6
BRNNe Chong
(-15.4)
SAID
24.9
2338
Gousian-De Det (Ours)
97.9
96.7
32.8
Goussian Det (Ours)
86.8 (-11.)
$1.5
Guusiaa Groqui
76.5
(154)
2D Reference
Ground Truth
Gaussian-Grouping
Gausstaa-Groupin--Our
Figure 7: Rendered qualitative results of open-vocabulary 3D instance segmentation.
We accordingly replaced the backbone of G-BRNet (Cheng et al. (021) with the umbrella variant of RepSurf. The results on ScanNet are shown in Table RepSurf prone to be severely distracted by the outlier Gaussians since it only considers the very local surfaces, which harms the detection quality drastically (-15.34% on AP25). Secondly, merely focusing on the very local (umbrella-shaped in this case) surface while ignoring the holistic surface modeling leads to sub-optimal results (-4.8% on AP This indicates that the comprehensive surface modeling strategy better captures the intricate geometry of objects, thereby contributing to sizably improved 3D detection accuracy. Such awareness of holistic surface representation is essential to prevent the distraction by outliers.
5 CONCLUSION
In this paper, we present Gaussian- Det for multi-view 3D object detection. Gaussian- -Det leverages Gaussian Splatting and exploits the continuous surface representation. To overcome the numerous outliers inherently introduced by the data modality, we devise a Closure Inferring Module (CIM) for comprehensive surface-bace based objectness deduction. CIM learns the probabilistic feature residual for partial surfaces and coalesces them into holistic representation on closure measurement, thereby serv- ing as prior on proposal reliability. Experimental results on both synthetic and real-world datasets validate the superiority of our Gaussian- over various existing mult-view based approaches.
Broader Impact: The proposed method utilizes continuous surface representations to enhance the applications like augmented reality and robotic navigation. While it offers cost-effective solutions via the use of RGB cameras instead of expensive lidar systems, it is largely affected by the quality of input data and may struggle handling occluded objects. It also raises privacy concerns due to the demand for surveillance devices. To mitigate these issues, the adoption of ethical guidelines is crucial to ensure responsible use and maintain the reliability of the technology.
Gaussian-Del may be struggling to handle occlusion scenes (as shown in Figure 5). which an inherent challenge in the setting multi-view 3D object detection. Specifically, the detectors are unaware of the appearances and detailed geometries of the occluded part, thereby further worsening the inherent ill-posedness of 3D perception given 2D images.
Limitations:
Future Works: Since the degree of surface closure of the objectness that contains joint objects also relatively high, such scenarios might be challenging for Gaussian-DD -Det. Future work can incorporate the color information to discriminate the objects and calculate the flux values separately. In addition, the exploration of large-scale datasets on autonomous driving is a promising practice.
Acknowledgment: This work supported in part by the National Natural Science Foundation of China under Grant 62206147, and in part by the China Postdoctoral Science Foundation under Grant 2024M761674.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Barron, B. Mildenhall, M. Tancik, Hedman, R. Martin-Brualll, and Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 5855-5864, 2021.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. T-PAMI, (5(8)):198-1888, 2013.
J. Cen, Z. Zhou, J. Fang, W. Shen, L. Xie, D. Jiang, X. Zhang, Q. Tian, et al. Segment anything in 3d with nerfs. NeurIPS, 36:25971 2023.
R. Chabra, J. E. Lenssen, E. 1lg, T. Schmidt,, J. Straub, S. Lovegrove, and R. Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In ECCV, pages 608-625, 2020.
D. Charatan, Li, Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR, 2024.
A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. ECCV, pages 333-350, 2022.
Y. Chen, Z. Chen, C. Zhang, F. Wang, X. Yang, Y. Wang, Z. Cai, L. Yang, H. Liu, and G. Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting, In CVPR, pages 21476-21444 21485, 2024a.
Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, Geiger, T.-J. Cham, and Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In ECCV, pages 370-386, 2024b.
B. Cheng. L. Sheng. S. Shi, M. Yang, and D. Xu. Back-tracing representative points for voting-based 3d object detection in point clouds. In CVPR, pages 8963-8972, 2021.
H. K. Cheng. S. W. Oh, B. Price, A. Schwing, and J.-Y. Lee. Tracking anything with decoupled video segmentation. In ICCV, pages 1316-1326, 2023.
A. Dai, A.X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotaal
3d reconstructions of indoor scenes. CVPR, pages 2432-2443, 2017.
Fridovich-Kan Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, pages 5501-5510, 2022.
H. Fu, B. Cai, Gao, L.-X. Zhang, J. Wang. C. Li, Q. Zeng. Sun, R. Jia, Zhao, al. 3d-front:
3d furnished rooms with layouts and semantics. In ICCV, pages 10933-10942, 2021.
A. Guédon and V. Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruc- tion and high-quality mesh rendering. In CVPR, pages 5354-5363, 2024.
B. Hu, J. Huang, Y. Liu, Y.W. Tai, and C.-K. Tang. Nerf-rpn: A general framework for object detection nerfs. In CVPR, pages
2023.
B. Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH, pages 1-11, 2024.
M. Z. Irshad, S. Zakharov, V. Guizilini, A. Gaidon, Z. Kira, and R. Ambrus. Nerf-mae: Masked autoencoders for self-supervised 3d representation learning for neural radiance fields. In ECCV, 2024.
Y. Jeong, S. Shin, J. Lee, C. Choy, A. Anandkumar, M. Cho, and J. Park. Perfception: Perception using radiance fields. NeurlPS, pages 26105-26121, 2022.
B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. 7OG, 42(4):1-14, 2023.
Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. In ICCV, pages 19729-19739, 2023.
11

---

Published as a conference paper at ICLR 2025
D. P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR, 2014.
Liu, C. Zhang, Y. Zheng, and Y. Duan. Semantic ray: Learning a generalizable semantic field with cross-reprojection attention. In CVPR, pages 17386-17396, 2023.
Z. Liu, Z. Zhang, Y. Cao, H. Hu, and X. Tong. Group-free 3d object detection via transformers, In ICCV, pages 2949-2958, 2021.
L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, pages 4460 04000-470, 2019.
B. Mildenhall, P. P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Repre- senting scenes neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020.
Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. TOG, 41(4):1-15, 2022.
J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, pages 165-174, 2019.
C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation.
CVPR, pages 652-660, 2017a.
C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In NeurIPS, pages 5099-5108, 2017b.
C. R. Qi, O. Litany, K. Hc, and L. Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, pages 9277-9286, 2019.
M. Qin, W. Li, Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. In CVPR. pages 20051-20060, 2024.
S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg. J. M. Turner, E. Under- sander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. matttat-matterpot 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In NeurlPS, 2021.
H. Ran, J. Liu, and C. Wang. Surface representation for point clouds. In CVPR, pages 18942-18952, 2022.
Z. Ren and E. B. Sudderth. 3d object detection with latent support surfaces. In CVPR, pages 937-946, 2018.
M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind. Hypersim: photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, pages 10912-10922, 2021.
D. Rukhovich, A. Vorontsova, and Konushin. Feaf3d: fully convolutional anchor-free 3d object detection. In ECCV, pages 477-493, 2022a.
D. Rukhovich, A. Vorontsova, and A. Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. WACV, pages 2397-2406, 2022b.
J.-C. Shi, M. Wang, H.-B. Duan, and S.-H. Guan. Language embedded 3d gaussians for open- vocabulary scene understanding, In CVPR, 2024.
S. Song and J. Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In CVPR, pages 808-816, 2016.
J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
Tian, Y. Duan, A. Wang, J. Guo, and Du. Semantic Flow: Learning semantic fields of dynamic scenes from monocular videos. In ICLR, 2024.
12

---

Published as a conference paper at ICLR 2025
Z. Tian, C. Shen, H. Chen, and T. He. Fcos: Fully convolutional one-stage object detection. In ICCV, pages 9627-9636, 2019.
T. Tu, S.-P. Chuang, Y.-L. Liu, C. Sun, K. Zhang, D. Roy, C.-H. Kuo, and M. Sun. Imgeonet: Image-induced geometry-aware voxel representation for multi-view 3d object detection. In ICCV, pages 6996-7007, 2023.
H. Wang, S. Dong, S. Shi, A. J. J. Li, Z. Li, L. Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. In NeurlPS, pages 29975 29975-29988, 2022.
Wang. L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurlPS, pages 27171-27183, 2021.
C. Xu, B. Wu. Hou, S. Tsai, R. J. Wang, W. Zhan, Z. He, P. Vajda, K. Keutzer, and M. Tomizuka. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. In ICCV, pages 23320-23330, 2023.
Q Xu, Z. Xu, J. Philip. S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann. Point-nerf: Point-based neural radiance fields. CVPR, pages 1438-5448 2022a.
X. Xu, Y. Wang. Y. Zheng. Y. Rao, J. Lu, and J. Zhou. Back to reality: Weakly-supervised 3d object detection with chape-guides label enhancement. In CVPR, pages 8438- 8447, 2022b.
M. Ye, M. Danelljan, Yu, and Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In ECCV, pages 162-179, 2024.
S. Zhang. X. Fei, F. Liu, H. Song, and Y. Duan. Gaussian graph network: Learning efficient and generalizable gaussian representations from multi-view images. In NeurIPS, 2024.
Y. Zheng, Y. Duan, Z. Li, J. Zhou, and J. Lu. Learning dynamic scene-conditioned 3d object detectors. TPAMI, 46(5):2911-2996, 2024.
S. Zhi, T. Laidlow, S. Leutenegger, and A. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, pages 15838-15847, 2021.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DETAILS ON THE DATASETS
A.1 DATASET OVERVIEW
In Figure 8 visualize the scenes in the 3D-FRONT dataset with ground-truth bounding boxes. Each column shows two different views of the same scene. A total of 159 usable rooms are manually selected, cleaned, and rendered to ensure have high quality for scene reconstruction and object detection, as done by NeRF-RPN (Hu et al. 2023
Figure 8: Snapshots of scenes in 3D-FRONT and the corresponding 3D bounding box annotations.
A.2 STATISTICAL DETAILS
In Figure [9] we have illustrated the statistics on the surface closure of two datasets measured by |$|. The bar charts show that the majority of the flux values are clustered near zero, which indicates that most objects in both datasets contain higher degree of surface closure and our assumption is applicable to the employed datasets.
3D-FRONT
ScanNet
0.5
Properion
0.3
0.0
0.55
0.50
0.75
1.00
1.25
.00
1.75
2:0
Surface Closure Measured by Absolute Flux Value
Figure 9: Statistics on surface closure two datasets (Fu et 2021 Dai al. 2017). Most objects in both datasets contain higher degree of surface closure. reconstructed 3D Gaussians via 3DGS+SuGAR,
The statistics are collected from our
B ADDITIONAL TECHNICAL DETAILS
B. MORE DETAILS ON NETWORK ARCHITECTURE
For corresponding feature poand generation, we use the same PointNet++ backbone in VoteNet QQi al. 2019),, which the backbone B in Eqn. 4 Specifically, it has four set abstraction (SA) layers and two feature cropagation/upsamplle 512 and 256 Gaussians respectively. The two FP layers up-sample the 4th SA layer's output back to 1024 Gaussians with 256-dim features and 3D coordinates.
(FP) layers. SA layers sub-sample the input to 2048, 1024,
The setpryingaand-pollie function in Eqn. 5 and Eqn. ha the output size of 128. Therefore, pert and fhol have the same feature channels 128 and f Tinal has the channel of 256.
The stacked MLP for proposal refinement Eqn. has output size 128, 128, 9. In these 9 channels, the first two are for objectness classification, the following is for heading angle refinement and the last six are for offsets refinement. The initial proposal has the same shape with refined proposal and is generated in the same way.
14

---

Published as a conference paper at ICLR 2025
For the point cloud based methods, the original 3D Gaussians G {gp (kpp Ep. ap. cp) (generated by 3DGS+SuGAR) were used as the input, where the Gaussian center Hp was regarded as concatenated and regarded the input point-wiss
the input point-wise coordinate. Ep. Op. Cp features. Except for the first layer whose input dimension was modified to fit the input Gaussians, faithfully followed the network and training configurations of the point cloud based methods (Qi
ct al. 2019 Liu et al. [021] Rukhovich et al. 2022a Xu et al. 2002bb) we compared with.
B.2
DETAILS ON CALCULLATING SURFACE NORMAL
We provide details calculating surface normal, which is determined by solving the eigenvalue problem:
Pp Aup
(14)
where E, R.S.S, Then we made identical deformation both sides of the equation.
R. An,
(15)
,, n R
We set R. As Rp is rotation matrix, we get R,T,, R. 1. Then:
S dx
(16)
The solution x, = [1,0,07 x1 [0, X2 = (0,0, 1]" Therefore:
np.1 = R x,, -,, = 0,1,
(17)
We define eigenvector with smallest eigenvalue as the Surface Normal np. The defination is consistent with (Guédon and Lepetit 2024). However, n, is unoriented, meaning it point either inside or outside. Theretore, we determine the surface normal direction by considering the object center (in practice,
proposal center). To elaborate, the orientation of surface normal will align with the orientation pointing from the object center to the Gaussian center. An alternative strategy is follow (Ran al., 2022) determine the orientation. Specifically, we ensure that the first element of the normal vector is non-negative. Then we determine the orientation using an instance-level random inversion with p = 50%. Ablation study in Table shows that the result of strategy of (Ran et al. 2022) is lower than the aforementioned strategy.
B.3 MORE DETAILS ON TRAINING In training stage, we optimize the residue loss by maximizing the evidence lower bound.
Cres og (Feand)
E (ound) [log pFFond FV) KL(g(FV [Feand)] (18)
After applying the reparameterization trick (Bengio et al. 2013), the total variational loss can be reduced to:
M
1 M
Lres =
log pa (Joand
(19)
logo (oooloo(!)) (o) (2L)) (o(
2M
Cree + Lappr
where C' refers to channel and M refers to the number of candidate Gaussians. Cree refers to reconstruction loss and Cappe refers Gaussian approximation loss. During training we implement square difference for simplicity.
B.4 PROOF OF THEOREM 1
Theorem is special case of Gauss' Flux Theorem by setting T as any constant vector field:
ndS
V TaV V
(20)
15

---

Published as a conference paper at ICLR 2025
Table 7: The hareae-rrrmeeter setup in Gaussian-
Name
Description
Value
Areoe
Weight of Cres
Apred
Weight of Epred
Arefine
Weight of Lrafine
Residue weight
0.1
8
Threshold of grouping the Gaussians
0.2
P
The number of imput Gaussians
40000
M
The number of candidate Gaussians Goadd
1024
Channel candidate feature Fraand
256
C ADDITIONAL ABLATION EXPERIMENTS AND ANALYSIS
C.I HYPER-PARAMER SETUP
We summarize the hyper-parameters used in Table These include hyper-parameters for loss function, grouping threshold, input and network setup. We have also tried different choices of Ayes We choose Ares 1 as the default setting.
Table 8: Experimental results of parameter settings Gaussian-Ded Ayes 0 denotes deterministic partial surface representation ablation study. Performances of detection on 3D-FRONT are reported.
Ares
Is Default
AP25
APso
95.7
74.7
0.1
95.5
76.1
0.5
96.2
76.6
96.7
77.7
2
96.2
75.7
We have also collected the statistics before and after removing Fy and illustrated the learning curve in Figure I0 We can observe that the probabilistic feature residual term contributed to a slightly faster initial convergence and less fluctuation throughout the training process. This suggests that Gaussian-Ded Det with FV more stable during training.
C.2 IMPACTS OF Fv ON THE TRAINING PROCESS
Learning Curve Comparison
40
Fv w.o. Fvv
30
5000
10000
15000
20000
25000
30000
35000
40000
Itnaaton
Figure 10: Training curves before and after removing Fy.
C.3 ANALYSIS ON UNDERLYING GAUSSIAN REPRESENTATION
To investigate how the underlying Gaussian representation, we have experimented on the ablation study by detecting from 3G Gaussians that were reconstructed via the original 3DGS (Kerbl et al. 2023) algorithm.
The quantitative results are displayed in Table B. While the detection accuracies drop with the input of lower quality 3D Gaussians, they still outnumber all the compared methods in Table 9 which consume higher-quality 3D Gaussians that were reconstructed via 3DGS+SuGaR. We additionally show experimental results using 2DGS for better surface modeling quality and improvements can be observed.
16

---

Published as a conference paper at ICLR 2025
Table 9: Quantitative results on the Gaussian representation.
Methods
FRONT
ScanNet
AP2s
APso
AP.s
Gaussian-Det (3DGS+SuGaf) SuGaR KKrtl et 2023 Guédon and Lepetit, 2024
96.7
71.7
71.7
24.5
Gaussian- Kerhl
Gaussian-Det (2DGS
Froning
95.3
79.9
70.1
24.6
D ADDITIONAL VISULIZATION RESULTS
D.1 ADDITIONAL QUALITATIVE RESULTS ON ABLATION STUDIES
Figure [I] we have accordingly illustrated qualitative comparisons of the ablation studies on different components (see Figure 11 in the Appendix of the revised manuscript). It can be seen that either discarding the holistic surface coalescence or using the deterministic partial surface representation leads to sub-optimal visual results, such as inaccurate sizes and locations.
Discard the Holistic Surface Conleseence
Determinisiic Partial Surface Representation
Ground Truth
Full Framework
Figure 11: Additional qualitative results on ablation studies.
Example 1
Example
Example 3
R33
0.02h
Figure 12: Detection results (upper) under challenging scenarios and corresponding ground-truths (bottom).
D.2 ADDITIONAL QUALITATIVE RESULTS UNDER CHALLENGING SCENARIOS
Apart from the failure case in Figure 5 we illustrate another case in Example 1, Figure 12 where Gaussian-Da successfully detects the occluded object. In this example, Gaussian-Det benefits from the prior that the objectness contains a high degree of surface closure ( = 0.074), thereby tackling the occlusion caused by the sofa (/1) 0.025).
On the other hand, for those challenging objects with higher that take up small proportion, have illustrated similar cases of ScanNet in Example 2&3 of Figure 12 Example 2, a bookshelf desk ($ 0.380) and two
(| 0.833) is close to suitcase and two walls. In Example garbage bins ( = 0.556, 0.218) are tightly positioned to each other and are against a wall. The absolute flux values ((T) of these exemplified objects are relatively higher due less exposure
17

---

Published as a conference paper at ICLR 2025
camera shooting. While assumption is weaker in these examples, Gaussian- Det is still able to detect the target objects.
In Figure 13 have included more qualitative results occlusions, where the compared methods predict inaccurate sizes or locations.
2D Reference
Ground Truth
Gaussian-Det (Ours)
NekF-RPN
G-FCAF3D
Figure 13: Additional qualitative results challenging cases and comparisons with other methods.
Another challenging scenario is detecting from low-quality 3D Gaussians that reconstructed from images with noised poses. From Figure [4] we can see that Gaussian- Det is still capable of detecting the target objects, especially the table which is visually blurred after rendering.
G-BRNet Figure 14: Qualitative results low-quality Gaussians reconstructed from images with noised poses.
2D Reference
Ground Truth
Gaussian- Det (Ours)
D.3
ADDITIONAL FAILURE CASES
Ground-trut
Gaussian- Det (Ours)
Ground truth
Gaussian- Det (Ours)
Figure 15: Additional failure cases.
Apart from the example provided, we have further presented more failure cases in Figure 15 Since surfaces of the objectness that contains jointed objects can also be regarded as closed, Gaussian-Det may mistake them as a whole. Future researches can focus on incorporating the color information to discriminate joint objects.
E ADDITIONAL QUANTITATIVE RESULTS
E.1 DETAILED COMPARISON WITH NERF-RPM AND NERF-MAE
In Table I0 report all the available officially reported results of NeRF-RPN and NeRF-MAE for a thorough comparison with Gaussian-Der Det. The checkmark indicates that NeRF-MAE employs self-supervised pre-training before finetuning the downstream detection task. F3D, HM3D and
18

---

Published as a conference paper at ICLR 2025
Table 10: Detailed quantitative comparison on 3D-FRONT and ScanNet.
Reference
Metheds
Pretrained
3D-PRONT
ScunNet
ARRa
AFa:
AlPo
ARR:
ARco
APs:
APpo
NeRF-RPN (AB, VGG19)
97.8
76.5
65.9
43.2
$5.7
42.4
40.7
14.4
NoRF-RPN (AB, R50)
96.3
70.5
65.7
45.1
96.2
32.0
34.1
9.0
al. 2023
NeRF-RPN Swin-S)
98.5
63.2
51.8
26.2
93.6
44.3
38.7
12.9
NcRF-RIN (AF: R50)
95.6
67.7
83.9
55.6
91.6
35.5
$5.2
NeRF-RPN (AF; Swin-S)
96.3
62.5
78.7
41.0
90.6
39.9
$7.5
20.5
NeRF-RPN
96.3
69.9
85.2
59.9
89.2
42.9
55.5
1844
Inshad
NcRF-MAR
NcRF-MAE (F3D. Aug)
96.3
74.3
59.1
90.5
39.1
51.3
15.5
NcRF-MAB (FSD&HM3D&HS,, Aug)
97.2
74.5
85.3
63.0
92.0
39.5
57.1
17.0
Guesiinn Det (Ouns)
97.9
82.3
96.7
77.7
87.3
43.0
71.7
24.5
Table 11: Comparison of model efficiency on the task of 3D instance segmentation.
Methods
Training Time
GPU Memory
Grousiinn Yee et al. 2024
1.33h
35.8GB
Gaussian Gaussian-Croupurg Crouping Ours
0.65h
12.8GB
HS refer to the 3D-FRONT, HM3D and Hypersim datasets for self-supervised pre-training. Aug. indicates augmentations done during pre-training. Note that without pre-training on exterior databases, Gaussian-Det still outperforms variants of both NeRF-RPN and NeRF-MAB most evaluation metrics.
E.2 MODEL EFFICIENCY ON 3D INSTANCE SEGMENTATION
Apart from the quantitative results in Table 6 we have additional measured model efficiency in Table [1] We can see that leveraging the surface closure substantially speed up the training period and significantly save the GPU memory footprint.
19


Reviews:
[{"review_id": "eylfgIWNck", "reviewer": "Reviewer_s6A7", "summary": "This paper proposed a 3D object detection method based on the Gaussian splatting. With the Gaussian splatting as input, the proposed deep model extracts the 3D object bounding boxes. Experiments show the proposed methods give higher recall and precision than the competing methods.", "questions": "See the questions in the weakness section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The proposed method takes advantage of the Gaussian splatting representation for 3D object detection. Gaussian splatting is a more compact representation than the point clouds.", "weaknesses": "- The paper combines several methods: (1) Official implementation of Gaussian splatting and (2) PointNet++. There is no new network structure\ndeveloped. The novelty is low.\n\n- The overall network is not explicitly defined in the paper. please add this, otherwise it is impossible to check the details.\n\n- The writing of the paper is quite unclear, especially the section 3.3. Theorem 1 is directly from the existing theorem. No need to include here. It is also confusing where this theorem is used anywhere in the paper. The equations are sloppy. Notations are a mess. The features\nused two notations F and f. F^{cand} is not defined. f^{part} is also never defined anywhere in the paper. The equations from (6), (7), (11) need motivation and clarification.\n\n- The experimental setting is not clearly defined. The authors used a dataset originally for 3D object detection in Nerf. It is not clear how the point cloud based methods, which need a point cloud input, are used in the experiment comparison.\n\n- The qualitative examples of the proposed method show the quality of the 3D boxes is sometimes a lot worse than other competing methods. The boxes are often bigger and quite off. The competing methods also show many overlapping bounding boxes. No non-max suppression This may cause the lower numbers from the competing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3PrF3j3BL", "reviewer": "Reviewer_XhiG", "summary": "In this paper is presented a novel method for multi-view 3D object detection, exploiting Gaussian splatting to obtain a continuous scene representation. Then, to handle outliers in the previous representation, a closure inferring module is included that learns a probabilistic feature residual for partial surfaces and coalesces them into holistic representation on closure measurement. Experimental evaluation is reported on both synthetic and real datasets, including an ablation study and some comparisons with respect to competing approaches.", "questions": "To be honest, I do not have many issues with the current submission. \n\nIn figure 6, maybe the authors could use a different color per object. \n\nHow could a noisy estimate of the cameras affect overall performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is addressing a complex and important problem. \n\nThe authors include most of the details to understand their proposal, being the technical section clear enough and well written. \n\nGaussian-Det is simple and effective. To this end, the overall framework is divided into the construction of surfaces based on Gaussian representations, an object proposal initialization, as well as a partial surface feature inference and holistic surface closure coalescence. \n\nBoth synthetic and real experiments are provided in the paper. The results are competitive with respect to several competing approaches.\n\nThe ablation study helps the reader.", "weaknesses": "- Camera poses exploit a type of ground truth (point cloud) to be inferred. In my opinion, this is clearly a strong limitation of the paper. \n\n- Lack of challenging cases. In my opinion, some qualitative instances could be included in the paper, showing the effectiveness of the method in complex cases, especially, in comparison with other approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M8Shn8UOBm", "reviewer": "Reviewer_nYGE", "summary": "This paper introduces a novel 3D object detection method, Gaussian-Det, which uses 3DGS to represent objects as continuous surfaces rather than NeRF discrete points. The method uses Gaussian as a surface descriptor and introduces the Closure Inference Module (CIM), which solves the outlier problem inherent in the 3DGS method by considering surface closure as a prior for objectness. The CIM operates in two phases: firstly, the probabilistic feature residuals are estimated for some of the surfaces, and then they are merged to form a holistic representation that measures the overall surface closure. The method is evaluated on synthetic (3D-FRONT) and real-world (ScanNet) datasets, and the results show a significant improvement over previous SOTA methods. The main innovations of the method are its surface-based representation and the use of surface closure as a geometric prior to improve the quality of detection.", "questions": "Improving readability and providing a visual comparison of the ablations would better present this work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The proposed method show SOTA performance compared with previous NeRF based methods.\n- Use 3DGS is faster than NeRF.\n- The CIM module effectively handles outliers in Gaussian Splatting.", "weaknesses": "- There is a lack of qualitative comparisons between different components, particularly regarding the use of Holistic Surface Coalescence and Residual Estimation. These comparisons are important for readers to better understand the effectiveness of each component.\n- It would be interesting to analyze how the underlying Gaussian representation affects the results. For example, what is the impact of using the original 3DGS or a 2DGS representation?\n- The overall writing is unclear and difficult to follow.\n\t- L.208 should be \"From G^{cand}\" \n\t- There is multiple M in FIg. 2, are they the same?\n\t- Figure 3 is confusing. It is unclear what “open surfaces” and “closed surfaces” refer to. In Figure 3 (a) and (b), the primary visual difference appears to be the incompleteness of the Gaussians in (b). Please clarify the relationship between open and closed surfaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4wp2jvztIt", "reviewer": "Reviewer_kAE9", "summary": "The paper presents Gaussian-Det, a method to detect 3D objects via Gaussian Splatting. The proposed method leverages Gaussian Splatting as surface representation for multi-view based 3D object detection. Gaussian-Det proposes Closure Inferring Module (CIM) to deal with the outliers present using Gaussian-Splatting technologies. The paper presents experiments on both synthetic and real 3D-object detection datasets where Gaussian-Det mostly outperforms previous methods.", "questions": "See Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Interesting solution using Gaussian-Splatting technologies for 3D object detection.\n- Proposing Closure Inferring Module (CIM) to deal with the noisy nature of Gaussian-Splatting representations.", "weaknesses": "- Clarity of the paper is not good and needs improvement (my most concerning weakness). First, overall the high-level idea is not clearly explained. Although, Fig. 2 tries to do this, it falls short. Figure 2 lacks a clear illustration of the goal using the Backbone, and most importantly, fails to explain clearly how the CIM module (in my opinion the main contribution). Second, given the lack of clarity, it is hard to judge the novelty and quality of the solution. For example, I don't find the description of the Theorem if the high-level idea is not explained clearly. In my opinion it is just using space that could've been used to add Figures to explain CIM in a better way.\n\n- Lack of details in the experiments. First, in line 320-321, the narrative says that the scenes are manually downsized into 159 usable rooms. However, it is not clear if the data was verified to ensure high quality. This is important since this data is used to measure quality.\n \n- Insufficient experiments. In my opinion, showing results only on 2 datasets is not that convincing. Given that 3D object detection has been around for some time, it would've been more informative to know the performance of this proposed method on more real-scenes benchmarks/datasets (e.g., Waymo Open datasets, Objectron, etc.). I think that these datasets could've revealed the practical benefit of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)