MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps
Authors: Yating Xu
Chen Li
Gim Hee Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: 3d object detection
plane sweep
Gaussian Splatting
Primary area: machine_vision

Abstract: The key challenge of multi-view indoor 3D object detection is to infer accurate geometry information from images for precise 3D detection. Previous method relies on NeRF for geometry reasoning. However, the geometry extracted from NeRF is generally inaccurate, which leads to sub-optimal detection performance. In this paper, we propose MVSDet which utilizes plane sweep for geometry-aware 3D object detection. To circumvent the requirement for a large number of depth planes for accurate depth prediction, we design a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume. We select multiple locations that score top in the probability volume for each pixel and use their probability score to indicate the confidence. We further apply recent pixel-aligned Gaussian Splatting to regularize depth prediction and improve detection performance with little computation overhead. Extensive experiments on ScanNet and ARKitScenes datasets are conducted to show the superiority of our model. Our code is available at https://github.com/Pixie8888/MVSDet.

MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps
Yating Xu
Chen Li
Gim Hee Lee'
Department of Computer Science, National University of Singapore' Institute of High Performance Computing, A*STAR Centre for Frontier Al Research, A*STAR"
xu. yatinngu. nus.
lichenCu. nus edu g www.eeee@.mm.... nus .du......gg
Abstract
The key challenge of multi-view indoor 3D object detection is to infer accurate geometry information from images for precise 3D detection. Previous method relies on NeRF for geometry reasoning. However, the geometry extracted from NeRF generally inaccurate, which leads to sub-optimal detection performance. In this paper, propose MVSDet which utilizes plane sweep for gcometry-awant 3D object detection. To circumvent the requirement for large number of depth planes for accurate depth prediction, we design a probabilistic sampling and soft weighting mechanism to decide the placement We select multiple locations that score top in the probability volume for each pixel and use their probability score to indicate the confidence. We further apply recent pixel-aligned Gaussian Splatting to regularize depth prediction and improve detection performance with little computation overhead. Extensive experiments on ScanNet and ARKitScenes datasets conducted show the superiority of our model. Our code available at ht thtpps:/ //t/t//gttt Com/Pixie8888, /MVSBttt
pixel features on the 3D volume.
Introduction
Indoor 3D object detection is a fundamental task in scene understanding and has wide applications in robotics, AR/VR equipment, etc. Although point cloud based 3D objection methods [13] [4] have achieved impressive performance, depth sensors are required to capture the data, which may not be available due budget limitation, form factor constraints, etc. Recently, the more economic pipeline of 3D object detection from only posed multi-view images is gaining increasing attention. However, is much
sophisticated estimate geometry information from 2D images alone.
straightforward solution to this problem is using ground truth geometry information, e.g. point cloud 19 or TSDF [17] to supervise the model. Built on the 3D volume representation [16). ImGeoNet 19 predicts the emptiness of each voxel by converting the ground truth point clouds to surface voxels as supervision. CN-RMA [17] first reconstructs 3D scenes using ground truth TSDF as supervision and then runs an existing point cloud based object detector to predict bounding boxes. Although they achieve promising performance, the precise ground truth scene geometry is hard to obtain and may not be available
An alternative way is to learn geometry via self-supervision. The pioneer work Im VoxeINet [16] unprojectes 2D image features to 3D volume representation. However, 2D features can propagate to irrelevant 3D locations since depth information is not known. NeRF-Det [22] relies on a Neural Radiance Field (NeRF) 12 learn density field and queries an opacity score for each voxel. The
"Chen Li was at the National University of Singapore when this work was done.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reference Scene
NeRF-Def
Ours
Figure 1: Comparison with NeRF-Det 22 The 3D voxel centers (grey dots) are overlaid with the reference scene. The red dots denotes the erroncous backprojection pixel features to the points in the free space. Compared to NeRF-Det, we show much less inaccurate backprojections.
opacity score is multiplied with voxel feature to decrease the influence of voxels in the empty space to the feature volume. Since the detection performance is completely determined by the quality of NeRF, enormous effort is spent on making NeRF generalizable and avoiding aliasing issue. Unfortunately. the geometry extracted from NeRF remains unsatisfactory due to insufficient surface constraints in the representation [21]. Consequently, wrongly backprojects features to voxels in the free space (as illustrated by the red dots in the example shown the middle of Fig. TT.
In this paper, we propose MVSDet to extract geometry information from only the input miew images. A straightforward way is to leverage multi-view stereo algorithms (24 25 to decide the accurate depths for correct placements of 2D features of each image the 3D volume. However, accurate depth estimation in the plane-sweeping algorithm [8] requires computationally expensive sampling of many depth planes over all the multi-view images. We mitigate the computational complexity by proposing a probabilistic sampling and soft weighting mechanism to decide the possible depth locations for each pixel. Specifically, we sample multiple top scoring depth proposals in the probability volume that are most likely covering the truc depth locations. Pixel features placed onto the 3D volume only when the backprojected ray intersects at the depth locations. Since the normalized probability score indicates the confidence of the the current depth location, we use it to weigh the pixel feature before assigning the feature to its backprojected voxel center.
To further improve the depth prediction accuracy, we utilize the recent pixel-aligned Gaussian Splatting (PAGS) 3 28 for novel view rendering as an additional supervision. PAGS predicts a 3D Gaussian primitive g for every pixel in the input views, and all the Gaussians are used to render novel views via rasterization-base splatting. Compare to NeRF that uses computation expensive volumetric sampling, Gaussian Splatting is fast and light-weight. A key to good rendering quality in PAGS is the correct positioning of the 3D Gaussians, which depends on an accurate depth prediction. As result, by putting the Gaussians according to the depth map computed from the probability volume in the plane sweep module, the rendering loss would guide the the Gaussian centers and consequently the depths to the correct values.
In summary, our contributions are as follows:
We propose a probabilistic sampling and soft weighting mechanism to efficiently learn ge- ometry without sampling many depth planes in multi-view stereo. Multiple depth proposals are sampled with the probability scores to guide the propagation of image features to 3D voxels.
2. We adopt pixel-aligned Gaussian Splatting to enhance depth prediction without much additional computation overhead, which consequently improves detection performance.
3. We conduct extensive experiments on the ScanNet and ARKitScene datasets to verify the effectivess of our method. Notably, we achieve significant improvements of +3.9 and +5.1 under the mAP@0.5 metric on ScanNet and ARKitScenes, respectively.
2 Related Work
Indoor 3D Object Detection. 3D object detection for indoor scenes predicts three dimensional bounding boxes and corresponding classes by taking in 3D or 2D inputs. Point cloud is the most popular choice of 3D data for detection as it provide accurate 3D information. To detect objects

---

from the irregularity and sparseness of the point cloud, VoteNet [14] utilizes Hough voting by points sampling, voting and grouping to generate object proposals. Later methods improve VoteNet
by either predicting geometric primitives [27 or using hierarchical graph network 4 3DETR [33
reduces hand-coded designs of VoteNet with transformer encoder-decoder blocks. Despite their promising performance, depth sensors are required to capture the data, which is not always available due to power consumption or budget limitation.
is a cheaper choice, but with a sacrifice of losing geometry information. Some methods guide the model to predict scene geometry using ground truth geometry, i.e. ground truth surface voxels [99] or TSDF [7], as supervision. However, obtaining ground truth scene geometry is troublesome. In contrast, Im VoxelNet [16] builds a 3D feature volume in the world coordinate, where each voxel center aggregate the corresponding features of its projected 2DD pixels. Subsequently, 3D U-Net is applied to refine the volume features and predict bounding boxes from each voxel center. However, voxels may wrongly aggregate irrelevant image features since depth is not known. Recently, NeRF-Det [2] uses NeRF to learn density field and predict an opacity score per voxel center to downweigh the influence of voxels in the empty space to the feature volume. However, the implicit modeling of geometry in NeRF leads to unsatisfactory performance. Moreover, the reliance of NeRF during training side-tracked the effort in making NeRF generalizable. Instead of implicitly modeling geometry with NeRF, we utilize plane-swept cost volumes for geometry aware scene reasoning. We propose a probabilistic sampling and soft weighting mechanism to accurately decide the placement of pixel features without sampling many depth planes.
Alternatively, detecting 3D objects from images only 16
View Depth Estimation.
Multi-view depth estimation has long been studied in the multi-
MVSNet (24] constructs 3D cost volume, regularize it with 3D CNN and regresses the depth map from the probability volume. However, MVSNet consumes
view stereo 24
20
25
5
26
large memory due the expensive 3D cost volume. Follow-up works reduce computation by replacing 3D CNN with recurrent network [23 25) or using coarse to fine depth estimation [5] 20 2 77. Recurrent methods only reduces cost regularization module to a 2D network, but still faces large 3D cost volume to predict accurate depth. Although coarse to fine pipeline can reduce the number of sampled depth planes, it still requires to sample certain amount of initial depth locations, e.g. 32 to 64 planes 20]. to get reasonable coarse depth map, which still leads to intractable computation in our mueti-view object detection task. Bac et al. (2] propose to sample extremely few number (i.e. 5) of initial depth planes based on pre-trained single view depth probability distribution. It has been further applied to outdoor multi-view object detection [TO. However, both of them need the guidance of ground truth depth to learn correct monocular depth estimation, and would otherwise fail as shown in the experiment section. Instead, we propose probabilistic sampling and soft weighting module to bypass the large 3D cost volume to decide the 3D locations of every pixel. We also novelly utilize Gaussian Splatting to enhance our depth prediction with little computation overhead.
3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) 9 is recent technique for novel view rendering. models 3D scene explicitly with Gaussian primitives, each of which defined by a 3D Gaussian center, covairance matrix, opacity and color. Compared to volume rendering based NeRF [12, it achieves real-time rendering with much lighter computation. To avoid per-scene optimization of 3DGS, several works (28 6 I8] propose to model a scene with pixcl-aligned Gaussians Splatting (PAGS), A Gaussian primitive is predicted per pixel, and all predicted Gaussians are then combined for novel view synthesis. A key to PAGS is the accurate 3D Gaussian center which is determined by the depth estimation. Thus, instead of targeting novel view synthesis, we novelly utilize it for 3D object detection through improving depth prediction. Our method is also significantly different from NeRF-Det. While NeRF plays a major role in NeRF-Det by predicting opacity scores, we use Gaussian Splatting as regularizer to our plane sweep algorithm with little computation overhead.
3 Our Method
The goal of multi-view indoor 3D object detection is to predict bounding box (B) C 7R of objects in the 3D scene and their corresponding classes from N posed images (I,,... IN] E TR x3 Each bounding box B is parameterized as (a,, y, z, w. h, I, Φ), where (z, 2) are the coordinates of the box center, (w, h () are the width, height, and length, and ø is the
Problem Definition.
3

---

Inpur Images
Feature
Cost
Probabilistic Sampling Soft Weighting
Mups
Volumes
Feature Extractor
Feature Volume
Depth Probubility Distribution
Detection Head
f
B
B
Select Nearby Views
Selected Depth
Vadd
C Iavalid
GT Location
Novel view
M
Mg Equ.8
based Rendering
Select Nearby Views
MLP
M
M
Figure Overview of our MVSDet. The upper branch shows the detection pipeline with our proposed probabilistic sampling and soft weighting. The backprojected ray intersects at points (shown as dots), but only the green point receives the pixel feature based on the selected depth proposals. The red points are denoted as invalid backprojection location and thus the pixel feature is not assigned them. "GT Location" is the ground truth 3D location of the pixel. The lower branch shows the pixel-aligned Gaussian Splatting (PAGS). We select nearby views for the novel image from the images input the detection branch and predict Gaussian maps on them. Note that PAGS removed during testing.
rotation angle around z-axis. The intrinsic and extrinsic matrix for each input image are denoted as K E R3x3 and P |R R3344, respectively.
Overview. Fig. shows our proposed MVSDet, a geometry-aware approach for indoor 3D object detection from posed multi-view images. Our MVSDet is built 3D volume-based object detection (cf. Sec. B.I. Instead of naively assigning the same 2D feature redundantly to every voxel center that intersect the backprojected ray, place the pixel features according to the estimated depth from proposed efficient plane sweep algorithm. We alleviate the costly sampling for accurate depth prediction by proposing probabilistic sampling and soft weighting mechanism (ef. Sec. 3.2). We further utilize pixel-aligned Gaussian Splatting to enhance our depth prediction module with little extra computation cost (cf. Sec. (.3).. Intuitively, depth-aware framework leads to more precise assignments of multi-view image features in the 3D volume and consequently better 3D object detection results.
many depth planes
3.1 Background: 3D Volume-Based Object Detection
Existing methods [6 22 estimate bounding boxes from 3D feature volume aggregated from
the multi-view image features. Image features F E R x are first extracted from every input image R
3D volume is defined the world space with the size of N,, N, x N₂ voxels. Voxel center p = [x. y, =] E TR3 projected to i-th input image to obtain the 2D coordinate
[ug, v;]T g2
Iul V/, 4,] KPP.pp
[u,,v.] v,, = [4//4, v./d.J,
(1)
where KM is the scaled intrinsic matrix according the image downsampling ratio. Subsequently, 2D feature f; RC is assigned to via nearest neighbour interpolation follow:
f, interpolate ((u,, vz) Ft)..
For that are projected outside the boundary of feature map or behind the image plane, the projection considered invalid and we set 0. Finally, averages all valid backprojected features as the voxel feature
En= f//n R where denotes the number of valid projection.

---

The feature volume is refined by 3D U-Net before being fed into the detection head to predict category, bounding box and centerness score on each voxel location. The training losses for detection consists of focal loss for classification Luls. cross-entropy loss for centerness Ceenter. and loU loss for location Lixe:
Laxt - Lols + Leenter Lloe-
(3)
Observation. We observe that the feature aggregation method in existing multi-view indoor 3D) object detection works causes 2D pixel features to duplicated on voxels intersecting with the ray emitted from camera origin though the pixel as illustrated in Fig. 3 This is a result from lack of depth-awareness where voxels can end up aggregating irrelevant pixel features that are either not on the surface or occluded.
Proposition. To circumvent this problem, propose: 1) efficient plane sweep to instill depth awareness. Our efficient plane sweep consists of a probabilistic sampling and soft weighting mech- anism based cost volume representation to evaluate the placement of pixel features the intersected voxel centers; 2) depth prediction regularizor based on 3D Gaussian Splatting to improve depth accuracy.
3.2 Efficient Plane Sweep
Cost Volume Construction. We build N cost volumes for N input images to predict N depth probability distribution maps. To construct the cost volume for the i-th view, set the image as the reference view and select nearby views the source views [1)) A raw matching volume for I; is constructed by backprojecting
Volume
GT
Location
Invalid
source feature map F FE2NB(()) E R 7 4 xC into the coordinate system defined by I; at a stack of fronto-paralled virtual planes. The virtual planes {di..... dd) are uniformly sampled on a pre- defined depth range. The coordinate mapping from the source feature map F, to the reference feature map F; at depth dm is determined by a planar homography transformation:
Valid
Im Voxeleeeee
Ours
Figure Comparison of different feature back- projection methods. The pixel ray intersects at voxel centers with the blue box denoting the ground truth 3D location of the pixel. Our method computes the placement of the pixel fea- tures based on the depth probability distribution (purple) and thus able to suppress incorrect inter- sections.
dm (K, (K T)
.1]"
4p. Kj [Ry
(4)
where q is the homogeneous coordinate of a pixel on F,, 9j.m is the projected homogeneous coordinate of q on Fy. Ríj [ty are the rotation and the translation of Iú relative to I, We use Eqn. 4 to warp every source feature map F, FjE2NB() into all
the depth planes and use a variance based metric 24 to generate a cost volume U E R xCxM Subsequently, we use shallow 3D CNN to refine U to generate probability volume (after softmax) B B R 4 x. We also predict offset per depth bin to account for the discretization error.
Probabilistic Sampling and Soft Weighting. Although we can place pixel features to the 3D volume by predicting depth using the weighted average based on B, it requires sampling sufficient depth planes, which is prohibitive for our task. To this end, we propose a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on its backprojected ray without the need for many depth planes.
For every pixel, we sample & depth locations that score top in the corresponding distribution in B as its depth proposals. We denote their depth values as [dhak] ... diax, ). and also use their corresponding probability score to evaluate its confidence with respect to the ground truth location. The scores denoted (Baa, Blcke where Baasa BuKs/ i= Bu Intuitively, this is equivalent to split one depth prediction to multiple depth prediction based on the scores. As a result, it is more likely cover the ground truth location when depth bins are insufficient.
Suppose a voxel center p E R3 intersects the backprojected ray of a pixel from the i-th image, and the depth of p under the i-th camera frustum is d(p). We consider the projection of point p to the i-th
5

---

image valid and set the indicator g = when d(p) resides near any of the top-k depth proposals {diak)..... dats, J. We assign the normalized probability score of the nearest depth proposal to as its confidence. The corresponding pixel feature is assigned to p weighted by the confidence score. The projection is invalid when d(p) is not close to any of the depth proposals, and we set the backprojected feature from i-th image as O. Thus, the feature f, RC backprojected from the i-th image to point p and its corresponding indicator 8; are given as follows:
f. {
B Boi f if d(p) C {didk. d
1 g. = 0
if d(p) C (didk datx, otherwise
(5)
otherwise
where (p)) is the index of depth proposal that is close to d(p). and B Bapp) the score of do(p)- After looping through every input image, we average all valid backprojected features for point p as its voxel feature V RC:
N= Σ gff. where T/
N=
V Tu
Σ g. p(p)
(6)
We also utilize the multi-view consistency to compute a surface score for point by averaging confidence scores B xp) from every valid projection. We set 0 when the point does not have any valid projection, which means point p is in the free space. is multiplied with voxel feature v the final feature E IRC for point as follows:
i=N
i= N
s=
Σ Bop)
where T/
ga+
V
sv.
The adoption of s shares similar spirit with existing methods [22 empty space voxels in the feature volume. Yet, we learn it directly from multi-view images instead of relying on NeRF 22
[9| decrease the influence of
ground truth supervision [9]
3.3 Enhancing Depth Prediction with Gaussian Splatting
We further utilize the recent pixel-aligned Gaussian Splatting (PAGS) 28
enhance our depth prediction module. PAGS takes in sparse views and predict Gaussian primitive per pixel. The parameters for each primitive are center μ Gaussian opacity a covariance Σ and color c. All the Gaussian primitives are combined together to render novel view via rasterization-bar based splatting.
We select 3 nearby views per novel view from the images input the detection branch and predict Gaussian maps (M,,, M., Ms., M.) for the selected views. The Gaussian center map Mμ E R 4 W x3 are directly estimated from the predicted depth based the probability volume B as follows:
(r) = o(r) + D(r)h(r), D BG
(8)
where G - G [d]...., dx] T is the virtual depth planes and D E R x x1 is the estimated depth map. o(r) is the camera origin, D(r) is the projected ray depth obtained from the depth map D, and h(r) is the ray direction for pixel r. The Gaussian opacity map M. R # 48 is predicted by taking the max probability score of B as follow:
M. max (B, dim = -1).
The Gaussian covariance map My; E # x 4h x 16 and color map M, E R x3 are predicted from MLP follows:
My. M. = MLP(F
f),
(10)
where || denotes concatenation and E R # x 4 x3 denotes the resized image map. Following 3DGS [9. Σ is predicted by rotation quaternion and scaling factors. Color is predicted by spherical harmonics coefficients.
We render the image color Cooter via blaha-bbending and the rendering loss is L2 loss as follows:
Premder = ||Ccolar Ccolar
(11)
One of the key factors for good rendering is accurate M,, which is directly related to correct depth estimation from our model (ef. Eqn. (). The rendering loss thus iteratively guides the Gaussians
to the correct 3D locations, and consequently benefits our detection pipeline. Our final loss is
L = Las + Lacader-

---

Table 1: Results on ScanNet. "GT Geo" denotes whether ground truth geometry used as super- vision during training.
Table 2: Results on ARKitScenes. "GT Geo" denotes whether ground truth geometry is used supervision during training.
Method
GT Geo
mAP@.2. .25
mAP@
Method
GT
mAP@ 225
mAP@.5
ImGeoNet
54.8
28.4
ImGeoNet
60.2
43.4
CN-RMA
58.6
36.8
CN-RMA
67.6
56.5
Im VoxelNet 16
46.7
23.4
Im VoxelNeet
27.3
4.3
NeRF-Det
22
53.5
27.4
NeRF-Det
22
39.5
21.9
Ours
56,2
31.3
Ours
42,9
27.0
Ground-truth
NERF-DA
Figure 4: Qualitative comparison on ScanNet dataset. Note that the mesh is not the input to the model and is only for visualization purpose.
4
Experiments
4.1 Datasets
We conduct experiments the ScanNet and ARKitScenes datasets. ScanNet has 1,201 and 312 scans training and testing, respectively. We detect axis-aligned bounding boxes for classes. ARKitScenes has 4,498 and 549 scans for training and testing, respectively. We detect oriented bounding boxes for 17 class. We adopt mean average precision (mAP) with thresholds of 0.25 and 0.5 the evaluation metrics.
4.2 Implementation Details
We use the same feature extractor, training and testing configurations as (22 for detection. Specifically, images are resized into (240, 320). During training, we input 40 images to the detection branch. During testing, the detection branch takes 100 images and the rendering branch removed. The size of the 3D volume is (N₂ 40, N, 40, N_ 16). with each voxel represents cube of 0. 0.16m x0.16m 0.2m The depth range empirically M is set to 12 and = 3 depth proposals are selected in the probabilistic sampling. We consider d(p) C [diax] dias,] d(p) is within +.02mm of any depth proposals. For the rendering branch, we select another two images as the target views that do not overlap with the images in the detection branch. We use AdamW optimizer with learning rate 0.0002, total epochs 12 and batchsize 1. All experiments conducted on two NVIDIA A6000 GPUs.
as [0.2m, 5m]. The number of depth planes
7

---

Table 3: Ablation study of probabilistic sampling and soft weighting, All methods are conducted without using rendering loss.
Prohabilistic Sampling
Soft Weighting
mAP@.25
mAP@ .5
50.0
24.8
36.9
13.5
56.0
29.7
Table 4: Ablation study of Gaussian Splatting. M denotes the number of depth planes in the plane sweep. "Gaussian" denotes using pixel-aligned Gaussian splatting. "RMSE" the depth evaluation metric. "Memory A" denotes the increased memory consumption during training.
M
Gaussian mAP@.1 25
mAP@.
RMSE
Memory A (GB)
56.0
29.7
0.674
0
/
56.2
31.3
0.374
+2.6
16
55.8
31.1
0,480
+9.4
4.3 Comparison with Baselines
We compare our method with ImGeoNet (9). CN-RMA 17]. Im VoxelNet [6] and NeRF-Det [22 We directly report their results from the CN-RMA [17] paper. Note that ImGeoNet and CN-RMA use ground truth 3D geometry as supervision during training. Tab. [ and Tab. 2 show the results on ScanNet and ARKitScenes, respectively. It is expected that using ground truth geometry as supervision can achieve good performance. However, ground truth geometry may not be accessible and therefore seek for a self-supervised approach that do not rely its supervision. Im VoxelNet and NeRF-Det are the two existing methods that leverage self-supervision to learn geometry for multi- view 3D detection. Compared to these works, our method achieve much better performance. It clearly shows the superiority of our efficient plane sweep method over the vanilla feature backprojection in Im VoxeNet and the density field in NeRF-Det. Fig. [ shows the qualitative results on ScanNet. The first two columns show that our model can detect more target objects. We also find that NeRF-Det tends to detect objects in the free space (last two colums), which is due inaccurate feature projection. In contrast, our model is able to place bounding boxes at more accurate locations.
4.4 Ablation Study
Effectiveness of Probabilistic Sampling and Soft Weighting. Tab. 3ub.3sshoge shows the ablation stdudy of the probabilistic sampling and soft weghting (PSSW). All methods are conducted without the rendering loss frenter to test the performance of PSSW alone. Removing "Probabilistic Sampling" means replacing top-k sampling with a single depth estimation computed by the weighted average of depth probability volume B. We use the max probability score the weight in this case. Removing
"Soft Weighting" means replacing B B'p(p) with value 1. As shown in the table, removing either "Probabilistic Sampling" or "Soft Weighting" causes drastic drop in the performance. Particularly, removing probabilistic sampling depth proposals drops by 19.1 mAP@ .25. It suggests that estimating accurate depth is very hard under insufficient depth planes. Furthermore, soft weighting is very important to our model can decrease the influence of wrong depth proposals introduced by the sampling. Overall, both probabilistic sampling and soft weighting are crucial to our model.
Effectiveness of Gaussian Splatting for Depth Prediction. Tab, 4 shows the ablation study for using pixel-aligned Gaussian splatting (PAGS) for detection. "RMSE" evaluates the average quality of depth prediction for the images in the detection branch. Increasing depth planes M 16 or using Gaussian Splatting both improve the depth estimation and bring similar improvement to the detection performance. However, using PAGS consumes much less memory during training than M = 16, f.e. adding only 28% memories compared to increasing depth planes. Moreover, PAGS does not bring any additional memory cost during testing because is removed for detection. We visualize the depth maps predicted by the probability volume B in Fig. 5 The depth maps are of 1/16 size of the original image since we estimate depth on the feature map level. shows that PAGS significantly improves the depth quality.
8

---

Imogo
Depth
Gomssian
wio Gauesonn
Image
Depth
Gausiaa
we Gametaa
Figure 5: Depth map visualization. "GT Depth" denotes ground truth depth map. Both "w/ Gaussian" and "w/o Gaussian" use M = 12 depth planes.
Table 5: Ablation study of Top-k depth proposals.
Analysis of the number of Top-k depth proposals. Tab. 5 shows the ablation study of number of depth proposals. Sampling top-1 depth proposal leads to severe performance decrease as the correct depth location is harder to be selected. Sampling too many depth proposals (k = 5) also leads to some decrease since more inaccurate locations sampled. We thus choose k = 3 in our model.
mAP@.2. 25
mAP@.
49.3
24.4
56.2
31.3
55.5
29.8
Comparison with Depth Estimation Methods. Fig. shows the comparison of different depth prediction methods to the detection performance.
60.5
MVSNet 24] performs one-time plane sweep by
54.9
56.2
using M = 16 depth planes. BEVStereo [IO] performs iterative depth estimation by sampling depth planes according to a monocular depth es- and
stad
35
Ours
33.2
timation. We set the number of iterations the number of depth planes in each iteration to
MVSNet
BEY Sterre
We remove ground truth depth supervision for BEVStereo for fair comparison. "Ground -tround-truth Depth" denotes placing pixel feature on the 3D volume according to the ground-truth depth loca- tion, which the upper bound of our method. We show the results of our approach using different number of depth planes (M = 12 and A 8). MVSNet performs badly even though samples more planes than us. This is because MVSNet requires sufficient depth planes to estimate depth correctly. BEVStereo also fails because requires the ground truth depth supervision to learn good initial monocular depth estimation. contrast, our model achieve close performance as the "Ground-truth Depth" using only 12 even 8 depth planes. This demonstrates that approach efficiently learns geometry without sampling many depth planes.
Grmend-truth
9.4
20
Depth
Figure 6: Comparison of different depth predic-
tion methods 3D object detection on ScanNet.
Time and memory comparison. Tab. 6 shows the comparison of time and memory in the training and testing stages on ScanNet, respectively. We omit comparison with ImGeoNet since it does not release any code. All models are ran on 2x A6000 GPUs. Due the complexity of CN-RMA (as mentioned in Sec 3.5 of their paper), requires much longer time to train and evaluate than other models. Furthermore, CN-RMA consumes much more memory in the training stage because it requires joint end-to-end training of the 3D reconstruction and detection network. Although NeRF-Det is efficient in time and memory of the training and testing stages, their performance much worse than ours as shown in Tab. and
Table 6: Time and memory comparison in training and testing stages ScanNet dataset, respectively.
Train
Test
Method
Time(hrs)
Memory(GB)
Time(min)
Memory(GB)
CN-RMA T7
121
43
10
12
NeRF-Det
7
13
12
Ours
35

---

Conclusion
In this paper, we propose MVSDet for multi-view image based indoor 3D object detection. We design probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume without the need of the computationally sampling of many depth planes. We further introduce the use of pixel-alignel Gaussian Splatting to improve depth prediction with little computation overhead. Extensive experiments on two benchmark datasets demonstrate the superiority of our method.
6 Limitation
Similar to existing multi-view stereo methods [24] 20 feature matching would fail on texture-less reflective surfaces. One possible solution is to combine with monocular depth estimation (2) However, estimating monocular depth is non-trivial and we leave it for future research.
This research work supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021).
Acknowledgement.
References
[1] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann, Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 7822-7831. 2021.
[2] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Multi-view depth estimation by fusing single-view depth probability with multi-view geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2842-2851, 2022.
[3] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixclsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint (rXiv:2312.12333) 2023.
[4] Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying. Danny Z Chen, and Jian Wu. A hierarchical graph network for 3d object detection on point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 392 401, 2020.
[5] Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Proceedings of the JEEE/CVF international conference on computer vision, pages 1538-1547, 2019.
[6] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint Xiv:222403. 14627, 2024.
[7] Shuo Cheng. Zexiang Xu, Shilin Zhu, Zhuwen Li Erran Ravi Ramamoorthi, Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2524-2534, 2020.
[8] David Gallup, Jan-Michael Frahm, Philippos Mordohai, Qingxiong Yang, and Marc Pollefeys. Real-time plane-sweeping stereo with multiple sweeping directions. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, 2007.
[9] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14. 14, 2023.
[10] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang. Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo. Proceedings the AAAI Conference on Artificial Intelligence, volume 37, pages 1486-1494, 2023.
[11] Xiaoxiao Long, Lingji Liu, Wei Li, Christian Thcobalt, and Wenping Wang. Multi-view depth estimation using epipolar spatio-temporal networks. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 8258-8267, 2021.
[12] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405 4211 Springer, 2020.
10

---

[13] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906-2917, 2021.
[14] Charles R Qi, Or Litany, Kaiming He, and I Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277-9286, 2019.
[15] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: Fully convolutional anchor-free 3d object detection. In European Conference on Computer Vision, pages 477 4933. Springer, 2022.
[16] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general purposs object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2397-2406. 2022.
[17] Guanlin Shen, Jingwei Huang, Zhihua Hu, and Bin Wang. Cn-rma: Combined network with ray marching aggregation for 3d indoors object detection from multi-view images. arXiv preprint M.XXix22433.4199 2024.
[18] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv preprint arXiv:2312. 2023.
[19] Tao Tu, Shun-Po Chuang, Yu-Lun Liu, Cheng Sun, Ke Zhang, Donna Roy, Cheng-Har Kuo, and Min Sun. Imgeonet: Image-induced geometry-aware voxel representation for multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6996-7007, 2023.
[20] Fangjinhua Wang. Silvano Galliani, Christoph Vogel, and Marc Pollefeys. Itermvs: Iterative probability estimation for efficient multi view stereo. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 8606 1616-8615,220 2022.
[21] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
[22] Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Jialiang Wang, Wei Zhan. Zijian He, Peter Vajda, Kurt Keutzer, et al. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2023.
[23] Jianfeng Yan, Zizhuang Wei, Hongwei Yi, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu- Wing Tai. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. In European conference on computer vision, pages 674-689. Springer, 2020.
[24] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings the European conference computer vision (ECCV), pages 767-783, 2018
[25] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high- resolution multi-view stereo depth inference. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5525-5534, 2019.
[26] Zehao and Shenghua Gao. Fast-mvsnet: Sparse-to-dense multi-view steren with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1949-1958, 2020.
[27] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 311-329 Springer, 2020.
[28] Shunyuan Zheng. Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Ligiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. arXiv preprint arXiv:2312.02155, 2023.
11

---

A Appendix Supplemental Material
A.1 Qualitative Results on ARKitScenes Dataset
shows the qualitative results on ARKitScenes dataset. Compared to NcRF-Det 22 more target objects.
Fig. 2
we are able to detect
Ground-truth
NERF-DO
Our
Figure 7: Qualitative comparison on ARKitScenes dataset, Note that the mesh is not the input to the model and is only for visualization purpose.
A.2 Novel View Synthesis Results
shows the novel view synthesis results on ScanNet test dataset. Our model gives reasonable rendering results, indicating that the Gaussian Splatting module successfully learn the geometry. Note that the Gaussian Splatting module is only regularizer to our plane sweep algorithm instead of the determining factor to learn geometry like NeRF in NeRF-Det 22
Fig.
Rendering
Readering
Rendering
Readering
Figure 8: Novel view synthesis results on ScanNet "Renderin" denotes the rendered image / depth from our Gaussian Splatting module. "GT" denotes the ground-truth image /depth of the novel view.
Table 7: Per-class results under AP@0.25 on ScanNet dataset.
Method
chair
door
pic
desk
curt
fridg
NeRF-Der
42.3
84.6
75.9
78.5
$6.3
33.4
21.4
49.9
50.6
73.9
21.3
54.3
62.5
90.9
$7.7
75.5
32.3
Ours
40.5
82.4
79,2
80.2
55.6
40.3
25.4
60.9
3.5
47.3
73.4
28.9
64.6
64.1
94.8
52.1
76.7
41.8
12

---

Table 8: Per-class results under AP@0.5 on ScanNet dataset.
Mothod
cab
bed
chair
sofa
tabl
door
wind
bishf
desk
fridg
showr
toil
sink
bath
other
NeRF-Dei
73.1
45.3
40.6
39.5
8.1
20.3
0.2
13.8
12.5
5.3
25.3
26.0
49.1
12.7
Ours
14.9
71.4
48,9
54.4
38.8
9.5
3.1
29.6
0.8
9.8
48.5
3.6
40.2
10.2
77.3
29.0
52.9
17.7
A.3
Per-Class Performance
Tab.
Tab.
Tab.
and Tab. I0 are the per-class results under AP@0.25 and AP@0.5 on ScanNet and
ARK datasets, respectively.
Table 9: Per-class results under AP@0.25 on ARKitScenes dataset.
Method
cab
fridg
shil'
bed
sink
wahr
lolt
bibib
dsbwstr
lsplce
stool
chr
[bl
TV
sofa
NeRF-Det
34.7
61.1
30.7
73.2
29.9
77.2
86.4
45.0
7.4
12.1
46.4
38.3
55.5
Ours
42.7
65.6
34.6
12.1
77.9
38.5
61.5
78.9
86.9
$1.5
13.6
5,4
13.2
50.0
40.7
0.2
59.0
Table 10: Per-class results under AP@0.5 on ARKitScenes dataset.
Method
cab
fridg
shif
stove
bed
sink
wahr
tolt
bchtb
oven
dsbwshr
fiplce
stool
chr
1bl
TV
sofa
NeRF-Det
10.8
36.1
46.3
Ours
17.5
9.2
51.9
51.6
65.0
70.6
27.8
29.1
20.2
36.5
13

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The experiments support our claims.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Please see Section 6 Limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not propose any theories.
Guidelines:
The NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
14

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include the implementation details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
the data and code, with sufficient instructions faithfully reproduce the main experimental results, described supplemental material?
Question: Does the paper provide open
Answer: [No]
Justification:
We will release our code upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... httpas:///iis.cc/putttesse cc/puliic/
quides/CodeSubmi ssionPolccy for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ :/nniss//.///////////e:::eee cc/public/ qu..dess/ooeeSuubllesssoool..... for more details.
15

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommenedd, but including URLs to data and code is permitted.
6. Experimental Setting/Detan Details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include the dataset and implementation details in the experiment section,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: We follow baselines to not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We include it in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
16

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:[Yor Yes]
Justification:
We confirm that follow NeurlPS Code of Ethics.
Guidelines:
The answer means that authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [No]
Justification: Our work useful for 3D scene understanding. We do not have any negative social impact.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example. the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement in
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We follow baselines deseribe safeguards. We do foresee any misuse cases.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
properly cite datasets we use.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[NA]
Justification:
We do not release any assets.
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: not conduct crowdsourcing or research with Human Subjects
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
18

---

Justification:
We do not have such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
19


Reviews:
[{"review_id": "oT27Wn0OTW", "reviewer": "Reviewer_N557", "summary": "This work presents a method for multi-view 3d object detection. The method computed a MVS cost volume using a few planes, it then samples k likely depth values per pixel and builds a 3d feature volume based on the voxels close to the samples depth values weighted by their confidence. Additionally, during training pixel aligned gaussian splatting (GS) is used to provide an additional rendering loss to guide the depth estimation.", "questions": "Why does it not improve with more planes? Is the 3d grid too coarse?\nIt seems that 2 views are used for MVS, but 3 for GS. Why not use more?\nHow is the depth offset predicted? Like in MVSNet? Is it ever used for the object detection?", "limitations": "Limitations were mentioned.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed 3d feature volume construction seems to outperform the existing baselines.\nThe use of GS during training seems to also slightly improve the 3d object detection.", "weaknesses": "It is unclear how the nearby views are selected. Is it the same as in the existing works?\nTo evaluate the contribution of the probabilistic sampling alone it would be good to add a line in table 3 without probabilistic sampling and soft weighting.\nWhat is the benefit of using the PAGS rendering loss over the classical MVS photometric losses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R1/Q1] How to select nearby views.** We compute the Euclidean distance between the camera location of reference view and input views to find the nearby views.\n\n**[R1/Q2] No probabilistic sampling or soft weighting.** By comparing Row 2 and 3 of Tab 3 in our paper, we can already evaluate the effectiveness of probabilistic sampling as it shows severe performance decrease without probabilistic sampling. We also conduct an experiment of removing probabilistic sampling and soft weighting in **Tab. R1-N557/Q2** (attached PDF). Compared with Row 2 of the same table, not using probabilistic sampling has large detection performance drop, which again verifies the effectiveness of probabilistic sampling.\n\n**[R1/Q3] Compare with photometric loss.** The first row of **Tab. R1-N557/Q3** (attached PDF) shows the result of replacing PAGS with photometric loss. Photometric loss assumes the consistent pixel colors across nearby views and would fail in the case of occlusion (e.g. cluttered objects) or non-Lambertian surface (e.g. the varnish layer of the furniture), which is common in the indoor scene. Consequently, it only brings marginal improvement. In contrast, the PAGS can reconstruct the scene with 3D Gaussians and the spherical harmonics of each Gaussian model the view-dependent colors.\n\n**[R1/Q4] Why not improve with more planes?** We conjecture the saturation of performance at mAP@.25 vs. increase of performance at mAP@.5 from 12 to 16 planes in **Tab. R1-N557/Q4** (attached PDF) is likely due to the less strict evaluation at mAP@.25, which requires less accurate bounding box localization and consequently does not gain more information from the increase in the number of depth planes.\n\n**[R1/Q5] why not use more views?** We follow MVSNet[23] to use 2 views for MVS. We set 3 views in GS because we empirically find 3 nearby views is enough to cover the local area and adding more views does not bring significant performance as shown in **Tab. R1-N557/Q5** of attached PDF. 'GS=5' means using 5 views for GS and 2 views for MVS.\n\n**[R1/Q6] How to predict and use depth offset.** The depth offset is predicted through a MLP on top of the refined cost volume for each depth bin. Each offset is added to its corresponding depth bin to adjust the discrete depth planes. They are used in the probabilistic sampling of $\\text{d}_{\\text{idx}_k}$ (Line 191) and the depth map D (Eqn 8) of Gaussian Splatting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3EochI7fYE", "reviewer": "Reviewer_YYFF", "summary": "The manuscript proposes MVSDet, a multiview 3d object detection model that is evaluated on indoor scene datasets. Multiview information is lifted to 3D via an efficient per-frame depth sampling scheme. The most probable top-k depth values per pixel are used to lift 2D features into a global feature volume in a weighted way. Based on the thus accumulated 3d feature volume a 3d network regresses 3d bounding box parameters. In order to regularize the depth regression which is essential for constructing a occlusion-aware 3d feature volume, the manuscript proposes leveraging pixel-aligned Gaussian splats to construct a rendering loss against nearby views. Both the probabilistic depth sampling and the rendering loss are shown to contribute significantly to the performance of the model. Overall the model outperforms other models that do not use GT depth supervision during training.", "questions": "- Fig 1: it is kind of hard to see what is going on with the red points in the renderings. \n- Fig 2: \"ray intersects at 3 points (shown as dots)\" -- do you mean as red triangles? \n- Fig 2: the arrows in the gaussian splats orange box of the diagram are not very clear. This could be polished more.\n- l 151: it is unclear what is ment by 27 location canidates are selected for each target object? \n- l 184: how is the depth offset predicted in more detail and how is it used during sampling? Is it used for example in Eq 8 to adjust the depth planes per pixel?\n- ablation that does supervise the proposed model with gt depth instead of Gaussian splats rendering?\n- Tab 4: is the memory change per batch? (I assume since B=1)?", "limitations": "The limitations are addressed adequately in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The probabilistic depth estimation shows strong performance improvement over the dense planesweep depth approach when considering memory. Tab 3 is very effective in convincing me of the need for the probabilistic depth sampling. I do wonder how the numbers change without the regressed depth offset correction?\n\nShowing that Gaussian Splat-based rendering loss supports the performance of object detection model is useful since not all 3d object detection datasets do have surface GT for training. Tab 4 shows a small improvement for the more strict mAP threshold.\n\nThese two contributions are strongly supported by the ablation studies in the experiment section and useful to be shared with the community.", "weaknesses": "I do wonder how much the difference is to supervising with GT depth instead of gaussian splats. (The experiment in Fig 6 is similar but not quite the same since placing features at GT depth locations does not need the probabilistic depth model to regress the depth). This last experiment might drive home the effectiveness of Gaussian splats and allow direct comparison to the ImGeoNet and CN-RMA related works in Tab 1 and 2. \n\nThe writing and illustrations are mostly clear and support the understanding of the manuscript. There are quite a few open questions (see questions) that should be addressed to improve the presentation of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R2/Q1] No depth offset and how to use depth offset.** The first row of **Tab. R2-YYFF/Q1** of attached PDF shows the result of removing depth offset, which is worse than our model. Please refer to **R1/Q6** on how to predict and use depth offset.\n\n**[R2/Q2] Use GT depth as supervision.** **Tab. R2-YYFF/Q2** of attached PDF shows the ablation study of replacing Gaussian splats with ground truth depth supervision. The performance of our model is very close to using ground truth depth as supervision, which strongly verifies the effectiveness of Gaussian Splats. In addition, the 'GT Depth' model still cannot directly be compared with ImGeoNet or CN-RMA because the two models require the dense 3D geometry to supervise, which need tedious 3D reconstruction procedures on top of the raw RGB-D data.\n\n**[R2/Q3] Red points in Fig 1.** The red points are the voxel centers of the 3D volume for detection (see Sec 3.1 in our main paper). They never appear in the rendering branch. The rendering is done by splatting the Gaussian primitives predicted from the selected nearby views.\n\n**[R2/Q4,Q5] Fig 2.** The red triangles denote the selected depth locations by our probabilistic sampling. The three points (one green and two red) are the intersections of the ray in the 3D volume. Only the green point receives the corresponding pixel feature because it resides near the selected depth locations. The red points are the invalid backprojection locations. We will refine Fig 2 in the final version.\n\n**[R2/Q6] 27 locations.** Following NeRF-Det and ImVoxelNet, we apply center sampling to select candidate voxels in the 3D volume that are responsible to regress each target object. For each target object, we sample 27 voxel locations that are closest to the target object center. We will add the clarification to this part in the final paper.\n\n**[R2/Q7] Memory change per batch.** Yes. We set batch size to 1 and Tab 4 of main paper shows the memory change per batch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z37QhP6HDo", "reviewer": "Reviewer_ZuP3", "summary": "This paper mainly focuses on the problem of 3D object detection form multi-view images. It introduces MVSNet-like method for depth prediction, brings out probabilistic sampling, soft weighting and pixel-aligned Gaussian Splatting to improve the correctness, robustness of depth prediction, especially with sparse images. Therefore, it improves the performance for 2D features to be projected to the 3D space, therefore improves the performance of 3D object detection.", "questions": "1. I do not know about the 2D feature extractor, the detection head, and the detection head of this work, are they the same as NeRF-Det?\n2. Please further explain the novelty of your method, or your novel discovery that leads you to your method.\n3. I do not agree with the opinion of the article that ground truth geometry for supervision on the training stage is difficult to obtain, please further explain this claim, since your results are not better than CN-RMA and ImGeoNet that used ground truth geometry for supervision. \n4. Following my question 3, introducing ground truth geometry may cause significant time and memory consumption in training, and makes the network far more complicated and therefore increases time and memory cost in inference. Could you give me detailed time and memory consumption of your own method and all the baselines you have run, on both training and inference stages?", "limitations": "Yes, the authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq.\n\nThis paper is clear-written. It introduces probabilistic sampling, soft weighting and pixel-aligned Gaussian Splatting to improve the performance of MVSNet-based depth prediction to improve the performance of 3D object detection without the need of ground truth geometry.", "weaknesses": "1. The 2D feature extractor, the detection head, and the detection head of this work is not clearly described.\n2. The MVSNet-like depth prediction with probabilistic sampling and soft weighting, as well as the pixel-aligned Gaussian Splatting are all common methods or from previous works, I consider the novelty to be limited.\n3. Also, I do not agree with the opinion of the article that ground truth geometry for supervision on the training stage is difficult to obtain. As far as I know, To obtain supervision for 3D object detection (AABB or OBB bounding boxes), you need to use a lidar or a RGBD camera to obtain the geometry of a 3D scene, and then label the bounding boxes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R3/Q1] Feature extractor and detection head.** Yes, they are the same with NeRF-Det. We use the ResNet50 to extract image features at multiple stages, and fuse them via a feature pyramid network as the final feature map for each image. The 3D U-Net (Line 149) outputs feature maps at three scales, all of which are sent into the detection head. The detection head consists of three 3D convolutional layers for classification, location, and centerness prediction, respectively. From each voxel center and each of the three scales, the head estimates a class probability, a centerness score, and 3D bounding box offset. Please refer to NeRF-Det for more details.\n\n**[R3/Q2.1] Limited novelty.** We do not agree. **(1)** Compared to other MVS models (which predict one depth location obtained from probability weighted sum over all depth planes), we are the first to _probabilistically sample_ multiple top-scored depth locations. As agreed by **Reviewer YYFF**, our probabilistic sampling and soft weighting shows strong superiority over the normal mvs models when there is only few number of depth planes. **(2)** The pixel-aligned Gaussian Splatting is originally only used in the novel view synthesis. We are the first to apply it in the detection task to regularize the depth prediction with light computation overhead.\n\n**[R3/Q2.2] Novel discovery leading to our method.** As shown in Fig 1 of our main paper, NeRF-Det shows many wrong backprojections of 2D pixel features to the points in the free space due to the inaccurate geometry learned from NeRF. Therefore, we propose to use plane sweep to better estimate geometry. Compared to NeRF which does not have sufficient surface constraints (only predict a density score per point), the plane sweep approach can accurately predict the surface. However, the standard plane sweep method require sampling many depth planes to estimate the depth accurately, which leads to intractable computation for our multi-view 3D detection task. To tame computation complexity while maintaining accuracy, we propose probabilistic sampling and soft weighting together with the novel use of pixel-aligned Gaussian Splatting.\n\n**[R3/Q3] GT geometry is needed for labeling bboxes and thus should be used in supervision.** We disagree. 3D bounding boxes can be annotated _without_ using ground-truth (dense) geometry obtained from lidar or rgb-d cameras. An example is the Objectron Dataset **[1]**, where it annotates the 3D bounding boxes on a _sparse point cloud_ obtained via feature tracking on a AR device, and refines bounding boxes by re-projecting them onto the multi-view images (see Section 3.2 and 3.3 of their paper). \nDue to the requirement of dense geometry to compute TSDF or surface voxels for supervision, CN-RMA and ImGeoNet are largely restricted to certain datasets such as Scannet and ARKitScenes, where tedious heavy postprocessing of the raw data are needed to ensure high quality dense geometry. In contrast, our proposed method offers better versatility on datasets without dense geometry since we do not rely on the GT geometry for supervision.  \n**[1] Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations. CVPR 2021**\n\n**[R3/Q4] Time and memory comparison.** **Tab. R3-ZuP3/Q4** of attached PDF shows the comparison of time and memory in training and testing stages on the ScanNet, respectively. We omit comparison with ImGeoNet since it does not release any code. All models are run on 2 A6000 GPUs. Due to the complexity of CN-RMA (as mentioned in Sec 3.5 of their paper), it requires much longer time to train and evaluate than other models. Furthermore, CN-RMA consumes much more memory in the training stage because it requires joint end-to-end training of the 3D reconstruction and detection network. Although NeRF-Det is efficient in time and memory of training and testing stages, their performance is much worse than ours as shown in Table 1 and 2 of our main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their affirmation of the effectiveness of the proposed probabilistic sampling and soft weighting and the use of Gaussian Splatting for multi-view 3D object detection without using ground-truth geometry as supervision.  We strongly agree with **Reviewer YYFF** that the two contributions deserve to be shared with the community.  **All the experiment tables are included in the attached PDF**.

Meta-review: All three reviewers appreciate the methods, mainly its efficiency. Issues and questioned raised by the reviewers were clarified in the rebuttal and the discussion.
Decision: Accept (poster)