Title: MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3d object detection

plane sweep

Gaussian Splatting
Abstract: The key challenge of multi-view indoor 3D object detection is to infer accurate geometry information from images for precise 3D detection. Previous method relies on NeRF for geometry reasoning. However, the geometry extracted from NeRF is generally inaccurate, which leads to sub-optimal detection performance. In this paper, we propose MVSDet which utilizes plane sweep for geometry-aware 3D object detection. To circumvent the requirement for a large number of depth planes for accurate depth prediction, we design a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume. We select multiple locations that score top in the probability volume for each pixel and use their probability score to indicate the confidence. We further apply recent pixel-aligned Gaussian Splatting to regularize depth prediction and improve detection performance with little computation overhead. Extensive experiments on ScanNet and ARKitScenes datasets are conducted to show the superiority of our model. Our code is available at https://github.com/Pixie8888/MVSDet.

MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps

Abstract

The key challenge of multi-view indoor 3D object detection is to infer accurate geometry information from images for precise 3D detection. Previous method relies on NeRF for geometry reasoning. However, the geometry extracted from NeRF is generally inaccurate, which leads to sub-optimal detection performance. In this paper, we propose MVSDet which utilizes plane sweep for geometry-aware 3D object detection. To circumvent the requirement for a large number of depth planes for accurate depth prediction, we design a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume. We select multiple locations that score top in the probability volume for each pixel and use their probability score to indicate the confidence. We further apply recent pixel-aligned Gaussian Splatting to regularize depth prediction and improve detection performance with little computation overhead. Extensive experiments on ScanNet and ARKitScenes datasets are conducted to show the superiority of our model. Our code is available at https://github.com/Pixie8888/MVSDet .

1 Introduction

Indoor 3D object detection is a fundamental task in scene understanding and has wide applications in robotics, AR/VR equipment, etc. Although point cloud based 3D objection methods [13, 14, 15] have achieved impressive performance, depth sensors are required to capture the data, which may not be available due to budget limitation, form factor constraints, etc. Recently, the more economic pipeline of 3D object detection from only posed multi-view images is gaining increasing attention. However, it is much more sophisticated to estimate geometry information from 2D images alone.

A straightforward solution to this problem is using ground truth geometry information, e.g. point cloud [19] or TSDF [17], to supervise the model. Built on the 3D volume representation [16], ImGeoNet [19] predicts the emptiness of each voxel by converting the ground truth point clouds to surface voxels as supervision. CN-RMA [17] first reconstructs 3D scenes using ground truth TSDF as supervision and then runs an existing point cloud based object detector to predict bounding boxes. Although they achieve promising performance, the precise ground truth scene geometry is hard to obtain and may not be available [1].

An alternative way is to learn geometry via self-supervision. The pioneer work ImVoxelNet [16] unprojectes 2D image features to a 3D volume representation. However, 2D features can propagate to irrelevant 3D locations since depth information is not known. NeRF-Det [22] relies on a Neural Radiance Field (NeRF) [12] to learn a density field and queries an opacity score for each voxel. The

NeRF-Det

Ours

opacity score is multiplied with voxel feature to decrease the influence of voxels in the empty space to the feature volume. Since the detection performance is completely determined by the quality of NeRF, enormous effort is spent on making NeRF generalizable and avoiding aliasing issue. Unfortunately, the geometry extracted from NeRF remains unsatisfactory due to insufficient surface constraints in the representation [21]. Consequently, it wrongly backprojects features to voxels in the free space (as illustrated by the red dots in the example shown on the middle of Fig. 1).

In this paper, we propose MVSDet to extract geometry information from only the input multi-view images. A straightforward way is to leverage multi-view stereo algorithms [24, 25] to decide the accurate depths for correct placements of 2D features of each image to the 3D volume. However, accurate depth estimation in the plane-sweeping algorithm [8] requires computationally expensive sampling of many depth planes over all the multi-view images. We mitigate the computational complexity by proposing a probabilistic sampling and soft weighting mechanism to decide the possible depth locations for each pixel. Specifically, we sample multiple top scoring depth proposals in the probability volume that are most likely covering the true depth locations. Pixel features are placed onto the 3D volume only when the backprojected ray intersects at the depth locations. Since the normalized probability score indicates the confidence of the the current depth location, we use it to weigh the pixel feature before assigning the feature to its backprojected voxel center.

To further improve the depth prediction accuracy, we utilize the recent pixel-aligned Gaussian Splatting (PAGS) [3, 28] for novel view rendering as an additional supervision. PAGS predicts a 3D Gaussian primitive [9] for every pixel in the input views, and all the Gaussians are used to render novel views via rasterization-based splatting. Compare to NeRF that uses computation expensive volumetric sampling, Gaussian Splatting is fast and light-weight. A key to good rendering quality in PAGS is the correct positioning of the 3D Gaussians, which depends on an accurate depth prediction. As a result, by putting the Gaussians according to the depth map computed from the probability volume in the plane sweep module, the rendering loss would guide the the Gaussian centers and consequently the depths to the correct values.

In summary, our contributions are as follows:

We propose a probabilistic sampling and soft weighting mechanism to efficiently learn geometry without sampling many depth planes in multi-view stereo. Multiple depth proposals are sampled with the probability scores to guide the propagation of image features to 3D voxels.

We adopt pixel-aligned Gaussian Splatting to enhance depth prediction without much additional computation overhead, which consequently improves detection performance.

2 Related Work

Indoor 3D Object Detection. 3D object detection for indoor scenes predicts three dimensional bounding boxes and corresponding classes by taking in 3D or 2D inputs. Point cloud is the most popular choice of 3D data for detection as it provide accurate 3D information. To detect objects

from the irregularity and sparseness of the point cloud, V oteNet [14] utilizes Hough voting by points sub-sampling, voting and grouping to generate object proposals. Later methods improve VoteNet by either predicting geometric primitives [27] or using hierarchical graph network [4]. 3DETR [13] reduces hand-coded designs of VoteNet with transformer encoder-decoder blocks. Despite their promising performance, depth sensors are required to capture the data, which is not always available due to power consumption or budget limitation.

Alternatively, detecting 3D objects from images only [16, 22, 19, 17] is a cheaper choice, but with a sacrifice of losing geometry information. Some methods guide the model to predict scene geometry using ground truth geometry, i.e. ground truth surface voxels [19] or TSDF [17], as supervision. However, obtaining ground truth scene geometry is troublesome. In contrast, ImVoxelNet [16] builds a 3D feature volume in the world coordinate, where each voxel center aggregate the corresponding features of its projected 2D pixels. Subsequently, 3D U-Net is applied to refine the volume features and predict bounding boxes from each voxel center. However, voxels may wrongly aggregate irrelevant image features since depth is not known. Recently, NeRF-Det [22] uses NeRF to learn a density field and predict an opacity score per voxel center to downweigh the influence of voxels in the empty space to the feature volume. However, the implicit modeling of geometry in NeRF leads to unsatisfactory performance. Moreover, the reliance of NeRF during training side-tracked the effort in making NeRF generalizable. Instead of implicitly modeling geometry with NeRF, we utilize plane-swept cost volumes for geometry-aware scene reasoning. We propose a probabilistic sampling and soft weighting mechanism to accurately decide the placement of pixel features without sampling many depth planes.

Multi-View Depth Estimation. Multi-view depth estimation has long been studied in the multiview stereo [24, 20, 25, 5, 26, 11, 7]. MVSNet [24] constructs a 3D cost volume, regularize it with a 3D CNN and regresses the depth map from the probability volume. However, MVSNet consumes large memory due the expensive 3D cost volume. Follow-up works reduce computation by replacing 3D CNN with recurrent network [23, 25] or using coarse to fine depth estimation [5, 20, 2, 7]. Recurrent methods only reduces cost regularization module to a 2D network, but still faces a large 3D cost volume to predict accurate depth. Although coarse to fine pipeline can reduce the number of sampled depth planes, it still requires to sample a certain amount of initial depth locations, e.g. 32 to 64 planes [7, 20], to get a reasonable coarse depth map, which still leads to intractable computation in our multi-view object detection task. Bae et al. [2] propose to sample extremely few number ( i.e. 5) of initial depth planes based on a pre-trained single view depth probability distribution. It has been further applied to outdoor multi-view object detection [10]. However, both of them need the guidance of ground truth depth to learn a correct monocular depth estimation, and would otherwise fail as shown in the experiment section. Instead, we propose a probabilistic sampling and soft weighting module to bypass the large 3D cost volume to decide the 3D locations of every pixel. We also novelly utilize Gaussian Splatting to enhance our depth prediction with little computation overhead.

3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) [9] is a recent technique for novel view rendering. It models a 3D scene explicitly with Gaussian primitives, each of which is defined by a 3D Gaussian center, covairance matrix, opacity and color. Compared to volume rendering based NeRF [12], it achieves real-time rendering with much lighter computation. To avoid per-scene optimization of 3DGS, several works [28, 3, 6, 18] propose to model a scene with pixel-aligned Gaussians Splatting (PAGS). A Gaussian primitive is predicted per pixel, and all predicted Gaussians are then combined for novel view synthesis. A key to PAGS is the accurate 3D Gaussian center which is determined by the depth estimation. Thus, instead of targeting novel view synthesis, we novelly utilize it for 3D object detection through improving depth prediction. Our method is also significantly different from NeRF-Det. While NeRF plays a major role in NeRF-Det by predicting opacity scores, we use Gaussian Splatting as a regularizer to our plane sweep algorithm with little computation overhead.

3 Our Method

Problem Definition. The goal of multi-view indoor 3D object detection is to predict bounding box {B} ⊆ R 7 of objects in the 3D scene and their corresponding classes from N posed images { I 1 , · · · , I N } ∈ R H × W × 3 . Each bounding box B is parameterized as ( x, y, z, w, h, l, ϕ ) , where ( x, y, z ) are the coordinates of the box center, ( w,h,l ) are the width, height, and length, and ϕ is the

Selected Depth

Invalid

Valid

Depth Probability

Distribution

!

!

" #

"

!

" #

#

!

" #

$

!

$

!

!

$

!

!

= " #

#

!

!

!

Feature Volume

Feature Extractor

Feature

Maps

Cost

MLP

Rasterization-

based

Rendering

Select Nearby Views

&

%

&

&

via Eqn.8

&

'

&

(

Novel view

Detection Head

…

…

…

D

Input Images

GT Location

Select Nearby Views

rotation angle around z-axis. The intrinsic and extrinsic matrix for each input image are denoted as K ∈ R 3 × 3 and P = [ R | t ] ∈ R 3 × 4 , respectively.

Overview. Fig. 2 shows our proposed MVSDet, a geometry-aware approach for indoor 3D object detection from posed multi-view images. Our MVSDet is built on 3D volume-based object detection ( cf. Sec. 3.1). Instead of naively assigning the same 2D feature redundantly to every voxel center that intersect the backprojected ray, we place the pixel features according to the estimated depth from our proposed efficient plane sweep algorithm. We alleviate the costly sampling of many depth planes for accurate depth prediction by proposing a probabilistic sampling and soft weighting mechanism ( cf. Sec. 3.2). We further utilize pixel-aligned Gaussian Splatting to enhance our depth prediction module with little extra computation cost ( cf. Sec. 3.3). Intuitively, our depth-aware framework leads to more precise assignments of multi-view image features in the 3D volume and consequently better 3D object detection results.

3.1 Background: 3D Volume-Based Object Detection

Existing methods [16, 22] estimate bounding boxes from a 3D feature volume aggregated from the multi-view image features. Image features F ∈ R H 4 × W 4 × C are first extracted from every input image I ∈ R H × W × 3 . A 3D volume is defined in the world space with the size of N x × N y × N z voxels. Voxel center p = [ x, y, z ] ⊤ ∈ R 3 is projected to i -th input image to obtain the 2D coordinate [ u i , v i ] ⊤ ∈ R 2 as:

where K ′ i is the scaled intrinsic matrix according the image downsampling ratio. Subsequently, 2D feature f i ∈ R C is assigned to p via nearest neighbour interpolation as follow:

For p that are projected outside the boundary of feature map or behind the image plane, the projection is considered as invalid and we set f i = 0 . Finally, it averages all valid backprojected features as the voxel feature v = ∑ n i =1 f i /n ∈ R C , where n denotes the number of valid projection.

The feature volume is refined by a 3D U-Net before being fed into the detection head to predict category, a bounding box and centerness score on each voxel location. The training losses for detection consists of focal loss for classification L cls, cross-entropy loss for centerness L center, and IoU loss for location L loc :

Observation. We observe that the feature aggregation method in existing multi-view indoor 3D object detection works causes 2D pixel features to be duplicated on voxels intersecting with the ray emitted from camera origin though the pixel as illustrated in Fig. 3. This is a result from lack of depth-awareness where voxels can end up aggregating irrelevant pixel features that are either not on the surface or occluded.

Proposition. To circumvent this problem, we propose: 1) an efficient plane sweep to instill depth awareness. Our efficient plane sweep consists of a probabilistic sampling and soft weighting mechanism based on a cost volume representation to evaluate the placement of pixel features on the intersected voxel centers; 2) a depth prediction regularizor based on 3D Gaussian Splatting to improve depth accuracy.

3.2 Efficient Plane Sweep

Cost Volume Construction. We build N cost volumes for N input images to predict N depth probability distribution maps. To construct the cost volume for the i -th view, we set the image I i as the reference view and select 2 nearby views as the source views I j ∈ 2NB ( i ) . A raw matching volume for I i is constructed by backprojecting source feature map F j ∈ 2NB ( i ) ∈ R H 4 × W 4 × C into the coordinate system defined by I i at a stack of fronto-parallel virtual planes. The virtual planes { d 1 , . . . , d M } are uniformly sampled on a predefined depth range. The coordinate mapping from the source feature map F j to the reference feature map F i at depth d m is determined by a planar homography transformation:

where q is the homogeneous coordinate of a pixel on F i , q j,m is the projected homogeneous coordinate of q on F j . [ R ij | t ij ] are the rotation and the translation of I j relative to I i . We use Eqn. 4 to warp every source feature map F j ∈ 2NB ( i ) into all the depth planes and use a variance based metric [24] to generate a cost volume U ∈ R H 4 × W 4 × C × M . Subsequently, we use a shallow 3D CNN to refine U to generate a probability volume (after softmax) B ∈ R H 4 × W 4 × M . We also predict an offset per depth bin to account for the discretization error.

Probabilistic Sampling and Soft Weighting. Although we can place pixel features to the 3D volume by predicting depth using the weighted average based on B, it requires sampling sufficient depth planes, which is prohibitive for our task. To this end, we propose a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on its backprojected ray without the need for many depth planes.

For every pixel, we sample k depth locations that score top in the corresponding distribution in B as its depth proposals. We denote their depth values as { didx 1 , . . . , didx k } , and also use their corresponding probability score to evaluate its confidence with respect to the ground truth location. The scores are denoted as { ˜ Bidx 1 , . . . , ˜ Bidx k } , where ˜ Bidx k = Bidx k / ∑ i = k i =1 Bidx i . Intuitively, this is equivalent to split one depth prediction to multiple depth prediction based on the scores. As a result, it is more likely to cover the ground truth location when depth bins are insufficient.

Suppose a voxel center p ∈ R 3 intersects the backprojected ray of a pixel from the i -th image, and the depth of p under the i -th camera frustum is d(p). We consider the projection of point p to the i -th

Valid

Invalid

Volume

ImVoxelNet

Ours

GT

Location

image as valid and set the indicator g i = 1 when d(p) resides near any of the topk depth proposals { didx 1 , . . . , didx k } . We assign the normalized probability score of the nearest depth proposal to p as its confidence. The corresponding pixel feature is assigned to p weighted by the confidence score. The projection is invalid when d(p) is not close to any of the depth proposals, and we set the backprojected feature from i -th image as 0. Thus, the feature ˜ f i ∈ R C backprojected from the i -th image to point p and its corresponding indicator g i are given as follows:

where ϕ ( p ) is the index of depth proposal that is close to d(p), and ˜ B i ϕ ( p ) is the score of d ϕ ( p ) . After looping through every input image, we average all valid backprojected features for point p as its voxel feature ˆ v ∈ R C :

We also utilize the multi-view consistency to compute a surface score s for point p by averaging confidence scores ˜ B i ϕ ( p ) from every valid projection. We set s = 0 when the point does not have any valid projection, which means point p is in the free space. s is multiplied with voxel feature ˆ v as the final feature v ∈ R C for point p as follows:

The adoption of s shares similar spirit with existing methods [22, 19] to decrease the influence of empty space voxels in the feature volume. Yet, we learn it directly from multi-view images instead of relying on NeRF [22] or ground truth supervision [19].

3.3 Enhancing Depth Prediction with Gaussian Splatting

We further utilize the recent pixel-aligned Gaussian Splatting (PAGS) [3, 28] to enhance our depth prediction module. PAGS takes in sparse views and predict a Gaussian primitive per pixel. The parameters for each primitive are center µ , Gaussian opacity α , covariance Σ and color c. All the Gaussian primitives are combined together to render a novel view via rasterization-based splatting.

We select 3 nearby views per novel view from the images input to the detection branch , and predict Gaussian maps { M µ , M α , M Σ , M c } for the selected views. The Gaussian center map M µ ∈ R H 4 × W 4 × 3 are directly estimated from the predicted depth based on the probability volume B as follows:

where G = [ d 1 , . . . , d M ] ⊤ is the virtual depth planes and D ∈ R H 4 × W 4 × 1 is the estimated depth map. o ( r ) is the camera origin, ˆ D ( r ) is the projected ray depth obtained from the depth map D, and h ( r ) is the ray direction for pixel r . The Gaussian opacity map M α ∈ R H 4 × W 4 × 1 is predicted by taking the max probability score of B as follow:

The Gaussian covariance map M Σ ∈ R H 4 × W 4 × 16 and color map M c ∈ R H 4 × W 4 × 3 are predicted from a MLP as follows:

where ∥ denotes concatenation and ˆ I ∈ R H 4 × W 4 × 3 denotes the resized image map. Following 3DGS [9], Σ is predicted by a rotation quaternion and scaling factors. Color is predicted by spherical harmonics coefficients.

We render the image color ˆ Ccolor via alpha-blending and the rendering loss is a L2 loss as follows:

One of the key factors for good rendering is accurate M µ , which is directly related to correct depth estimation from our model ( cf. Eqn. 8). The rendering loss thus iteratively guides the Gaussians to the correct 3D locations, and consequently benefits our detection pipeline. Our final loss is L = L det + L render .

Ground-truth

NeRF-Det

Ours

4 Experiments

4.1 Datasets

We conduct experiments on the ScanNet and ARKitScenes datasets. ScanNet has 1,201 and 312 scans for training and testing, respectively. We detect axis-aligned bounding boxes for 18 classes. ARKitScenes has 4,498 and 549 scans for training and testing, respectively. We detect oriented bounding boxes for 17 class. We adopt mean average precision (mAP) with thresholds of 0.25 and 0.5 as the evaluation metrics.

4.2 Implementation Details

Weuse the same feature extractor, training and testing configurations as [22] for detection. Specifically, images are resized into (240 , 320) . During training, we input 40 images to the detection branch. During testing, the detection branch takes in 100 images and the rendering branch is removed. The size of the 3D volume is ( N x = 40 , N y = 40 , N z = 16) , with each voxel represents a cube of 0.16m × 0.16m × 0.2m The depth range is empirically set as [0 . 2 m , 5 m ] . The number of depth planes M is set to 12 and k = 3 depth proposals are selected in the probabilistic sampling. We consider d(p) ⊂ { didx 1 , . . . , didx k } if d(p) is within ± 0 . 2 mof any depth proposals. For the rendering branch, we select another two images as the target views that do not overlap with the images in the detection branch. We use AdamW optimizer with learning rate 0.0002, total epochs of 12 and batchsize of 1. All experiments are conducted on two NVIDIA A6000 GPUs.

4.3 Comparison with Baselines

We compare our method with ImGeoNet [19], CN-RMA [17], ImVoxelNet [16] and NeRF-Det [22]. We directly report their results from the CN-RMA [17] paper. Note that ImGeoNet and CN-RMA use ground truth 3D geometry as supervision during training. Tab. 1 and Tab. 2 show the results on ScanNet and ARKitScenes, respectively. It is expected that using ground truth geometry as supervision can achieve good performance. However, ground truth geometry may not be accessible and therefore we seek for a self-supervised approach that do not rely its supervision. ImV oxelNet and NeRF-Det are the two existing methods that leverage self-supervision to learn geometry for multiview 3D detection. Compared to these works, our method achieve much better performance. It clearly shows the superiority of our efficient plane sweep method over the vanilla feature backprojection in ImVoxeNet and the density field in NeRF-Det. Fig. 1 shows the qualitative results on ScanNet. The first two columns show that our model can detect more target objects. We also find that NeRF-Det tends to detect objects in the free space (last two colums), which is due to inaccurate feature projection. In contrast, our model is able to place bounding boxes at more accurate locations.

4.4 Ablation Study

Effectiveness of Probabilistic Sampling and Soft Weighting. Tab. 3 shows the ablation stdudy of the probabilistic sampling and soft weghting (PSSW). All methods are conducted without the rendering loss L render to test the performance of PSSW alone. Removing 'Probabilistic Sampling' means replacing topk sampling with a single depth estimation computed by the weighted average of depth probability volume B. We use the max probability score as the weight in this case. Removing 'Soft Weighting' means replacing ˜ B i ϕ ( p ) with value 1. As shown in the table, removing either 'Probabilistic Sampling' or 'Soft Weighting' causes drastic drop in the performance. Particularly, removing probabilistic sampling depth proposals drops by 19.1 at mAP@.25. It suggests that estimating accurate depth is very hard under insufficient depth planes. Furthermore, soft weighting is very important to our model as it can decrease the influence of wrong depth proposals introduced by the sampling. Overall, both probabilistic sampling and soft weighting are crucial to our model.

Effectiveness of Gaussian Splatting for Depth Prediction. Tab. 4 shows the ablation study for using pixel-aligned Gaussian splatting (PAGS) for detection. 'RMSE' evaluates the average quality of depth prediction for the images in the detection branch. Increasing depth planes to M = 16 or using Gaussian Splatting both improve the depth estimation and bring similar improvement to the detection performance. However, using PAGS consumes much less memory during training than M = 16 , i.e. adding only 28% memories compared to increasing depth planes. Moreover, PAGS does not bring any additional memory cost during testing because it is removed for detection. We visualize the depth maps predicted by the probability volume B in Fig. 5. The depth maps are of 1/16 size of the original image since we estimate depth on the feature map level. It shows that PAGS significantly improves the depth quality.

Image

GT Depth

Image

GT Depth

Analysis of the number of Topk depth proposals. Tab. 5 shows the ablation study of number of depth proposals. Sampling top-1 depth proposal leads to severe performance decrease as the correct depth location is harder to be selected. Sampling too many depth proposals ( k = 5 ) also leads to some decrease since more inaccurate locations are sampled. We thus choose k = 3 in our model.

Comparison with Depth Estimation Methods.

Fig. 6 shows the comparison of different depth prediction methods to the detection performance. MVSNet [24] performs one-time plane sweep by using M = 16 depth planes. BEVStereo [10] performs iterative depth estimation by sampling depth planes according to a monocular depth estimation. We set the number of iterations to 3 and the number of depth planes in each iteration to 5. We remove ground truth depth supervision for BEVStereo for fair comparison. 'Ground-truth Depth' denotes placing pixel feature on the 3D volume according to the ground-truth depth location, which is the upper bound of our method. We show the results of our approach using different number of depth planes ( M = 12 and M = 8 ). MVSNet performs badly even though it samples

56.2

33.2

9.4

60.5

54.9

5

15

25

35

45

55

65

0

4

8

12

16

20

Number of Depth Planes

M

mAP@.25

Ours

MVSNet

BEVStereo

Ground-truthDepth

more planes than us. This is because MVSNet requires sufficient depth planes to estimate depth correctly. BEVStereo also fails because it requires the ground truth depth supervision to learn a good initial monocular depth estimation. In contrast, our model achieve close performance as the 'Ground-truth Depth' using only 12 or even 8 depth planes. This demonstrates that our approach efficiently learns geometry without sampling many depth planes.

Time and memory comparison. Tab. 6 shows the comparison of time and memory in the training and testing stages on ScanNet, respectively. We omit comparison with ImGeoNet since it does not release any code. All models are ran on 2 × A6000 GPUs. Due to the complexity of CN-RMA (as mentioned in Sec 3.5 of their paper), it requires much longer time to train and evaluate than other models. Furthermore, CN-RMA consumes much more memory in the training stage because it requires joint end-to-end training of the 3D reconstruction and detection network. Although NeRF-Det is efficient in time and memory of the training and testing stages, their performance is much worse than ours as shown in Tab. 1 and 2.

5 Conclusion

In this paper, we propose MVSDet for multi-view image based indoor 3D object detection. We design a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume without the need of the computationally sampling of many depth planes. We further introduce the use of pixel-aligned Gaussian Splatting to improve depth prediction with little computation overhead. Extensive experiments on two benchmark datasets demonstrate the superiority of our method.

6 Limitation

Similar to existing multi-view stereo methods [24, 20], feature matching would fail on texture-less or reflective surfaces. One possible solution is to combine with monocular depth estimation [2]. However, estimating monocular depth is non-trivial and we leave it for future research.

Acknowledgement. This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021).

References

Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7822-7831, 2021.

Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Multi-view depth estimation by fusing single-view depth probability with multi-view geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2842-2851, 2022.

David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312.12337 , 2023.

Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z Chen, and Jian Wu. A hierarchical graph network for 3d object detection on point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 392-401, 2020.

Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1538-1547, 2019.

Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.14627 , 2024.

Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2524-2534, 2020.

David Gallup, Jan-Michael Frahm, Philippos Mordohai, Qingxiong Yang, and Marc Pollefeys. Real-time plane-sweeping stereo with multiple sweeping directions. In 2007 IEEE Conference on Computer Vision and Pattern Recognition , 2007.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4):1-14, 2023.

Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1486-1494, 2023.

Xiaoxiao Long, Lingjie Liu, Wei Li, Christian Theobalt, and Wenping Wang. Multi-view depth estimation using epipolar spatio-temporal networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8258-8267, 2021.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , pages 405-421. Springer, 2020.

Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2906-2917, 2021.

Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9277-9286, 2019.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Fcaf3d: Fully convolutional anchor-free 3d object detection. In European Conference on Computer Vision , pages 477-493. Springer, 2022.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2397-2406, 2022.

Guanlin Shen, Jingwei Huang, Zhihua Hu, and Bin Wang. Cn-rma: Combined network with ray marching aggregation for 3d indoors object detection from multi-view images. arXiv preprint arXiv:2403.04198 , 2024.

Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv preprint arXiv:2312.13150 , 2023.

Tao Tu, Shun-Po Chuang, Yu-Lun Liu, Cheng Sun, Ke Zhang, Donna Roy, Cheng-Hao Kuo, and Min Sun. Imgeonet: Image-induced geometry-aware voxel representation for multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6996-7007, 2023.

Fangjinhua Wang, Silvano Galliani, Christoph Vogel, and Marc Pollefeys. Itermvs: Iterative probability estimation for efficient multi-view stereo. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8606-8615, 2022.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, et al. Nerf-det: Learning geometry-aware volumetric representation for multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23320-23330, 2023.

Jianfeng Yan, Zizhuang Wei, Hongwei Yi, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu-Wing Tai. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. In European conference on computer vision , pages 674-689. Springer, 2020.

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV) , pages 767-783, 2018.

Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for highresolution multi-view stereo depth inference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5525-5534, 2019.

Zehao Yu and Shenghua Gao. Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1949-1958, 2020.

Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16 , pages 311-329. Springer, 2020.

Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. arXiv preprint arXiv:2312.02155 , 2023.

A Appendix / Supplemental Material

A.1 Qualitative Results on ARKitScenes Dataset

Ground-truth

NeRF-Det

Ours

A.2 Novel View Synthesis Results

Fig. 8 shows the novel view synthesis results on ScanNet test dataset. Our model gives reasonable rendering results, indicating that the Gaussian Splatting module successfully learn the geometry. Note that the Gaussian Splatting module is only a regularizer to our plane sweep algorithm instead of the determining factor to learn geometry like NeRF in NeRF-Det [22].

Rendering

GT

Rendering

GT

Rendering

GT

Rendering

GT

A.3 Per-Class Performance

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The experiments support our claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Section 6 Limitation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not propose any theories.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include the implementation details in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release our code upon paper acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include the dataset and implementation details in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We follow baselines to not report error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include it in the implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer:[Yes]

Justification: We confirm that we follow NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Our work is useful for 3D scene understanding. We do not have any negative social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We follow baselines to not describe safeguards. We do not foresee any misuse cases.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite the datasets we use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer:[NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not conduct crowdsourcing or research with Human Subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not have such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly focuses on the problem of 3D object detection form multi-view images. It introduces MVSNet-like method for depth prediction, brings out probabilistic sampling, soft weighting and pixel-aligned Gaussian Splatting to improve the correctness, robustness of depth prediction, especially with sparse images. Therefore, it improves the performance for 2D features to be projected to the 3D space, therefore improves the performance of 3D object detection.\n\nSTRENGTHS:\nA substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq.\n\nThis paper is clear-written. It introduces probabilistic sampling, soft weighting and pixel-aligned Gaussian Splatting to improve the performance of MVSNet-based depth prediction to improve the performance of 3D object detection without the need of ground truth geometry.\n\nWEAKNESSES:\n1. The 2D feature extractor, the detection head, and the detection head of this work is not clearly described.\n2. The MVSNet-like depth prediction with probabilistic sampling and soft weighting, as well as the pixel-aligned Gaussian Splatting are all common methods or from previous works, I consider the novelty to be limited.\n3. Also, I do not agree with the opinion of the article that ground truth geometry for supervision on the training stage is difficult to obtain. As far as I know, To obtain supervision for 3D object detection (AABB or OBB bounding boxes), you need to use a lidar or a RGBD camera to obtain the geometry of a 3D scene, and then label the bounding boxes.\n\nQUESTIONS:\n1. I do not know about the 2D feature extractor, the detection head, and the detection head of this work, are they the same as NeRF-Det?\n2. Please further explain the novelty of your method, or your novel discovery that leads you to your method.\n3. I do not agree with the opinion of the article that ground truth geometry for supervision on the training stage is difficult to obtain, please further explain this claim, since your results are not better than CN-RMA and ImGeoNet that used ground truth geometry for supervision. \n4. Following my question 3, introducing ground truth geometry may cause significant time and memory consumption in training, and makes the network far more complicated and therefore increases time and memory cost in inference. Could you give me detailed time and memory consumption of your own method and all the baselines you have run, on both training and inference stages?\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes MVSDet, a multiview 3d object detection model that is evaluated on indoor scene datasets. Multiview information is lifted to 3D via an efficient per-frame depth sampling scheme. The most probable top-k depth values per pixel are used to lift 2D features into a global feature volume in a weighted way. Based on the thus accumulated 3d feature volume a 3d network regresses 3d bounding box parameters. In order to regularize the depth regression which is essential for constructing a occlusion-aware 3d feature volume, the manuscript proposes leveraging pixel-aligned Gaussian splats to construct a rendering loss against nearby views. Both the probabilistic depth sampling and the rendering loss are shown to contribute significantly to the performance of the model. Overall the model outperforms other models that do not use GT depth supervision during training.\n\nSTRENGTHS:\nThe probabilistic depth estimation shows strong performance improvement over the dense planesweep depth approach when considering memory. Tab 3 is very effective in convincing me of the need for the probabilistic depth sampling. I do wonder how the numbers change without the regressed depth offset correction?\n\nShowing that Gaussian Splat-based rendering loss supports the performance of object detection model is useful since not all 3d object detection datasets do have surface GT for training. Tab 4 shows a small improvement for the more strict mAP threshold.\n\nThese two contributions are strongly supported by the ablation studies in the experiment section and useful to be shared with the community.\n\nWEAKNESSES:\nI do wonder how much the difference is to supervising with GT depth instead of gaussian splats. (The experiment in Fig 6 is similar but not quite the same since placing features at GT depth locations does not need the probabilistic depth model to regress the depth). This last experiment might drive home the effectiveness of Gaussian splats and allow direct comparison to the ImGeoNet and CN-RMA related works in Tab 1 and 2. \n\nThe writing and illustrations are mostly clear and support the understanding of the manuscript. There are quite a few open questions (see questions) that should be addressed to improve the presentation of the method.\n\nQUESTIONS:\n- Fig 1: it is kind of hard to see what is going on with the red points in the renderings. \n- Fig 2: \"ray intersects at 3 points (shown as dots)\" -- do you mean as red triangles? \n- Fig 2: the arrows in the gaussian splats orange box of the diagram are not very clear. This could be polished more.\n- l 151: it is unclear what is ment by 27 location canidates are selected for each target object? \n- l 184: how is the depth offset predicted in more detail and how is it used during sampling? Is it used for example in Eq 8 to adjust the depth planes per pixel?\n- ablation that does supervise the proposed model with gt depth instead of Gaussian splats rendering?\n- Tab 4: is the memory change per batch? (I assume since B=1)?\n\nLIMITATIONS:\nThe limitations are addressed adequately in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This work presents a method for multi-view 3d object detection. The method computed a MVS cost volume using a few planes, it then samples k likely depth values per pixel and builds a 3d feature volume based on the voxels close to the samples depth values weighted by their confidence. Additionally, during training pixel aligned gaussian splatting (GS) is used to provide an additional rendering loss to guide the depth estimation.

STRENGTHS:
The proposed 3d feature volume construction seems to outperform the existing baselines.
The use of GS during training seems to also slightly improve the 3d object detection.

WEAKNESSES:
It is unclear how the nearby views are selected. Is it the same as in the existing works?
To evaluate the contribution of the probabilistic sampling alone it would be good to add a line in table 3 without probabilistic sampling and soft weighting.
What is the benefit of using the PAGS rendering loss over the classical MVS photometric losses?

QUESTIONS:
Why does it not improve with more planes? Is the 3d grid too coarse?
It seems that 2 views are used for MVS, but 3 for GS. Why not use more?
How is the depth offset predicted? Like in MVSNet? Is it ever used for the object detection?

LIMITATIONS:
Limitations were mentioned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12