MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps

Authors: yating  xu
chen  li
gim hee lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 09:59:34

Abstract: The key challenge of multi-view indoor 3D object detection is to infer accurate geometry information from images for precise 3D detection. Previous method relies on NeRF for geometry reasoning. However, the geometry extracted from NeRF is generally inaccurate, which leads to sub-optimal detection performance. In this paper, we propose MVSDet which utilizes plane sweep for geometry-aware 3D object detection. To circumvent the requirement for a large number of depth planes for accurate depth prediction, we design a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume. We select multiple locations that score top in the probability volume for each pixel and use their probability score to indicate the confidence. We further apply recent pixel-aligned Gaussian Splatting to regularize depth prediction and improve detection performance with little computation overhead. Extensive experiments on ScanNet and ARKitScenes datasets are conducted to show the superiority of our model. Our code is available at https://github.com/Pixie8888/MVSDet.

Main paper: 
 introduction: 
 Indoor 3D object detection is a fundamental task in scene understanding and has wide applications in robotics, AR/VR equipment, etc. Although point cloud based 3D objection methods [13, 14, 15] have achieved impressive performance, depth sensors are required to capture the data, which may not be available due to budget limitation, form factor constraints, etc. Recently, the more economic pipeline of 3D object detection from only posed multi-view images is gaining increasing attention. However, it is much more sophisticated to estimate geometry information from 2D images alone.A straightforward solution to this problem is using ground truth geometry information, e.g. point cloud [19] or TSDF [17] , to supervise the model. Built on the 3D volume representation [16] , ImGeoNet [19] predicts the emptiness of each voxel by converting the ground truth point clouds to surface voxels as supervision. CN-RMA [17] first reconstructs 3D scenes using ground truth TSDF as supervision and then runs an existing point cloud based object detector to predict bounding boxes. Although they achieve promising performance, the precise ground truth scene geometry is hard to obtain and may not be available [1] .An alternative way is to learn geometry via self-supervision. The pioneer work ImVoxelNet [16] unprojectes 2D image features to a 3D volume representation. However, 2D features can propagate to irrelevant 3D locations since depth information is not known. NeRF-Det [22] relies on a Neural Radiance Field (NeRF) [12] to learn a density field and queries an opacity score for each voxel. The Figure 1 : Comparison with NeRF-Det [22] . The 3D voxel centers (grey dots) are overlaid with the reference scene. The red dots denotes the erroneous backprojection pixel features to the points in the free space. Compared to NeRF-Det, we show much less inaccurate backprojections. opacity score is multiplied with voxel feature to decrease the influence of voxels in the empty space to the feature volume. Since the detection performance is completely determined by the quality of NeRF, enormous effort is spent on making NeRF generalizable and avoiding aliasing issue. Unfortunately, the geometry extracted from NeRF remains unsatisfactory due to insufficient surface constraints in the representation [21] . Consequently, it wrongly backprojects features to voxels in the free space (as illustrated by the red dots in the example shown on the middle of Fig. 1 ).In this paper, we propose MVSDet to extract geometry information from only the input multi-view images. A straightforward way is to leverage multi-view stereo algorithms [24, 25] to decide the accurate depths for correct placements of 2D features of each image to the 3D volume. However, accurate depth estimation in the plane-sweeping algorithm [8] requires computationally expensive sampling of many depth planes over all the multi-view images. We mitigate the computational complexity by proposing a probabilistic sampling and soft weighting mechanism to decide the possible depth locations for each pixel. Specifically, we sample multiple top scoring depth proposals in the probability volume that are most likely covering the true depth locations. Pixel features are placed onto the 3D volume only when the backprojected ray intersects at the depth locations. Since the normalized probability score indicates the confidence of the the current depth location, we use it to weigh the pixel feature before assigning the feature to its backprojected voxel center.To further improve the depth prediction accuracy, we utilize the recent pixel-aligned Gaussian Splatting (PAGS) [3, 28] for novel view rendering as an additional supervision. PAGS predicts a 3D Gaussian primitive [9] for every pixel in the input views, and all the Gaussians are used to render novel views via rasterization-based splatting. Compare to NeRF that uses computation expensive volumetric sampling, Gaussian Splatting is fast and light-weight. A key to good rendering quality in PAGS is the correct positioning of the 3D Gaussians, which depends on an accurate depth prediction. As a result, by putting the Gaussians according to the depth map computed from the probability volume in the plane sweep module, the rendering loss would guide the the Gaussian centers and consequently the depths to the correct values.In summary, our contributions are as follows:1. We propose a probabilistic sampling and soft weighting mechanism to efficiently learn geometry without sampling many depth planes in multi-view stereo. Multiple depth proposals are sampled with the probability scores to guide the propagation of image features to 3D voxels.2. We adopt pixel-aligned Gaussian Splatting to enhance depth prediction without much additional computation overhead, which consequently improves detection performance.3. We conduct extensive experiments on the ScanNet and ARKitScene datasets to verify the effectivess of our method. Notably, we achieve significant improvements of +3.9 and +5.1 under the mAP@0.5 metric on ScanNet and ARKitScenes, respectively. 
related work: 
 Indoor 3D Object Detection. 3D object detection for indoor scenes predicts three dimensional bounding boxes and corresponding classes by taking in 3D or 2D inputs. Point cloud is the most popular choice of 3D data for detection as it provide accurate 3D information. To detect objects from the irregularity and sparseness of the point cloud, VoteNet [14] utilizes Hough voting by points sub-sampling, voting and grouping to generate object proposals. Later methods improve VoteNet by either predicting geometric primitives [27] or using hierarchical graph network [4] . 3DETR [13] reduces hand-coded designs of VoteNet with transformer encoder-decoder blocks. Despite their promising performance, depth sensors are required to capture the data, which is not always available due to power consumption or budget limitation.Alternatively, detecting 3D objects from images only [16, 22, 19, 17] is a cheaper choice, but with a sacrifice of losing geometry information. Some methods guide the model to predict scene geometry using ground truth geometry, i.e. ground truth surface voxels [19] or TSDF [17] , as supervision. However, obtaining ground truth scene geometry is troublesome. In contrast, ImVoxelNet [16] builds a 3D feature volume in the world coordinate, where each voxel center aggregate the corresponding features of its projected 2D pixels. Subsequently, 3D U-Net is applied to refine the volume features and predict bounding boxes from each voxel center. However, voxels may wrongly aggregate irrelevant image features since depth is not known. Recently, NeRF-Det [22] uses NeRF to learn a density field and predict an opacity score per voxel center to downweigh the influence of voxels in the empty space to the feature volume. However, the implicit modeling of geometry in NeRF leads to unsatisfactory performance. Moreover, the reliance of NeRF during training side-tracked the effort in making NeRF generalizable. Instead of implicitly modeling geometry with NeRF, we utilize plane-swept cost volumes for geometry-aware scene reasoning. We propose a probabilistic sampling and soft weighting mechanism to accurately decide the placement of pixel features without sampling many depth planes.Multi-View Depth Estimation. Multi-view depth estimation has long been studied in the multiview stereo [24, 20, 25, 5, 26, 11, 7] . MVSNet [24] constructs a 3D cost volume, regularize it with a 3D CNN and regresses the depth map from the probability volume. However, MVSNet consumes large memory due the expensive 3D cost volume. Follow-up works reduce computation by replacing 3D CNN with recurrent network [23, 25] or using coarse to fine depth estimation [5, 20, 2, 7] . Recurrent methods only reduces cost regularization module to a 2D network, but still faces a large 3D cost volume to predict accurate depth. Although coarse to fine pipeline can reduce the number of sampled depth planes, it still requires to sample a certain amount of initial depth locations, e.g. 32 to 64 planes [7, 20] , to get a reasonable coarse depth map, which still leads to intractable computation in our multi-view object detection task. Bae et al. [2] propose to sample extremely few number (i.e. 5) of initial depth planes based on a pre-trained single view depth probability distribution. It has been further applied to outdoor multi-view object detection [10] . However, both of them need the guidance of ground truth depth to learn a correct monocular depth estimation, and would otherwise fail as shown in the experiment section. Instead, we propose a probabilistic sampling and soft weighting module to bypass the large 3D cost volume to decide the 3D locations of every pixel. We also novelly utilize Gaussian Splatting to enhance our depth prediction with little computation overhead.3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) [9] is a recent technique for novel view rendering. It models a 3D scene explicitly with Gaussian primitives, each of which is defined by a 3D Gaussian center, covairance matrix, opacity and color. Compared to volume rendering based NeRF [12] , it achieves real-time rendering with much lighter computation. To avoid per-scene optimization of 3DGS, several works [28, 3, 6, 18] propose to model a scene with pixel-aligned Gaussians Splatting (PAGS). A Gaussian primitive is predicted per pixel, and all predicted Gaussians are then combined for novel view synthesis. A key to PAGS is the accurate 3D Gaussian center which is determined by the depth estimation. Thus, instead of targeting novel view synthesis, we novelly utilize it for 3D object detection through improving depth prediction. Our method is also significantly different from NeRF-Det. While NeRF plays a major role in NeRF-Det by predicting opacity scores, we use Gaussian Splatting as a regularizer to our plane sweep algorithm with little computation overhead.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please see Section 6 Limitation.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
our method: 
 Problem Definition. The goal of multi-view indoor 3D object detection is to predict bounding box {B} ⊆ R 7 of objects in the 3D scene and their corresponding classes from N posed images , but only the green point receives the pixel feature based on the selected depth proposals. The red points are denoted as invalid backprojection location and thus the pixel feature is not assigned to them. "GT Location" is the ground truth 3D location of the pixel. The lower branch shows the pixel-aligned Gaussian Splatting (PAGS). We select nearby views for the novel image from the images input to the detection branch and predict Gaussian maps on them. Note that PAGS is removed during testing. rotation angle around z-axis. The intrinsic and extrinsic matrix for each input image are denoted as{I 1 , • • • , I N } ∈ R H×W ×3 .! ! " # " ! " # # ! " # $ ! $ ! ! $ ! ! = " # # ! ! !K ∈ R 3×3 and P = [R | t] ∈ R 3×4 , respectively.Overview. Fig. 2 shows our proposed MVSDet, a geometry-aware approach for indoor 3D object detection from posed multi-view images. Our MVSDet is built on 3D volume-based object detection (cf. Sec. 3.1). Instead of naively assigning the same 2D feature redundantly to every voxel center that intersect the backprojected ray, we place the pixel features according to the estimated depth from our proposed efficient plane sweep algorithm. We alleviate the costly sampling of many depth planes for accurate depth prediction by proposing a probabilistic sampling and soft weighting mechanism (cf. Sec. 3.2). We further utilize pixel-aligned Gaussian Splatting to enhance our depth prediction module with little extra computation cost (cf. Sec. 3.3). Intuitively, our depth-aware framework leads to more precise assignments of multi-view image features in the 3D volume and consequently better 3D object detection results.Existing methods [16, 22] estimate bounding boxes from a 3D feature volume aggregated from the multi-view image features. Image featuresEQUATIONwhere K ′ i is the scaled intrinsic matrix according the image downsampling ratio. Subsequently, 2D feature f i ∈ R C is assigned to p via nearest neighbour interpolation as follow:EQUATIONFor p that are projected outside the boundary of feature map or behind the image plane, the projection is considered as invalid and we set f i = 0. Finally, it averages all valid backprojected features as thevoxel feature v = n i=1 f i /n ∈ R C, where n denotes the number of valid projection.The feature volume is refined by a 3D U-Net before being fed into the detection head to predict category, a bounding box and centerness score on each voxel location. The training losses for detection consists of focal loss for classification L cls , cross-entropy loss for centerness L center , and IoU loss for location L loc :L det = L cls + L center + L loc .(3) Observation. We observe that the feature aggregation method in existing multi-view indoor 3D object detection works causes 2D pixel features to be duplicated on voxels intersecting with the ray emitted from camera origin though the pixel as illustrated in Fig. 3 . This is a result from lack of depth-awareness where voxels can end up aggregating irrelevant pixel features that are either not on the surface or occluded.Proposition. To circumvent this problem, we propose: 1) an efficient plane sweep to instill depth awareness. Our efficient plane sweep consists of a probabilistic sampling and soft weighting mechanism based on a cost volume representation to evaluate the placement of pixel features on the intersected voxel centers; 2) a depth prediction regularizor based on 3D Gaussian Splatting to improve depth accuracy. Cost Volume Construction. We build N cost volumes for N input images to predict N depth probability distribution maps. To construct the cost volume for the i-th view, we set the image I i as the reference view and select 2 nearby views as the source views I j∈2NB(i) . A raw matching volume for I i is constructed by backprojecting source feature mapValid Invalid Volume ImVoxelNet Ours GT LocationF j∈2NB(i) ∈ R H 4 × W4 ×C into the coordinate system defined by I i at a stack of fronto-parallel virtual planes. The virtual planes {d 1 , . . . , d M } are uniformly sampled on a predefined depth range. The coordinate mapping from the source feature map F j to the reference feature map F i at depth d m is determined by a planar homography transformation:EQUATION)where q is the homogeneous coordinate of a pixel on F i , q j,m is the projected homogeneous coordinate of q on F j . [R ij | t ij ] are the rotation and the translation of I j relative to I i . We use Eqn. 4 to warp every source feature map F j∈2NB(i) into all the depth planes and use a variance based metric [24] to generate a cost volume U ∈ R Probabilistic Sampling and Soft Weighting. Although we can place pixel features to the 3D volume by predicting depth using the weighted average based on B, it requires sampling sufficient depth planes, which is prohibitive for our task. To this end, we propose a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on its backprojected ray without the need for many depth planes.For every pixel, we sample k depth locations that score top in the corresponding distribution in B as its depth proposals. We denote their depth values as {d idx1 , . . . , d idx k }, and also use their corresponding probability score to evaluate its confidence with respect to the ground truth location. The scores are denoted as { Bidx1 , . . . , Bidx k }, where Bidx k = B idx k / i=k i=1 B idxi . Intuitively, this is equivalent to split one depth prediction to multiple depth prediction based on the scores. As a result, it is more likely to cover the ground truth location when depth bins are insufficient. Suppose a voxel center p ∈ R 3 intersects the backprojected ray of a pixel from the i-th image, and the depth of p under the i-th camera frustum is d(p). We consider the projection of point p to the i-th image as valid and set the indicator g i = 1 when d(p) resides near any of the top-k depth proposals {d idx1 , . . . , d idx k }. We assign the normalized probability score of the nearest depth proposal to p as its confidence. The corresponding pixel feature is assigned to p weighted by the confidence score. The projection is invalid when d(p) is not close to any of the depth proposals, and we set the backprojected feature from i-th image as 0. Thus, the feature fi ∈ R C backprojected from the i-th image to point p and its corresponding indicator g i are given as follows:EQUATIONwhere ϕ(p) is the index of depth proposal that is close to d(p), and Bi ϕ(p) is the score of d ϕ(p) . After looping through every input image, we average all valid backprojected features for point p as its voxel feature v ∈ R C :EQUATIONWe also utilize the multi-view consistency to compute a surface score s for point p by averaging confidence scores Bi ϕ(p) from every valid projection. We set s = 0 when the point does not have any valid projection, which means point p is in the free space. s is multiplied with voxel feature v as the final feature v ∈ R C for point p as follows:EQUATION)The adoption of s shares similar spirit with existing methods [22, 19] to decrease the influence of empty space voxels in the feature volume. Yet, we learn it directly from multi-view images instead of relying on NeRF [22] or ground truth supervision [19] .We further utilize the recent pixel-aligned Gaussian Splatting (PAGS) [3, 28] to enhance our depth prediction module. PAGS takes in sparse views and predict a Gaussian primitive per pixel. The parameters for each primitive are center µ , Gaussian opacity α , covariance Σ and color c. All the Gaussian primitives are combined together to render a novel view via rasterization-based splatting.We select 3 nearby views per novel view from the images input to the detection branch , and predict Gaussian maps {M µ , M α , M Σ , M c } for the selected views. The Gaussian center map M µ ∈ R H 4 × WThe Gaussian covariance map We render the image color Ĉcolor via alpha-blending and the rendering loss is a L2 loss as follows:EQUATIONL render = || Ĉcolor -C color || 2 .(11) One of the key factors for good rendering is accurate M µ , which is directly related to correct depth estimation from our model (cf. Eqn. 8). The rendering loss thus iteratively guides the Gaussians to the correct 3D locations, and consequently benefits our detection pipeline. Our final loss is 4 ExperimentsL = L det + L render .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: We do not propose any theories.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
datasets: 
 We conduct experiments on the ScanNet and ARKitScenes datasets. ScanNet has 1,201 and 312 scans for training and testing, respectively. We detect axis-aligned bounding boxes for 18 classes. ARKitScenes has 4,498 and 549 scans for training and testing, respectively. We detect oriented bounding boxes for 17 class. We adopt mean average precision (mAP) with thresholds of 0.25 and 0.5 as the evaluation metrics.We use the same feature extractor, training and testing configurations as [22] for detection. Specifically, images are resized into (240, 320 We compare our method with ImGeoNet [19] , CN-RMA [17] , ImVoxelNet [16] and NeRF-Det [22] . We directly report their results from the CN-RMA [17] paper. Note that ImGeoNet and CN-RMA use ground truth 3D geometry as supervision during training. Tab. 1 and Tab. 2 show the results on ScanNet and ARKitScenes, respectively. It is expected that using ground truth geometry as supervision can achieve good performance. However, ground truth geometry may not be accessible and therefore we seek for a self-supervised approach that do not rely its supervision. ImVoxelNet and NeRF-Det are the two existing methods that leverage self-supervision to learn geometry for multiview 3D detection. Compared to these works, our method achieve much better performance. It clearly shows the superiority of our efficient plane sweep method over the vanilla feature backprojection in ImVoxeNet and the density field in NeRF-Det. Fig. 1 shows the qualitative results on ScanNet. The first two columns show that our model can detect more target objects. We also find that NeRF-Det tends to detect objects in the free space (last two colums), which is due to inaccurate feature projection.In contrast, our model is able to place bounding boxes at more accurate locations.Effectiveness of Probabilistic Sampling and Soft Weighting. Tab. 3 shows the ablation stdudy of the probabilistic sampling and soft weghting (PSSW). All methods are conducted without the rendering loss L render to test the performance of PSSW alone. Removing "Probabilistic Sampling" means replacing top-k sampling with a single depth estimation computed by the weighted average of depth probability volume B. We use the max probability score as the weight in this case. Removing "Soft Weighting" means replacing Bi ϕ(p) with value 1. As shown in the table, removing either "Probabilistic Sampling" or "Soft Weighting" causes drastic drop in the performance. Particularly, removing probabilistic sampling depth proposals drops by 19.1 at mAP@.25. It suggests that estimating accurate depth is very hard under insufficient depth planes. Furthermore, soft weighting is very important to our model as it can decrease the influence of wrong depth proposals introduced by the sampling. Overall, both probabilistic sampling and soft weighting are crucial to our model.Effectiveness of Gaussian Splatting for Depth Prediction. Tab. 4 shows the ablation study for using pixel-aligned Gaussian splatting (PAGS) for detection. "RMSE" evaluates the average quality of depth prediction for the images in the detection branch. Increasing depth planes to M = 16 or using Gaussian Splatting both improve the depth estimation and bring similar improvement to the detection performance. However, using PAGS consumes much less memory during training than M = 16, i.e. adding only 28% memories compared to increasing depth planes. Moreover, PAGS does not bring any additional memory cost during testing because it is removed for detection. We visualize the depth maps predicted by the probability volume B in Fig. 5 . The depth maps are of 1/16 size of the original image since we estimate depth on the feature map level. It shows that PAGS significantly improves the depth quality. Analysis of the number of Top-k depth proposals. Tab. 5 shows the ablation study of number of depth proposals. Sampling top-1 depth proposal leads to severe performance decrease as the correct depth location is harder to be selected. Sampling too many depth proposals (k = 5) also leads to some decrease since more inaccurate locations are sampled. We thus choose k = 3 in our model. Comparison with Depth Estimation Methods. Fig. 6 shows the comparison of different depth prediction methods to the detection performance. MVSNet [24] performs one-time plane sweep by using M = 16 depth planes. BEVStereo [10] performs iterative depth estimation by sampling depth planes according to a monocular depth estimation. We set the number of iterations to 3 and the number of depth planes in each iteration to 5. We remove ground truth depth supervision for BEVStereo for fair comparison. "Ground-truth Depth" denotes placing pixel feature on the 3D volume according to the ground-truth depth location, which is the upper bound of our method. We show the results of our approach using different number of depth planes (M = 12 and M = 8).MVSNet performs badly even though it samples more planes than us. This is because MVSNet requires sufficient depth planes to estimate depth correctly. BEVStereo also fails because it requires the ground truth depth supervision to learn a good initial monocular depth estimation. In contrast, our model achieve close performance as the "Ground-truth Depth" using only 12 or even 8 depth planes. This demonstrates that our approach efficiently learns geometry without sampling many depth planes. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We include the implementation details in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this paper, we propose MVSDet for multi-view image based indoor 3D object detection. We design a probabilistic sampling and soft weighting mechanism to decide the placement of pixel features on the 3D volume without the need of the computationally sampling of many depth planes. We further introduce the use of pixel-aligned Gaussian Splatting to improve depth prediction with little computation overhead. Extensive experiments on two benchmark datasets demonstrate the superiority of our method.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will release our code upon paper acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitation: 
 Similar to existing multi-view stereo methods [24, 20] , feature matching would fail on texture-less or reflective surfaces. One possible solution is to combine with monocular depth estimation [2] . However, estimating monocular depth is non-trivial and we leave it for future research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We include the dataset and implementation details in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow baselines to not report error bars. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We include it in the implementation details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite the datasets we use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer:[NA] Justification: We do not release any new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We do not conduct crowdsourcing or research with Human Subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]  
 
 Appendix: A.1 Qualitative Results on ARKitScenes Dataset Fig. 2 shows the qualitative results on ARKitScenes dataset. Compared to NeRF-Det [22] , we are able to detect more target objects. OursFig. 8 shows the novel view synthesis results on ScanNet test dataset. Our model gives reasonable rendering results, indicating that the Gaussian Splatting module successfully learn the geometry. Note that the Gaussian Splatting module is only a regularizer to our plane sweep algorithm instead of the determining factor to learn geometry like NeRF in NeRF-Det [22] . A.3 Per-Class Performance Tab. 7, Tab. 8, Tab. 9 and Tab. 10 are the per-class results under AP@0.25 and AP@0.5 on ScanNet and ARKitScenes datasets, respectively. Justification: The experiments support our claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 


Decision: All three reviewers appreciate the methods, mainly its efficiency. Issues and questioned raised by the reviewers were clarified in the rebuttal and the discussion.

Reviews: review: paper_summary: paper_summary: This work presents a method for multi-view 3d object detection. The method computed a MVS cost volume using a few planes, it then samples k likely depth values per pixel and builds a 3d feature volume based on the voxels close to the samples depth values weighted by their confidence. Additionally, during training pixel aligned gaussian splatting (GS) is used to provide an additional rendering loss to guide the depth estimation.
strength_weakness: strength_weakness: It is unclear how the nearby views are selected. Is it the same as in the existing works?
To evaluate the contribution of the probabilistic sampling alone it would be good to add a line in table 3 without probabilistic sampling and soft weighting.
What is the benefit of using the PAGS rendering loss over the classical MVS photometric losses?
questions: questions: Why does it not improve with more planes? Is the 3d grid too coarse?
It seems that 2 views are used for MVS, but 3 for GS. Why not use more?
How is the depth offset predicted? Like in MVSNet? Is it ever used for the object detection?
limitations: limitations: Limitations were mentioned.
paper_summary: paper_summary: The manuscript proposes MVSDet, a multiview 3d object detection model that is evaluated on indoor scene datasets. Multiview information is lifted to 3D via an efficient per-frame depth sampling scheme. The most probable top-k depth values per pixel are used to lift 2D features into a global feature volume in a weighted way. Based on the thus accumulated 3d feature volume a 3d network regresses 3d bounding box parameters. In order to regularize the depth regression which is essential for constructing a occlusion-aware 3d feature volume, the manuscript proposes leveraging pixel-aligned Gaussian splats to construct a rendering loss against nearby views. Both the probabilistic depth sampling and the rendering loss are shown to contribute significantly to the performance of the model. Overall the model outperforms other models that do not use GT depth supervision during training.
strength_weakness: strength_weakness: I do wonder how much the difference is to supervising with GT depth instead of gaussian splats. (The experiment in Fig 6 is similar but not quite the same since placing features at GT depth locations does not need the probabilistic depth model to regress the depth). This last experiment might drive home the effectiveness of Gaussian splats and allow direct comparison to the ImGeoNet and CN-RMA related works in Tab 1 and 2. 

The writing and illustrations are mostly clear and support the understanding of the manuscript. There are quite a few open questions (see questions) that should be addressed to improve the presentation of the method.
questions: questions: - Fig 1: it is kind of hard to see what is going on with the red points in the renderings. 
- Fig 2: "ray intersects at 3 points (shown as dots)" -- do you mean as red triangles? 
- Fig 2: the arrows in the gaussian splats orange box of the diagram are not very clear. This could be polished more.
- l 151: it is unclear what is ment by 27 location canidates are selected for each target object? 
- l 184: how is the depth offset predicted in more detail and how is it used during sampling? Is it used for example in Eq 8 to adjust the depth planes per pixel?
- ablation that does supervise the proposed model with gt depth instead of Gaussian splats rendering?
- Tab 4: is the memory change per batch? (I assume since B=1)?
limitations: limitations: The limitations are addressed adequately in the paper.
paper_summary: paper_summary: This paper mainly focuses on the problem of 3D object detection form multi-view images. It introduces MVSNet-like method for depth prediction, brings out probabilistic sampling, soft weighting and pixel-aligned Gaussian Splatting to improve the correctness, robustness of depth prediction, especially with sparse images. Therefore, it improves the performance for 2D features to be projected to the 3D space, therefore improves the performance of 3D object detection.
strength_weakness: strength_weakness: 1. The 2D feature extractor, the detection head, and the detection head of this work is not clearly described.
2. The MVSNet-like depth prediction with probabilistic sampling and soft weighting, as well as the pixel-aligned Gaussian Splatting are all common methods or from previous works, I consider the novelty to be limited.
3. Also, I do not agree with the opinion of the article that ground truth geometry for supervision on the training stage is difficult to obtain. As far as I know, To obtain supervision for 3D object detection (AABB or OBB bounding boxes), you need to use a lidar or a RGBD camera to obtain the geometry of a 3D scene, and then label the bounding boxes.
questions: questions: 1. I do not know about the 2D feature extractor, the detection head, and the detection head of this work, are they the same as NeRF-Det?
2. Please further explain the novelty of your method, or your novel discovery that leads you to your method.
3. I do not agree with the opinion of the article that ground truth geometry for supervision on the training stage is difficult to obtain, please further explain this claim, since your results are not better than CN-RMA and ImGeoNet that used ground truth geometry for supervision. 
4. Following my question 3, introducing ground truth geometry may cause significant time and memory consumption in training, and makes the network far more complicated and therefore increases time and memory cost in inference. Could you give me detailed time and memory consumption of your own method and all the baselines you have run, on both training and inference stages?
limitations: limitations: Yes, the authors have addressed the limitations.

Comments: title: Answer to Rebuttal
comment: Thank you for addressing my questions and the two evaluations that support the claims in the paper (depth offset helps, GS rendering is nearly the same as GT depth supervision). From looking at the other reviews I dont see any other problems that were not addressed. The consensus seems to move to weak accept which I support.
Thank you for your detailed answers and additional evaluations. As all of my questions have been answered I would consider raising my rating to weak accept, if no further discussions arise.
The rebuttal clearly addresses my questions. While its performance does not surpass CN-RMA, it is efficient in terms of time and memory cost during both training and testing stages. I strongly recommend including Tab.R3-ZuP3/Q4 in the final version. I would like to raise my rating.