RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion
Authors: Bardienus Pieter Duisterhof
Jan Oberst
Bowen Wen
Stan Birchfield
Deva Ramanan
Jeffrey Ichnowski
Venue: NeurIPS 2025 poster ()
Keywords: 3D Vision
3D completion
Primary area: applications

Abstract: 3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, are computationally expensive, and struggle to capture sharp object boundaries. 
Our work (RaySt3R) addresses these limitations by recasting 3D shape completion as a novel view synthesis problem. 
Specifically, given a single RGB-D image, 
and a novel viewpoint (encoded as a collection of query rays),
we train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for those query rays. 
RaySt3R fuses these predictions across multiple query views 
to reconstruct complete 3D shapes. 
We evaluate RaySt3R on synthetic and real-world datasets, and observe it achieves state-of-the-art performance,
outperforming the baselines on all datasets by up to 44% in 3D chamfer distance.

RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion
Bardienus P. Duisterhof
Jan Oberst Camegie Mellon University
Bowen Wen NVIDIA
Stan Birchfield NVIDIA
Carnegie Mellon University
Deva Ramanan
Jeffrey Ichnowski Carnegie Mellon University
Carnegie Mellon University
Incomplete 3D input
Masked RGB
Masked dopth
RaySERR
Side
Back
Incomplete 3D input
Masked depth
RaySBR
Side
Back
Figure 1: RaySt3R a method for zero-shot shape completion from single foreground- RGB-D image. It predicts depth maps, object masks, and per-pixel confidence scores for novel viewpoints, and fuses them to reconstruct complete 3D shape. RaySt3R is able to recover the geometry of full objects in cluttered real-world scenes, despite only being trained synthetic data.
Abstract
3D shape completion has broad applications robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, computationally expensive, and struggle to capture sharp object boundaries. Our work (RaySt3R) addresses these limitations by recasting 3D shape completion novel view synthesis problem. Specifically, given single RGB-D image and a novel viewpoint (encoded as a collection of query rays), we train feedforward transformer predict depth maps, object masks, and per-pixel confidence scores for those query rays. RaySt3R fuses these predictions across multiple query views reconstruct complete 3D shapes. We evaluate RaySt3R on synthetic and real- world datasets, and observe it achieves state-of-the-art performance, outperforming the baselines on all datasets by up 44 % in 3D chamfer distance. Project page: rayst3r. github.ic io
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
3D shape completion is an enabling tool for visual reasoning and physical interaction with partially visible objects, and facilitates a wide range of downstream tasks such as robot grasping in cluttered en- vironments 229 55). obstacle avoidance [3 41 mechanical search [7]] digital-twin reconstruction, and Augmented or Virtual or Extended Reality (AR/VR/XR) applications.
Challenges. We focus on the robotics-driven setting where an RGB-D image is provided as input for multi-object shape completion. While object-centric methods achieve high reconstruction quality for single objects, multi-object scenes require instance segmentation and alignment procedures that tend to be brittle in practice. Generative approaches use 2D image generation models [59] 49] to
generate images from novel viewpoints, but these can be sensitive to large viewpoint changes and are computationally inefficient at inference time, which hinders robot and XR deployment. Other methods scale up 3D prediction on abundant synthetic scene data (18), but the resolution for the 3D representations (such as 3D MAE voxel grids) is too coarse to capture sharp object shapes with high-frreuency geometry details.
Approach. We propose Ray Stereo 3D Reconstruction (RaySt3R), novel method for addressing the above challenges. Given a single masked RGB-D image as input, our key insight to recast shape completion as novel view-synthesis task, then aggregate multiple view predictions to generate complete 3D shape. Our approach draws inspiration from recent work that casts 3D reconstruction as point map regression via multi-view transformers 48
We similarly use a vision transformer [33 features extracted from the input image.
9
(ViT) [22) architecture defined over visual DINOv2
However, instead of requiring second image as additional input, we input the novel view to be synthesized in the form a camera ray map. Specifically, RaySt3R is trained to predict depth maps, confidence maps, and foreground masks for each queried ray via cross- attssntttennoon We then merge RaySt3R's geometric predictions from multiple novel views using the per-ray confidence and mask predictions.
Data. Since RaySt3R can be seen as a view-synthesis engine, we can train at scale on pairs of RGB-D images without requiring volumetric 3D supervision (as required by prior work (18). We train RaySt3R on a large-scale augmented synthetic dataset with 251 k unique scenes and 11 million novel depth maps (training pairs). Across synthetic and real-world benchmarks, RaySt3R outperforms prior art by up to 44 % (in shape completion accuracy). Despite never being trained on real data, RaySt3R generalizes well to real-world cluttered scenes.
This paper contributes:
RaySt3R, a method for view-hased 3D shape completion that learns confidence-aware depth maps and object masks from novel views, and uses novel formulation of merging multi-view prediction. A new curated large-scale dataset with I1 million novel depth maps and masks, which we will open-source facilitate future research.
Evaluations of RaySt3R on synthetic and real-world datasets that show RaySt3R achieves state- of-the-art accuracy for 3D shape completion, and successfully generalizes to real-world cluttered scenes after training on only synthetic data.
2
Related work
3D shape completion has seen impressive progress over the last years. We explore related works categorized by their reasoning space, i.e., volumetric approaches and view-based approaches.
Volumetric reasoning Volumetric methods operate directly in 3D space and provide strong geomet- ric priors. Some approaches directly predict point clouds [I]] 30 43). while others rely on implicit geometry representations. The latter infer 3D structure at test time by querying the representation with spatial point and a partial observation (c.g.. an RGB or RGB-D scan). Prior work uses signed
distance functions for such representations 34 23 or voxel occupancy grids 35 16
31
53 24
OctMAE [18] builds on the idea of MAE [3 from the image synthesis domain, and applies it to next-token prediction natively in 3D. Although these methods yield promising results, their resolution
This work was generously supported by the Center for Machine Learning and Health (CMLH) at CMU, the NVIDIA Academic Grant Program, and the Pittsburgh Supercomputing Center.

---

is constrained by the cubic cost of their volumetric resolution, leading to coarse grid and smoothed structures lacking fine details.
Another strategy decomposes the scene at the object level | 58 SceneComplete 1 constructs a 3D scene by chaining together foundation models for object segmentation, occlusion inpainting, 3D shape retrieval, and pose estimation. Despite its modular design, our experiments (Section suggest that this reliance on multiple components introduces brittleness and several single points failure.
Recent work like TRELLIS [35] instead learns a 3D latent representation from text or image, which can be decoded into various formats such as meshes. However, as shown in Section experiments suggest it struggles in real-world multi-object scenes. In contrast, RaySt3R adopts a view-based strategy that is specifically designed for robust 3D completion in cluttered environments.
View-based reasoning Diffusion models [4 and their extensions
[5
39
38
have enabled
unprecedented performance in generative tasks such as image synthesis, inpainting, and video prediction. Several works leverage off-the-shelf generative models for 3D generation tasks [27] [59] 44 [2]. ViewCrafter (39] leverages these advances by iteratively completing scene point cloud
using a point-conditioned video diffusion model. Similar to RaySt3R, LVSM (0] synthesizes novel views by querying a transformer with Plücker Rays and by conditioning on input views. RaySt3R predicts depths and object masks instead of images, and does not require full Plücker rays for querying novel views. RayZer [19] is a self-supervised large view synthesis model, using its self-predicted camera poses to eliminate the need for any ground-truth camera annotations.
In the object-centric domain, Li al. 25 predicts layered depth maps for constructing object-level and scene-level 3D geometries. While this method yields promising results on single-object shape completion, it struggles with predicting accurate geometries real-world scenes containing multiple objects. Unique3D [54] tries to strike balance between fidelity and inference speed by predicting multi-view images, generating corresponding normal maps and textured mesh within seconds.
While these models often yield visually appealing results, they lack geometric consistency, especially for cluttered real world environments. The inference time of large diffusion models may also hinder deployment in robotics or XR settings. In contrast, RaySt3R predicts geometrically accurate depth maps from novel views, for fast and accurate 3D shape completion in cluttered real-world scenes.
Problem statement
Given single RGB-D image, jimput Hxxx x3, Dimput HRW foreground mask M {0, 1 x and a camera with known intrinsics, Kiipptt E R3x3, the goal is to predict the full 3D surface geometry of all masked foreground objects. We frame the prediction goal as set of points Q E R x3 that is both accurate and complete w.r.t. the ground-truth points QEE e Sx3 sampled on the surfaces (c.g.. meshes) of all objects the scene. We measure accuracy as the shortest distance from predicted point to the nearest ground-truth point, averaged over all predicted points. We measure completeness as the shortest distance from ground-truth point to the nearest predicted point, averaged over all ground truth points.
4 Methods
An overview of our approach is illustrated in Figure We propose to train transformer that, given the partial capture from single RGB-D image and foreground mask, predicts depth maps and per- pixel confidence scores, and foreground masks for novel views. We first present the model architecture (Section 4.1), then the training objectives (Section 4.2). We then describe the procedure for querying novel views (Section 4.30 and conclude with the prediction merging strategy (Section 4.4
4.1 Network architecture
The RaySt3R network architecture is inspired by DUSt3R 48 and successors 47
Here, we leverage VIT with point map, ray map, and depth map representations for 3D object completion.
49 45
The inputs to RaySt3R (Figure 2) are foreground RGB-D image and novel query view. First, we unproject the input depth map Dilpput to point map Xiput [ Hxxx3 using the given
3

---

Inputs
RaySt3R Inference
View Merging
Novel View
Self Attention
Cross Attention
Ray Maps
Depth
Queries
Conf
DPT
Depth
Keys
Mask
views
DINOv
RGB
Figure 2: The architecture of RaySt3R. RaySt3R takes a single RGB-D image and foreground mask as input, and predicts depth maps, object masks, and per-pixel confidence scores for novel views. First, we apply the foreground mask the RGB and point map input. Next, we self-attention layers for the point map and ray map inputs, and feed RGB image cross-attention layers followed by two separate DPT heads (36] for depth and mask predictions. Finally, we provide confidence- and occlusion-awe -warre multi-view merging formulation.
the frozen DINOv2 (33 encoder. We feed all features into
(j : D Dimpat We convert the 1. query view into ray map R E IR Xxxx2 with R.,; = [ -2)/ff (j ////fl where Cz, Cy is the image center and fz fy are the focal lengths of the novel target view.
input image intrinsics Kinput thus X-input = - input
Because we would like to pass information between the input and query views via cross attention, transform the input point map Xingu into the target camera coordinate frame ease information pquery input h(X xipuu), where pquery inpat
sharing. That X context
lR 3x transforms from the input query
camera coordinate frame, and (a, y. 2) + (x, y, 2, 1).
We compute features for the point map and ray query with layers of self-attention (SA).
ppaint_mup (A((Xooneet)) Fra SA(R)
(1)
We mask out the background in Xooneet and replace it with single learned background token. We process the RGB image by first masking out the background, and subsequently passing through frozen DINOv2 33 encoder. Recent work has shown that combination of features from different layers of pre-trained VIT is useful for downstream tasks [10], hence we concatenate the features from intermediate layers of the DINOv2 encoder, and use a linear layer to project them to FDINO
For the cross-attention (CA) layers, construct the keys of the first layer by concatenating Frpoint map and DINO The queries are the ray features paay.
G CA(F"M concat( map DINO))
(2)
Finally, we use DPT head 36 to predict depth maps, and its confidence scores. A separate DPT head predicts the object mask.
4.2 Training objectives
We train RaySt3R to predict confidence-awand depth maps and object masks.
Depth loss Inspired by DUS13R 48
we define the confidence-aware depth loss:
Ldepth
Σ
Σ
M (Ciss ||de, df, a log Ci.))
Here, Oi, is the confidence score of each pixel in the predicted depth map, a is hyper parameter, ds is the predicted depth, df, is the ground-truth depth, and M is the ground-truth mask. The

---

confidence scores are enforced to be strictly positive by setting C2. + 1 + exp(Ci,)). This enables a confidence estimate without explicit supervision.
We also predict binary object masks from novel viewpoints, and use a binary cross entropy loss to supervise it during training.
Lmask
Σ
Σ
(-m-t, log(ma.) (1 -1- log -mi))) m
=E - [0
Here, maj is the predicted object mask after sigmoid operation, and m is the ground-trute object mask. Finally, we combine the depth and mask losses with sum weight Amask.
Listal = Ldepth
Amusk femask
(5)
4.3
View sampling
To construct a set of query views to sample, we a tight bounding box to the input view point map and sample points on sphere with radius Agres around the box's 's center. Here, Ass is tunable parameter, and r66 is half the length of the bounding-bel where the bounding box is too small, thus we clip the radius to be at least Acam/cam- where ream is the distance from the camera to the center of the bounding box and Acam is tunable hyperparameter. We sample points evenly
diagonal. We found degenerate cases
cylindrical equal-area projection of the sphere to improve the coverage of the scene. We don't it include the input point map in our predictions as it likely contains noise and artifacts. Instead, query RaySt3R with the input view and include it in predictions.
4.4 Merging predictions
After predicting depth maps and object masks for all novel views, we merge them to produce complete 3D shape. We merge the depth maps by accounting for occlusions, RaySt3R's predicted masks, and the confidence scores.
Occlusion handling: First, we filter the points in each novel view to only parts of the scene that were visible in the input image (i.e., those points occluded by the input view's foreground mask inpppt and depth map ) Each point Gn.i.j is defined as the point predicted by the n-th novel view at pixel (i. j). Its projection in the input view is given by Pa,i KinputPP hh transforms points from the nith novel view to the input view. We define each entry of the mask as:
where plaput
(Pn,i.))= > inipur and M =
occ m
R, 0
otherwise
RaySt3R predicted masks: Even with the occlusion constraint, the object mask from a novel view largely unknown. For example, any observed surface could be a thin plate or rich 3D object. We use the predicted mask from RaySt3R to filter the points, by thresholding the predicted mask RayS:3
mi.j
[0, at 0.5.
RaySt3R's architecture enables unsupervised confidence scores for each pixel in the predicted depth maps. Confidence scores are typically used reduce edge-bleeding dense ViT predictions 448 45), or to exclude out-of-distribution objects such as specularities. With the same RaySI3R at for all experiments, more analysis is provided in Section 5.8
Confidence scores:
objective, we threshold
[.
Final mask: The final valid mask for a given novel view is obtained by setting
Mn,i.f
ace 1 m nii.
RaySt3R
0.5
RayStaR %
T
We obtain our final 3D reconstruction by aggregating valid points across all novel views.
Results
5.1 Training dataset
RaySt3R's training procedure requires large number of camera pairs to scale zero-shot to the real world. It requires an RGB image for the input view and depth maps, intrinsics, and extrinsics for
5

---

both cameras. We leverage existing synthetic datasets from FoundationPose [22] and OctMAE [18] OctMAE [88] places GSO 8 and Objaverse 77 objects synthetic scenes, and provide a single rendered image and depth map for each scene. Foundation has separate GSO and Objaverse splits, we only use the GSO split. For both datasets, we use the Objaverse and GSO meshes to render depth maps from novel views. Our dataset spans 251 k unique scenes, 12 k objects, and 11 M novel depth maps rendered for supervision.
5.2 Evaluation datasets
We evaluate RaySt3R on synthetic and real-world datasets. Following OctMAE 18 we evaluate on subsets of evaluation splits of the YCB-Vides [56] (900 frames), HOPE 42
(50 frames), and
HomebrewedDB [21] (1,000 frames) datasets. They are real-world 6D pose estimation datasets with noisy depth maps and imperfect masks, including common objects such as boxes and cylinders, as well as items of complex geometries such as metal parts. For results on synthetic data, we evaluate on evaluation split of the OctMAE [18] (1,000 frames) dataset test split. We notice edge artifacts in the masks introduced due to data compression in the original work [18].
5.3 Data augmentation
Synthetic training data lacks noise and other artifacts as present in the real world. We therefore apply data augmentation during training to better bridge the sim-to-real gap. Inspired by [52] [T) 66, we apply a set of augmentations to the input views training time, For depth maps, we randomly apply Gaussian noise, add holes, and shift the pixel coordinates [2] G For the RGB image, we randomly vary brightness and contrast, and apply a per-channel salt and pepper noise and Gaussian noise.
5.4 Implementation details
We train RaySt3R on 8x 80-GB A 100 GPUs for 18 epochs, totaling approximately 20 million scene iterations. We set the batch size to 10 per GPU, and a learning rate of 1.5 x 10 with half-cosine learning-rate schedule, starting with one warm-up epoch and using an Adam optimizer (28). We use a ViT-B model with patch size 16, embedding dimension 768, 12 heads, 12 cross-attention layers, but 4 self-attention layers to save on compute. We select the VIT-L with registers for DINOv2 33
We set A②5 = 1.3 and Aeam = 0.7 for all real-world datasets, and Ass = 2.5 and Acam 1.2 for the OctMAE dataset. The parameters are chosen to be larger for the OctMAE dataset, as the input view is typically placed very close to the objects with severe occlusions. We set the confidence threshold = 5 for all experiments, and sample 22 views in total. During training we the confidence parameter a = 0.2, Amask = 0.1. Inference takes less than 1.2 seconds on a single RTX 4090 GPU, and can be further reduced by querying fewer views.
5.5 Baselines
We compare RaySt3R against the -fate-of-the-art in 3D shape completion. OctMAE introduced novel 3D MAE algorithm, and also trained prior shape completion models on their novel dataset. We compare against OctMAE, and the prior works they trained, i.e., VoxFormer (26), ShapeFormer (57).
MCC [53). ConvONet 335 POCO 3 AICNet [24), Minkowski [5] and OCNN [46].
We also compare against SceneComplete [1. which uses combination of foundation models to produce complete geometry. The authors leverage a VLM and Grounded-SAM (37] to produce object-level masks, image inpainting fill in occluded regions, image-to-30 model to produce 3D mesh and finally FoundationPose for 3D alignment.
We also benchmark against Unique3D 54 and TRELLIS 55 which are recent image to 3D models. Finally, we compare against 'Layered Ray Intersections' (LaRI) 225,, which introduced the concept of layered point maps to predict multiple points on each camera ray. Unique3D, Trellis, and Lari predict points in canonical coordinates, we align the predictions with the ground truth 3D using first brute-force search for similarity transform, followed by ICP 40) Note that we do not perform such a registration for RaySt3R, but provide this to baselines to give them the benefit of the doubt.
LaRI [25]. Unique3D [54] and TRELLIS [55] were not trained on cluttered scenes.

---

While LaRi [25] and 3 [34 do not require foreground masks for reconstructing objects, we observe that TRELLIS [55] tends reconstruct the entire scene. Therefore, we compare TRELLIS with masked image input and a raw image input. We also attempted to evaluate ViewCrafter [59] on this task, but the images produced by the video diffusion model were of too poor quality to perform the evaluation. We provide more details in the supplementary material.
5.6 Quantitative results
Following prior work [8]. we evaluate the zero-shot generalization performance of all methods using chamfer distance (CD) and F1-Score@ 10mm (FI). Detailed formulations of the metrics are provided in the supplemental material. We present the quantitative results of this evaluation in Table T RaySt3R consistently outperforms all baselines across both synthetic and real-world scenes. The strongest baseline, OctMAE (18), performs competitively, however RaySt3R surpasses across all metrics, by 20 % to 44 % in CD. SceneComplete [ proves to be a fragile pipeline, therefore not yielding competitive results. We were unable to produce SceneComplete results on HOPE, as the pipeline requires intractable amount of VRAM for cluttered scenes. LaRI 255 Unique3D [54] and TRELLIS [55] show better performance than SceneComplete [1, but also do not produce competitive results. Feeding masked RGB images to TRELLIS [55] outperforms raw image inputs on all datasets
except HomebrewedDB [21]. We show common failure modes Section 5.7
We also compute the standard deviation of the chamfer distance across all real-world datasets for each method and observe that our model exhibits the lowest standard deviation (1.74 mm). followed by OctMAE [8 (2.38 mm) and Unique3D (54) (8. mm).
Table I: Quantitative evaluation of multi-object scene completion on synthetic and world datasets. evaluate on the test split OctMAF [18]. and the BoP henehmarks YCB- Video [56), HOPE (42, and Home- YCB-Video
brewedDB (21) We report chamfer distance (CD) [mm] and F1-Score@ 10mm (F1). The first section contains numbers copied from OctMAE [88, the second section contains recent works evaluated. For alignment of
LaRI (25]. Unique3D [S4] and TRELLIS [55] with the ground truth mesh, we apply brute force search followed
by ICP [40. We evaluate TRELLIS [55] with masked and unmasked RGB inputs. SceneComplete runs out
of VRAM on HOPE 422.
The results suggest RaySt3R outperforms all baselines.
Synthetic
Real
OctMAE
18
YCB Video
56
21 HB
HOPE
42
Method
CD4
Fif
CD}
F1
CD4
FIf
CDJ
FIf
VoxFormer
26
44.54
0.382
30.32
0.438
34.84 0.366
47.75 0.323
ShapeFormer
57
39.50
0.401
38.21
0.385
40.93 0.328
39.54 0.306
MCC [53
43.37 0.459
35.85
0,289
19.59 0.371
17.53 0.357
ConvONet
35
23.68 0.541
32.87
0.458
26.71 0.504
20.95 0.581
POCO 3
21.11 0.634
0.587 15.45
13.17 0.624
13.20 0.602
AICNet
24
15.64 0.573
0.545 12.26
11.87 0.557
11.40 0.564
Minkowski
11.47 0.746
8.04 0.761
8.81 0.728
8.56 0.734
OCNN 46
0.782
7.10 0.778
7.02 0.792
0.742
OctMAE 18
6.48 0.839
0.800 6,40
6.14
6.97 0.803
LaRI 25
39.22
0.283
11.41
0.658
22.23 0.414
18.64 0.528
Unique3D 54
44.62
0.244
17.56
0.468
25.41 0.329
26.37 0.322
TRELLIS (w/ mask) 55
65.43
0.224
22.44
0.454
36.12 0.364
19.46 0.470
TRELLIS (w/o mask) [5
69.45
0.221
31.45
0.345
29.98 0.360
20.87 0.438
SceneComplete
0.289 81.57
96.63
0.359
85.81 0.416
N/A N/A
RaySt3R (ours)
5.21
0.893
3.56
0.930
4.75 0.889
3.92 0.926
5.7 Qualitative results
Figure showws qualitative results of RaySt3R on real-worle datasets. The results suggest that RaySt3R is capable of generating high-quality 3D predictions, generating more consistent and complete results compared to the baselines. The most competitive baseline, OctMAE [8], produces viable but oversmoothed object shapes due to its low-resolution 3D MAE grid. Furthermore, TRELLIS (55). Unique3D (54) and LaRI (55 struggle with relative object placement and aspect ratios. They also fail for certain ont-of-distribution objects, and occasionally fail to register to the ground truth point cloud.

---

Tnputs
Predictions
LaRI
TRELLIS mask
Ours
Unique3D
SceneComplete
OctMAR
LaRl
TRELLIS with mask
OctMA
Ours
Unique3D
TRELLIS with mask
OcIMAE
Ours
Unique3D
TRELLIS without mask
OcIMAE
Ours
LaRI
TRELLIS
OcIMAE
Ours
Figure 3: Qualitative results of RaySt3R in real-world multi-object scenes. For each scene, we select a subset of methods to preserve visual clarity, but we share all method predictions in the supplemental material. The results suggest RaySt3R produces the most geometrically accurate shapes compared the basclines. The most competitive baseline, OctMAE [8] tends to predict softer shapes as result of its coarse 3D MAE grid. We observe TRELLIS (55). Unique3D 54), and LaRI (25) struggle with relative object placing, aspect ratios, and out-of-distribution objects, occasionally leading to incorrect registration to the ground truth points. Finally, SceneComplete I proves to be hrittle to single points of failure such as missing object masks.
Interestingly, TRELLIS 55 may predict table surfaces even for masked input images with no visible table. SceneComplete 1 proves to be brittle to single-point failures such as missing object masks.
5.8 Ablation studies
Training ablations: Table summarizes our training ablation results, with all models trained under the same setup for roughly 20 million scene iterations. Training a ViT-S model (384-dim, 6 heads) leads to performance drop. Disabling data augmentation further degrades zero-shot

---

Table 2:
Ablation study RaySt3R train-
Table 3: Ablation study RaySt3R view merging on the Oct- MAE test dataset [T8). We ablate querying the input view (Sec- tion 4.3 occlusion masking with the imput mask, and finally RaySt3R's predicted masks. The results suggest all steps con- tribute to performance, especially the predicted masks.
ing the Video dataset [56]. The sults suggest data scale, data diversity, DI- NOv2 [33] features, data augmentation, and model size affect RaySt3R's performance.
Method
CD L
f
Query Input
Occ. Mask
Pred. Mask CDL FIf
RaySt3R (propesed)
3,56
0,930
5,21 0.893
VIT-S
3,70
0,920
7.55 0.836
No data mugmentation
3.89
0.916
7,69 0.855
Train 100k scenes
4.30
0.894
10.12 0.825
w/o DINOv2 (8
4,81
0.877
X
X
73.17 0.641 x
Train on 226k USO scene
5.34
Distance
Completeness
Accuracy
Metric [mm]
1.25
6.25
8.75
10
3
Confidence threshold
Conf threshold
Conf threshold
metrics averaged over all real-world datasets. The results suggest increasing the confidence threshold im- proves accuracy and degrades completeness.
Figure 4: Confidence threshold
Figure 5: The impact of the confidence threshold on the predicted 3D points. This experiment suggests increas- ing the confidence threshold can aid in reducing the edge bleeding issue.
generalization to real-world data, highlighting its importance. We also compare training 100k uniformly sampled scenes versus the full 226k GSO set. The smaller but more diverse 100k set performs better, emphasizing the value of data diversity. Finally, removing DINOv2 33) inputs causes an additional decline, underscoring the benefit of pretrained features. We also train a model exclusively on the OctMAE split of our dataset, and find achieves CD of 5.87, and F1 of 0.893 on the OctMAE test set. This result suggests RaySt3R outperforms the baselines even when trained on the same dataset.
View merging ablations: Table 3 shows the ablations on our view merging formulation. We ablate querying the input view (Section 4.3), using the input mask to detect occluded regions, and finally the use of RaySt3R's 's predicted mask. The results suggest that each component of our formulation contributes to shape completion performance. especially the predicted masks.
Confidence RaySt3R predicts a per-pixel confidence value, which can be used to filter the predictions. To understand the impact of the confidence threshold, we report chamfer distance, Completeness, and Accuracy for a range of thresholds, as depicted in Figure The results suggest that confidence is good proxy for eтror and that confidence can be effectively used to trade off accuracy and completeness. Depending on the application, the threshold can therefore be tuned accordingly, as some applications may be less tolerant to outliers requiring high accuracy, while others may require more complete predictions. For all prior experiments, we set confidence threshold T to 5 for a balance between accuracy and completeness. Figure 5 shows a visualization of the impact of changing the confidence threshold on the predicted 3D points.
6 Conclusion and future work
We present RaySt3R, a novel approach to 3D shape completion from a single RGB-D image and foreground mask. RaySt3R learns to predict depth maps, object masks, and per-pixel confidence scores for novel viewpoints, which are fused to reconstruct complete 3D shapes. We benchmark RaySt3R on real-world and synthetic datasets, and compare it to the theee-ff in volumetric

---

and view-based methods. The results suggest that RaySt3R is capable of generating high-quality 3D predictions, outperforming the baselines across the board.
While the results suggest our view-based approach generates high-quality reconstructions, it also suffers from the common edge bleeding issue, adding noise predictions. Our formulation also requires inferring high-quality novel view poses, which is a non-trivial task beyond objects on table tops. An advantage RaySt3R's view-based approach is that it enables training on real-world data without the need for ground truth meshes. Training on real-world data may help generalize to more objects and adapt real-world noise. Future work may also explore scaling up compute by exploring other architectures like diffusion transformers.
7
Acknowledgements
This work was generously supported by the Center for Machine Learning and Health (CMLH) at CMU, the NVIDIA Academic Grant Program, and the Pittsburgh Supercomputing Center. The authors would like to thank Mandi Zhao, Shun Iwase, Balázs Gyenes, Gerhard Neumann, Sergey Zakharov, Nikhil Varma Keetha, Jeff Tan and all members of the Momentum Robotics lab at CMU for providing useful feedback.
References
[1] A. Agarwal, G. Singh, B. Sen, T. Lozano-Pérez, and L. P. Kaelbling. Scenecomplete: Open- world 3d scene completion in complex real world environments for robot manipulation, 2024.
[2] T. Barron and J, Malik. Intrinsic scene properties from a single rgb-d image. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 17-24, 2013.
[3] A. Boulch and R. Marlet. Poco: Point convolution for surface reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 6302-6314 131,, 2022.
[4] A. Bozic, P. Palafox, J. Thies, A. Dai, and M. NieBner. Transformerfusion: Monocular rgb scene reconstruction using transformers. Advances in Neural Information Processing Systems, 34:1403-1414, 2021.
[5] C. Choy, J. Gwak, and Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019.
[6] M. Dalal, M. Liu, W. Talbott, C. Chen, D. Pathak, J. Zhang, and R. Salakhutdinov. Local policies enable zero-shot horizoorizon manipulation. International Conference of Robotics and Automation, 2025.
[7] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. CVPR, 2022.
[8] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and Vanhoucke, Google scanned objects: high-quality dataset of 3d scanned household items. In ICRA, 2022.
[9] B. Duisterhof. L. Zust, P. Weinzaepfel, V. Leroy, Y. Cabon, and J. Revaud. Mast3r-sfm: fully-integrated solution for unconstrained moucture-from-aaee motion, 2024.
[10] A. El-Nouby, M. Klein, S. Zhai, M. A. Bautista, V. Shankar, A. Toshev, J. M. Susskind, and
A. Joulin. Scalable pre-training of large autoregressive image models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
[11] H. Fan, H. Su. and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-6153 613, 2017.
[12] R. Gao A. Holynski*, P. Henzler, A. Brussee, R. Martin-Brualle P. P. Srinivasan, J. T. Barron, and * Cat3d: Create anything 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
10

---

[13] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
[14] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
[15] Ho, I. Salimans, Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[16] J. Hou, A. Dai, and M. Nießner. Revealnet: Seeing behind objects in rgb-d scans. In CVPR, 2020.
[17] H. Huang, L. Fu, M. Danielczuk, C. M. Kim, Z. Tam, J. Ichnowski, A. Angelova, B. Ichter, and K. Goldberg. Mechanical search on shelves with efficient stacking and destacking of objects. In Robotics Research, pages 205-221, Cham, 2023. Springer Nature Switzerland.
[18] S. Iwase. K. Liu, V. Guizilini, A. Gaidon, K. Kitani, R. Ambrus, and S. Zakharov. Zero-shot multi-object scene completion, 2024.
[19] H. Jiang, H. Tan, P. Wang, H. Jin, Y. Zhao, S. Bi, K. Zhang, F. Luan, K. Sunkavalli, Q. Huang, and G. Pavlakos. Rayzer: A self-superve sulfrsuue large view synthesis model. 2025.
[20] H. Jin, H. Jiang, H. Tan, K. Zhang, S. Bi, T. Zhang, F. Luan, N. Snavely, and Z. Xu. Lvsm: large view synthesis model with minimal 3d inductive bias. In The Thirteenth International Conference on Leurning Representations, 2025.
[21] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilie. Homebreweddb: Rgb-d dataset for 6d pose estimation of 3d objects. ICCVW, 2019.
[22] A. Kolesnikov, A. Dosovitskiy, D. Weissenborn, G. Heigold, J. Uszkoreit, L. Beyer, M. Minderer, M. Dehghani, N. Houlsby, S. Gelly, T. Unterthiner, and X. Zhai. An image worth 16x 16 words: Transformers for image recognition at scale. 2021.
[23] L. Ladicky, O. Saurer, Jeong, F. Maninchedda, and M. Pollefeys. From point clouds mesh using regression. In Proceedings of the IEEE International Conference on Computer Vision, pages 1893-3902,-2 2017.
[24] Li, K. Han, P. Wang. Y. Liu, and X. Yuan. Anisotropic convolutional networks for 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3351-3359, 2020.
[25] R. Li, B. Zhang, Z. Li, F. Tombari, and P. Wonka. Lari: Layered ray intersections for single-view 3d geometric reasoning. In arXiv preprint arXiv:250.18424, 2025.
[26] Y. Li, Z. Yu, C. Choy, C. Xiao, J. M. Alvarez, Fidler, C. Feng, and A. Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In CVPR, 2023.
[27] R. Liu, R. WU, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298-9309, 2023.
[28] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2019.
[29] J. Mahler, M. Matl, V. Satish, M. Danielczuk. B. DeRose, S. McKinley, and K. Goldberg. Learning ambidextrous robot grasping policies. Science Robotics, 4(26):caau4984, 2019.
[30] I.. Melas-Kyriazi, C. Rupprecht, and A. Vedaldi. Pc2: Projection-condiiionee point cloud diffusion for single-image 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12923-12932, 2023.
[31] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction function space. Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 4460-4470, 2019.

---

[32] I. Mishani, H. Feddock, and M. Likhachev. Constant-time motion planning with anytime refine- ment for manipulation. In 2024 IEEE International Conference on Roboties and Automation (ICRA), page 10337-10343. IEER, May 2024.
[33] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang. H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[34] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: signed distance functions for shape representation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 165-174, 2019.
Learning continuous
[35] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[36] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. In ICCV, 2021.
[37] T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang. H. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open-wole world models for diverse visual tasks, 2024.
[38] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[39] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion mod- els with deep language understanding. Advances in neural information processing systems, 35:36479-36494,, 2022.
[40] A. Somani, T. S. Huang, and S. D. Blostein. Least-squares fitting of two 3-D point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 698-700, 1987.
[41] Z. Tang, B. Sundaralingam, J. Tremblay, B. Wen, Y. Yuan, S. Tyree, C. Loop. A. Schwing, and S. Birchfield. Rgb-only reconstruction of tabletop scenes for collision-freee manipulator control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1778-1785. IEEE, 2023.
[42] S. Tyree, Tremblay, T. To, J. Cheng, T. Mosier, Smith, and S. Birchfield. 6-dof pose estimation of household objects for robotic manipulation: An accessible dataset and benchmark. In International Conference on Intelligent Robots and Systems (IROS), 2022.
[43] A. Vahdat, F. Williams, Z. Gojeic. O. Litany, S. Fidler, K. Kreis, al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35: 1021-10099,, 2022.
[44] V. Voleti, C.-H. Yao, M. Boss, A. Letts, D. Pankratz, D. Tochilkin, C. Laforte, R. Rombach, and V. Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer Vision, pages 439-457. Springer, 2024.
[45] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2025.
[46] P.-S. Wang, Y. Liu, and X. Tong. Deep octree-based cnns with for 3d shape and scene completion. In CVPRW, 2020.
skip connections
[47] Q. Wang, Y. Zhang, A. Holynski, A. A. Efros, and A. Kanazawa. Continuous 3d perception model with persistent state. arXiv preprint arXiv:2501.12387, 2025.
12

---

[48] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
[49] E. Weber, Müller, Y. Kant, V. Agrawal, M. Zollhöfer, A. Kanazawa, and C. Richardt. Fillerbuster: Multi-view scene completion for casual captures, 2025, arXiv:2502.05175.
[50] B. Wen, W. Lian, K. Bekris, and S. Schaal. Catgrasp: Learning category-level task-relevant grasping in clutter from simulation. In 2022 International Conference on Robotics and Automa- fion (ICRA), pages 6401-6408 IEEE, 2022.
[51] B. Wen, C. Mitash, B. Ren, and K. E. Bekris. se (3)-tracnnet: Data-driven 6d pose tracking by calibrating image residuals in synthetic domains. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10367-10373, IEEE, 2020.
[52] B. Wen, W. Yang. J. Kautz, and Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024.
[53] C.-Y. Wu, J. Johnson, J. Malik, C. Feichtenhofer, and G. Gkioxari. Multiview compressive coding for 3D reconstruction. In CVPR, 2023.
[54] K. Wu, F. Liu, Z. Cai, R. Yan, H. Wang. Y. Hu, Y. Duan, and K. Ma. Unique3d: High-quality and efficient 3d mesh generation from a single image, 2024.
[55] Xiang, Z. Lv, S. Xu, Y. Deng, R. Wang, B. Zhang, D. Chen, X. Tong, and J. Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint PxXiv:2412.0100 2024.
(56) Y. Xiang, T. Schmidr, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. 2018.
[57] X. Yan, L. Lin, N. J. Mitra, D. Lischinski, D. Cohen-Or, and H. Huang. Shapeformer: Transformer- based shape completion via sparse representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[58] K. Yao, L. Zhang, X. Yan, Y. Zeng, Q. Zhang, L. Xu, W. Yang, J. Gu, and J. Yu. CAST: Component-aligned 3d scene reconstruction from RGB image. In 25Xi::2502./2894 2025.
[59] W. Yu, J. Xing, L. Yuan, W. Hu, X. Li, Z. Huang, X. Gao, T.-T. Wong, Shan, and Y. Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint Nil::2409000449 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Yes, we have precisely highlighted the contribution of our model without overelaiming.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
Yes, we discuss the limitation of our method in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We do not provide any theorems or proofs in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We will provide all necessary hyper parameters and details to reproduce the results. We also plan to release the code, model checkpoints, and dataset.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to release the code, model checkpoints and dataset of this paper, with useful instructions to help adoption.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http://icc/giiie////eeeuubiiibii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
Yes, we believe we have provided all necessary details to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We present metrics suitable for the experiments we execute. For training the model, rerunning the training several times would be an impractical cost terms of compute and energy expenditure. For evaluation, we partially adopt results from prior papers, and unfortunately do not have access to the raw metrics or checkpoints necessary to compute error bars. For the methods we compute metrics for, discuss the methods with the largest variance in our writing and provide a full set of error bars in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
17

---

Justification:
Yes, we provide the compute required for training our model.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, work, to the best of our knowledge, conforms with the NeurlPS code ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential impact on robotics and extended reality. There an ongoing dehate about the positive and negative societal impact of these fields.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
18

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not foresee a need for safeguards in our system.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have accredited the original owners of the assets used in this work, specifically Objaverse, GSO, OctMAE, and FoundationPose.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsswieeesseeeeeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
19

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No crowdsourcing or human subjects involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not involve LLMs in our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standaadd components.
Please refer to LLM policy for what should or should not be described.
20


Reviews:
[{"review_id": "I9LqCoAoM0", "reviewer": "Reviewer_cWX6", "summary": "The paper addresses the task of reconstruction complete 3D objects/scenes from a single view RGB-D observation thereof.\nTo this end, the paper proposes to leverage transformer-based reconstruction network, which predicts depth maps, foreground masks and confidences for novel views. In order to complete an object/scene 22 novel views are sampled, and merged into a final point cloud prediction. The merging considers occlusions, confidence and mask predictions. \n\nTo train the model, the author propose a new large-scale synthetic dataset, which will be released to the public.\n\nTo evaluate the model the authors follow the evaluation procedure on OctMAE and copy their numbers and baselines. Additionally the authors add comparisons to LaRI, TRELLIS and Unique3D, all of which can only use RGB as input, while the proposed method sees RGB-D as input.\n\nFinally, the authors ablate the importance of their merging scheme, and show ablations about ViT backbone architecture and training dataset options and usage of DINO-v2 features.", "questions": "- As mentioned above, I am mainly concerned about the current evaluations. The following comparisons are important for the paper:\n    - Either train OctMAE on the newly proposed data, or train the proposed method on the original OctMAE training dataset using their augmentations. Otherwise, the experimental comparisons are completely inconclusive. (Not sure if the 226k GSO dataset ablation goes into a similar direction. If you the authors can elaborate, since the paper does not present all relevant details about it).\n    - I believe the comparisons to TRELLIS and LaRI are not fair, since these models these models are designed for RGB inputs. For a fairer comparisons, the authors could for example train their method without the depth conditioning. Similarly, TRELLIS is mainly trained on single objects (for LaRI it depends which checkpoints the authors used), but evaluated on cluttered scenes. A fairer comparison that does not involve training TRELLIS on the proposed dataset, would be to evaluate single object reconstructions, e.g. using a subset of the current evaluation data, or datasets like CO3D.\n\n- That being said, the formulation is certainly interesting and could prove valuable to the community in the future, e.g. LVSM follows a similar formulation is influential for NVS methods. Compared to other feed-forward reconstruction transformers, one distinction of LVSM is the lack of internal 3D representation and rendering formulation. Have the authors considered/experimented with adding an internal 3D representation to their model? After all it seems like an important design decision, which could be briefly discussed in the paper.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- (**S.a**) The paper casts 3d object/scene completion as a depth prediction task from novel views. This is a novel formulation, and especially interesting since methods like LVSM show great success with a similar formulation for novel-view RGB prediction.\n- (**S.b**) The generated training dataset will likely benefit the community. I am not sure wether the effect of the new data is properly ablated though, since some information on the dataset ablations are missing (i.e. is 226k GSO, the data that OctMAE used?).\n- (**S.c**) The paper is well-written and easy to follow.\n- (**S.d**) The paper shows string experimental results, and, like OctMAE, successful manages the synthetic2real domain gap at inference time.\n\n\n**Weaknesses**\n- (**W.a**) OctMAE not trained on same dataset, so in the current state of the paper, it is not clear whether improvements are coming from the model/formulation or from the training data.\n- (**W.b**) The paper does not mention LVSM, which follows a similar formulation for novel-view RGB synthesis. This should at least be referenced and/or discussed somewhere. \n- (**W.c**) The comparisons to TRELLIS, LaRI etc. are not fair, because they don't get depth as input, and e.g. TRELLIS is used to model the distribution over single objects. The authors made no efforts of constructing fairer comparisons, like evaluating single object completion or training a version of their model without depth as input. In the current state, these added experiments merely present expectable results.\n\n**Conclusion:**\nThe paper introduces an interesting paradigm for object/scene completion. However, the experimental validation of the presented ideas are lacking. Some comparisons are not fair because of vastly different training data, while other baselines receive only RGB instead of RGB-D inputs. Due to this lacking rigor of the evaluation, I am currently leaning towards rejection.", "was_revised": true, "final_justification": "The authors have significantly improved the paper, by adding a crucial ablation experiment, additional clarifications, and comparisons on single objects. With these additions the experimental evaluation was raised above the acceptance threshold.\n\nOverall, the paper presents significant value to the community (although the depth input is slightly niche), as it is the first method (to the best of my knowledge), which approach geometry reconstruction/completion using a view-based approach. While this results in less smooth geometry, e.g. compared to volumetric approaches, it is still a highly relevant research direction, especially, in times where view-based methods (LVSM, RayZer, GENIE3, VideoDiffusion models with camera control) are very successful for NVS. Therefore, the novelty aspect is also above acceptance threshold.", "rebuttal": "Thank you cWX6 for these great questions, we will modify the paper to reflect our answers below:\n\n**(W.a) Comparison against OctMAE**:  \tFollowing your suggestion, we trained RaySt3R only on the OctMAE dataset and with only the OctMAE augmentations. The table below shows RaySt3R trained on either dataset outperforms OctMAE (**bold** is best, _italics_ is second best).\n\n| Method             | Trained on | CD $\\downarrow$ | F1 $\\uparrow$ |\n| ------------------ | ---------- | --------:| -----------:|\n| **OctMAE**         | OctMAE     | 6.48     | 0.839       |\n| **Ours (RaySt3R)** | OctMAE (new) | _5.87_   |  _0.877_    |\n| **Ours (RaySt3R)** | RaySt3R    | **5.21** | **0.893**   |\n_Table 1. Model and training dataset comparision_\n\nWhile we did not have the resources to train OctMAE on our dataset, we believe that this experiment nevertheless shows the contribution of our method.\n\nIn retrospect, our description of the datasets may have been unclear.\nOur dataset uses the exact same objects as OctMAE, and only 10\\% more input images.  \"1.1 million scenes\", refers to the unique RGB-D input images; but the actual scenes are only 251k.  Also, \"12 M views\" includes both the input images and the depth maps used for supervision.  See table below.\n\n|  | **OctMAE** dataset | **RaySt3R** dataset | \n|---|---:|---:|\n| # RGB-D Input Images | 1 M | 1.1 M |\n| # Objects |   12 k   |    12 k  |\n| # Scenes (object configurations) | 25 k | 251 k |\n| Supervision | Sampled mesh | 11 M depth maps |\n_Table 2. Training dataset size comparision_\n\n**(W.b) LVSM** We agree LVSM is a relevant method with impressive performance, and commit to discussing it in detail in the final version of the paper. We cast 3D shape completion as a novel view synthesis problem which allows leveraging ideas from LVSM, DUSt3R and other recent works. A key difference between LVSM and RaySt3R is indeed the scene encoding, where we encode the query view extrinsics into the input view point map. The advantage of our approach is that we do not need full plucker coordinates in the query tokens, and instead only encode the intrinsics (HxWx2). A key advantage of the LVSM approach would be improved efficiency for many query views, as the encoder has to be run only once. We agree an ablation on this aspect would be useful for the community, but it goes beyond the compute resources available to us during the rebuttal.\n\n**(W.c) Comparisons vs TRELLIS and LaRI** \nThank you for raising concerns about fair evaluation of the TRELLIS and LaRI baselines. First of all, our use case is to complete cluttered scenes based on a RGB-D image, which is a common while challenging setup in robotics. We believe the mentioned baselines are valuable in putting RaySt3R's results into perspective of recent and popular image to 3D pipelines, and help motivate depth conditioning for RaySt3R. We also made our best attempt to improve their results, by aligning the results of the baselines to the GT mesh with a brute force search + ICP, including a hyper parameter ablation (supp 1.2.1). RaySt3R outperforms the baselines, even when they are provided access to the GT mesh for alignment, which is unavailable to our method.\n\nHowever, following your suggestion, we additionally evaluate RaySt3R and baselines on a single object setting from YCB-V dataset. For each frame, we evaluate on the largest object with no occlusions which is often assumed by single-view RGB-based object reconstruction methods such as TRELLIS. \n\n\n| Method   | CD $\\downarrow$ | F1 $\\uparrow$ |\n| -------- | ----:| -----:|\n| TRELLIS  | 22.37 | 0.56  |\n| LaRI     | 13.75     |    0.51  |\n| OctMAE   |  8.34    |  0.74     |\n| Ours (RaySt3R)  | **4.06** | **0.91** |\n_Table 3. YCB-V single-object evaluation_\n\n**Clarifications**\n- **226k GSO in Table 2**: This ablation trains RaySt3R on the subset of our dataset curated from Foundation Pose, containing GSO objects only."}, {"review_id": "14ZtwmNrBV", "reviewer": "Reviewer_bxUS", "summary": "The authors proposed a method to recover complete 3D (without texture) from masked RGB-D image. They train a feed-forward  transformer to predict depth maps, object masks, and per-pixel confidence scores for novel views, and use information to help recover the missing part.\n\nContributions:\n1. a good method to predict novel view information. \n2. a new curated large-scale dataset that can help future research.\n3. comprehensive experiment on different dataset and good performance.\n\nOverall I think this model is easy to use and can contribute to the community.", "questions": "1. as shown in the teaser, the input depth may have some invalid or missing value in the corresponded RGB pixel, but they seems to share the same mask (line 87-88). Will this introduce noisy value to the network?\n\n2. the model sample 22 views (line 194) in total to predict the full 3D. What will happen if sampling more or less views?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The writing is clear and easy to understand.\n\n2. Good alignment. As shown in Fig. 3, although TRELLIS has better visual result, it's not as good aligned as the proposed model.\n\n3. Good performance in both quantivive and qualitive result, as shown in the main paper and supplementary.\n\n4. The paper did comprehensive ablations, like the range of hyper-parameters.\n\nWeaknesses:\n\n1. The model does not handle texture, which is also important for many real-world applications.\n\n2. The objects in the experiment seems lies on a flat plane, and there are no heavy occlusion or stacking. It would be interesting to test model performance on those extreme cases.\n\n3. The reconstructed object is not very smooth, and the reconstructed scene does not have individual instance segmentations.", "was_revised": true, "final_justification": "Thanks the authors to respond to my comments and the discussion from other reviewers. The authors test their pipeline in extra dataset (OctMAE) and shows good result. I read the comments from FXY9 and cWX6 and I think the threshold is not a determinant factor to reject this paper. Overall I think they propose an easy and straight-forward pipeline and I think the community can get some inspriation from this work.", "rebuttal": "Thank you bxUS for these great questions, we will modify the paper to reflect our answers below:\n\nQ1. **Noisy masks**: Indeed the network may be challenged by noisy depth masks as present in the datasets. We try to mimic this situation in the data augmentations (Sec 5.3), and show it improves performance (Table 2).\n\nQ2. **Sampling views**: When sampling fewer views there is a higher chance of missing part of the geometry while inference will be faster. This is an application-specific tradeoff, e.g. when reconstructing only a single object fewer views may be sufficient.\n\nBelow we reply to weaknesses raised in your review:\n\nW1. **Texture**: Geometry is the essential modality needed for a variety of robotic tasks such as grasping, 3D occupancy reasoning and motion planning, which is the focus of this work. In those applications, texture is not commonly used. However, we agree it would be an exciting future direction to also predict texture for broader applications such as digital twin creation.\n\nW2. **Non-planar predictions**: While we did not show these examples in the paper, the HOPE dataset contains several scenes where objects are placed in a non-planar fashion with heavy occlusions. For example, in HOPE's scene 3 objects are placed on the different parts of a chair (seat, handrest, headrest). In HOPE's scene 9, objects are placed on top of a cabinet as well as inside the drawers.\n\nW3. **Improving smoothness**: The lack of smoothness and occasional noisy points can be limited by setting a higher confidence threshold, sampling fewer views, or by fitting a TSDF to the depth map predictions. We will include an off-the-shelf TSDF fitting method in the open-source release. \n\n**Instance segmentation**: We agree individual instance segmentation would be a useful future direction, allowing users to export individual objects to a simulation environment. Given a modified dataset, a future version of RaySt3R may leverage ideas from PanSt3R [1] to also achieve instance segmentation.\n\n[1] PanSt3R: Multi-view Consistent Panoptic Segmentation, Lojze Zust, Yohann Cabon, Juliette Marrie, Leonid Antsfeld, Boris Chidlovskii, Jerome Revaud, Gabriela Csurka"}, {"review_id": "FFDggtXdUx", "reviewer": "Reviewer_G7Ap", "summary": "This paper proposes a supervised learning method for predicting a full 3D model of an object from a single masked RGB-D image (the so-called shape completion problem). It does so by learning to predict new depth maps of a scene from different viewpoints, and combining multiple such views into the final 3D model.", "questions": "I think it would be important to clarify what new insights (besides its results) are to be gained from the paper. Are its methodological aspects, for example, likely to be usesful in other tasks, and why?\n\nPlease clarify the \"zero-shot\" aspects of the presentation since this paper has nothing to do with scene categories, and is just about reconstruction.\n\nYou appear to predict a point cloud, but the illustrations seem to be mesh renderings. Please clarify.", "limitations": "A very bried discussion of the limitations in the Conclusion section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper is well written, the proposed approach is simple and sound, and it gives good results, outperforming all baselines it is compared to.\n\nWeaknesses:\n\nThe prediction of novel views appears to be a simple  transformer-based approach to depth map regression controlled by the choice of viewpoint, in the line of DUSt3R and related techniques, that regress point clouds to input views. This limits the novelty of the approach. The authors should justify the choice of  learning to render novel views, and aggregate the results, instread of directely regressing the depth of back-facing points, or any other proxy to the shape completion task.", "was_revised": true, "final_justification": "I remain unconvinced by the importance of the problem addressed in this paper, but the proposed solution for this type of problems is reasonable, and the authors have done a good job wth the rebuttal. I stand by my original rating  and have not problem having the paper accepted.", "rebuttal": "Thank you G7Ap for these great questions, we will modify the paper to reflect our answers below:\n\nQ1. With RaySt3R, we have shown that casting 3D shape completion as a novel view synthesis task enables leveraging recent insights from DUSt3R for object-centric tasks. With mask prediction and multi-view merging, we believe our method may be applied to other object-centric tasks such as 3D-complete tracking or panoptic segmentation. Besides the results, an important aspect is the ability to train on real-world data without GT meshes.\n\nQ2. In our setting, zero-shot transfer means we train RaySt3R only on synthetic data and evaluate the model on real-world scenes outside the training set, consisting of unseen objects, scene configurations, and noisy depth sensing input.\n\nQ3. The 3D visualizations show dense point clouds (not meshes), where colors are assigned based on each point's 3D location using a Turbo colormap.\n\n**Weaknesses**\n\nIn addition to the weaknessess we addressed in answering the questions above, we note\n\nW1. **Justification of learning novel views vs other 3D-completion proxies**: Layered ray intersections (LaRI) is an example of an alternative proxy of 3D completion where effectively the back of an object is predicted. Benchmarking against LaRI and other volumetric approaches, the results suggest RaySt3R produces better completion results. RaySt3R can also be trained on any pair of (real-world) RGB-D views, all other baselines require access to the full scene geometry.  \n\n**Limitations**: Thank you for pointing out the limited discussion in the conclusion, we commit to expanding the limitations and future work in the camera ready version. We will discuss the pros and cons of view-based multi-object completion. The advantages we see are the ability to query an arbitrary number of views to trade off quality and inference speed, the ability to supervise directly on real-world data without GT meshes, and empirically more accurate results compared to the volumentric baselines. View-based methods suffer from the common edge bleeding issue, and deciding where to sample novel views can be challenging. Finally, we will highlight RaySt3R's limitation to objects far beyond the current training dataset, such as transparent objects."}, {"review_id": "whQWMg5Uec", "reviewer": "Reviewer_FXY9", "summary": "This paper proposes a view-based architecture followed by view-based sampling and point filtering to perform objection completion from a single RGB-D image.  It also proposes a new simulated multi-object dataset to help improve performance.", "questions": "The key topics that I would like clarifaction on are:\n\n1.\tOctMAE has slightly worse performance across metrics while being trained on significantly less data (25k scenes and 1M images) compared to the proposed work (1.1M scenes and 12M images).  Since the two main contributions of the paper are the RaySt3R architecture and the dataset, these should be decoupled to understand how much performance is gained from each individually.  Ideally, OctMAE would be trained on this new dataset and RaySt3R would be trained on the OctMAE dataset so that this is clear.  \n\n2.\tThe Chamfer distance metric raises several questions.  In Figure 4, the optimal Chamfer distance comes at a confidence threshold of 0.  However, this was not the threshold that was used, and instead a confidence threshold of 5 is used despite giving worse performance.  Visually, Figure 5 shows that a confidence threshold of 0 gives very noisy results with many points far from the surface.  This raises the question that the metrics used are not indicative of good results.  One potential issue is that L1-Chamfer is far too lenient with outliers, and L2 and RMSE should be reported instead.  Furthermore, completeness and accuracy are only shown in Figure 4, but this could be reported in Table 1 as there is a trade-off between the two as shown in the plot.  \n\n3.\tMore discussion on limitations of view-based methods compared to 3D representations would benefit the paper.  The limitations section at the end of the conclusion does not dive into the method, but is rather a generic statement on the need for real-data and scaling that could apply to any machine learning method.  Using a view-based method is an interesting data point, but the benefits and drawbacks could be discussed in much greater detail.  For example, the abstract says the approach addresses 3D consistency that other methods lack, but it is unclear why a ray-based approach, which requires additional occlusion filtering here, is more consistent than a volumetric one.  The first clip in the video also demonstrates that  views cannot recover the bottom of the objects which are left empty.", "limitations": "As mentioned above, more discussion on limitations of view-based methods would benefit the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Overall, the paper is well-written and the method is straightforward to understand.  The visualizations are also clear and visually appealing.\n\nI will discuss some questions and clarifications that I have below:\n\n•\tIn general, there should be more discussion on the drawbacks of view-based approaches and how they differ compared to 3D representations.  The proposed method claims to address 3D consistency, but this is a potential problem with view-based methods compared to 3D representations, such as voxel grids.  Furthermore, view-based methods have the problem that the views must be selected to ensure coverage, but in some cases, this may be extremely challenging.  Even in the video, the bottom of objects and part of the pitcher are missing.  In very cluttered scenes, views have to be selected such that the rays pass between two objects, which with contact may be impossible.  \n\n•\tWhile it is mentioned that RaySt3R does not need volumetric supervision, it still requires complete geometry,  It does not seem like a significant drawback that volumetric methods need to precalculate from the complete geometry since this is at training time and has no effect at test time.\n\n•\tAs the paper claims the approach is efficient, a runtime comparison with others would be helpful.  View-based methods will also have complexity linear in the number of target views, which may need to vary for different scenes depending on the amount of clutter.  22 views in 1.2 seconds is fairly efficient, but comparing to other approaches would strengthen this.\n\n•\tHow does the random sampling guarantee coverage? Some results in the video for example show insufficient coverage.\n\n•\tIt is mentioned that the input point map is not included in the predictions.  Does this mean there is noise in the depth maps or do the predictions not align with the input point map?  The occlusion filter is also used to remove points that would be visible in the original view, but is this due to some predicted points not being 3D consistent due to the view-based method? \n\n•\tIt is difficult to judge how OctMAE compares with the proposed method, as they are trained on very different amounts of data with different augmentations.  RaySt3r uses 1.1 million scenes and 12 million views, while OctMAE is trained on 25k scenes and 1M images.  Since OctMAE is the closest in terms of performance, evaluating on equal footing with respect to training data would demonstrate if the data, the architecture, or both are the key to improved performance.\n\n•\tSampling parameters are varied between real and OctMAE datasets.  Do other methods use dataset specific parameters? If not, a unified set of parameters should be used for fairness.\n\n•\tL1-Chamfer is used in Figure 4.  Is this the Chamfer distance used for all experiments? In Figure 4, the best Chamfer distance is with a confidence threshold of 0, but a worse performing confidence threshold of 5 is used for experiments instead.  In Figure 5, a 0 threshold gives inconsistent and noisy results.  This leads to the question whether the metrics used are indicative of good 3D completion, as the best performing threshold leads to worse visual results.  L1-Chamfer is likely too forgiving for outliers and noisy points, so L2 and RMSE should potentially be used instead.", "was_revised": true, "final_justification": "I appreciate that the authors covered many points of discussion during the rebuttal, specifically:\n\n - Training RaySt3R on the OctMAE dataset for fair comparison\n - Using a unified set of parameters across all datasets\n - Clarifications of the dataset compared to the one proposed in OctMAE\n - Agreeing to give discussion on view-based vs. volumetric representations for scene completion\n - Inclusion of a limitations section beyond the generic ones listed in the conclusion\n - Additional details on metrics\n\nI have upgraded my score to borderline accept, as overall the paper provides valuable contributions.\n\nThe main factor for not raising the score beyond borderline accept is that it is still unclear why a confidence threshold that gives worse average results across all datasets is selected (Table 6 in the response to my review).  This somewhat diminishes the Chamfer distance as a reliable metric. It is used to signal how RaySt3R outperforms baselines, but then RaySt3R is designed to not fully optimize this metric.", "rebuttal": "Thank you FXY9 for these great questions, we will modify the paper to reflect our answers below:\n\nQ1. **Comparison against OctMAE**:  \tFollowing your suggestion, we trained RaySt3R only on the OctMAE dataset and with only the OctMAE augmentations. The table below shows RaySt3R trained on either dataset outperforms OctMAE (**bold** is best, _italics_ is second best).\n\n| Method             | Trained on | CD $\\downarrow$ | F1 $\\uparrow$ |\n| ------------------ | ---------- | --------:| -----------:|\n| **OctMAE**         | OctMAE     | 6.48     | 0.839       |\n| **Ours (RaySt3R)** | OctMAE (new) | _5.87_   |  _0.877_    |\n| **Ours (RaySt3R)** | RaySt3R    | **5.21** | **0.893**   |\n_Table 1. Model and training dataset comparision_\n\nWhile we did not have the resources to train OctMAE on our dataset, we believe that this experiment nevertheless shows the contribution of our method.\n\nIn retrospect, our description of the datasets may have been unclear.\nOur dataset uses the exact same objects as OctMAE, and only 10\\% more input images.  \"1.1 million scenes\", refers to the unique RGB-D input images; but the actual scenes are only 251k.  Also, \"12 M views\" includes both the input images and the depth maps used for supervision.  See table below.\n\n|  | **OctMAE** dataset | **RaySt3R** dataset | \n|---|---:|---:|\n| # RGB-D Input Images | 1 M | 1.1 M |\n| # Objects |   12 k   |    12 k  |\n| # Scenes (object configurations) | 25 k | 251 k |\n| Supervision | Sampled mesh | 11 M depth maps |\n_Table 2. Training dataset size comparision_\n\nQ2. **Evaluation Metrics**: We follow prior work by computing the Chamfer distance with the L2 norm (the Euclidean distance) for both accuracy and completeness.\nSee Equations (1)-(3) in the supplemental material. \n(In the paper, the reference to \"Chamfer-L1\" is an error which we will correct in the final version of the paper.)\n\nAs requested, to further study the impact of the introduced noise, the tables below provide the CD computed based on RMSE, as well as the accuracy and completeness. \nThese additional metrics show that RaySt3R consistently outperforms the baselines.\n\n\n| Method            | OctMAE | YCB-Video | HomebrewedDB | HOPE  |\n| ----------------- | ------:| ---------:| ------------:| -----:|\n| Unique3D          |  62.23 |     24.16 |        34.27 | 36.12 |\n| TRELLIS with mask |  84.94 |     31.55 |        45.55 | 27.34 |\n| TRELLIS           |  88.32 |     42.26 |        40.32 | 30.53 |\n| LaRI              |  55.88 |     15.72 |        30.54 | 25.26 |\n| OctMAE            |  21.72 |     11.31 |        10.30 | 14.52 |\n| Ours (RaySt3R)           |   **9.33** |      **6.17** |         **7.85** |  **6.37** |\n_Table 3. RMSE results_\n\n| Method            | OctMAE | YCB-Video | HomebrewedDB | HOPE  |\n| ----------------- | ------:| ---------:| ------------:| -----:|\n| Unique3D          |  47.29 |     20.27 |        27.72 | 30.18 |\n| TRELLIS with mask |  48.12 |     22.43 |        26.22 | 20.74 |\n| TRELLIS           |  45.34 |     28.20 |        23.23 | 20.28 |\n| LaRI              |  41.21 |     11.74 |        24.60 | 19.66 |\n| OctMAE            |  20.27 |      8.49 |         8.69 |  7.69 |\n| Ours (RaySt3R)           |   **7.80** |      **3.67** |         **5.93** |  **4.66** |\n_Table 4. Completeness results_\n\n\n\n| Method            | OctMAE | YCB-Video | HomebrewedDB | HOPE  |\n| ----------------- | ------:| ---------:| ------------:| -----:|\n| Unique3D          |  45.87 |     14.86 |        23.09 | 22.57 |\n| TRELLIS with mask |  84.11 |     23.46 |        42.71 | 19.77 |\n| TRELLIS           |  94.42 |     35.27 |        38.19 | 23.82 |\n| LaRI              |  42.61 |     11.08 |        19.87 | 17.62 |\n| OctMAE            |   5.93 |      5.06 |         4.28 |  7.21 |\n| Ours (RaySt3R)           |   **2.63** |      **3.45** |         **3.58** |  **3.19** |\n_Table 5. Accuracy results_\n\n **Confidence threshold:** \nFig 4 shows a tradeoff: a lower confidence threshold yields better Chamfer distance but worse accuracy. To avoid unfairly choosing hyperparameters from the test data, we arbitrarily selected a threshold of 5 for all experiments, qualitative and quantiative. The table below shows the Chamfer distance for each scene with different confidence thresholds. Overall, RaySt3R is not sensitive to the choice of confidence threshold and consistently outperforms the baselines under all confidence thresholds.\n\n| Method | Conf | YCB-Video | HomebrewedDB | HOPE |\n|--------|------:|------------:|--------------:|------:|\n| Ours (RaySt3R) | 0.00 | 3.59 | **4.49** | 3.99 |\n| Ours (RaySt3R) | 1.25 | **3.53** | 4.50 | 3.99 |\n| Ours (RaySt3R) | 2.50 | 3.55 | 4.58 | 3.98 |\n| Ours (RaySt3R) | 3.75 | 3.56 | 4.67 | 3.94 |\n| Ours (RaySt3R) | 5.00 | 3.56 | 4.75 | **3.92** |\n| Ours (RaySt3R) | 6.25 | 3.57 | 4.85 | 3.94 |\n| Ours (RaySt3R) | 7.50 | 3.58 | 4.95 | 3.97 |\n| Ours (RaySt3R) | 8.75 | 3.62 | 5.11 | 4.05 |\n| Ours (RaySt3R) | 10.00 | 3.65 | 5.27 | 4.16 |\n| OctMAE  | N/A  | 6.40 | 6.14 | 6.97 |\n_Table 6. Confidence ablation_ \n\nQ3. **View-based vs volumetric approachs**: We agree the pros and cons of view-based completion vs 3D representations should be emphasized more. The advantages of view-based completion are: (1) the ability to query an arbitrary number of views to trade off quality and inference speed; (2) the ability to supervise directly on real-world data without GT meshes; and (3) pixel-aligned and more accurate prediction compared to volumetric methods. In terms of limitations, while view-based methods may suffer from multi-view consistency issue such as edge bleeding, our proposed  confidence-based thresholding and multi-view fusion strategies effectively enhance the raw prediction from the network. Reconstruction results evaluated under multiple metrics indicate our method consistently outperforms volumetric approaches such as OctMAE. In the original abstract, the limited multi-view consistency refers to the other view-based methods such as ViewCrafter, which has been compared against and outperformed by our method in the supplemental material. However, we do admit there are a few other limitations which we will discuss more in the updated version of the paper, including: (1) deciding where to sample novel views involves more designs than volumetric methods; (2) when a relatively large number of views are queried, it leads to slower inference times; (2) generalizing to objects far beyond the current training dataset, such as transparent\nobjects.\n\n**Weaknesses**\nWe respond to other points raised below:\n* **Video animation**: We added the first animation to the video only for illustrative purposes, showing only 3 queried views instead of 22. To see the full reconstruction, please skip to 2:21 in the video or Figure 15 in the supplemental material. RaySt3R successfully reconstructs the bottom of the object because it samples from the viewing sphere (even though the bottom is not shown).\n* **Training without complete geometry**: While we train RaySt3R on a synthetic dataset, our method (unlike volumetric methods) does not require ground-truth meshes and can be trained on 2 or more depth images of a scene (as would be available from real-world depth cameras). We will clarify this in the final version of the paper.\n* **Random sampling**: We worded this incorrectly.  We did not use random sampling, since we found it is more likely to miss areas of the scene and introduce non-determinism.  We place query views on a grid evenly spaced on a cylindrical equal-area projection of a sphere.\n* **Noisy input depth**: We do not use the 3D points from the input depth map. In the real datasets they tend to be noisy, and introduce floaters especially at edges of objects. There are no 3D consistency issues. We conduct an experiment by appending the input depth map to RaySt3R's predictions in the YCB-V dataset, and observe the Chamfer distance increases from 3.56 [mm] to 3.66 [mm]. \n* **Runtime**: We appreciate your suggestion to provide a more comprehensive runtime analysis. In the table below, we provide additional information as benchmarked on our own hardware. SceneComplete is more challenging to benchmark, as it relies on an OpenAI API call to use their VLM. We find RaySt3R with flash attention runs faster than vanilla torch attention (568 ms vs 1200 ms). \n\n\n| Method            |  Runtime [ms]      |\n| ----------------- | ------------------:|\n| OctMAE            |        56            |\n| LaRI              |       283            |\n| TRELLIS           |  11,409               |\n| SceneComplete* |              $\\sim$ 30,000     |\n| Ours (RaySt3R) w Torch Atnn |              1200 |\n| Ours (RaySt3R) w Flash Atnn |              568 |\n_Table 7. Runtime comparison on 1 $\\times$ RTX 4090. *Per object, as reported by original paper._ \n\n- **Unified set of parameters**: Not all code for the listed baselines is available, hence we are unsure whether hyperparameters were varied for each dataset. Nevertheless, we study RaySt3R's performance with a single set of hyperparameters instead of two sets for synthetic and real datasets. This experiment (see table below) suggests RaySt3R outperforms the baselines even with a single set of hyperparameters.\n\n| Method                          | OctMAE | YCB-Video | HomebrewedDB | HOPE |\n| ------------------------------- | ------:| ---------:| ------------:| ----:|\n| OctMAE                          |   6.48 |      6.40 |         6.14 | 6.97 |\n|   RaySt3R                       |   **5.21** |      **3.56** |        **4.75**  | **3.92** |\n| RaySt3R w single parameter set  |   **5.21** |     4.19  |         5.20     |  4.66    |\n_Table 8. Hyper parameter study, Chamfer Distance [mm]_"}]

Author rebuttal:

Meta-review: The final ratings are 2 borderline accepts, 2 accepts. The AC have read the reviews and rebuttal, and discussed the submission with the reviewers. The reviewers raised a number of points during the review phase including discussions of limitations, evaluation protocol, datasets, and comparisons. The authors were able to address many of these points during the rebuttal and discussion phases by including additional experiments on OctMAE dataset and clarification of view-based and volumetric representations in relation to the proposed method. While there are still some concerns remaining such as the confidence threshold raised by FXY9, the AC and reviewers reached a positive consensus. The AC recommends the authors to incorporate the feedback and suggestions provided by the reviewers, and the materials presented in the rebuttal (and additionally revisit the issues raised on confidence threshold), which would improve the next revision of the manuscript.
Decision: Accept (poster)