Title: RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Vision

3D completion
Abstract: 3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, are computationally expensive, and struggle to capture sharp object boundaries. 
Our work (RaySt3R) addresses these limitations by recasting 3D shape completion as a novel view synthesis problem. 
Specifically, given a single RGB-D image, 
and a novel viewpoint (encoded as a collection of query rays),
we train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for those query rays. 
RaySt3R fuses these predictions across multiple query views 
to reconstruct complete 3D shapes. 
We evaluate RaySt3R on synthetic and real-world datasets, and observe it achieves state-of-the-art performance,
outperforming the baselines on all datasets by up to 44% in 3D chamfer distance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a view-based architecture followed by view-based sampling and point filtering to perform objection completion from a single RGB-D image.  It also proposes a new simulated multi-object dataset to help improve performance.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper is well-written and the method is straightforward to understand.  The visualizations are also clear and visually appealing.\n\nI will discuss some questions and clarifications that I have below:\n\n•\tIn general, there should be more discussion on the drawbacks of view-based approaches and how they differ compared to 3D representations.  The proposed method claims to address 3D consistency, but this is a potential problem with view-based methods compared to 3D representations, such as voxel grids.  Furthermore, view-based methods have the problem that the views must be selected to ensure coverage, but in some cases, this may be extremely challenging.  Even in the video, the bottom of objects and part of the pitcher are missing.  In very cluttered scenes, views have to be selected such that the rays pass between two objects, which with contact may be impossible.  \n\n•\tWhile it is mentioned that RaySt3R does not need volumetric supervision, it still requires complete geometry,  It does not seem like a significant drawback that volumetric methods need to precalculate from the complete geometry since this is at training time and has no effect at test time.\n\n•\tAs the paper claims the approach is efficient, a runtime comparison with others would be helpful.  View-based methods will also have complexity linear in the number of target views, which may need to vary for different scenes depending on the amount of clutter.  22 views in 1.2 seconds is fairly efficient, but comparing to other approaches would strengthen this.\n\n•\tHow does the random sampling guarantee coverage? Some results in the video for example show insufficient coverage.\n\n•\tIt is mentioned that the input point map is not included in the predictions.  Does this mean there is noise in the depth maps or do the predictions not align with the input point map?  The occlusion filter is also used to remove points that would be visible in the original view, but is this due to some predicted points not being 3D consistent due to the view-based method? \n\n•\tIt is difficult to judge how OctMAE compares with the proposed method, as they are trained on very different amounts of data with different augmentations.  RaySt3r uses 1.1 million scenes and 12 million views, while OctMAE is trained on 25k scenes and 1M images.  Since OctMAE is the closest in terms of performance, evaluating on equal footing with respect to training data would demonstrate if the data, the architecture, or both are the key to improved performance.\n\n•\tSampling parameters are varied between real and OctMAE datasets.  Do other methods use dataset specific parameters? If not, a unified set of parameters should be used for fairness.\n\n•\tL1-Chamfer is used in Figure 4.  Is this the Chamfer distance used for all experiments? In Figure 4, the best Chamfer distance is with a confidence threshold of 0, but a worse performing confidence threshold of 5 is used for experiments instead.  In Figure 5, a 0 threshold gives inconsistent and noisy results.  This leads to the question whether the metrics used are indicative of good 3D completion, as the best performing threshold leads to worse visual results.  L1-Chamfer is likely too forgiving for outliers and noisy points, so L2 and RMSE should potentially be used instead.\n\nQUESTIONS:\nThe key topics that I would like clarifaction on are:\n\n1.\tOctMAE has slightly worse performance across metrics while being trained on significantly less data (25k scenes and 1M images) compared to the proposed work (1.1M scenes and 12M images).  Since the two main contributions of the paper are the RaySt3R architecture and the dataset, these should be decoupled to understand how much performance is gained from each individually.  Ideally, OctMAE would be trained on this new dataset and RaySt3R would be trained on the OctMAE dataset so that this is clear.  \n\n2.\tThe Chamfer distance metric raises several questions.  In Figure 4, the optimal Chamfer distance comes at a confidence threshold of 0.  However, this was not the threshold that was used, and instead a confidence threshold of 5 is used despite giving worse performance.  Visually, Figure 5 shows that a confidence threshold of 0 gives very noisy results with many points far from the surface.  This raises the question that the metrics used are not indicative of good results.  One potential issue is that L1-Chamfer is far too lenient with outliers, and L2 and RMSE should be reported instead.  Furthermore, completeness and accuracy are only shown in Figure 4, but this could be reported in Table 1 as there is a trade-off between the two as shown in the plot.  \n\n3.\tMore discussion on limitations of view-based methods compared to 3D representations would benefit the paper.  The limitations section at the end of the conclusion does not dive into the method, but is rather a generic statement on the need for real-data and scaling that could apply to any machine learning method.  Using a view-based method is an interesting data point, but the benefits and drawbacks could be discussed in much greater detail.  For example, the abstract says the approach addresses 3D consistency that other methods lack, but it is unclear why a ray-based approach, which requires additional occlusion filtering here, is more consistent than a volumetric one.  The first clip in the video also demonstrates that  views cannot recover the bottom of the objects which are left empty.\n\nLIMITATIONS:\nAs mentioned above, more discussion on limitations of view-based methods would benefit the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper proposes a supervised learning method for predicting a full 3D model of an object from a single masked RGB-D image (the so-called shape completion problem). It does so by learning to predict new depth maps of a scene from different viewpoints, and combining multiple such views into the final 3D model.

STRENGTHS AND WEAKNESSES:
Strengths:
The paper is well written, the proposed approach is simple and sound, and it gives good results, outperforming all baselines it is compared to.

Weaknesses:

The prediction of novel views appears to be a simple  transformer-based approach to depth map regression controlled by the choice of viewpoint, in the line of DUSt3R and related techniques, that regress point clouds to input views. This limits the novelty of the approach. The authors should justify the choice of  learning to render novel views, and aggregate the results, instread of directely regressing the depth of back-facing points, or any other proxy to the shape completion task.

QUESTIONS:
I think it would be important to clarify what new insights (besides its results) are to be gained from the paper. Are its methodological aspects, for example, likely to be usesful in other tasks, and why?

Please clarify the "zero-shot" aspects of the presentation since this paper has nothing to do with scene categories, and is just about reconstruction.

You appear to predict a point cloud, but the illustrations seem to be mesh renderings. Please clarify.

LIMITATIONS:
A very bried discussion of the limitations in the Conclusion section.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-28

comments: SUMMARY:
The authors proposed a method to recover complete 3D (without texture) from masked RGB-D image. They train a feed-forward  transformer to predict depth maps, object masks, and per-pixel confidence scores for novel views, and use information to help recover the missing part.

Contributions:
1. a good method to predict novel view information. 
2. a new curated large-scale dataset that can help future research.
3. comprehensive experiment on different dataset and good performance.

Overall I think this model is easy to use and can contribute to the community.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The writing is clear and easy to understand.

2. Good alignment. As shown in Fig. 3, although TRELLIS has better visual result, it's not as good aligned as the proposed model.

3. Good performance in both quantivive and qualitive result, as shown in the main paper and supplementary.

4. The paper did comprehensive ablations, like the range of hyper-parameters.

Weaknesses:

1. The model does not handle texture, which is also important for many real-world applications.

2. The objects in the experiment seems lies on a flat plane, and there are no heavy occlusion or stacking. It would be interesting to test model performance on those extreme cases.

3. The reconstructed object is not very smooth, and the reconstructed scene does not have individual instance segmentations.

QUESTIONS:
1. as shown in the teaser, the input depth may have some invalid or missing value in the corresponded RGB pixel, but they seems to share the same mask (line 87-88). Will this introduce noisy value to the network?

2. the model sample 22 views (line 194) in total to predict the full 3D. What will happen if sampling more or less views?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the task of reconstruction complete 3D objects/scenes from a single view RGB-D observation thereof.\nTo this end, the paper proposes to leverage transformer-based reconstruction network, which predicts depth maps, foreground masks and confidences for novel views. In order to complete an object/scene 22 novel views are sampled, and merged into a final point cloud prediction. The merging considers occlusions, confidence and mask predictions. \n\nTo train the model, the author propose a new large-scale synthetic dataset, which will be released to the public.\n\nTo evaluate the model the authors follow the evaluation procedure on OctMAE and copy their numbers and baselines. Additionally the authors add comparisons to LaRI, TRELLIS and Unique3D, all of which can only use RGB as input, while the proposed method sees RGB-D as input.\n\nFinally, the authors ablate the importance of their merging scheme, and show ablations about ViT backbone architecture and training dataset options and usage of DINO-v2 features.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- (**S.a**) The paper casts 3d object/scene completion as a depth prediction task from novel views. This is a novel formulation, and especially interesting since methods like LVSM show great success with a similar formulation for novel-view RGB prediction.\n- (**S.b**) The generated training dataset will likely benefit the community. I am not sure wether the effect of the new data is properly ablated though, since some information on the dataset ablations are missing (i.e. is 226k GSO, the data that OctMAE used?).\n- (**S.c**) The paper is well-written and easy to follow.\n- (**S.d**) The paper shows string experimental results, and, like OctMAE, successful manages the synthetic2real domain gap at inference time.\n\n\n**Weaknesses**\n- (**W.a**) OctMAE not trained on same dataset, so in the current state of the paper, it is not clear whether improvements are coming from the model/formulation or from the training data.\n- (**W.b**) The paper does not mention LVSM, which follows a similar formulation for novel-view RGB synthesis. This should at least be referenced and/or discussed somewhere. \n- (**W.c**) The comparisons to TRELLIS, LaRI etc. are not fair, because they don't get depth as input, and e.g. TRELLIS is used to model the distribution over single objects. The authors made no efforts of constructing fairer comparisons, like evaluating single object completion or training a version of their model without depth as input. In the current state, these added experiments merely present expectable results.\n\n**Conclusion:**\nThe paper introduces an interesting paradigm for object/scene completion. However, the experimental validation of the presented ideas are lacking. Some comparisons are not fair because of vastly different training data, while other baselines receive only RGB instead of RGB-D inputs. Due to this lacking rigor of the evaluation, I am currently leaning towards rejection.\n\nQUESTIONS:\n- As mentioned above, I am mainly concerned about the current evaluations. The following comparisons are important for the paper:\n    - Either train OctMAE on the newly proposed data, or train the proposed method on the original OctMAE training dataset using their augmentations. Otherwise, the experimental comparisons are completely inconclusive. (Not sure if the 226k GSO dataset ablation goes into a similar direction. If you the authors can elaborate, since the paper does not present all relevant details about it).\n    - I believe the comparisons to TRELLIS and LaRI are not fair, since these models these models are designed for RGB inputs. For a fairer comparisons, the authors could for example train their method without the depth conditioning. Similarly, TRELLIS is mainly trained on single objects (for LaRI it depends which checkpoints the authors used), but evaluated on cluttered scenes. A fairer comparison that does not involve training TRELLIS on the proposed dataset, would be to evaluate single object reconstructions, e.g. using a subset of the current evaluation data, or datasets like CO3D.\n\n- That being said, the formulation is certainly interesting and could prove valuable to the community in the future, e.g. LVSM follows a similar formulation is influential for NVS methods. Compared to other feed-forward reconstruction transformers, one distinction of LVSM is the lack of internal 3D representation and rendering formulation. Have the authors considered/experimented with adding an internal 3D representation to their model? After all it seems like an important design decision, which could be briefly discussed in the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}