Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly

Authors: junsheng  zhou
yu-shen  liu
zhizhong  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 04:07:03

Abstract: Large language and vision models have been leading a revolution in visual computing. By greatly scaling up sizes of data and model parameters, the large models learn deep priors which lead to remarkable performance in various tasks. In this work, we present deep prior assembly, a novel framework that assembles diverse deep priors from large models for scene reconstruction from single images in a zero-shot manner. We show that this challenging task can be done without extra knowledge but just simply generalizing one deep prior in one sub-task. To this end, we introduce novel methods related to poses, scales, and occlusion parsing which are keys to enable deep priors to work together in a robust way. Deep prior assembly does not require any 3D or 2D data-driven training in the task and demonstrates superior performance in generalizing priors to open-world scenes. We conduct evaluations on various datasets, and report analysis, numerical and visual comparisons with the latest methods to show our superiority. Project page: https://junshengzhou.github.io/DeepPriorAssembly.

Main paper: 
 introduction: 
 Reconstructing scenes from images is a vital task in 3D computer vision and computer graphics. It bridges the gap between the 2D images that can be easily captured by phone cameras and the 3D geometries of scenes for various real-world applications, e.g., autonomous driving, augmented/virtual reality and robotics. Reconstructing scenes from multi-view images [60, 67] is well-explored to recover 3D geometries with multi-view consistency and camera poses. However, reconstructing a scene from a single RGB image is still challenging, which is extremely difficult due to inadequate information. Recent works [10, 42] try to solve this task as a reconstruction problem which leverages neural networks with an encoder-decoder architecture to draw supervisions from pairs of images and 3D ground truth geometries and layouts. Nevertheless, due to the limited amount of image-scene pairs, these methods struggle to generalize to out-of-distribution images in open world.Large language and vision models have been extensively studied in the past few years, which revolutionized neural language processing [58, 6] , 2D/3D representation learning [15, 77] and content generation [51, 26] , etc. By greatly scaling up sizes of training samples and model parameters, large models show brilliant capabilities and remarkable performance. However, they are limited in a specific task, which limits their capability in high level perception tasks. Driven by the observation, we raise an interesting question: can we assemble series of deep priors from large models, which are experts with different modalities in different tasks, to solve an extremely challenging task that none of them can accomplish alone? Figure 1 : An illustration of our work. We assemble diverse deep priors from large models with frozen parameters for scene reconstruction from single images in a zero-shot manner.In this work, we propose deep prior assembly, a novel framework which assembles diverse deep priors from large models for scene reconstruction from single images in a zero-shot manner. We rethink this task from a new perspective, and decompose it into a set of sub-tasks instead of seeking to a data-driven solution. We narrow down the responsibility of each deep prior on a sub-task that it is good at, and introduce novel methods related to poses, scales, and occlusion parsing to enable deep priors to work together in a robust way. Specifically, we first detect and segment the instances in the input image with Grounded-SAM [29, 33] , which is a variation of Segment-Anything Model [29] . For the segmented instances that are often corrupted due to occlusions or of low resolution, we leverage Stable-Diffusion [51] to enhance and inpaint images containing the segmented instances. However, the Stable-Diffusion often produces some predictions which drift away from the input instances and do not conform to the original appearance. To solve this issue, we introduce to use Open-CLIP [47, 23] to filter out the bad samples and select the ones matching the input instance most. We then generate the 3D models for each instance with Shap•E [26] using the amended instance images as input. Additionally, we estimate the depth of the origin image with Omnidata [13] as the 3D scene geometry prior. To recover a layout consistent to the image, we propose an approach to optimize the location, orientation and size for each 3D instance to fit it to the estimated segmentation masks and the depths.Deep prior assembly merely generalizes deep priors and does not require additional data-driven training for extra prior knowledge. Our evaluations on various open-world scenes show our capability of reconstructing diverse objects and recovering plausible layout merely from a single view angle. Our main contributions can be summarized as follows.• We propose deep prior assembly, a flexible framework that assembles diverse deep priors from large models together for reconstructing scenes from single images in a zero-shot manner.• We introduce a novel approach to optimizing the location, orientation and scale of instances by matching them with both 2D and 3D supervision.• We evaluate deep prior assembly for generating diverse open-world 3D scenes, and show our superiority over the state-of-the-art supervised methods.2 Related Work 
large models in different modalities: 
 Recently, it has been drawing significant attention on scaling up deep models for much more powerful representations and higher performances with different modalities (e.g. NLP, 2D vision).Starting from NLP, recent works in scaling up pre-trained language models [6, 34, 48] have largely revolutionized natural language processing. Some researches translates the progress from language to 2D vision [47, 12, 3, 20, 15] and 3D vision [77] via model and data scaling.Except for the large foundation models which focus on producing large-scale representations for language, 2D images or 3D point clouds, some researches explore large models for specific tasks (e.g. text-to-image generation [51] , image segmentation [29] , 3D analysis [62, 78, 76, 31, 30] and 3D object generation [80, 61, 35, 79] ) and have shown remarkable performance. Stable Diffusion trains a large model of latent diffusion models and achieves commercially available 2D content generationMatching Error = 0.18 × × √ Sofa Table Table Figure 2:The overview of deep prior assembly. Given a single image of a 3D scene, we detect the instances and segment them with Grounded-SAM. After normalizing the size and center for the instances, we attempt to amend the quality of the instance images by enhancing and inpainting them. Here, we take a sofa in the image for example. Leveraging the Stable-Diffusion model, we generate a set of candidate images through image-to-image generation, with additional guidance from a text prompt of the instance category predicted by Grounded-SAM. We then filter out the bad generation samples with Open-CLIP by evaluating the cosine similarity between the generated instances and original one. After that, we generate multiple 3D model proposals for this instance with Shap•E from the Top-K generated instance images. Additionally, we estimate the depth of the origin input image with Omnidata as a 3D geometry prior. To estimate the layout, we propose an approach to optimize the location, orientation and scale for each 3D proposal by matching it with the estimated segmentation masks and the depths (the ⋆ for the example sofa). Finally, we choose the 3D model proposal with minimal matching error as the final prediction of this instance, and the final scene is generated by combining the generated 3D models for all detected instances. effects. Segment Anything Model (SAM) [29] revolutionize the field of image segmentation by training models with large-scale annotated data. Omnidata [13] trains the large depth estimation model with various data sources for bringing robust 3D awareness to pure RGB images. In the 3D domain, the recent works Point•E [40] and Shap•E [26] collect millions of 3D objects to train large 3D models for generating 3D geometries from rendering-style images or texts. In this work, we aim at leveraging the powerful capabilities of the large models in different modalities and different domains to solve a challenging task, i.e. scene generation from single images, by assembling deep priors together.Recovering the underlying 3D surfaces of scenes from images [60, 70, 22, 19, 41] or point clouds [72, 75, 73, 25, 36, 74, 24, 43] is a long-standing task in 3D computer vision. Most of the previous works focus on the multi-view reconstruction with the input dense images captured around the scene. Classic multi-view stereo methods [1, 5, 4] mainly represent the scene by estimating depths for dense images with feature matching. Inspired by NeRF [39] which performs volume rendering for scene representation, a series of works [60, 44, 65, 59, 68, 69] introduce the neural implicit surface reconstruction by learning signed distance fields [45] or occupancy fields [38] for scenes from multi-view images. NeuRIS [59] proposes to use normal priors for indoor scene reconstruction, and MonoSDF [67] further introduces monocular depth cues for improving scene geometries. 
method: 
 Overview. The overview of deep prior assembly is shown in Fig. 2 . We will start from an introduction of the task decomposition in Sec. 3.1 and then present the pipeline for solving each of the decomposed sub-tasks using a deep prior from a specific large model in Sec. 3.2. Finally, we introduce an optimization-based approach for layout estimation in Sec. 3.3.Revealing 3D scene geometries from a single image is an extremely challenging task duo to limited context and supervisions. Instead of using a data-driven strategy to learn priors [42, 10] , we reformulate this task from a new perspective. We decompose it into a set of sub-tasks, each of which can be done using one deep prior without a need of learning extra knowledge. More specifically, we can progressively resolve the task by: 1) First, performing detection and segmentation on the input image to acquire the segmentation images, masks and category labels for all detected instances.2) Amending instance images through enhancing and inpainting to improve the image qualities.3) Generating a set of 3D model proposals for each instance from 2D segmented images.4) Estimating the layout by predicting the location, rotation, and scale for each 3D proposal to put them to the correct position of the 3D scene.5) Producing a scene reconstruction by applying the estimated layout and shape poses with reconstructed instances.Inspired by the remarkable performances of recent large models, we propose to assign an expert large model in each sub-task, which maximizes their abilities for modeling a scene in a zero-shot manner.Detect and Segment 2D instances. To reveal a scene S from a single image I, we first detect the instances in I and separate multiple objects into single instances. In this way, we can reconstruct a scene at shape level, which simplifies the task.Mask R-CNN vs. SAM vs. Grounded SAM. Detecting and segmenting images have been widely explored in the past few years. Mask R-CNN [21] is widely adopted as a popular and robust backbone. However, the performance of Mask R-CNN does not generalize well since it is only trained under a relative small dataset. Recently, the large SAM [29] have shown promising segmentation accuracy by scaling up parameters and using more training samples, nevertheless, it only predicts fine-grained masks but with few semantic concepts. Thus, we adopt Grounded-SAM, which is an improved version of SAM by introducing Grounding-DINO [33] as an open-set detector and using SAM to jointly predict detection boxes, segmentation masks and category labels for each instance, formulated as:EQUATION{m i , c i , d i } includes the predicted mask m i , category c i and the detection confidence score d i for the i-th instance, and N is the number of instances in this scene. We only keep the predictions with a high confidence score larger than a threshold σ. The low confident instances often contain large occlusions or wrong category predictions.Enhance and Inpaint 2D instances. With the predicted masks {m i } N i=1 , we achieve the segmented instance images {t i } N i=1 by masking the original input image I. We then normalize each instance image by centering it at the origin and normalizing its scale in {t i } N i=1 to 0.6 of the max width or height of I. As shown in Fig. 2 , the segmented instance images often suffer from occlusions or low-resolution of small instances. The low-quality images have a large negative impact on the followup 3D generation. Therefore, we propose to first improve the quality of instance images by enhancing and inpainting them with the large model Stable-Diffusion [51] .Specifically, we adopt the image-to-image generation [37] from Stable-Diffusion. We take the instance image t i as the initialization, and add noises on it and then subsequently denoise the noise corrupted image to increase the realism through the guidance of the text prompt description from the predicted categories c i . We find the prompt template 'High quality, authentic style {category}' works fine for most of the indoor instances. For other situations, we directly leverage the category as the prompt. We observe that Stable-Diffusion may produce some unreliable predictions which are 'too creative' and can not faithfully improve the image quality but turn it into another image, as shown in Fig. 3 . To solve this issue, we generate multiple enhanced images {e j i } M j=1 for each instance image t i and filter out the bad generation samples with the approaches described next. Filter Out Bad Generation Samples.To filter out the bad generation samples produced by Stable-Diffusion and select the top K enhanced images for the following 2D-to-3D generation, we propose to leverage the CLIP models as a judge to determine which generated images {e j i } M j=1 from Stable-Diffusion can conform to the original appearance of the instance t i . Specifically, we adopt the large open-sourced CLIP [47] model Open-CLIP [23] as the implementation.We use cosine similarities {z j i } M j=1 between the generated instance image {e j i } M j=1 and the original one t i as a metric for the selection, formulated as:EQUATION)where f θ is the frozen image encoder from Open-CLIP. We use the Top-K generated instance images with the largest similarities to the original t i as the amended images.As shown in Fig. 3 , Open-CLIP successfully filters out the bad generations.Generate 3D models from 2D instances. The object-level 3D generation from single images [38, 61] is a well-explored task in 3D computer vision, however, most previous works show limited generation performance on open-world images. Shap•E [26] is a large model for 3D generation by training a 3D hyper diffusion model with millions of non-public 3D objects. Therefore, we leverage Shap•E to provide the deep prior to convert 2D instance images into 3D reconstructions. By this way, we obtain K 3D reconstruction proposals {s k i } K k=1 for each 2D instance t i by employing Shap•E on the top K amended images.The final step is to select the most accurate 3D model proposal ŝi from the K candidates {s k i } K k=1 and put it to the right position in a 3D scene to recover the scene layout in the input image. To achieve this, we propose a novel approach to optimize the location, orientation and size for each 3D proposal in {s k i } K k=1 by matching it with the estimated segmentation mask and the estimated depth. We further introduce a RANSAC-like solution for robust position optimizing. We select the reconstruction with the minimum matching error as the reconstruction ŝi of t i . Depth Estimation. For more accurate layout estimations, we first estimate the depth map of the input image I as a 3D geometry prior. We leverage the large model Omnidata [13, 27] as the depth estimator which is trained under a collected huge depth dataset [13] containing 14-million indoor, outdoor, scenelevel and object-level samples.An issue here is that the depth D estimated for the input image I with Omnidata is not scale-aware, which can not be directly used as supervisions. We solve this problem by estimating the scale h and shift q of the predicted depth using one pair of predicted and real depth of a selected scene for each dataset. Specifically, we leverage least-squares criterion [14, 49] which has a closed-form solution to solve the depth scale and shift by matching the pair of predicted and real depth with a specific scene camera intrinsic parameter C K . After transforming D with the scale h and shift q, we can now back-project D into the 3D space with camera intrinsic parameter C K , achieving a 3D scene depth point cloud. The example in Fig. 4 shows that the depth point cloud produced using the transformed depth maps precisely aligns with the ground truth scene. Alternatively, we can use metric depth estimation methods [63, 66, 64, 28] , which naturally handle depth scale and shift, to replace Omnidata and lessen the reliance on ground truth depth data. The depth point cloud d i for each instance t i is further achieved by masking the back-projected 3D point cloud.Pose/Scale Optimization. We further estimate the scale and pose of each 3D model proposal s k i to put them into the right position in the 3D scene. We propose to solve this problem with an optimization-based approach on the location, rotation and size for s k i per the mask m i from the Grounded-SAM and the depth point cloud d i from Omnidata.We model this problem as a 7-DoF shape registration task with 3-DoF of translation (tx, ty, tz), 3-DoF of rotation (rx, ry, rz) and one DoF of object scale (v). Specifically, we first sample a point cloud p k i from the mesh of a 3D proposal s k i and initialize a 7-DoF transform as a transformation function f ϕ with learnable 7-DoF parameters ϕ. We then project p k i with f ϕ to achieve the transformed prediction pk i by:EQUATIONWe obtain pk i by optimizing the 7-Dof parameters ϕ with supervisions. With the estimated depth d i , we can draw the direct 3D matching supervision by minimizing the Chamfer Distance Loss between the transformed pk i and the depth points d i . However, merely with the 3D matching constraint, the pose/scale optimization do not always converge stably since the predicted depth d i is usually with noises in complex scenes, which significantly affects the registration on shapes.To resolve this issue, we get inspirations from [8] to leverage the mask information predicted by Stable-Diffusion as an extra matching supervision in 2D space. Specifically, we project the transformed 3D point cloud pk i to the 2D space with the camera intrinsic parameters C K , resulting in a 2D point cloud pk i . Meanwhile, we form another 2D point cloud mi from the mask m i by randomly sampling dense 2D points on the occupied region of the mask m i . We then use the 2D matching constraint to minimize the 2D Chamfer Distance between pk i and mi . We illustrate the effect of 2D Matching constraint with an optimization example in Fig. 5 . The final loss for pose/scale optimization of 3D reconstruction s k i is then formulated as: And the green 2D points donate the 2D projection of transformed 3D point clouds sampled from the generated shape of this chair instance. More robust registration is achieved with the proposed 2D matching constraint. The total 1,000 iterations take 9.2 seconds on a single 3090 GPU.EQUATIONWith the optimization-based 7-DoF registration, we are now able to put the generated 3D object proposals into the 3D scene. However, if the misregistration is quite large, especially in the rotation, the optimization may be trapped in a local optimum and fail to produce accurate registrations. We further introduce a RANSAC-like solution to enhance the robustness of pose/scale optimization. Specifically, we repeat the optimization r times with randomly initialized rotation matrices for f ϕ each time. The final transform for the 3D proposal s k i is selected as the one with minimum matching loss in Eq. ( 4) among r optimal optimizations, and we define the matching error w k i of s k i as the minimal matching loss. Table 1 : Comparisons on scene reconstruction from single images. Lower is better for CDL1 (i.e., Chamfer Distance), higher is better for F-Score. CDL1-S is the single-direction Chamfer Distance from the generated objects to the ground truth meshes. Datasets. We evaluate deep prior assembly under four widely-used 3D scene reconstruction benchmarks 3D-Front [17] , Replica [55] , BlendSwap [2] and ScanNet [11] .3D-Front BlendSwap Replica Method CDL1-S ↓ CDL1 ↓ F-Score↑ CDL1-S ↓ CDL1 ↓ F-Score↑ CDL1-S ↓ CDL1 ↓ F-Score↑ Mesh R-3D-Front [17] is a synthetic 3D dataset of indoor 3D scenes. We adopt the data pre-processed by PanoRecon [10] and randomly select 1,000 scene images from the test set as the single-image dataset. Note that all the images are captured parallel to the ground with camera locations at 0.75m height above the floor in the 3D-Front dataset. We follow PanoRecon to achieve the corresponding ground truth mesh for each image by only keeping the geometry at the same view and cull anything outside of the view frustum.The Replica [55] dataset is an indoor scene dataset which contains 8 scanned 3D indoor scene with highly photo-realistic 3D indoor scene reconstruction at both room and flat scale. We adopt the pre-processed data provided by MonoSDF [67] and sample one image for each scene as the single-image dataset. The ground truth meshes are obtained with the same way as 3D-Front.The BlendSwap [2] dataset is a high-fidelity synthesis 3D scene dataset collected by Neural-RGBD [2] , containing 9 scenes with complex geometries. We collect single-view images and corresponding ground truth meshes with the same way as Replica dataset.The ScanNet [11] dataset is a real-word 3D scene dataset captured by RGB-D cameras. We select 7 scenes from the test set of ScanNet and sample one image from each scene as the input.Baselines. We mainly compare our method with the state-of-the-art methods in scene reconstruction from single images, i.e., Mesh R-CNN [18] , Total3D [42] and PanoRecon [10] . Note that all these methods are data-driven methods and trained under 3D datasets with ground truth 3D annotations, while our method solves the task in a zero-shot manner. This means that we do not require any data-driven training on any 3D or 2D datasets, which is a much more flexible and general solution for the single image reconstruction task. We direct evaluate these methods with the official codes and the pre-trained models for numerical and visual comparisons.Metrics. We use Chamfer Distance and F-Score with the default threshold of 0.1 following [42, 46] as metrics. Since Mesh R-CNN and Total3D only predicts the 3D objects and do not generate the backgrounds (e.g. wall and floor), we further report the single-direction Chamfer Distance from the generated objects to the ground truth meshes, i.e., CDL1-S, to only evaluate the accuracy of generated objects. Note that Total3D can generate the scene layout which can roughly represent the background, however, we find that Total3D generates layouts with large errors on all the three datasets. Therefore we do not keep the layout of Total3D for evaluation. While we achieve the background points by back-projecting the segmented background depth maps. We sample 10k points on the ground truth meshes and the generated scenes of each methods for evaluation. Please refer to the appendix for more details on evaluation. 
scene reconstruction on 3d-front: 
 Tab. 1 reports numerical comparisons on the 3D-Front dataset. We achieve the best performance among the state-of-the-art methods. Specifically, PanoRecon is trained under the 3D-Front dataset, therefore it shows convincing results in this dataset. Mesh R-CNN and Total3D are trained under Pix3D [57] /ShapeNet [7] and SUN-RGBD [54]/Pix3D datasets, respectively.The qualitative comparison is shown in Fig. 6 , where we remove the background geometries for PanoRecon, ours and GT for a clear visual comparison on the generated instances among all the methods. We further show the colored scene since the used object generator Shap•E is able to generate textured 3D objects. The visualization demonstrates our superior performance to produce accurate and visual-appealing scene reconstruction from merely a single image in a zero-shot manner.We further evaluate our method under the open-world images from the BlendSwap dataset and the indoor scene dataset Replica. The quantitative comparisons in these two datasets are shown in Tab. 1, where we achieve the best performance over all the baseline methods. Note that the performance of PanoRecon [10] largely degrades under open-world scene images compared to the performance under 3D-Front dataset. The reason is that PanoRecon fails to generalize to the out-of-distribution inputs and can only handle the specific image patterns in the trained 3D-Front dataset.The visual comparison is shown in Fig. 7 , where we significantly outperform the previous works in the generation accuracy and completeness. Specifically, as shown in the 3-rd and 5-th row in Fig. 7 , our method generates more accurate geometries for the table with thin legs and the chair with a complex back. While Mesh R-CNN and Total3D can only generate the coarse 3D shapes and also fail to estimate accurate layout.We further evaluate deep prior assembly under the ScanNet [11] using real images. For a qualitative comparison with other methods, we select 7 scenes from the test set of ScanNet and sample one image from each scene as the input. We compare deep prior assembly with the state-of-the-art methods in scene reconstruction from single images, e.g., Mesh R-CNN [18] and Total3D [42] . We do not compare with PanoRecon [10] here since it fails to generalize to the out-of-distribution inputs and can only handle the specific image patterns in the trained 3D-Front dataset, as demonstrated in Fig. 7 .We show the visual comparisons in Fig. 8 , where we successfully reconstruct scenes from real images and significantly outperform the previous works in the reconstruction accuracy and completeness. This demonstrates the huge potentials of the assembled deep priors in reconstructing real-world 3D scenes. Note that the real-world images are often blurry and corrupted when the camera doesn't focus well, e.g., the blurry input image shown in the 5-th row in Fig. 8 . While our proposed deep prior assembly can also handle these challenging situations due to the powerful and robust deep priors from the large vision models. Table 3 : Ablation studies on the effect of Open-CLIP filtering and RANSAC-like solution.Effect of Open-CLIP and RANSAC-like solution. We further evaluate the effectiveness of filtering bad samples with Open-CLIP and the RANSAC-like solution for robust pose / scale optimization. The results is shown in Tab. 3, where both components improve the scene reconstruction accuracy. 
conclusion: 
 We introduce deep prior assembly, a novel framework that assembles diverse deep priors from large models for scene reconstruction from single images in a zero-shot manner. This approach breaks down the task into several sub-tasks, each of which is handled by a deep prior. We do not rely on any 3D or 2D data-driven training, and provide the key solutions on layout estimation and occlusion parsing to make all deep priors work together robustly. We report analysis, numerical and visual comparisons to show remarkable performance over the latest methods. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification:We use the open-sourced datasets under their licenses. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Alternatives on sub-tasks.We explore the effectiveness of our chosen solutions in each sub-task by comparing them with the alternatives. Specifically, we conduct ablations to replace Shap•E [26] with One-2-3-45 [32] , replace Open-CLIP [47, 23] with EVA-CLIP [56] and replace Omnidata [13] with MiDaS [50] in Tab. 4.We visually compare Shap•E with One-2-3-45 for shape generation in Fig. 9 , where the results demonstrate that Shap•E is a more robust solution for generating 3D models from 2D instances.We further conduct ablation studies to explore the effect of the instance scales to the generation qualities of Shap•E as described in "Enhance and Inpaint 2D instances" of Sec. 3.2 in our paper.We provide an visual comparison of the generations with different instance scales in Fig. 10 . The results show that Shap•E is quite sensitive to the scale of instances in the images, where a too small or too large scale will lead to inaccurate generations with unreliable geometries and appearances. We set the scale to 6 where the Shap•E performs the best in shape generation from instance images according to our experiences.A.3 The Effect of Confidence Threshold.We further conduct ablations to evaluate the effect of confidence threshold σ as described in "Detect and Segment 2D instances." of Sec.3.2 in our paper. As shown in Fig. 11 , a too large σ will drop too many instances and a too small σ struggles to filter bad instances. We choose σ = 0.4 as the suitable confidence threshold.We additionally compare our method with SOTA data-driven scene reconstruction works PanoRecon [10] , BUOL [9] and Uni-3D [71] . We show the visual comparisons under 3D-Front and ScanNet datasets in Fig. 12 , where our method achieves better and more visual-appealing results under both 3D-Front and ScanNet datasets. Specifically, our method significantly outperforms other methods using real-world images in ScanNet. The reason is that all the three methods are trained under 3D-Front, and struggle to generalize on real-world images.We further compare deep prior assembly with ScenePrior [41] in Fig. 13 . As shown, our method clearly outperforms ScenePrior in terms of the quality of scene geometries. Moreover, ScenePrior can only reconstruct the geometry, whereas deep prior assembly is capable of recovering high-fidelity scene appearances as well.We demonstrate that deep prior assembly can also reconstruct the background geometries from the scene images. We show two scene reconstructions with backgrounds (i.e. floor, wall) in BlendSwap and Replica datasets in We further evaluate the efficiency of our proposed deep prior assembly by reporting the average runtime of each sub-pipeline in our framework. The results in Tab. 5 show that reconstructing a scene from a single image takes less than 3 minutes in total, where the inference of Grounded-SAM, Open-CLIP and Omnidata takes only about 1 second. The most time consuming parts include the StableDiffusion, Shap•E and the RANSAC-like pose/scale optimization. For these three parts, we process all instances of the scene in parallel, resulting in significant time savings.PanoRecon. For evaluating PanoRecon [10] , we adopt the official code and pretrained models for inference and directly report the performance under 3D-Front [16] dataset by evaluating the metrics (e.g. Chamfer Distance and F-Score) between the reconstructions and the ground truth meshes. For the Replica [55] and BlendSwap [2] datasets where the scene location and orientation do not match the 3D-Front dataset where the PanoRecon is trained, we first normalize the center of the predicted scenes to the ground truth scenes and then register the predicted scene reconstructions to the ground truth scenes. Specifically, we first predict an initial alignment by a global registration algorithm based on feature matching [53] with RANSAC and then leverage ICP (Iterative Closest Point) registration algorithm [52] to obtain the fine registration based on the initial alignment. The metrics are reported with the registered reconstructions and the ground truth meshes.Total3D. We leverage the official code and the pretrained models for predicting scene reconstructions with Total3D [42] . We evaluate Total3D under 3D-Front, Replica and BlendSwap with a similar way as we evaluating PanoRecon to first normalize the predicted scenes and register them to the ground truth ones before computing metrics. Total3D only predicts 3D objects and do not generate backgrounds (e.g. wall and floor). Therefore, we further report the single-direction chamfer distance from the generated objects to the ground truth meshes, i.e., Chamfer-L1 (S), to only evaluate the accuracy of the generated objects. Note that Total3D can generate the scene layout which can roughly represent the background, however, we find that Total3D generates layouts with large errors on all the three datasets. Therefore we do not keep the layout of Total3D for evaluation.Mesh R-CNN. We adopt the official code and the pretrained models for predicting scene reconstructions with Mesh R-CNN [18] . We notice that Mesh R-CNN produces reconstructions with larger scales than the predictions of other methods and the ground truths. Therefore, we first normalize both the center and scale of the predicted scenes to the ground truth scenes and then register the predicted scene reconstructions to the ground truth scenes with a similar way as we evaluate Total3D.Deep Prior Assembly. We evaluate our proposed deep prior assembly in the same way. We follow the same settings as we evaluate other baselines to first normalize the center of the predicted scenes to the ground truth scenes, and then register the predicted scenes to the ground truth scenes. The background points (e.g. wall and floor) of deep prior assembly are obtained by back-projecting the background depth maps, i.e., the areas where no instances exists.One limitation of our method is that it may sometimes produce reconstructions with 3D instance models that are not perfectly aligned with the 2D instance segmentations in the input images. For example, the left chair generation in the last row of Fig. 8 exhibits a different color from the 2D instance of the input image. The reason is that we use Stable-Diffusion to enhance and inpaint the 2D instances, and then leverage Shap•E to generate 3D reconstructions. This process can introduce some randomness in the generated textures. The randomness primarily affects the appearances, while the geometries remain accurate. However, we justify that most reconstructions can faithfully recover the consistent scene appearances and geometries from the input images. Another limitation of deep prior assembly is that scene reconstructions may exhibit distortion due to inaccurate depth scale and shift. This issue can be addressed by replacing Omnidata with recent advances in metric depth estimation methods [63, 28] .• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. We discuss the applications and potential impacts of our method in the introduction.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).† The corresponding author is Yu-Shen Liu. 


Decision: All reviewers generally felt that reasons for acceptance outweigh those for rejection, although they agree that this feels like an engineering effort. Nevertheless, the AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: A multistage pipeline for single image 3D reconstruction is proposed, leveraging multiple off-the-shelf models. To begin, SAM is used to segment and decompose the input image. Stable diffusion is then leveraged to complete instance segments with potentially missing information, and failures of this process are filtered out by CLIP. Finally, Shap-E is applied to generate 3D models, which are then registered to the image for a final 3D reconstruction.
strength_weakness: strength_weakness: The method seems overly complicated and unlikely to be robust. 
* The argument about heuristic selection of depth shift seems unconvincing; in practice, the correct depth shift from a scale and shift invariant monodepth estimator can vary widely between multiple images even in the same scene. Why not use multiple pairs of images to compute the appropriate depth shift, or metrically ground the depth shift as in RealmDreamer?
* In order to go from a SAM instance to mask, a very complex pipeline is proposed. Essentially, it consists of inpainting with SD, followed by filtering with CLIP, followed by shape estimation with Shap-E, followed by a likely unstable alignment procedure. This seems error-prone. Why not just train an LRM-like model that accepts potentially off-center and partially occluded instance images and produces 3D objects aligned to the input camera’s location? This seems not too difficult to train, there are 10M+ object datasets publicly available to achieve it, and it would certainly be more robust than the proposed pipeline.
questions: questions: Since the method consists of chaining several foundation models together, it should be possible to show "zero-shot" performance on scenes not limited to the simple synthetic or indoor room scenes shown here. Can the proposed method work on more complex scenes, such as real images, outdoor scenes or scenes containing animals or people?
limitations: limitations: Yes, there is a thorough limitations discussion.
paper_summary: paper_summary: The method uses leverages multiple off-the-shelf models to parse a scene represented by a single image into 3D assets in their respective layout. Concretely they use a segmentation model to locate objects, then a diffusion model to enhance the image quality, use Shape-E to generate 3D proposals, and estimate the layout use depth estimation and point cloud matching. The method outperforms existing methods.
strength_weakness: strength_weakness: The method relies on the quality of foundational models to perform well. The comparison to other baselines is not apples to apples since the other methods are not trained on the large datasets leveraged by the foundational model. This isn't necessarily a flaw in this work, since leveraging additional data could be seen as a potential advantage, but it does create an unequal comparison. 

Like I mentioned in the strengths section, I think it's unclear what the motivation for the paper is. A little further discussion to anchor this work to an application would be helpful since the method novelty is not particularly high.
questions: questions: Does this work for other types of scenes such as outdoor? Or is restricted to indoor scenes with furniture? 

**Suggestions:**

Figure 2 has a lot of elements and is hard to parse. Maybe simplifying the figure to illustrate the main idea would help.
limitations: limitations: Yes limitations are discussed.
paper_summary: paper_summary: This work introduces a system named "Deep Prior Assembly" for zero-shot scene reconstruction from a single image. It breaks down the single-image scene reconstruction task into several steps that can be solved utilizing pretrained large models, such as SAM for segmentation, Shap-E for 3D object generation, and Omnidata for depth estimation. Additionally, an optimization-based approach is also proposed for pose estimation.
strength_weakness: strength_weakness: **1. Technical contribution of this work is limited.**

The work seems to be more of an engineering trial that combines pretrained models to build a reconstruction system, rather than an insightful research endeavor. The authors highlight that they are exploring "deep prior" from large pretrained models, but the priors are not well integrated and instead function independently as separate modules. Additionally, naively combining many large models could lead to high computational demand and potential error accumulation. For example, the proposed system will run a diffusion generative model several times for each object in a scene, and includes a 9.2-second optimization for pose estimation of each object.

**2. Missing state-of-the-art baseline methods.**

Although there are thorough evaluation results provided in the paper, all the baseline methods are proposed before 2021. Has there been any new progress in single-view scene reconstruction after 2021? For example, ScenePrior [1] published in CVPR'23 introduces a conditional autoregressive generative method for single-view reconstruction.

---
[1] Learning 3D Scene Priors with 2D Supervision. Nie et al. CVPR 2023.
questions: questions: **1. What time does each stage of the proposed method take? And what time do the baseline methods take?**

**2. What's the relation between the entire running time of reconstruction and the number of objects in the scene? And how about the baseline methods?**

**3. Have you considered using methods like 3D object detection for layout estimation? It could potentially offer faster and more accurate results than the proposed depth-based method.**
limitations: limitations: **1. A naive combination of pretrained large models might cause error accumulation.**

**2. The proposed method has a much longer running time compared to baseline methods.**
paper_summary: paper_summary: This paper introduces a framework called "Deep Prior Assembly," which combines various deep priors from large models to reconstruct scenes from single images in a zero-shot manner. By breaking down the task into multiple sub-tasks and assigning an expert large model to handle each one, the method demonstrates the ability to reconstruct diverse objects and plausible layouts in open-world scenes without additional data-driven training.
strength_weakness: strength_weakness: 1.	One obvious drawback of this methodology is the huge memory cost caused by assembling many pre-trained modules if we aim to apply this method on mobile edge device. Can the authors think out of a way to mitigate this problem?
2.	The reconstruction quality of this framework relies mainly on the capacity of single-image reconstruction model, e.g. Shap-E in this paper. However, according to practical experience, Shap-E is really sensitive to the scale of input image. Is there a universal appropriate scale that fits to all the instances, as mentioned in the paper the scale is set to 6?
3.	The methodology tries to fit the 3D proposal into the scene by aligning them with the depth estimated by Omnidata. According to the ablation, 3D matching plays the most important role compared to SD and 2D-matching in the whole framework. So how to ensure the quality of estimated depth? 
4.  I am curious about how will the methodology perform on the outdoor scenes.
questions: questions: Please refer to the weaknesses part.
limitations: limitations: Please refer to the weaknesses part.

Comments: title: Response to Reviewer osDy
Response to Reviewer DEJS
Rebuttal
Response to Reviewer veoH
Response to Reviewer 67ke
Response to Rebuttal
Responses to Reviewer veoH
We will be happy to take any questions
comment: Dear Reviewer osDy,

Many thanks for all the helpful comments and positive assessment. We will include the new experiment results and more reconstruction sampls in the revised version of the manuscript. 
Following your suggestions, we consider training a reconstruction model similar to ZeroShape for predicting complete shapes as a future work of DeepPriorAssembly. We will include the insights to Sec.5 of the revised paper and will conduct experiments to explore the effectiveness of this alternative.

We really appreciate you for upgrading the score. 

Best regards,

Authors
Dear Reviewer DEJS,

Thank you for your response and the positive assessment on our rebuttal. We really appreciate your expertise and all the invaluable feedback. The technical contributions of DeepPriorAssembly lie in the the exploration of the robust and optimal approach to robustly integrating available deep priors for a challenging task, rather than simply combining them.

We show that simply combining several large models is not enough to solve the difficult task of zero-shot scene reconstruction. While our work is the first to propose assembling large models and decomposing tasks to tackle this challenge, we also make key technical contributions to ensure deep priors work together effectively and robustly.

The naive approach involves segmenting scene images and generating 3D objects from these segments, but it fails because (1) occlusions and low resolution often lead to incomplete or poor-quality 3D objects, and (2) existing techniques can't recover the 3D scene layout.
To overcome these issues, we propose two main techniques:

1. To enhance the quality of 3D instances and handle occlusions and low resolution, we introduce the StableDiffusion model to refine and inpaint segmented images, followed by CLIP models to filter and select the best-matching instances. These innovations in task decomposition and deep prior integration are crucial for producing accurate and high-quality 3D shapes.
2. For recovering the scene layout, we introduce a novel method to optimize the location, orientation, and size of instances by matching them with both 2D and 3D supervision. These supervisions come from estimated segmentation masks and predicted depths, generated by the assembled deep priors. Additionally, a RANSAC-like solution further improves the robustness of pose and scale optimization. This method effectively links the deep priors and is key to achieving robust zero-shot scene reconstruction.

Thank you for your valuable feedback and the time you spent evaluating our work. We truly appreciate your insights. 

Best regards,

Authors
Thanks for the thoughtful rebuttal. I'll change my score to borderline accept. 

I'm still unconvinced about the pipeline setup. You might consider https://zixuanh.com/projects/zeroshape.html (in particular how they handle their training data) as an alternative and competing approach which would be likely to have strong performance if trained in the specified way. 

The results on in the wild images are impressive! I would definitely suggest featuring these (and more + other examples) more prominently in a revised version of the manuscript, since the existing datasets are rather monotonous and supervised methods would probably perform well, so the "zero-shot" aspect is not really emphasized.
I appreciate the detailed response to the review and have also read the comments from other reviewers. While the proposed method presents impressive experimental results, it is more like an engineering effort by combining several pretrained modules to build a software pipeline, which is not quite suitable for the NeurIPS research community. Therefore, I maintain my original score of "5: Borderline accept" and would like to participate actively in the subsequent reviewer discussion period.
Dear reviewer,

Please read the author rebuttal and the other reviews and post a comment as to how your opinion has or has not changed and why.
Dear Reviewer veoH,

Thank you for your response and the positive assessment. We really appreciate your expertise and all the invaluable feedback. We response to each of your additional questions below. 

**Discussion-Q5:Efforts on the combination way of deep priors.**

We have invested extensive efforts in exploring the optimal approach to robustly integrating available deep priors for a challenging task, rather than merely combining them. 

We demonstrate that the naive approach of simply combining several large models fails to solve the challenging task of zero-shot scene reconstruction. Beyond the contribution of firstly proposing to assemble large models and the task decomposition for this challenge task, we offer additional significant technical contributions on solving the critical challenge of making deep priors work together robustly. Specifically, the naive solution involves segmenting the input scene images and then generating 3D objects from the segmented instances. However, the solution fails dramatically due to: (1) the instances are often corrupted by occlusions and low-resolution, leading to failures for reconstructing complete and high-quality 3D objects, and (2) none of the existing techniques are capable of recovering the scene layout for 3D objects.

To address these challenges and robustly assemble deep priors for zero-shot 3D scene reconstruction, we propose two significant technical contributions on improving the quality of 3D instances and accurately recovering the scene layouts.

(1) To improve the robustness of the framework and overcome the challenges of occlusions and low-solution in segmented instances, we novelly introduce the StableDiffusion model to enhance and inpaint the instance images, followed by CLIP models to filter out the poor-quality samples and select the ones matching the instance most. The novel designs on task decomposition and introducing suitable deep priors are the key contributors for achieving accurate and high-quality geometries and appearances of the generated shapes.

(2) For the challenging task of recovering the layout of the scene containing the reconstructed shapes, we propose a novel approach on optimizing the location, orientation and size of instances by matching them with both 2D and 3D supervisions. The supervisions are novelly introduced from the estimated segmentation masks and the predicted depths, which are also obtained by the assembled deep priors. Moreover, a RANSAC-like solution is proposed to further improve the robustness of the pose/scale optimization. This approach is the link among deep priors and plays the key role in robustly assembling deep priors for the final target of zero-shot scene reconstruction.

Following your suggestions, we will conduct more experiments on the combination way of deep priors in our revision.

**Discussion-Q6:Technical contributions.**

Our technical contributions lie in the novel task decomposition, the deep priors chosen and their assembly, and the proposed RANSAC-like pose/scale optimization for recovering scene layouts. We summarize the main contributions as follows.

1. We propose the first framework which assembles diverse deep priors from large models together for the extremely difficult task of reconstructing scenes from single images in a zero-shot manner.
2. To improve the robustness of the framework and overcome the challenges in this task (e.g. occlusions and low-solution of instances), we novelly utilizes the StableDiffusion model for image enhancement and inpainting, combined with the CLIP model to filter out poor-quality samples.
3. We introduce a novel approach on optimizing the location, orientation and size of instances by matching them with both 2D and 3D supervisions. Moreover, a RANSAC-like solution is proposed to further improve the robustness of the pose/scale optimization. The approach is the link among deep priors and plays the key role in robustly assembling deep priors for the final target of zero-shot scene reconstruction.

Only with our designs on the task decomposition, deep prior chosen and our novel approach on RANSAC-like pose/scale optimization through both 2D and 3D matching to recover the scene layout, can the assembly of deep priors from large models effectively succeed in the extremely challenging task of zero-shot scene reconstruction.

We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further.

Best regards,

Authors
Dear Reviewer 67ke,

Many thanks for the positive assessment. We really appreciate your expertise and all the invaluable feedback. Our method focuses on how to robustly join available deep priors for a challenging task. For the technical contributions, we make additional clarifications below. 

We justify that the naive approach of simply introducing several large models fails to solve the challenging task of zero-shot scene reconstruction. Beyond the contribution of firstly proposing to assemble large models and the task decomposition for this challenge task, we offer additional significant technical contributions on solving the critical challenge of making deep priors work together robustly. Specifically, the naive solution involves segmenting the input scene images and then generating 3D objects from the segmented instances. However, the solution fails dramatically due to: (1) the instances are often corrupted by occlusions and low-resolution, leading to failures for reconstructing complete and high-quality 3D objects, and (2) none of the existing techniques are capable of recovering the scene layout for 3D objects.

To address these challenges and robustly assemble deep priors for zero-shot 3D scene reconstruction, we propose two significant technical contributions on improving the quality of 3D instances and accurately recovering the scene layouts. 

(1) To improve the robustness of the framework and overcome the challenges of occlusions and low-solution in segmented instances, we novelly introduce the StableDiffusion model to enhance and inpaint the instance images, followed by CLIP models to filter out the bad samples and select the ones matching the instance most. The novel designs on task decomposition and introducing suitable deep priors are the key contributors for achieving accurate and high-quality geometries and appearances of the generated shapes.

(2) For the challenging task of recovering the layout of the scene containing the reconstructed shapes, we propose a novel approach on optimizing the location, orientation and size of instances by matching them with both 2D and 3D supervisions. The supervisions are novelly introduced from the estimated segmentation masks and the predicted depths, which are also obtained by the assembled deep priors. Moreover, a RANSAC-like solution is proposed to further improve the robustness of the pose/scale optimization. This approach is the link among deep priors and plays the key role in robustly assembling deep priors for the final target of zero-shot scene reconstruction. 

Our technical contributions can be summarized as follows.

1. We propose the first framework which assembles diverse deep priors from large models together for the extremely difficult task of reconstructing scenes from single images in a zero-shot manner. 
2. To improve the robustness of the framework and overcome the challenges in this task (e.g. occlusions and low-solution of instances), we novelly utilizes the StableDiffusion model for image enhancement and inpainting, combined with the CLIP model to filter out poor-quality samples.
3. We introduce a novel approach on optimizing the location, orientation and size of instances by matching them with both 2D and 3D supervisions. Moreover, a RANSAC-like solution is proposed to further improve the robustness of the pose/scale optimization. The approach is the link among deep priors and plays the key role in robustly assembling deep priors for the final target of zero-shot scene reconstruction. 

Only with our designs on the task decomposition, deep prior chosen and our novel approach on RANSAC-like pose/scale optimization through both 2D and 3D matching to recover the scene layout, can the assembly of deep priors from large models effectively succeed in the extremely challenging task of zero-shot scene reconstruction. 

We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further.

Best regards,

Authors
Thanks for your response. I appreciate your effort on ablation studies for different foundation models.

I think the ability of "zero-shot" within your method comes from the off-the-shelf foundation models(mainly) and the divide-and-conquer thought. So this should not be a prior advantage over other data-driven methods. In my opinion, the authors should design  extensive experiments to demonstrate the advantages of the way of their combination, rather than "combination" itself.

Overall, I think the technical contribution of this assembly is still limited, so I will keep my score as 5.
I thank the authors for their response.

 After reading other reviews and other responses to reviewers I will maintain my rating of borderline, 5. I think the technical contribution is limited as foundational models are being fused together and the techniques for combining don't seem particularly general. I think the motivation for matching assets to images of scenes is rather limited in its current form. In robotics, meshes with 6 DoF are estimated from images for grasping, but this task is a bit far from those in robotics.  

For the positives the paper improves over current methods, the figures and qualitative results are well done, and the writing and organization is clear. Therefore I maintain my rating of 5.
Dear reviewers,

As the reviewer-author discussion period is about to end, we are looking forward to your feedback on our rebuttal. Please let us know if our responses address your concerns. We would be glad to make any further explanation and clarification.

Thanks,

The authors
Dear Reviewer veoH,

Thank you for your response and the helpful comments. We respond to each of your additional questions below. Please do not hesitate to let us know if you have any additional questions.

**Discussion-Q1: Ablation studies on the choice of foundation models.**

We fully recognize the importance of evaluating each sub-task by substituting the foundational models with  similar alternatives. Actually, we have conducted the ablation study to explore the effectiveness of our chosen solutions in each sub-task by comparing them with the alternatives. The results and analisis are presented in Sec.C.1 and Table 4 of the Appendix. Specifically, we conducted ablations to replace Shap·E with One-2-3-45, replace Open-CLIP with EVA-CLIP and replace Omnidata with MiDaS, observing performance degradation with the alternative models. We further visually compare Shap·E with One-2-3-45 for shape generation in Fig. 8, where the results demonstrate that Shap·E is a more robust solution for generating 3D models from 2D instances. These ablation studies validate the effectiveness of each choice made within our framework.

**Discussion-Q2: The claim of zero-shot scene reconstruction.**

Actually, the term "zero-shot" signifies that no specific data or data-driven training is required for the task of scene reconstruction from single images. Unlike previous works in this field, such as PanoRecon and Total3D, which require specific image-scene pair data, our approach does not rely on such data.  The training data used in Shap·E is only capable of shape reconstruction learning, and it is impossible to train a model for single-view scene reconstruction solely by relying on the Shap·E data.

We justify that for the "zero-shot" tasks, using large-scale data from a different domain or task is not prohibited. For example, CLIP models are widely used for zero-shot image classification without requiring specific image-class pair data, though they do require massive amounts of image-text pair data for contrastive learning. Similarly, the data used in Shap·E does not undermine our claim of "zero-shot" single-view scene reconstruction. 

We will provide further clarification on the tasks and claims to enhance the understanding of DeepPriorAssembly's capabilities and the data used in each foundational model integrated into our framework.

**Discussion-Q3: Performance on outdoor scenes.**

We would like to justify that none of the previous works can successfully reconstruct outdoor scenes from single images. All prior approaches are trained on indoor scenes and struggle to generalize to real-world outdoor images, which contain out-of-distribution objects such as trees, buildings, and animals. Leveraging powerful large foundational models, DeepPriorAssembly is the first to demonstrate the capability for zero-shot scene reconstruction in these complex real-world outdoor scenes. As illustrated in Fig.A of the rebuttal PDF, DeepPriorAssembly accurately reconstructs scene geometries even in challenging outdoor scenes. However, the texture produced by Shap·E may not be optimal for the outdoor shapes. In the future, we may consider replacing Shap·E with a shape reconstruction method that performs better on outdoor shapes to further enhance the performance of outdoor scene reconstruction.

**Discussion-Q4: Optimize the connection and compatibility between different priors.**

Thanks for your suggestions. In the revision, we will improve the connection and compatibility among the foundation models by conducting more ablation studies on the choice of each foundation model within our framework, supplementing the analysis presented in Sec.C.1 and Table 4.

We are deeply grateful for your invaluable feedback and the time you dedicated to evaluating our work. Your comments and expertise are sincerely appreciated. Please let us know if there is anything we can clarify further. 

Best regards, 

Authors
Thanks for your explanation.

1. As this paper dedicated to assemble different priors from large foundation models, I think it's more convincing to show the robustness of the pipeline by substitute different parts with similar foundation models. 
2. Moreover, I think it's inappropriate to claim your method's superior performance on zero-shot manner, because this capacity actually comes from the massive training data of Shap-E. Also, as we can observe from the figure A in the rebuttal PDF, the method fails to perform well on outdoor scene, which is limited by Shap-E. 

Rationally, I would like to categorize this method as a straight-forward engineering effort to assemble the foundation models. Hope the authors can optimize the connection and compatibility between different priors.
Dear reviewers,

We appreciate your comments and expertise. Please let us know if there is anything we can clarify further. We would be happy to take this opportunity to discuss with you.

Thanks,

The authors