Title: Restore3D: Breathing Life into Broken Objects with Shape and Texture Restoration
Venue: NeurIPS 2025
Accepted: no
Keywords: Diffusion Models

3D object completion

Multi-view Image Generation

Multi-view Image Inpainting
Abstract: Restoring incomplete or damaged 3D objects is crucial for cultural heritage preservation, occluded object reconstruction, and artistic design.
Existing methods primarily focus on geometric completion, often neglecting texture restoration and struggling with relatively complex and diverse objects.
We introduce Restore3D, a novel framework that simultaneously restores both the shape and texture of broken objects using multi-view images. To address limited training data, we develop an automated data generation pipeline that synthesizes paired incomplete-complete samples from large-scale 3D datasets. 
Central to Restore3D is a multi-view model, enhanced by a carefully designed Mask Self-Perceiver module with a Depth-Aware Mask Rectifier.
The rectified masks, learned through the self-perceiver, facilitate an image integration and enhancement phase that preserves shape and texture patterns of incomplete objects and mitigates the low-resolution limitations of the base model, yielding high-resolution, semantically coherent, and view-consistent multi-view images. 
A coarse-to-fine reconstruction strategy is then employed to recover detailed textured 3D meshes from refined multi-view images. Comprehensive experiments show that Restore3D produces visually and geometrically faithful 3D textured meshes, outperforming existing methods and paving the way for more robust 3D object restoration. Project page: https://nip-ss.github.io/NIPS-anonymous/ .


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a multi-step method to inpaint the full 3D geometry and texture for partial object from multiview images. They use pretrained models to inpaint the 2D image, lift to 3D, finetune the mesh, and finally optimize the texture. \n\nContribution:\n1.  introduce an automated data synthesis pipeline that generates paired incomplete and complete shapes and texture\n2. introduce Restore3D, a pipeline to recover full 3D from partial multiview images\n3. validate the effectiveness by various experiments\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Good visual result. The model performs better than baselines shown in Fig. 6 and Fig. 7\n2. Real-world applications. It is interesting that the model could reconstruct the full objects from occlusion, as shown in Fig. 10, which shows some level of generalizations even using blender to create the training data.\n\nWeakness:\n\n1. The pipeline has 6 different stage, which looks quite complex to me: (1) Inpaint multiview images to complete ones (2) use pretrained  Real-ESRGAN to do super-resolution (3) use ControlNet-Tile to handle the inconsistency during super-resolution (4) use pretrained LRM[69]. model to genrate coarse mesh and render normal (5) use pretrained StableNormal to refine the previous mesh (6) optimize the texture with the input image. \n\nI feel these pretrained model are not tightly associated. For example, the input image after super-resolution (1024*1024) are resized back to low resolution (320*320) and feed to LRM, making the previous super-resolution a bit of unnecessray. And why not train a model to inpaint model and do super-resolution at the same time? Since super-resolution introduces many artifacts, the authors need to use another model to rectify that.\n\nAnd I think currently there are many end to end models that can handle step 4-6 with good performance. For example, what's the performance compared with other single-view to 3D pipeline? like [Direct3D](https://nju-3dv.github.io/projects/Direct3D/) and [TRELLIS](https://microsoft.github.io/TRELLIS/)? Though they have different number of inputs, they could generate high quality textured 3d objects too.\n\n2.  Does the pipeline require known camera during evalutation? The source of \"Camera Embeddings\" in figure 3 is not clear. And line 223-225 mentions that the model \"need to convert these normals into world space using camera extrinsic parameters\". If camera matrix are provided by user, then it will make the application . If the camera matrix are predicted, then this step may be sensitive to the inaccruate camera pose.\n\n3. The authors creating incomplete dataset using random size ico sphere or cube (line 138), which may causes the cross-section to be too smooth and regular, and may not able to fully reflect the missing appearance of real objects. \n \n4. (Minor Weakness) Writing issue.\nIn figure 1, what does \"Complete3D\" refer to? it is \"Restore3D\" (the proposed method)?\n\nQUESTIONS:\nAmong all weakness in above sections, I think the author should compare the multiview-to-3d steps with current end-to-end baselines, like single RGB to 3D pipelines etc. My main concern is that these end-to-end models may be more straightforward but reach comparable or better result.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors have proposed a method, Restore3D, for shape and texture completion of incomplete 3D objects, using consistent multi-view inpainting and Large Reconstruction Models (LRMs). At the core of the method is a mask prediction module which predicts multi-view consistent masks from a single image, crucial for consistent multi-view inpainting. A LRM uses the completed multi-view images to produce 3D reconstruction of objects. Finally, a quality enhancement pipeline, renders the recovered object into normal images and employs normal image enhancement and texture enhancement modules to improve the shape and texture quality of the 3D reconstruction. The method achieves improved performance in comparisons with recent SOTA for the task of 3D object completion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The multi-view mask prediction with the Mask Self-Perceiver module is a novel ideal as far as I am aware, for multi-view consistent inpainting. The implementation is interesting and according to the provided results is seems to be working.\n\n- Multi-view inpainting demonstrates improved results compared to SOTA.\n\n- The authors provided an ablation study which helps to understand the individual effect of the employed design choices.  \n\nWeaknesses:\n- The motivation of the paper, as well as the problem statement is not aligned with the conducted work and experiments. The paper is presented as a method for 3D reconstruction of fractured objects and states the importance of restoring incomplete or damaged objects. However the work is a 3D object completion method and the conducted experiments mainly focus on synthetic occlusions rather than real. \n\n- The evaluation part of the paper lacks of  experiments on real or synthetically fractured objects, again diverging the method from the initial motivation. \n\n- The method produces higher quality 3D reconstructions than LRM and InstantMesh, however coarse-to-fine quality improvement is not a novel idea as has been done in a similar way in [*1]. Also, recent related methods such as [*1, 2, 3] about enhancing an initial 3D reconstruction are not discussed or compared with in the paper. \n\n[*1] Wu K, Liu F, Cai Z, Yan R, Wang H, Hu Y, Duan Y, Ma K. Unique3d: High-quality and efficient 3d mesh generation from a single image. In The Thirty-eighth Annual Conference on Neural Information Processing Systems 2024 Jan 1.\n\n[*2] Luo Y, Zhou S, Lan Y, Pan X, Loy CC. 3DEnhancer: Consistent Multi-View Diffusion for 3D Enhancement. In Proceedings of the Computer Vision and Pattern Recognition Conference 2025 (pp. 16430-16440).\n\n[*3] Edelstein Y, Patashnik O, Cohen-Bar D, Zelnik-Manor L. Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation. In Proceedings of the Computer Vision and Pattern Recognition Conference 2025 (pp. 21458-21468).\n\nQUESTIONS:\n- Regarding the reconstruction results in section 4.2, what is the input of the other methods? Do you use your inpainting to first complete the image and pass the same input to all methods? \n\n- Line 277. Training time should be Inference time?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCollaborative restoration of complex object shape and texture.\nthis work designs a mask self-perceiver module with a depth-aware mask corrector to autonomously perceive and reconstruct missing parts, ensuring that the original features are preserved, and develops an image fusion enhancement pipeline to restore fine details.\nAt the same time, a large number of comparative experiments and ablation studies were conducted to prove that the above method is reasonable and efficient.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The first batch of work exploring the collaborative restoration of complex object shapes and textures\n- No need for manual mask annotation, with automatic mask generation mechanism, to achieve higher pipeline automation\n\nWeaknesses:\n- The experimental part lacks some comparisons with other related works. For example, in the inpaint experiment, there should be comparisons with other multiview methods, not just comparisons based on SD.\n- The fonts and diagrams in the charts are poor, and many contents need to be compared with the article repeatedly to understand them.\n- A key comparison was not made: after processing with the single-image mask inpaint solution, the SOTA multiview generation solution was used for generation and reconstruction.\n\nQUESTIONS:\n- For the inpainting experiment, other multiview inpainting methods are needed for comparison, such as using controlnet's mvpaint instead of the single-view version of SD\n- We need to add more experiments. For example, in the image infilling stage, use a single view to infill the first image, and use the SOTA multiview method to generate the rest. For example, if we get multi-view input at the very beginning, then directly use LRM to reconstruct, and then use the mask of the front view to project to other views, what will be the result (using the above method to infill)?\n- The training data are all synthetic data, but in actual use, it is very likely that they are not synthetic data. Are there any experimental results for this data gap? Please provide some analysis of failure cases.\n- As far as I know, the current high-resolution multi-view reconstruction has achieved a resolution of 1k to 2k (that is, 2k to 4k multi-view). How is the difference in details between the multi-stage coarse-to-fine scheme used in this work and these high-resolution schemes?\n\nLIMITATIONS:\n- From the pipeline, the input set in this work is multi-view input, which is a rare case. In this case, it should be convenient to perform a geometric reconstruction first and then inpaint, but the author did not do this.\n- The coarse-to-fine solution only increases the resolution to 512, which cannot reach the 1k or 2k resolution level of SOTA.\n- This work mentions that this method is needed in areas such as cultural relics restoration, but there is no relevant content in the experimental results and applications", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a system for reconstructing 3D objects from multi-view images of partial (aka broken) objects. I find this to be an interesting problem due to its applicability in heritage restoration and occluded object reconstruction which is highly relevant for domains like robotic vision. Moreover, the system is capable of generating texture in addition to geometry. The paper proposes to achieve this by leveraging several vision foundation models and combining them with custom made modules like Mask Self-perceiver and Depth-aware Mask Rectifier to achieve state of the art performance. Extensive comparisons with state of the art methods demonstrate the superiority of Restore3D. Several ablations are also included to help understand component contributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The problem of reconstructing complete 3D assets from multiple images of broken objects is a novel problem with several applications. Restore3D generates visually superior geometry and texture results compared to prior state of the art. Table 1 and Figure 6 corroborate paper's claims of being SOTA for multi-view image inpainting.\n\n2. Clever use of Foundation Models to achieve reconstruction of geometry and texture. The paper proposes Mask self-perceiver and depth-aware mask rectifier for effective combination of priors from foundation models to perform image Inpainting. Their contributions are mentioned in Table 2 and Figure 8. \n\n3. The method uses standard tricks in reconstruction like iterative refinement to get better results. Moreover the lack of need for manually defining the masks further helps with the usability of the method. \n\n\nWeakness:\n\n1. Lack of real-world examples both for training as well as inference. I am afraid that the paper hasn't explored the real world use case a bit even though this is one of the core motivations for their problem. It would have been interesting to see model trained on synthetic data deployed on real world multi-view images of broken objects. \n\n2. Material synthesis is not physically based. Given that the model is trained on synthetic dataset, I wonder what is the difficulty in generating albedo, specularity, etc. \n\n3. Needing to run multiple VFMs, means heavy GPU requirements. It would be good to have a discussion on the efficiency viz baseline methods.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}