InfiniteMesh: View Interpolation using Multi-view Diffusion for 3D Mesh Reconstruction

Authors: Zhiyuan Wu, Xibin Song, Steve Wang, Weizhe Liu, Ziang Cheng, Jiayu Yang, Weixuan Sun, Shenzhou Chen, Taizhang Shang, Shan Luo, Pan Ji
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: View Interpolation, Multi-view Diffusion, 3D Mesh Reconstruction

Abstract:
We present InfiniteMesh, a feed-forward framework for efficient high-quality image-to-3D generation with view interpolation. Recent advancements in Large Reconstruction Model (LRM) have demonstrated significant potential in extracting 3D content from multi-view images produced by 2D diffusion models. Nevertheless, challenges remain as 2D diffusion models often struggle to generate dense images with strong multi-view consistency, and LRMs often exacerbate this multi-view inconsistency during 3D reconstruction. To address these issues, we propose a novel framework based on LRM that employs 2D diffusion-based view interpolation to enhance the quality of the generated mesh. Leveraging multi-view images produced by a 2D diffusion model, our approach introduces an Infinite View Interpolation module to generate interpolated images from main views. Subsequently, we employ a tri-plane-based mesh reconstruction strategy to extract robust tokens from these multiple generated images and produce the final mesh. Extensive experiments indicate that our method generates high-quality 3D content in terms of both texture and geometry, surpassing previous state-of-the-art methods.

Decision: reject
Reviewer scores: 5, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730453399236

thread_type:
official_review

<thread_object>
creator: reviewer_ystn

<content>
summary:
This paper proposed an LRM-based image-to-3D generation with view interpolation. The proposed method consists of two key components: (1) an interpolation model that effectively expands 4 input views to 16 views by synthesizing intermediate viewpoints from multiview diffusion model outputs, and (2) a final LRM model that processes these interpolated views for 3D reconstruction. Experimental results demonstrate that this approach generates high-quality 3D content with superior texture and geometry, outperforming previous state-of-the-art methods.

strengths:
1. The paper's idea is easy to follow.
2. The proposed method demonstrates improvements in both accuracy and computational efficiency while enhancing the continuity and quality of the generated 3D contents.

weaknesses:
1. The proposed method is merely a combination of existing methods, lacking novelty.
2. The model lacks evaluation of OOD (out-of-distribution) datasets or real-world datasets, making its generalization capability questionable.
3. The experimental settings are unclear in the comparative experiments. LGM is a model that converts 4 views to 3D Gaussians, yet it appears in Table 2 in single-image-to-3D comparison experiments. The authors need to further clarify the evaluation details of other models.

questions:
1. When training the multi-view enhanced InstantMesh, are the input multiple views derived from the previous interpolation step or directly using ground truth ？
2. Have you considered performing secondary interpolation on the already interpolated views to obtain more viewpoint data to further improving the model's performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607059050

thread_type:
official_review

<thread_object>
creator: reviewer_he5a

<content>
summary:
This paper presents an effcient image-to-3d generation method with improved multi-view consistency. Building on LRM and 2d diffusion model, it introduces an Infinite View Interpolation (IVI) module, designed to create two interpolated images between main views to improve the quality and consistency of the generated 3D mesh. A tri-plane-based mesh reconstruction is then used to convert these multi-view images into a final 3D mesh with superior texture and geometry. The method demonstrates great generation results.

strengths:
1. The paper is well-written and easy to follow. The author has effectively organized the content and presented ideas in a clear and coherent manner.
2. Consistency in novel view synthesis is at the core of many image-to-3D task. The proposed method is able to outperform V3D and SV3D in multi-view consistency by a large margin. It seems that the introduced IVI module is efficient for generating more consistent meshes.
3. The paper demonstrates its effectiveness through substantial qualitative and quantitative experiments to verify its method outperforms baselines.

weaknesses:
1. The framework primarily builds on existing methods, incorporating previously developed components like the multi-view diffusion model Wonder3D and tri-plane-based Large Reconstruction Model (LRM). I expect the performance gain is great. However, this work does not show video results, and the results in Figure 3 is only marginally better than baselines. It is hard to validate the performance of this work.
2. simple comparison cases: The experimental comparisons mainly focus on simple cases. Including more complex and diverse cases would better showcase the model's strengths and highlight its potential advantages over competing methods.

I am open to further dicussion and willing to raise the score if the concerns are solved.

questions:
As shown in the experiments, InfiniteMesh generates 3D textured meshes based on unnatural images. In contrast, if natural images are used, will the results still maintain good geometry consistency? Additionally, how does the proposed method perform in more complex cases?
It would be beneficial to present some ablation studies on the IVI module, particularly examining the design of separated reference and condition image.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692763503

thread_type:
official_review

<thread_object>
creator: reviewer_d6ww

<content>
summary:
This paper introduces a pipeline for reconstructing or generating 3D textured object meshes from a single image. The pipeline is built on the recent framework of multi-view diffusion combined with large reconstruction models (LRMs). If I understand correctly, the central claim of this paper is that "interpolated view synthesis using image diffusion models can provide more information and enhance 3D consistency and lead to improved object reconstruction results in LRMs." The motivation, as outlined in the paper, is that (1) Sparse image-to-multi-view models are not enough for LRM models, and (2) previous dense image-to-multi-view diffusion models (like video models) lack pixel-accurate 3D consistency, which can negatively impact the reconstruction quality of LRM models.

The authors specifically propose an infinite-view generation model that is conditioned on continuous camera poses rather than discrete ones. The images generated from these interpolated camera poses are then used as inputs to the LRM model to create a 3D mesh.

Overall, I find the paper's main points and motivations challenging to follow. From my experience, using dense views as input images does not benefit LRM models. Given this, it's unclear to me how interpolating between these inconsistent views would improve 3D consistency. I struggle to see how this approach addresses the core issue.

strengths:
- This paper is easy to read with clean writing.
- The problem definition is clear.
- The paper compares the recent state-of-the-art image to 3D models, which is good.

weaknesses:
- As stated in the summary section, I do not think for the object reconstruction task, adding dense views for LRM models will help the reconstruction.
- The paper lacks in video results, or dense image results to show the quality of this “infinite” multi-view diffusion model. This part is crucial to help readers appreciate the whole method.
- The paper lacks quantitative ablation study on the view-interpolation design. There is only a qualitative comparison in Fig. 5, which is not convincing enough. Actually, I suspect the rationale of such multi-view interpolation design.

questions:
- In equation 3, where $n$ is defined?
- As the number of input images increases for the LRM model, how does the time required for training and inference compare to that of the sparse-view version of the LRM model?
- Also, as the number of output images increase for the multi-view generation model,  how does the time required for training and inference compare to that of the sparse-view version of multi-view generation model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723426267

thread_type:
official_review

<thread_object>
creator: reviewer_8fjs

<content>
summary:
The paper introduces InfiniteMesh, a framework for generating high-quality 3D meshes from a single image by combining multi-view diffusion-based view interpolation and mesh reconstruction techniques. The main task is to create consistent, high-fidelity 3D representations from limited initial views, addressing common issues like inconsistency and low detail in existing methods. Key contributions include the Infinite View Interpolation (IVI) module, which enhances multi-view consistency by generating interpolated views between initial main views, and a tri-plane-based reconstruction model that processes these views to produce accurate 3D meshes. Experimental results show that InfiniteMesh outperforms state-of-the-art approaches in both geometric and textural quality, especially on the Google Scanned Objects dataset.

strengths:
1. The IVI module is a novel contribution that improves multi-view consistency by generating interpolated views between 4 main views generated by Wonder3D. This approach minimizes inconsistencies across views, addressing a common challenge in 3D mesh generation. 

2. The combination of multi-view interpolation and a tri-plane-based reconstruction model results in superior mesh quality, with enhanced detail in both geometry and texture. The framework effectively captures intricate features and realistic textures, outperforming current state-of-the-art models both qualitatively and quantitatively.

weaknesses:
1. Inference Time missing: Infinite Mesh reports stronger quantitative performance and slightly better geometry and texture synthesis than several baselines. However, I am concerned about the extra computation time incurred by the IVI module for synthesizing sufficient novel views for multiview consistency. Instant Mesh constructs a mesh in 10 seconds, I doubt whether this method can get close to such speeds given the IVI overhead. For a fair comparison, the authors should report and compare average mesh creation times with the baselines considered in the paper.
2. The ablation with the camera trajectories for the IVI module does not seem too convincing, both qualitatively and quantitatively. The NVS results for the 2 hand-picked novel views are mostly similar, and the meshes have similar geometry details.
3. More details on mesh reconstruction: The authors should ideally provide more details and the motivation behind specific loss functions for mesh reconstruction in Equation 7, either in the main paper or the supplementary. This would save the reader (unfamiliar with all related literature, like the RODIN paper) from spending too much time reading other papers to understand the method correctly.
4. No 360 reconstruction videos: The authors should ideally share 360 reconstruction videos in the supplementary so that the multiview consistency introduced by the LRM and IVI modules can be better evaluated. Renderings of a few hand-picked novel views do not do much to convince the reader. The IVI module is introduced as a more compute-friendly alternative to video diffusion models to ensure multiview consistency. However, current qualitative results do not support that entirely.

**[Update post rebuttal]**: I would like to thank the authors for their efforts during the rebuttal phase and for responding to the reviewers' concerns earnestly.  

- Some of my doubts and questions were answered in the rebuttal, while others remain. For example, I am still unsure about the total number of main and interpolated views used for mesh reconstruction. An exact number can be easily cited.

 - Additional qualitative results in the supplementary show the benefits of the IVI module for multiview consistency, but still, most of the images (real-world/photographic) are inclined towards a synthetic setting. The author's reluctance to share reconstructions with more challenging real-world images as in competing methods weakens the experimental evaluation. For a more convincing comparison, the authors could easily have picked images used in works like Syncdreamer / Wonder3D rather than having a custom evaluation protocol.

- I would like to thank Reviewer Ystn for pointing out the discrepancy in previously reported performance of competing methods with the numbers reported in Table 2. I do not find the author's response satisfactory in changing the mesh texture evaluation protocol, whereas methods like Wonder3D report the performance of existing works as is. By following established experimental settings, this confusion could have been easily avoided. 

- I would encourage the authors to tweak some of their experiments for more convincing comparisons and resubmit to a future conference as I believe the proposed IVI module has merits. But for now, I would have to decrease my score.

questions:
1. How many interpolated views are used in total to optimize the mesh reconstruction? From what I understand, Wonder3D initially produces 4 views from a single image, and then the IVI module generates n = 2 interpolated views each on left and right for these 4 main views - so in total, 20 views are used for reconstruction (16 interpolated and 4 from Wonder3D) Is that correct?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents InfiniteMesh to generate 3D textured meshes from a single image. An IVI module is proposed to generate interpolated views so that it enhances the 3D consistency for the following tri-plane-based mesh reconstruction step. The 3D inconsistency in multi-view diffusion is challenging but important for image-to-3D tasks, and the quality of the generated meshes is good. However, there are some concerns raised by the reviewers and still remain after rebuttal. The experiments should be further revised, including a clearer explanation of the comparison with previous works as there are significant discrepancies in the reported PSNR, missing details in experimental settings, evaluations on challenging real-world images instead of the simple cases, etc.. Also, the novelty of the key module IVI should be further highlighted as it is somehow engineering and makes the whole pipeline more like a combination of existing methods. Due to the concerns, I recommend a decision of rejection of this paper.