GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement

Authors: Peiye Zhuang, Songfang Han, Chaoyang Wang, Aliaksandr Siarohin, Jiaxu Zou, Michael Vasilkovsky, Vladislav Shakhrai, Sergei Korolev, Sergey Tulyakov, Hsin-Ying Lee
Primary area: generative models
Keywords: 3D Reconstruction, Feed-forward

Abstract:
We propose a novel approach for 3D mesh reconstruction from multi-view images. We improve upon the large reconstruction model LRM that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images. We introduce three key components to significantly enhance the 3D reconstruction quality. First of all, we examine the original LRM architecture and find several shortcomings. Subsequently, we introduce respective modifications to the LRM architecture, which lead to improved multi-view image representation and more computationally efficient training. Second, in order to improve geometry reconstruction and enable supervision at full image resolution, we extract meshes from the NeRF in a differentiable manner and fine-tune the NeRF model through mesh rendering. These modifications allow us to achieve state-of-the-art performance on both 2D and 3D evaluation metrics on Google Scanned Objects (GSO) dataset and OmniObject3D dataset. Finally, we introduce a lightweight per-instance texture refinement procedure to better reconstruct complex textures, such as text and portraits on assets. To address this, we introduce a lightweight per-instance texture refinement procedure. This procedure fine-tunes the triplane representation and the NeRF's color estimation model on the mesh surface using the input multi-view images in just 4 seconds. This refinement achieves faithful reconstruction of complex textures. Additionally, our approach enables various downstream applications, including text/image-to-3D generation.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730477182074

thread_type:
official_review

<thread_object>
creator: reviewer_5yvj

<content>
summary:
This paper builds on the LRM framework with three main improvements: modifications to the LRM model structure, end-to-end geometry refinement with NeRF initialization, and per-instance texture refinement. These enhancements contribute to improved geometry reconstruction and texture quality.

strengths:
1. The proposed methods are well-grounded, and the ablation studies thoroughly support the authors' claims.
2. The visual results appear to achieve superior geometry and texture compared to previous methods.
3. The paper is well-written overall, making it easy to follow.

weaknesses:
1. The approach involves multiple stages, which may require more time compared to the baseline.
2. The paper lacks evaluation metrics for generation quality, such as FID or CLIP scores; including these metrics would strengthen the evaluation.

questions:
Aside from the mentioned weaknesses, I have the following questions:
1. Using DiffMC for geometry could be time-consuming. The paper highlights that the color refinement stage only takes 4 seconds, but it does not mention the time required for the geometry stage. I would like to see a breakdown of the time taken for each stage compared to the baseline.
2. In Section 3.2, where do the depth loss and normal loss supervision come from? If this is a single real-world image, how are depth and normal information obtained?
I would be glad to raise my rating if my concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730585515687

thread_type:
official_review

<thread_object>
creator: reviewer_4evv

<content>
summary:
The paper proposed an improved 3D mesh geometry and texture refinement model based on large reconstruction model (LRM) structures.

The authors analyze the limitation of the LRM model and update this with modified LRM architecture, geometry refinement with NeRF initialization, and texture refinement stages.

strengths:
The authors analyze and discuss well about why the previous LRM model has limitations and worse results. 

The qualitative results of the paper show improvement over the previous methods, especially for the text.

weaknesses:
There are some typos.

In L. 66, theoptimization → the optimization

In L. 195, Lhuillier & Quan (2005)., → Lhuillier & Quan (2005),

In Fig. 4, LRM takes one front-view image, but GTR (the authors’ method) takes 4 views, which is not a fair comparison. The authors need to use the same front-view image.

The paper lacks discussion and limitations of their method. The authors need to discuss the limitations of their method and the possibilities for further development.

questions:
In L. 91, the authors mention that “this enables us to render full-resolution images for supervision” for the advantage of the DiffMC pipeline. The reviewer wonders why it enables full-resolution images, and the authors need to add the reason; for instance, NeRF MLP needed heavy computation and was hard to use full-resolution, but mesh rasterization had effective computation.

In L. 97, the authors state, “fine-tune both the triplane representation and the color estimation model …”. The reviewer suggests replacing representation with feature, which is consistent with the word used later in the paper and less confusing with the triplane generator.

In L. 243, the reviewer wonders why the authors use Plucker coordinates for their camera ray embedding. Is there any insight or reason?

For the dataset, what is the source of an internal 3D asset dataset, and when retained, 26k superior assets of high quality are used?

In the qualitative results, IoU means 3D mIoU? The authors need to denote 3D if it is 3D metric. In Table 1 and Table 2, the left three columns for 2D metrics and the right two columns for 3D metrics are not specified. Authors need to add these details in the caption.

For the CD metric, the authors need to state their full name (maybe Chamfer Distance) first and use an abbreviation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622623135

thread_type:
official_review

<thread_object>
creator: reviewer_s5xh

<content>
summary:
This work proposes a novel approach for 3D mesh reconstruction from multi-view images. The authors improve upon the large reconstruction model LRM that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images. They introduce three key components to significantly enhance the 3D reconstruction quality. First of all, they examine the original LRM architecture and find several shortcomings. Subsequently, they introduce respective modifications to the LRM architecture, which lead to improved multi-view image representation and more computationally efficient training. Second, in order to improve geometry reconstruction and enable supervision at full image resolution, they extract meshes from the NeRF in a differentiable manner and fine-tune the NeRF model through mesh rendering. The method enables various downstream applications, including text/image-to-3D generation.

strengths:
1. The results are good in terms of both qualitative and quantative result.
2. The experiments are solid. Many baselines are compared.
3. The paper in written very well.

weaknesses:
1. This work has a large improvement in terms of quantative resutls. However, this work differs from previous works only from some incremental improvements in architecture. So what bring this such a large improvement? An ablation study of the proposed tricks will be very appreciated.
2. How long does it take to generate a single shape in total?
3. Will this work be open-sourced?
4. What is the common failure case of this method?
5. In table 1, some baselines has better results than yours (CD, IoU), however, you mark your results in bold font.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661845495

thread_type:
official_review

<thread_object>
creator: reviewer_dbu7

<content>
summary:
This paper proposed a feedforward method for extracting meshes from multi-view images 
based on large reconstruction models. The paper proposes several improvements in 
terms of network architecture, including replacing pretrained DINO encoder with
simple convolution image encoders to capture image details,  and replacing deconvolution 
layers with pixelshuffle layers. To extract meshes, the paper proposes to apply 
Differentiable Marching Cubes on the density grid. To improve geometry quality, the 
paper applies additional losses on depth and normal maps. The paper compares to 
previous methods such as LRM and InstantMesh and shows that the proposed method achieves 
better geometry reconstructions and renderings.

strengths:
1. The paper proposes several designs that improve the quality of LRM.  The paper performs 
ablation studies to validate the effectiveness of such designs.

weaknesses:
1. The overall pipeline of the paper is similar to that of InstantMesh, which also applies 
LRM for mesh reconstruction with differentiable iso-surface methods. Both papers apply
losses on depth and normal maps to improve the quality of the geometry.  While the proposed designs 
are helpful, they do not provide very significant technical contributions. Recent works 
such as MeshLRM, GS-LRM, and LGM have adopted similar strategies to improve the network design 
and should be discussed here.  

2. The paper does not provide quantitative evaluations on the effectiveness of the network 
design choices such as pixelshuffle layers and encoders. Ideally, the numbers should be provided 
for the ablation models in Table 1 for better clarity.

3. In the ablation models, is the DiNO encoder frozen or trainable during training? It's not 
fully clear why the ViT cannot capture image details considering the existence of residual 
connections.

Overall, I am not convinced that the paper presents enough technical novelty to be accepted. 
Therefore, I vote for a borderline rejection.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678124991

thread_type:
official_review

<thread_object>
creator: reviewer_gvuk

<content>
summary:
This paper presents GTR, a model for improved 3D mesh reconstruction from multi-view images, which refines both geometry and texture to achieve state-of-the-art quality. It first finds several shortcomings of previous large reconstruction models (LRM) and introduces respective modifications to the LRM architecture, achieving improved quality and more efficient training. The modifications include using a new convolution encoder to replace the DiNO ViT used by previous work, replacing the deconvolution upsampling with a linear layer
followed by a pixelshuffle, etc. Additionally, it presents an efficient per-instance texture refinement process, leveraging input images to
enhance texture details. GTR significantly outperforms baseline models on datasets like GSO and OmniObject3D, supporting downstream tasks such as text/image-to-3D generation.

strengths:
1. Rapid Texture Refinement: The per-object texture refinement is lightweight and achieves faithful texture reconstruction requiring a mere 4 seconds on an A100 GPU. 

2. Architecture Modifications: By replacing the DINO ViT transformer with a convolutional encoder and deconvolution layers with a linear layer followed by a pixelshuffle, GTR reduces artifacts of results and enhances high-frequency detail. These changes improve both the visual quality and efficiency of the model training. 

3. GTR achieves better results on major 3D reconstruction benchmarks including GSO and OmniObject3D datasets, showing clear quantitative and qualitative improvements over baseline models in 2D and 3D evaluation metrics.

weaknesses:
1. Missing baseline: The paper does not compare with some stronger baselines, such as Mesh-LRM, which has released an online demo from its first author before the ICLR submission deadline.

2. The mesh quality is not satisfactory. In the abstract, the authors said this approach was for "3D mesh reconstruction". However, according to the videos in the supplementary, the video quality is not satisfactory. The surfaces have a lot of bumpy and grid-like artifacts.

3. The overall novelty is limited. The texture refinement part is not very attractive and has been explored in many other related 3D generation works, such as One-2-3-45++, Mesh-LRM, etc. The authors put forward some modifications to the original LRM in this paper.  The original team for the LRM paper has also released several follow-ups for the original LRM, including Mesh-LRM and GS-LRM. Mesh-LRN has shown better results than this work and its modifications seem simpler Therefore, the effectiveness of the proposed modifications should be compared with Mesh LRM's modification.

questions:
1. Please compare with Mesh-LRM

2. Considering Mesh-LRM has shown better results, I am curious about the actual effectiveness of this method's modifications to the original LRM, compared with Mesh-LRM's.
For example, Mesh-LRM replaces DINO ViT with a simple patchify operation; it uses simple "Linear &  Unpatchify" to attach the triplane; it also uses separate MLPs but its MLPs have smaller sizes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present a large 3D reconstruction model for objects that takes one or multiple images and produces a textured mesh. The model is evaluated against several baseline and convincingly shows superior results.

The paper received mixed borderline reviews of 6/6/6/5/5 after the discussion phase. The reviewers appreciate the high quality results and good evaluations but noticed the limited technical contribution of this work.

After reading the paper as well, I come to similar conclusions. The technical contribution in this work is indeed limited, making a set of smaller adjustments to an existing model. However, the work also addresses a very relevant task that many people are interested in and clear improvements like shown here can have significant impact. The authors also committed to code and model release, which is necessary for this work to have relevant impact.

This paper is truly borderline, due to the above reasons. After some careful consideration I decided to recommend to accept it and would strongly suggest that the authors follow up on their commitment on model and code release.