GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement
Authors: Peiye Zhuang
Songfang Han
Chaoyang Wang
Aliaksandr Siarohin
Jiaxu Zou
Michael Vasilkovsky
Vladislav Shakhrai
Sergei Korolev
Sergey Tulyakov
Hsin-Ying Lee
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Reconstruction
Feed-forward
Primary area: generative models
TL;DR: We propose a novel feed-forward approach for 3D mesh reconstruction from multi-view images.

Abstract: We propose a novel approach for 3D mesh reconstruction from multi-view images. We improve upon the large reconstruction model LRM that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images. We introduce three key components to significantly enhance the 3D reconstruction quality. First of all, we examine the original LRM architecture and find several shortcomings. Subsequently, we introduce respective modifications to the LRM architecture, which lead to improved multi-view image representation and more computationally efficient training. Second, in order to improve geometry reconstruction and enable supervision at full image resolution, we extract meshes from the NeRF in a differentiable manner and fine-tune the NeRF model through mesh rendering. These modifications allow us to achieve state-of-the-art performance on both 2D and 3D evaluation metrics on Google Scanned Objects (GSO) dataset and OmniObject3D dataset. Finally, we introduce a lightweight per-instance texture refinement procedure to better reconstruct complex textures, such as text and portraits on assets. To address this, we introduce a lightweight per-instance texture refinement procedure. This procedure fine-tunes the triplane representation and the NeRF's color estimation model on the mesh surface using the input multi-view images in just 4 seconds. This refinement achieves faithful reconstruction of complex textures. Additionally, our approach enables various downstream applications, including text/image-to-3D generation.

Published as a conference paper at ICLR 2025
GTR: IMPROVING LARGE 3D RECONSTRUCTION MOD- ELS THROUGH GEOMETRY AND TEXTURE REFINEMENT
Peiye Zhuang
Songfang Han
Chaoyang Wang
Aliaksandr Siarohin
Jiaxu Zou
Michael Vasilkovsky
Vladislav Shakhrai
Sergei Koroley
Sergey Tulyakov
Ving Lee
https: snap wwesearrch.iihheee i
hard
but Snap
Sne
Input LRM (Hong et al. 2024
InstantMesh (Xu et al. 2024a)
Ours
Our normals
Figure 1: Comparison with the baseline methods on the sparse-view reconstruction task. We visualize the novel view results generated from the GSO dataset (Downs et al. (022) (row 1-3), or a generated image (row 4). Note that for the generated image, use Zero 123++ (Shi et al.) (2023) to generate 6 views, which are used as input for InstantMesh (Xu et al. 2024a) and GTR (Ours).
ABSTRACT
We propose novel approach for 3D mesh reconstruction from multi-view images. We improve upon the large reconstruction model LRM (Hong al.) 2024 that use a transformer-basee triplane generator and Neural Radiance Field (NeRF) model trained multi-view images. We introduce three key components to significantly enhance the 3D reconstruction quality. First of all, we examine the original LRM architecture and find several shortcomings. Subsequently, modifications to the LRM architecture, which lead to improved multi-view image representation and more computationally efficient training. Second, in order improve geometry reconstruction and enable supervision at full image resolution, we extract meshes from the NeRF in a differentiable manner and fine-tune the NeRF model through mesh rendering. These modifications allow to achieve state-of- the-art performance on both 2D and 3D evaluation metrics on Google Scanned Objects (GSO) dataset (Downs al. 2022) and OmniObject3D dataset (Wuj ct al. 2023). Finally, we introduce a lightweight per-instance texture refinement procedure better reconstruct complex textures, such text and portraits
introduce respective

---

Published as a conference paper at ICLR 2025
assets. To address this, we introduce lightweight per-instance texture refinement procedure. This procedure fine-tunes the triplane representation and the NeRF's color estimation model the mesh surface using the input multi-view images in just seconds. This refinement achieves faithful reconstruction of complex textures. Additionally, our approach enables various downstream applications, including text/imaee-o-3D generation.
INTRODUCTION
The task of generating 3D assets from text or images has wide applications in digital content generation and virtual reality (VR) (Lin al. 2023a b). Because of the scarcity of 3D data, efforts have been
made to leverage pre-trained large-scale text-to-image diffusion models. Some works learn 3D) asset generation from the pre-trained image diffusion models via score distillation techniques (Poole et al.
2023 Wang et al. 2023b at Chen et al. 2023; Lin et al. 2023a Zhu & Zhuang 2023 Qian et al.
2024.. which, however, require from several minutes several hours for single asset, since the
optimization algorithm requires many iterations. Other works propose to fine-tune tune image diffusion
models into multi-view image diffusion model (Liu al..
2023; Shi et al.) 2024 Liu al. 2024]
Long et al. 2023) using 3D asset datasets (Deitke et al.
2023). These multi-view images serve as an intermediate 3D representation, which is taken as input by large 3D reconstruction models (LRMs) (Hong al. 2024 Li et al., 2024 Xu al. 2024c| Wang et al. 2024: Tang et al. (2024) for asset generation. However, these previous approaches struggle to reconstruct faithful textures and high-quality geometry when using the Marching Cube (MC) algorithm (Lorensen & Cline. 1998).
Moreover, after extracting meshes using MC, the texture quality degrades even further.
In contrast, start developing the feed-forward mesh generation model by carefully examining the standard LRM architecture. First, we observed that DINO features tend to discard high-frequency image details, which are important for the precise reconstruction, from the input images. Thus replace the pre-trained DINO transformer (Caron et al.) (021) used previous LRMs (Hong et al.) 2024 Li et al.) 2024] Xu al. 2024c; Wang et al. 2024 Tang et al. 2024) with a convolutional encoder for the multi-view images. Moreover, because of the high computation requirements of the transformers, previous methods usually run transformer at 32 triplane resolution and use a deconvolution to upsample this triplane. However, we noticed that reconstruction from standard LRM often exhibits regular grid artifacts (see Fig. [ We speculate that the nature of these artifacts is similar to grid-shape artifacts observed in 2D deconvolutional generators (Odena et al. (2016). To address this, we replace all deconvolution layers with Pixelshuffle layers (Shi et al. | 2016)) Finally, employ two shallow Multi-layer Perceptrons (MLPs) to separately predict density and colors, which is beneficial for our following fine-tuning stages, which we will explain shortly.
Learning meshes is significantly harder than learning NeRF, Thus, we begin by training this modified architecture using NeRF volume rendering. With the trained NeRF model in place, we proceed fine-tune the pipeline using mesh rendering (also known rasterization). To achieve this, we employ Differentiable Marching Cubes (DiffMC) (Wei et al. (023) to extract meshes from the signed distance function (SDF) representation. This enables us to render full full-resolution images for supervision. Additionally, we employ depth loss to guide the geometry extraction. Our foed-forward mesh generation pipeline significantly boosts the quality of the reconstructions compared to the results extracted from NeRFs using MC.
NeRF density fields by transferring the densities
While the feed-forward mesh generation pipeline achieves advanced 3D reconstruction quality, it still faces challenges in accurately reconstructing intricate textures, such as text and complex patterns. To address this limitation and further enhance texture quality, we introduce a straightforward yet highly effective texture refinement procedure. Specifically, we fine-tune both the triplane feature and the color estimation model for each instance using sparse multi-view input images. As mentioned before, the shallow and separate density and color estimation models enable efficient updating of colors for surface points on the extracted meshes. In other words, the heavy triplane generator remains fixed in this texture refinement stage. This enables to achieve rapid optimization, reaching 5 iterations per second on an 100 GPU. Remarkably, our method achieves faithful texture reconstruction with just 20 steps of fine fine-tuning on v-eeew images, requiring a mere 4 seconds
A100 GPU.
Our proposed approach enables faithful 3D reconstruction from the multi-view input images, as shown in Fig- [] We conduct comprehensive comparison of our method with multiple concurrent
2

---

Published as a conference paper at ICLR 2025
Input Views
Rendered Views
Images
Depths
RGB
Mask Ray)
Normals
Conv. Encoder
(256 Res.)
Rasterization
Triplane
Transformer T
Decoder
(f fa)
(256' Res.)
Triplane Tokens
Triplane
Differentiable Mesh
Figure 2: Overview of our proposed approach for 3D reconstruction. Our pipeline consists of convolutional decoder &, cansformer-basee triplane generator, T, and NeRF-based triplane decoder that contains two MLPs, fe and fa. for color and density prediction, respectively. In practice, the triplane resolution is set to 256, and the mesh representation has grid size resolution of 256.
works (Hong et al. 2024 Xu et al. 2024a: Tang et al. 2024) using the Google Scanned Object (GSO) (Downs et 2022) and the OmniObject3D [Wu al. 2023) datasets, employing various evaluation metrics. For instance, in the 4-view reconstruction task using GSO dataset (Downs et al., 2022), our approach achieves Peak Signal-to-Noise Ratio (PSNR) of 29.79, a Structural Similarity Index (SSIM) of 0.94 and Learned Perceptual Image Patch Similarity (LPIPS) score of 0.059.
Extensive experiments show that our feed-forward model achieves superior results compared to the baseline approaches, while our per-instance refinement approach enables further texture improvement on text and complex patterns. See Fig. T for visual examples.
We summarize our technical contributions to two key components of multiview-to--31 reconstruction: (1) mesh generation and (2) accurate texture reconstruction. These contributions are outlined as follows:
We introduce a holistic design multi-view image to 3D reconstruction enhance the quality of the generated meshes. This includes modifications to the existing LRM model and fine-tuning the NeRF model with a differentiable mesh representation.
We present an efficient per-insse per-instance texture refinement process, leveraging input images to enhance texture details.
Furthermore, our model can be adapted to various downstream applications, such text/image-to- to-3 generation tasks.
RELATED WORK
Optinization-based 3D generation aims to use pre-trained large-scale text-to-image diffusion models Rombach et al. (2022): Saharia al. (2022) for generation, given the insufficient scale and diversity of existing 3D datasets. To distill 3D knowledge from the text-to-imase models, a Score Distillation Sampling (SDS) approach and its variants Poole et al. (023); Wang et al. (2023a b): Chen et al. (2023): Zhu & Zhuang (2033) have been proposed. In these methods, noise is added to an image rendered from 3D models like NeRFs and subsequently denoised by pre-trained text-to-image generative model Rombach al. (2022). The SDS approach aims to minimize the Kullback-Leibler (KL) divergence between a prior noise distribution and the estimated noise distribution from the text-to-image model. However, these SDS- methods time-consuming, usually taking up to hours to generate single instance. Alternatively, forward 3D generation models have been proposed to achieve faster generation.
3

---

Published as a conference paper at ICLR 2025
Render
Tex
fc
Input Conditional Images
Color MLP
Triplane Feature
Extracted Mesh
Figure 3: Texture refinement for extracted meshes. We refine the texture by fine-tuning the triplane feature of the asset and the color MLP, fe- using the input images. The Iearnable components are marked as 4. We use an L2 loss on the images, defined as Lrex
Feed-forward 3D generation has gained increasing attention recently due to its speed advantage. Most existing feed-forward 3D generation pipelines consist of two stages: (i) a text prompt (or single image) to multi-view generation and (ii) multi-view images to 3D shape reconstruction. In the first stage, multi-view images are generated from text or image input using multi-view generator Liu et al. (2023); Shi et al. (2024); Liu et al. (2024); Long et al. (2023), , which are usually fine- tunettned from image or video diffusion generation models Rombach et al. (2022); Blattmann et al. (2023). In
the second stage, 3D instance reconstructed from input multi view images. In this case, various 3D representations are used such as neural implicit fields Li ct al. (2024); Xu et al. (2024c): Hong al. 2024). Gaussian Splatting Xu al. (2024b); Tang al. (2024). or meshes Wang al. (2024);
Xu et al. (2024a). However, existing methods encounter cha lenges to either faithfully reconstruct textures when using implicit field representations Li et al.] (2024); Xu al. (2024c); Hong al.) (2024), or difficult to extract explicit geometries when using Gaussian Spatting Xu et al. (2024b);
Tang al. (2024). In this work, we focus on the second stage of the pipeline, i.e., reconstructing 3D shapes from multi-view images. To address those challenges, we propose a novel 3D reconstruction approach that improves the 3D quality through geometry and texture refinement.
Differentiable mesh is hybrid 3D representation that combines implicit and explicit surface repre- sentations, i.e., SDFs and meshes, and is suitable for 3D optimization. Recent popular representations include DMTet, Flexicubes, Differentiable Marching Cubes (DiffMC) [Shen et al. (0221 2023);
Wei et al. (003). In our work, we chose DiffMCWWer et al. (2023) as it doesn't 't require additional components beyond our existing model pipeline, contrast to using DMTet and Flexicubes Shen
et al. 2021 2023). where a deformation and weight prediction net are required.
MVS-based 3D reconstruction aims to generate novel views from sparse-vise -iew input images using Multi-view Stereo (MVS) techniques. Classic MVS methods leverage cost volumes Kutulakos & Seitz (0000]/Seitz & Dyer (1999), point clouds Stereopsis (2010); Lhuillier Quan (005)) or depth maps Campbell et al. (2008); Gallup et al. (2007) to learn blending weights for input sparse- view pixels for generating novel views. Recently. learning based methods Gu al.) (2020); Ma al.] (202T); Wang et al. 2021); Wei et al. (2021); Yi et al. (200)) have been proposed that can generalize to novel scenes. However, these methods require input views to have dense local overlap and struggle to generate 360-degree views of 3D assets. It is even more challenging when input multi-view images are generated without precise pixel-level alignment. Alternatively, in our work, we propose simple yet effective texture refinement procedure that enables high-quality texture reconstruction from vier input and robust to synthetic images.
3 METHOD
explain the modifi- cations made to the existing LRM architecture. In Sec. 3.2 we present a training procedure for our forwadd mesh generation model. The overview of our feed-forward mesh generation pipeline is illustrated in Fig. 2 Finally, in Sec. 3.3 we introduce our per- per-instance texture refinement procedure. This procedure is highlighted in Fig.
We separate the technical details of our method into three parts. In Sec. 3.1
4

---

Published as a conference paper at ICLR 2025
III
Sunn
ddp
VEN
Input
LRM (Hong et al. 2024
InstantMesh (Xu et al. 2024a)
Ours
Figure 4: Comparison with the baseline methods on OmniOhject3D dataset (Wu et al. 2023) We compare the novel view reconstruction results with the baseline methods. LRM (Hong
2024) takes the front-view image as input while Instantmesh (Xu et al. 2024a) and our method take views.
3.1 IMPROVING THE LARGE 3D RECONSTRUCTION ARCHITECTURE
A standard LRM (Hong et
architecture consists of an image encoder, transformer- triplane generator with a deconvolutional triplane upsampler, and a NeRF-based triplane decoder. In work, propose several modifications the LRM architecture outlined next.
2024
Convolutional image encoder E. LRM (Hong et al. 2024 Li et al. 2024 Xu al. 2024c) models typically utilize pre-trained transformer network DINO ViT (Caron et al. 2021 However, observe that since DiNO VIT was designed for semantic understanding. it tends to ignore some local details irrelevant image semantics but required for accurate reconstruction. Thus we propose to replace the DiNO (Caron et al.) 2021) architecture with a convolutional encoder. Since this encoder trained from scratch along with other components does not exhibit the bias pre-trained DiNO, An additional advantage of this encoder is that it does not require any modifications to consume additional inputs useful for 3D reconstruction. To this end, we complement the input images with binary foreground mask and Plücker coordinates for camera rays. We show comparison of the training process using different encoders in Appendix
Triplane upsampler T. One of the LRM architecture shortcomings is the tendency to generate grid-shaped artifacts. We attribute this issue to the deconvolution operation utilized a triplane upsampler. Indeed, reduce the computational requirements, the original LRM proposed to run transformer-basee triplane generator on 322 resolution and later utilize deconvolution operation to upsample the triplane. The deconvolution operation is widely studied in GAN literature, for example, Odena et al. (Odena et al.] (016) show that for 2D generators deconvolutions are the main source of grid-shaped artifacts. To this end, we replace the deconvolution upsampling with linear layer followed by pixelshuffle (Shi et al. 2016). This simple modification helps to alleviate grid-shaped texture artifacts.
NeRF decoders fe, fa. Unlike previous LRMs (Hong et al. 2024 ct al. 2024). we utilize two separate MIPs, defined as and fa. to estimate colors and density, respectively. This modification does not impact performance; however, it serves more practical purpose. For instance, we can train the color model fa and freeze fa when fine-tuning asset texture, or vice versa if fine-tuning asset geometry.
3.2 FEED FORWARD MESH GENERATION MODEL
The optimization through mesh representation may pose significant challenge. Indeed, the gradients for backpropagation through mesh exist only in small local neighborhood and, thus, convergence heavily depends accurate initialization. To tackle this, develop two-stage training procedure,

---

Published as a conference paper at ICLR 2025
Input
LGM (Tang al. 2024
Ours
GT
Figure 5: Comparison with LGM (Tang et al. 2024) on GSO DDwns et al. 2022) dataset. Both
LGM (Tang et al. 2024) and our method take 4-view images input and reconstruct novel views.
where in the first stage utilize the volumetric rendering and optimize NeRF, and, in the second stage, we perform geometry refinement by optimizing through a mesh representation.
NeRF training stage. In this stage, we simply optimize modified architecture (see Sec. 3.1 using a mean squared error (MSE) loss and an LPIPS loss on images, defined as
LEERF
Ligb + Xo LLPIPS
where Ap denotes the loss weight.
Geometry refinement with NeRF initialization. In the geometry refinement stage, we fine-tune the entire pipeline using mesh rendering. Specifically, we transfer the density field to an SDF field:
sdf ( (dd-)), s),
where d the estimated density and pre-defined level set for DifIMC, 10, practice.
Next, we render images, depths, and masks for training via mesh rendering (a.k.a rasterization). We fine-tune pipeline using an MSE loss and an LPIPS loss on images, MSE losses on depths, masks, and normal maps. Formally, define the loss as:
M
Ligh + Ap LLPIPS + Ad Ldepth + Am Lmask + + +hp Lnormal
(3)
where Ap. Au Am and An are the loss weights.
3.3 TEXTURE REFINEMENT FOR MESH REPRESENTATIONS
Inspired by previous works that utilize the Gaussian Splatting (Tang et al.) 2024 Xu et al. 2024b) representation, where colors of the input images can be easily retained Gaussian features, we notice disparity in our pipeline, which lacks this color memorization scheme. Thus, we refine the triplane feature of an asset and color MLP, fc- using the input multi-view images, Lcandy for surface points on the extracted mesh. We illustrate the refinement procedure in Fig. 3 The separated density and color MLPs benefit the texture refinement procedure, as we only fine-tune a single MLP. We use an MSE loss on input images for texture refinement:
LIex
Lz(Icund f
where Icond and Foond denote the ground-truth and predicted input images, respectively. Note that this stage, the image encoder & and the triplane generator are fixed and used to generate the initial triplane features of assets. The density estimation MI.P, fa, is also fixed and used to extract meshes.

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison on GSO dataset
Downs al.
2022
Method
PSNR (t)
SSIM (t)
LPIPS (1)
CD (1)
IoU (1)
LRM (Hong et al. 2024)
20.446
0.904
0.126
6,383
0.352
SV3D
Voleti et al.
2024
22.098
0.898
0.119
1.770
0.682
CRM Wang et al. 2024)
22.195
0.891
0.150
1.252
0.617
LGM
lang al.
2024)
25.227
0.925
0.066
1.373
0.601
InstantMesh (NcRF)
(Xu et al.
2024a
24.740
0.923
0.080
1.101
0.635
InstantMesh (Mesh)
(uu et al. 2024a)
24.444
0.920
0.08
1.115
0.645
Ours (Feed-forward)
28.673
0,946
0.055
0.740
0.708
Ours (Tex. refine)
29.788
0.960
0.047
EXPERIMENTS
In Sec. 4.1
we explain training datasets, implementation details, and evaluation with the baseline methods. We present both quantitative and qualitative results in Sec. 4.2 and the ablation study in Additional implementation details and comparisons are presented in Appendix.
Sec. 4.3
4.1
EXPERIMENT SETTINGS
Dataset: Our model is trained on a 140k asset dataset, which merges the filtered Objaverse dataset (Deitke et al., 2023) with internal 3D asset dataset. We filter the high-quality Obja- verse dataset to retain 26k superior assets of high quality. We render 32 random views for each training asset. In Appendix A we also provide additional ablation studies with model trained solely on the Objaverse dataset containing 100k assets.
Implementation Details. In practice, the loss weights I and An = 1. In practice, we found that adding the normal loss can lead to unstable training. Therefore, during the geometry refinement stage, add the normal loss once the model has stabilized, and we freeze the generator when incorporating the normal loss. Input multi-view images are of 512 resolution. The triplane transformer contains 24 attention blocks with a hidden dimension of 1024. Each attention layer has 16 attention heads and each head has a dimension of 64. During the NeRF training stage, images are rendered at 512 resolution, and the NeRF model trained using a patch size of 128 We uniformly sample 256 points along each camera ray. The density and color MLPs consist of 3 and 4 layers, respectively, with a hidden size of 512.
to Ap 0.5, Ao 0.5,
In the NeRF training stage, we use an AdamW optimizer with learning rate le - 4 and a weight decay of 0.05. Cosine scheduling is employed to gradually reduce the learning rate to after 150k training iterations. We a batch size of 512 on 32 A100 GPUs. For each asset, randomly choose 4 views as input and another 4 views for supervision. In the geometry refinement stage, we choose grid size of 256 during mesh extraction using DiffMC. We use another AdamW optimizer with learning rate 5e - 5. The batch size is 192 on 32 100 GPUs. For each asset, we randomly choose the per-instance texture refinement stage, the learning rates for the triplane feature and the color MLP, fc, are 0.15 and le - 4, respectively.
views as input and another 8 views for supervision.
Evaluation. We evaluate our method alongside baseline methods, including LRM (Hong SVD (Voleti et al., 2004), CRM (Wang et al. 2024), InstantMesh (Xu et al. 2024a and LGM (Tang al. 2024) using the Google Scanned Objects (GSO) (Downs al. 2022 and OmniObject3D Wu et al. 2023) dataset. We use the identical data lists of the GSO and OmniObject3D dataset and render camera orbits as outlined in Instantmesh (Xu et al., 2024a). Specifically, 300 GSO assets and 130 OmniObject3D assets (from 30 classes) are used for evaluation. We render 20 images for each asset in trigonometric orbiting trajectory, i.e., maintaining uniform azimuths and elevations in (-30%, 00, 30"), J. We use PSNR, SSIM, and LPIPS as image evaluation metrics, while Chamfer Distance (CD) and mloU are utilized for 3D geometry evaluation. To evaluate 3D geometry, we follow the mesh processing steps in InstantMesh (Xu al. (022a)). Specifically, reposition the generated meshes to the origin and align the coordinate system with the ground truuh meshes. We then rescale all meshes into [- 1] cube. We also use Iterative Closest Point (ICP) registration
2024
7

---

Published as a conference paper at ICLR 2025
Table Quantitative comparison on OmniObject3D dataset (Wu et
2023
Method
PSNR (1)
SSIM (1)
LPIPS (D)
CD (U)
IoU (1)
LRM (Hong et al.
2024)
18.082
0.888
0.125
4.347
0.448
LGM (GS) (Tang et al.
2024)
22.826
0.913
0.068
0.893
0.626
InstantMesh (NeRF) (Xu et al.
2024a
22.609
0.914
0.076
0.660
0.671
InstantMesh (Mesh) (Xu
2024a
22.141
0.910
0.079
0.603
0.675
Ours (Frearforward)
25.372
0.931
0.06
0.504
0.728
Ours (Tex. refine)
25.400
0.937
0.059
align the generated meshes to the ground truth meshes, We sample 16000 points on the asset surface to compute the CD and IoU scores.
4.2 RESULTS
Qualitative evaluation. We compare our method with baseline approaches using the GSO (Downs et al. (022) and OmniObject3D (Wu et al. 2023) datasets in Fig T Fig 5 method achieves more faithful texture reconstruction with finer details and more accurate geometry. For instance, in Fig. our model can generate clear text the first example and complex texture patterns on the second example. In the third example, our model enables the reconstruction of portraits printed on the asset. In Fig [] we compare our method with LGM (Tang et al. 20224, which uses Gaussian Splatting as a 3D representation. We observe that while LGM (Tang et al. 2024)
We observe that our
generate high-quality textures, it often tends to generate floating Gaussian points in inaccurate regions, even when the input images are ground-truth multi-view images. In Fig. 1T, we present a comparison between results and those of the concurrent MeshLRM approach Wei et al. (2024). The visual results indicate that our methods produces outcomes comparable to those achieved by MeshLRM. We present additional visual results in Appendix
2022 and Quantitative evaluation. We present the evaluation scores for the GSO (Downs et al.
OmniObject3D (Wu et al. 2023 dataset in Tab. D and Tab. 2] respectively. The CD scores are
presented by multiplying a rescale factor of 100. Note that both LGM Tang et al. (024) and InstantMesh (Xu al., 2024a) basclines are concurrent works. For InstantMesh (Xu et al.) 2024a)).
we compare both their results using either neural rendering or mesh rendering. Since LRM (Hong) al., (024)) takes single image as input, we provide a front-view image to Our full approach achieves the best evaluation results in 2D and 3D evaluations. We also present the evaluation scores using our feed-forward model, i.e., without the texture refinement procedure. These results still show significant improvements in texture and geometry quality.
Additionally, the LGM (Tang et al. 2024) method serves as a strong baseline as we directly compare against their generated images rather than rendered images from extracted meshes. We note that the baselines that do not incorporate some explicit geometry representation during training typically yield inferior rendering results when extracted meshes. However, results than their directly generated images. Furthermore, it takes minute to extract meshes using LGM (Tang al., 2024) from Gaussian Splatting. contrast, our method enables generate meshes within second, plus an additional 4 seconds for texture refinement, which in total is still faster than LGM TTag et al. 2024) inference.
method achieves better
Applications. Our method enables downstream tasks such as text/image-to-31 generation. In this case, we generate multi-view images using pre-trained text-to-image and/or maagett--multiviee diffusion models (Rombach et al.) 2022 Shi et al.) 2023). In practice, we use Zerol 12+++ (Shi et al. 2023) to generate 6-view images as the imput for our model. We show the generated results in Fig. and additional results in the Appendix C
4.3 ABLATION STUDY
Geometry refinement, In Fig. we compare results generated using NeRF+MC and NeRF with geometry refinement. We observe that directly extracting meshes from the NeRF field using MC leads to blurry texture results and significant drop 2D evaluation scores. In contrast, after fine-tuning at the geometry refinement stage, the rendered images show improved gigh-frequency details. The results in 9 present that the normal loss significantly improves the surface quality.
8

---

Published as a conference paper at ICLR 2025
Input
Synthetic novel views
Figure 6: Image-to-3D generation. Our method can adapt to the text/image-to-Da generation tasks. We visualize the input image (column 1) and generated RGB and normal images from novel views (column 2-7).
PSNR (1) SSIM (f)
LPIPS (4)
CD (L)
IoU (f)
19.810
0.903
0.116
0.775
0.652
3
NeRF + MC
NeRF+ Geo.
GT
Figure 7: Ablation study of geometry refinement. On the left, visualize the comparison between NeRF+MC, NeRF+Geometry refinement, and the ground-truth images. On the right, we present the evaluation scores using NeRF+MC.
Texture refinement. We visualize the novel view wesh-rendering results generated without and with per-instance texture refinement in Fig. T0 The results of the texture refinement appear to have superior detailed textures on mesh surfaces.
CONCLUSION
In this work, we introduce GTR, a large 3D reconstruction model that takes multi-view images as input. Our approach enables the generation of high-quality meshes with faithful texture reconstruction within seconds. We achieve this through three key contributions: modifications to the current LRM model architecture, the integration of end-to-end geometry refinement with NeRF initialization. and the implementation per-instance texture refinement procedure.

---

Published as a conference paper at ICLR 2025
NcRF+MC
Ours
GT
Figure 8: Additional visual comparison with Marching Cube (MC). When the rendering results using MC lack details (marked in red rectangles) and grid artifacts exist on surfaces. In contrast, our geometry refinement achieves better mesh rendering quality and geometry quality than the meshes extracted from NeRFs.
Input
w/o Loonnal
w/ Lnormal
Input
w/o Enormal
w/ Cnormal
Figure 9: Ablation on the normal loss. The visual results show that using the normal loss could produce higher-quality surfaces.
Igoap strn Dizli
NESCH laster Choia
Nikice Taster Choice
w/o Tex. refine w/ Tex. refine
GT
w/o Tex. refine w/ Tex. refine GT
Figure 10: Ablation study of the texture refinement procedure. We visualize the mesh rendering results without (column 1, 4) or with (column 2, 5) the texture refinement procedure, and correspond- ing ground-truth images (column 3, 6).
+
+
GT
MeshLRM
GT
Ours
GT
Ours
Figure 11: Visual comparison with MeshLRM Wei al. 2024 GSO dataset Downs et al. 2022). Our method generates better (at least comparable) concurrent work (Wei al. 2024).
textures with more details than the
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512) 2023.
Neill DF Camphell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla. Using multiple hypotheses to improve depth-maps for multi-view stereo. In ECCV, 2008.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
David Gallup, Jan-Michael Frahm, Philippos Mordohai, Qingxiong Yang, and Marc Pollefeys.
Real-time plane-sweeping stereo with multiple sweeping directions. In CVPR, 2007.
Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR, 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2024.
Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. IJCV, 2000.
Maxime Lhuillier and Long Quan. A quasi-dense approach to surface reconstruction from uncalibrated images. TPAMI, 2005.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In ICI.R, 2024.
Chen-Hsuaa Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yi Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023a.
Chich Hubert Lin, Hsin Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Hsuan Yang, and Sergey Tulyakov. Infinicity: Infinite-scale city synthesis. In ICCV, 2023b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot one image to 3d object. In ICCV, 2023.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consister images from single-view image. In ICLR, 2024.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In CVPR, 2023.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pp. 347-353. 1998.
Xinjun Ma. Yue Gong, Qirui Wang, Jingwei Huang, Lei Chen, and Fan Yu. Epp-mvsnet: Epipolar- assembling based depth prediction for multi-view stereo. In ICCV, 2021.
11

---

Published as a conference paper at ICLR 2025
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard arti- facts. Distill, 2016. doi: 0022391/diitiil.000033..
URL http:///dss:il1..pp//211..
deconv-checkerboary
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. diffusion. In ICLR, 2023.
Dreamfusion: Text-to-3d using 2d
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurlPS, 2022.
Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. IJCV, 1999.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. NeurlPS, 2021.
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-bass mesh optimization. TOG, 2023.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. +:: single image consistent multi-view diffusion base model. In arXiv preprint arXiv:2310.15/10, 2023.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. ICLR, 2024.
Robust Multiview Stereopsis. Accurate dense, and robust multiview stereopsis. TPAMI, 2010.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05004. 2024.
Vikram Voleti, Chun-Har Yao. Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. arXiv preprint arXiv:2403. /2008, 2024.
Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. Patch- matchnet: Learned multi-view patchmatch stereo. In CVPR, 2021.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurlPS, 2023b.
Zhengyi Wang. Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.0500: 2024.
12

---

Published as a conference paper at ICLR 2025
Xinyue Wei, Fanbo Xiang, Sai Bi, Anpei Chen, Kalyan Sunkavalli, Zexiang Xu, and Hao Su. Neumanifold: Neural watertight manifold reconstruction with efficient and high-quality rendering support. arXiv preprint 23xv:2305..7134. 2023.
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprini arXiv:2404. 12385, 2024.
Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, and Guoping Wang. Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network. In ICCV, 2021.
Tong Wu, Jiarui Zhang. Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulan 3d object dataset for realistic perception, reconstruction and generation. In CVPR, 2023.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07/99 2024a.
Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 2403::2003.1422111 2024b.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In ICLR, 2024c
Hongwei Yi, Zizhuang Wei, Mingyu Ding, Runze Zhang, Yisong Chen, Guoping Wang, and Yu- Yu-Wing Tai. Pyramid multi-view stereo net with self-adaptive view aggregation. ECCV, 2020.
Joseph Zhu and Peiye Zhuang, Hifa: High-fidelity text-to-3d with advanced diffusion guidance. In ICLR, 2023.
13

---

we present the Published as a conference paper at ICLR 2025
We present additional ablation study, limitation, and visual results in the Appendix manuscript. Please also refer to our website demo in the supplementary material for comprehensive overview.
A ABLATION STUDY
DINO encoder. We conduct experiments using different encoders. In Fig. 12
validation PSNR curves during training. when using either convolutional image encoder pre-trained DiNO ViT (Caron et al. 2021). Specifically, our convolutional encoder is a single layer that downsamples input images from 512 to 32. The triplane generator is self-attention transformer, identical in both settings. We train both models on 8 80G A100 GPUs. We observe that the DiNO experiment did show improved convergence during the initial iterations. Alternatively, more careful designs could optimize the use of DiNO VIT, which we leave for future study.
Objaverse training dataset. In Fig. [2] we also show the training process with a dataset consisting solely of 100k Objaverse (Deitke et al. (023) images. We did not observe a performance drop in the early stage compared the other experiments in the figure, which were trained on our mixed dataset.
Vae encoder. In Fig 13 we show preliminary results using pretrained VAE encoder" from an SD model (Rombach et al., (022). To enable the VAE encoder to handle multi-channel input, we separately provide images, masks, and the camera rays to the encoder, then assemble the output features using convolution layer. Experiments are run on 32 80G A100 GPUs. observe that using a pretrained VAE encoder leads to better convergence in the early training stage. We attribute this to the good initialization provided by VAEs compared to training the convolutional encoder from scratch.
205
1800
2040
0000
30000
30009
7500
Figure 12: Ablation study on image encoders Figure 13: Ablation study image encoders Conv. vs DiNO) and dataset.
(Conv. vs VAE).
B LIMITATION
Current triplane features at resolution of 256 are sometimes insufficient to capture fine geometric details. Meanwhile, alternative representations - explicit (e.g., sparse voxels) or implicit (e.g.. vector sets) - offer promising avenues for exploration. Additionally, our approach currently requires camera-conditioned input. would be more advantageous to develop methods capable of handling unposed input. Moreover, work focuses object-centric assets, leaving the exploration of complex scenes with composite objects as a valuable direction for future research.
C ADDITIONAL RESULTS
We show additional results generated by our approach in Fig. 14 16
In practice, we the pretrained SD VAF from http://tnngiinficccccoaahhhhhhhh
14

---

Published as a conference paper at ICLR 2025
Figure 14: Additional visual 3D reconstruction results on GSO (Dwnss et al.. 2022) dataset. The input of our model is orthogonal views. We show the novel view generated RGB images (column 1, 4) and normal images (column 5), and the ground-truth images (column 3, 6).
15

---

Published as a conference paper at ICLR 2025
Fublic
Solll
Additional visual 3D reconstruction results on GSO (Dwwns et al., 2022) dataset. The input of our model is orthogonal views. We show the novel view generated RGB images (colunn 4) and normal images (columm 2, 5), and the ground-truth -ruth images (column 3, 6).
Figure 15:
16

---

Published as a conference paper at ICLR 2025
Figure 16: Additional visual 3D asset generation results. The input images (column 1) either generated from text using pre-trained text-to ext-too-image diffusion models (Rombach et al. 2022) or online generated images.
17


Reviews:
[{"review_id": "hwWGkxTlPX", "reviewer": "Reviewer_GvuK", "summary": "This paper presents GTR, a model for improved 3D mesh reconstruction from multi-view images, which refines both geometry and texture to achieve state-of-the-art quality. It first finds several shortcomings of previous large reconstruction models (LRM) and introduces respective modifications to the LRM architecture, achieving improved quality and more efficient training. The modifications include using a new convolution encoder to replace the DiNO ViT used by previous work, replacing the deconvolution upsampling with a linear layer\nfollowed by a pixelshuffle, etc. Additionally, it presents an efficient per-instance texture refinement process, leveraging input images to\nenhance texture details. GTR significantly outperforms baseline models on datasets like GSO and OmniObject3D, supporting downstream tasks such as text/image-to-3D generation.", "questions": "1. Please compare with Mesh-LRM\n\n2. Considering Mesh-LRM has shown better results, I am curious about the actual effectiveness of this method's modifications to the original LRM, compared with Mesh-LRM's.\nFor example, Mesh-LRM replaces DINO ViT with a simple patchify operation; it uses simple \"Linear &  Unpatchify\" to attach the triplane; it also uses separate MLPs but its MLPs have smaller sizes.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Rapid Texture Refinement: The per-object texture refinement is lightweight and achieves faithful texture reconstruction requiring a mere 4 seconds on an A100 GPU. \n\n2. Architecture Modifications: By replacing the DINO ViT transformer with a convolutional encoder and deconvolution layers with a linear layer followed by a pixelshuffle, GTR reduces artifacts of results and enhances high-frequency detail. These changes improve both the visual quality and efficiency of the model training. \n\n3. GTR achieves better results on major 3D reconstruction benchmarks including GSO and OmniObject3D datasets, showing clear quantitative and qualitative improvements over baseline models in 2D and 3D evaluation metrics.", "weaknesses": "1. Missing baseline: The paper does not compare with some stronger baselines, such as Mesh-LRM, which has released an online demo from its first author before the ICLR submission deadline.\n\n2. The mesh quality is not satisfactory. In the abstract, the authors said this approach was for \"3D mesh reconstruction\". However, according to the videos in the supplementary, the video quality is not satisfactory. The surfaces have a lot of bumpy and grid-like artifacts.\n\n3. The overall novelty is limited. The texture refinement part is not very attractive and has been explored in many other related 3D generation works, such as One-2-3-45++, Mesh-LRM, etc. The authors put forward some modifications to the original LRM in this paper.  The original team for the LRM paper has also released several follow-ups for the original LRM, including Mesh-LRM and GS-LRM. Mesh-LRN has shown better results than this work and its modifications seem simpler Therefore, the effectiveness of the proposed modifications should be compared with Mesh LRM's modification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "76w6bWXps4", "reviewer": "Reviewer_dBU7", "summary": "This paper proposed a feedforward method for extracting meshes from multi-view images \nbased on large reconstruction models. The paper proposes several improvements in \nterms of network architecture, including replacing pretrained DINO encoder with\nsimple convolution image encoders to capture image details,  and replacing deconvolution \nlayers with pixelshuffle layers. To extract meshes, the paper proposes to apply \nDifferentiable Marching Cubes on the density grid. To improve geometry quality, the \npaper applies additional losses on depth and normal maps. The paper compares to \nprevious methods such as LRM and InstantMesh and shows that the proposed method achieves \nbetter geometry reconstructions and renderings.", "questions": "Please see the weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The paper proposes several designs that improve the quality of LRM.  The paper performs \nablation studies to validate the effectiveness of such designs.", "weaknesses": "1. The overall pipeline of the paper is similar to that of InstantMesh, which also applies \nLRM for mesh reconstruction with differentiable iso-surface methods. Both papers apply\nlosses on depth and normal maps to improve the quality of the geometry.  While the proposed designs \nare helpful, they do not provide very significant technical contributions. Recent works \nsuch as MeshLRM, GS-LRM, and LGM have adopted similar strategies to improve the network design \nand should be discussed here.  \n\n2. The paper does not provide quantitative evaluations on the effectiveness of the network \ndesign choices such as pixelshuffle layers and encoders. Ideally, the numbers should be provided \nfor the ablation models in Table 1 for better clarity.\n\n3. In the ablation models, is the DiNO encoder frozen or trainable during training? It's not \nfully clear why the ViT cannot capture image details considering the existence of residual \nconnections.\n\nOverall, I am not convinced that the paper presents enough technical novelty to be accepted. \nTherefore, I vote for a borderline rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZ3iygzBQ1", "reviewer": "Reviewer_S5XH", "summary": "This work proposes a novel approach for 3D mesh reconstruction from multi-view images. The authors improve upon the large reconstruction model LRM that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images. They introduce three key components to significantly enhance the 3D reconstruction quality. First of all, they examine the original LRM architecture and find several shortcomings. Subsequently, they introduce respective modifications to the LRM architecture, which lead to improved multi-view image representation and more computationally efficient training. Second, in order to improve geometry reconstruction and enable supervision at full image resolution, they extract meshes from the NeRF in a differentiable manner and fine-tune the NeRF model through mesh rendering. The method enables various downstream applications, including text/image-to-3D generation.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The results are good in terms of both qualitative and quantative result.\n2. The experiments are solid. Many baselines are compared.\n3. The paper in written very well.", "weaknesses": "1. This work has a large improvement in terms of quantative resutls. However, this work differs from previous works only from some incremental improvements in architecture. So what bring this such a large improvement? An ablation study of the proposed tricks will be very appreciated.\n2. How long does it take to generate a single shape in total?\n3. Will this work be open-sourced?\n4. What is the common failure case of this method?\n5. In table 1, some baselines has better results than yours (CD, IoU), however, you mark your results in bold font.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ThWyfPxKj", "reviewer": "Reviewer_4eVv", "summary": "The paper proposed an improved 3D mesh geometry and texture refinement model based on large reconstruction model (LRM) structures.\n\nThe authors analyze the limitation of the LRM model and update this with modified LRM architecture, geometry refinement with NeRF initialization, and texture refinement stages.", "questions": "In L. 91, the authors mention that “this enables us to render full-resolution images for supervision” for the advantage of the DiffMC pipeline. The reviewer wonders why it enables full-resolution images, and the authors need to add the reason; for instance, NeRF MLP needed heavy computation and was hard to use full-resolution, but mesh rasterization had effective computation.\n\nIn L. 97, the authors state, “fine-tune both the triplane representation and the color estimation model …”. The reviewer suggests replacing representation with feature, which is consistent with the word used later in the paper and less confusing with the triplane generator.\n\nIn L. 243, the reviewer wonders why the authors use Plucker coordinates for their camera ray embedding. Is there any insight or reason?\n\nFor the dataset, what is the source of an internal 3D asset dataset, and when retained, 26k superior assets of high quality are used?\n\nIn the qualitative results, IoU means 3D mIoU? The authors need to denote 3D if it is 3D metric. In Table 1 and Table 2, the left three columns for 2D metrics and the right two columns for 3D metrics are not specified. Authors need to add these details in the caption.\n\nFor the CD metric, the authors need to state their full name (maybe Chamfer Distance) first and use an abbreviation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors analyze and discuss well about why the previous LRM model has limitations and worse results. \n\nThe qualitative results of the paper show improvement over the previous methods, especially for the text.", "weaknesses": "There are some typos.\n\nIn L. 66, theoptimization → the optimization\n\nIn L. 195, Lhuillier & Quan (2005)., → Lhuillier & Quan (2005),\n\nIn Fig. 4, LRM takes one front-view image, but GTR (the authors’ method) takes 4 views, which is not a fair comparison. The authors need to use the same front-view image.\n\nThe paper lacks discussion and limitations of their method. The authors \bneed to discuss the limitations of their method and the possibilities for further development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qcX0YAp1p6", "reviewer": "Reviewer_5Yvj", "summary": "This paper builds on the LRM framework with three main improvements: modifications to the LRM model structure, end-to-end geometry refinement with NeRF initialization, and per-instance texture refinement. These enhancements contribute to improved geometry reconstruction and texture quality.", "questions": "Aside from the mentioned weaknesses, I have the following questions:\n1. Using DiffMC for geometry could be time-consuming. The paper highlights that the color refinement stage only takes 4 seconds, but it does not mention the time required for the geometry stage. I would like to see a breakdown of the time taken for each stage compared to the baseline.\n2. In Section 3.2, where do the depth loss and normal loss supervision come from? If this is a single real-world image, how are depth and normal information obtained?\nI would be glad to raise my rating if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed methods are well-grounded, and the ablation studies thoroughly support the authors' claims.\n2. The visual results appear to achieve superior geometry and texture compared to previous methods.\n3. The paper is well-written overall, making it easy to follow.", "weaknesses": "1. The approach involves multiple stages, which may require more time compared to the baseline.\n2. The paper lacks evaluation metrics for generation quality, such as FID or CLIP scores; including these metrics would strengthen the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)