MeshLRM: Large Reconstruction Model for High-Quality Meshes

## Abstract

Abstract We propose MeshLRM, a novel LRM-based approach that can reconstruct a high-quality mesh from merely four input images in less than one second. Different from previous large reconstruction models (LRMs) that focus on NeRF-based reconstruction, MeshLRM incorporates differentiable mesh extraction and rendering within the LRM framework. This allows for end-to-end mesh reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering. Moreover, we improve the LRM architecture by simplifying several complex designs in previous LRMs. MeshLRM’s NeRF initialization is sequentially trained with low- and high-resolution images; this new LRM training strategy enables significantly faster convergence and thereby leads to better quality with less compute. Our approach achieves state-of-the-art mesh reconstruction from sparse-view inputs and also allows for many downstream applications, including text-to-3D and single-image-to-3D generation.

###### Keywords:

## 1 Introduction

High-quality 3D mesh models are the core of 3D vision and graphics applications and 3D editing, rendering, and simulation tools are specifically optimized for them.
3D mesh assets are usually created manually by expert artists or alternatively reconstructed from multi-view 2D images.
Traditionally this has been done with complex photogrammetry systems [, , , ], though recent neural approaches, such as NeRF [], offer a simpler end-to-end pipeline through per-scene optimization.
However, these neural methods typically produce volumetric representations [, , , , , ]; converting them into meshes requires additional optimization in post [, , , ].
Furthermore, both traditional and neural approaches require a large number of input images and long processing time (up to hours), limiting their applicability in time-sensitive design workflows.

Our goal is efficient and accurate 3D asset creation via few-shot mesh reconstruction with direct feed-forward network inference and no per-scene optimization.
We base our approach on recent large reconstruction models (LRMs) [, , , ] for 3D reconstruction and generation.
Existing LRMs use triplane NeRFs [] as the 3D representation for high rendering quality.
While these NeRFs can be converted into meshes via a Marching Cube (MC) [] post-processing, this leads to a significant drop in rendering quality and geometric accuracy.

We address this with MeshLRM, a novel transformer-based large reconstruction model, designed to directly output high-fidelity 3D meshes from sparse-view inputs.
Specifically, MeshLRM incorporates differentiable surface extraction and rendering into a NeRF-based LRM. We apply a recent Differentiable Marching Cube (DiffMC) [] technique to extract an iso-surface from the triplane NeRF’s density field and render the extracted mesh using a differentiable rasterizer []. This enables end-to-end training of MeshLRM using mesh rendering losses, to optimize it to produce high-quality meshes that render realistic images.

We train MeshLRM by initializing the model using volumetric NeRF rendering; since our meshing components do not introduce any new parameters they can start from these pre-trained weights.
However, we find that training a mesh-based LRM with differentiable MC is still highly challenging.
The primary issue lies in the (spatially) sparse gradients from the DiffMC operation, that only affect surface voxels and leave the vast empty space untouched.
This leads to poor local minima for model optimization and manifests as floaters in the reconstructions.
We address this with a novel ray opacity loss that ensures that empty space along all pixel rays maintains near-zero density, effectively stabilizing the training and guiding the model to learn accurate floater-free surface geometry.
Our end-to-end training approach reconstructs high-quality meshes with rendering quality that surpasses NeRF volumetric rendering (see Tab. 5).

Our DiffMC-based meshing technique is general and can potentially be applied to any NeRF-based LRMs for mesh reconstruction.
In this work, we present a simple and efficient LRM architecture comprising of a large transformer model that directly processes concatenated multi-view image tokens and triplane tokens with purely self-attention layers to regress final triplane features for NeRF and mesh reconstruction.
In particular, we simplify many complex design choices used in previous LRMs [, , ], including the removal of pre-trained DINO modules in image tokenization and replacing the large triplane decoder MLP with small two-layer ones.
We train MeshLRM on the Objaverse dataset [] with a novel low-resolution pre-training and high-resolution fine-tuning strategy.
These design choices lead to a state-of-the-art LRM model with faster training and inference and higher reconstruction quality.

In summary, our key contributions are:

- •
A novel LRM-based framework that integrates differentiable mesh extraction and rendering for end-to-end few-shot mesh reconstruction.
- •
A novel ray opacity loss for stabilizing DiffMC-based generalizable training.
- •
An efficient LRM architecture and training strategies to enable fast and high-quality reconstruction.

We benchmark MeshLRM for 3D reconstruction (on synthetic and real data-sets [, ]) and 3D generation (in combination with other multi-view generation methods). Fig. 1 showcases high-quality mesh outputs from MeshLRM all reconstructed in less than one second.

## 2 Related Work

Mesh Reconstruction. Despite the existence of various 3D representations, meshes remain the most widely used format in industrial 3D engines. Reconstructing high-quality meshes from multi-view images is a long-standing challenge in computer vision and graphics. Classically, this is addressed via a complex multi-stage photogrammetry pipeline, integrating techniques such as structure from motion (SfM) [, , ], multi-view stereo (MVS) [, ], and mesh surface extraction [, ]. Recently, deep learning-based methods have also been proposed to address these problems for higher efficiency and accuracy [, , , , , ]. However, the photogrammety pipeline is not only costly, requiring dense input images and long processing time, but also often suffers from low-quality mesh rendering. Our approach enables sparse-view mesh reconstruction through direct feed-forward network inference in an end-to-end manner. We leverage neural volumetric representations and differentiable rendering, training our model to output high-quality meshes that can render realistic images.

Neural Reconstruction. Neural rendering techniques have recently gained significant attention for their ability to produce high-quality 3D reconstructions for realistic rendering [, , , ].
Most recent methods are based on NeRF [] and reconstruct scenes as various volumetric radiance fields [, , , , ] with per-scene rendering optimization. Though radiance fields can be converted into meshes with Marching Cubes [], the quality of the resulting mesh is not guaranteed. Previous methods have aimed to transform radiance fields into implicit surface representations (SDFs) [, , ], enhancing mesh quality from Marching Cubes. In addition, some methods [, , , ] attempt to directly extract meshes from radiance fields using differentiable surface rendering. However, these methods all require dense input images and long per-scene optimization time, which our approach avoids.

On the other hand, generalizable neural reconstruction has been explored, often leveraging MVS-based cost volumes [, , ] or directly aggregating multi-view features along pixels’ projective epipolar lines [, , ]. While enabling fast and few-shot reconstruction, these methods can only handle small baselines, still requiring dense images with local reconstruction in overlapping windows to model a complete object.
More recently, transformer-based large reconstruction models (LRMs) [, , , ] have been proposed and achieved 3D NeRF reconstruction from highly sparse views. In this work, we propose a novel efficient LRM, incorporating DiffMC techniques [], to enable high-quality sparse-view mesh reconstruction via direct feed-forward inference.

3D Generation.
Generative models have seen rapid progress with GANs [] and, more recently, Diffusion Models [] for image and video generation.
In the context of 3D generation, many approaches utilize 3D or multi-view data to train 3D generative models with GANs [, , , ] or diffusion models [, , , , ], which are promising but limited by the scale and diversity of 3D data.
DreamFusion [] proposed to leverage the gradient of a pre-trained 2D diffusion model to optimize a NeRF for text-to-3D generation with a score distillation sampling (SDS) loss, achieving diverse generation results beyond common 3D data distributions. This approach inspired many follow-up approaches, to make the optimization faster or improve the results quality [, , , , ]. However, these SDS-based methods rely on NeRF-like per-scene optimization, which is highly time-consuming, often taking hours.
More recently, a few attempts have been made to achieve fast 3D generation by utilizing pre-trained 2D diffusion models to generate multi-view images and then perform 3D reconstruction [, , ].
In particular, Instant3D [] leverages an LRM to reconstruct a triplane NeRF from four sparse generated images.
In this work, we focus on the task of sparse-view reconstruction and propose to improve and re-target a NeRF LRM to directly generate meshes with higher quality.
Our model can be naturally coupled with a multi-view generator, such as the ones in [, ], to enable high-quality text-to-3D and single-image-to-3D generation (see Fig. 5 and Fig. 6).

## 3 Method

In this section, we present our MeshLRM that reconstructs high-quality meshes within 1 second.
We start with describing our backbone transformer architecture (Sec. 3.1),
which simplifies and improves previous LRMs.
We then introduce our two-stage framework for training the model:
we first (Sec. 3.2) train the model to predict NeRF from sparse input images by supervising volume renderings at novel views,
followed by refining mesh surface extractions (Sec. 3.3)
by performing differentiable marching cubes on the predicted density field
and minimizing a surface rendering loss with differentiable rasterization.

Figure: Figure 2: The model architecture of MeshLRM. The images are first patchified to tokens. The transformer takes the concatenated image and triplane tokens as input. The output triplane tokens are upsampled with the unpatchifying operator while the output image tokens are dropped (not drawn in Fig.). With two tiny MLPs for density and color decoding, this model supports both the volumetric rendering (Sec. 3.2) and the DiffMC fine-tuning (Sec. 3.3). We render our final mesh output on the rightmost of the figure.

### 3.1 Model Architecture

As shown in Fig. 2, we propose a simple transformer-based architecture for MeshLRM, mainly consisting of a sequence of self-attention-based transformer blocks over concatenated image tokens and triplane tokens, similar to PF-LRM []. In contrast to PF-LRM and other LRMs [, , ],
we simplify the designs for both image tokenization and triplane NeRF decoding, leading to fast training and inference.
We next describe the model in detail.

Input posed image tokenization. MeshLRM adopts a simple tokenizer for posed images, inspired by ViT[].
We convert the camera parameters for each image into Plücker ray coordinates [] and concatenate them with the RGB pixels (3-channel) to form a 9-channel feature map.
Then we split the feature map into non-overlapping patches, and linearly transform them as transformer’s input.
With this process, the model does not need additional positional embedding as in ViT, since Plücker coordinates contain spatial information.

Note that our image tokenizer is much simpler than previous LRMs that use a pre-trained DINO ViT [] for image encoding.
We empirically find that the pre-trained DINO ViT is unnecessary, possibly because DINO is mainly trained for intra-view semantic reasoning, while 3D reconstruction mainly requires inter-view low-level correspondences.
By dropping this per-view DINO encoding, the model also enables a better flow between raw pixels and 3D-related information.

Transformer. We concatenate multi-view image tokens and learnable triplane (positional) embeddings, and feed them into a sequence of transformer blocks [], where each block is comprised of self-attention and MLP layers.
We add layer normalization [] before both layers (i.e., Pre-LN architecture), and use residual connections.
This deep transformer network enables comprehensive information exchange among all the tokens, and effectively models intra-view, inter-view, and cross-modal relationships.
The output triplane tokens, contextualized by all input views, are then decoded into the renderable triplane NeRF.
The output image tokens are dropped.
Lastly, each triplane token is unprojected with a linear layer and further unpatchified to $8\times 8$ triplane features via reshaping. (^1^11This operator is logically the same to a shared 2D-deconvolution with stride $8$ and kernel $8$ on each plane.)
All the predicted triplane features are then assembled into the final triplane NeRF.

Tiny density and color MLPs. Previous LRMs [, , , ] all use a heavy shared MLP (e.g., 9 hidden layers with a hidden dimension of 64) to decode densities and colors from triplane features.
This design leads to slow rendering in training.
This is a bottleneck for MeshLRM since we need to compute a dense density grid, using the MLP and triplane features, for extracting the mesh with DiffMC.
Therefore, we opt to use tiny MLPs with a narrower hidden dimension of $32$ and fewer layers.
In particular, we use an MLP with one hidden layer for density decoding and another MLP with two hidden layers for color decoding.
Compared to a large MLP, our tiny MLPs can lead to $~{}50\%$ speed-up in training without compromising quality (see Tab. 4).
We use separate MLPs since the density MLP and color MLP are used separately in the Marching Cubes and surface rendering processes, respectively.
We empirically find this MLP separation is necessary for DiffMC fine-tuning (described later in Sec. 3.3) as it largely improves the optimization stability which is critical to large-scale training.

While being largely simplified, our transformer model can effectively transform input posed images into a triplane NeRF for density and color decoding.
The density and color are then used to achieve both radiance field rendering for 1st-stage volume initialization (Sec. 3.2) and surface extraction+rendering for 2nd-stage mesh reconstruction (Sec. 3.3).

### 3.2 Stage 1: Efficient Training for Volume Rendering

We train our model with ray marching-based radiance field rendering (as in NeRF []) to bootstrap our model, providing good initialized weights for mesh reconstruction.
Instead of training directly using high-res ($512\!\times\!512$) input images like prior LRM works [, ], we develop an efficient training scheme, inspired by ViT [].
Specifically, we first pretrain our model with $256\!\times\!256$ images until convergence and then finetune it for fewer iterations with $512\!\times\!512$ images.
This training scheme cuts the computing cost significantly, hence obtaining much better quality than training at 512-res from scratch given the same amount of compute (see Tab. 1).

256-res pretraining.
The pre-training uses 256-resolution images for both model input and output.
We use a batch size of 8 objects per GPU and sample 128 points per ray during ray marching.
In contrast to training with 512-res images from scratch (like previous LRMs), our training efficiency benefits from the low-res pre-training from two factors: shorter sequence length for computing self-attention and fewer samples per ray for volume rendering (compared to our high-res fine-tuning).

512-res finetuning.
For high-res fine-tuning, we use 512-resolution images for the model input and output.
We use a batch size of 2 per GPU and densely sample 512 points per ray, which is $4\times$ the number in low-res pretraining.
Here, we compensate for the increased computational costs (from longer token sequences and denser ray samples) by reducing the batch size 4 times, achieving a similar training speed (per iteration) to the low-res pretraining.
It’s also worth noting that our resolution-varying pretraining and finetuning scheme benefits from the use of Plücker coordinates for camera conditioning.
It avoids the positional embedding interpolation [, ] and has inherent spatial smoothness.

Loss. We use an L2 regression loss $L_{v,r}$ and a perceptual loss $L_{v,p}$ (proposed in []) to supervise the renderings from both phases. Since rendering full-res images is not affordable for volume rendering, we randomly sample a $128\!\times\!128$ patch from each target 256- or 512-res image for supervision with both losses. We also randomly sample 4096 pixels per target image for additional L2 supervision, allowing the model to capture global information beyond a single patch. The loss for volume rendering training is expressed by:

$$ $L_{v}=L_{v,r}+w_{v,p}*L_{v,p}$ (1) $$

where we use $w_{v,p}=0.5$ for both 256- and 512-res training.

### 3.3 Stage 2: Stable Training for Surface Rendering

Once trained with volume rendering, our model already achieves high-fidelity sparse-view NeRF reconstruction, which can be rendered with ray marching to create realistic images.
However, directly extracting a mesh from the NeRF’s density field results in significant artifacts as reflected by Tab. 2.
Therefore, we propose to fine-tune the network using differentiable marching cubes
and differentiable rasterization, enabling high-quality feed-forward mesh reconstruction.

Mesh extraction and rendering. We compute a $256^{3}$ density grid by decoding the triplane features and adopt a recent differentiable marching cubes (DiffMC) technique [] to extract mesh surface from the grid.
The DiffMC module is based on a highly optimized CUDA implementation, much faster than existing alternatives [, ], enabling fast training and inference for mesh reconstruction.

To compute the rendering loss we need to render our generated mesh. We do so using a differentiable rasterizer Nvdiffrast [] and utilize the triplane features to (neurally) render novel images from our extracted meshes. This full rendering process is akin to deferred shading where we first obtain per-pixel XYZ locations via differentiable rasterization before querying the corresponding triplane features and regressing per-pixel colors using the color MLP.
We supervise novel-view renderings with ground-truth images, optimizing the model for high-quality end-to-end mesh reconstruction.

However, using the rendering loss alone leads to high instability during training and severe floaters in the meshes (see Fig. 3). This is mainly caused by the sparsity of density gradients during mesh rendering: unlike volume rendering which samples points throughout the entire view frustum, mesh (rasterization) rendering is restricted to surface points, lacking gradients in the empty scene space beyond the surface.

Ray opacity loss. To stabilize the training and prevent the formation of floaters, we propose to use a ray opacity loss.
This loss is applied to each rendered pixel ray, expressed by:

$$ $L_{\alpha}=||\alpha_{\mathbf{q}}||_{1},\quad\alpha_{\mathbf{q}}=1-\exp(-\sigma _{\mathbf{q}}||\mathbf{p}-\mathbf{q}||)$ (2) $$

where $\mathbf{p}$ represents the ground truth surface point along the pixel ray, $\mathbf{q}$ is randomly sampled along the ray between $\mathbf{p}$ and camera origin, and $\sigma_{\mathbf{q}}$ is the volume density at $\mathbf{q}$; when no surface exists for a pixel, we sample $\mathbf{q}$ inside the object bounding box along the ray and use the far ray-box intersection as $\mathbf{p}$.

In essence, this loss is designed to enforce the empty space in each view frustum to contain near-zero density. Here we minimize the opacity value $\alpha_{\mathbf{q}}$, computed using the ray distance from the sampled point to the surface. This density-to-opacity conversion functions as an effective mechanism for weighting the density supervision along the ray with lower loss values for points sampled closer to the surface. An alternative approach would be to directly regularize the density $\sigma_{\mathbf{q}}$ using an L1 loss, but we found this to lead to holes in surfaces as all points contribute the same, regardless of their distance to the surface.

Combined losses.
To measure the visual difference between our renderings and ground-truth (GT) images, we use an L2 loss $L_{m,r}$ and a perceptual loss $L_{m,p}$ proposed in [] similar to stage 1.
To compute the ray opacity loss $L_{\alpha}$, we obtain surface points using the GT depth maps.
In addition, to further improve our geometry accuracy and smoothness, we apply an L2 normal loss $L_{n}$ to supervise the face normals of our extracted mesh with GT normal maps in foreground regions.
Our final loss for mesh reconstruction is

$$ $L_{m}=L_{m,r}+w_{m,p}*L_{m,p}+w_{\alpha}*L_{\alpha}+w_{n}*L_{n}$ (3) $$

where we use $w_{m,p}=2$, $w_{\alpha}=0.5$ and $w_{n}=1$ in our experiments.
Since mesh rasterization is significantly cheaper than volume ray marching,
we render the images at full resolution (e.g., $512\!\times\!512$ in our experiment) for supervision,
instead of the random patches+rays used in Stage 1 volume training.

## 4 Experiments

### 4.1 Dataset and Evaluation Protocols

Our model is trained on the Objaverse dataset [] (730K objects) for the 1st-stage volume rendering training and is subsequently fine-tuned on the Objaverse-LVIS subset (46K objects) for the 2nd-stage surface rendering finetuning.
Empirically, the Objaverse-LVIS subset has higher quality and previous work [, , ] shows that fine-tuning favors quality more than quantity.
We thus follow this common practice.
We evaluate the reconstruction quality of MeshLRM alongside other existing methods on the GSO [], NeRF-Synthetic [], and OpenIllumination [] datasets, employing PSNR, SSIM, and LPIPS as metrics for rendering quality and bi-directional Chamfer distance (CD) as the metric for mesh geometry quality.
For the CD metric, since we cannot expect the model to precisely reconstruct unseen parts, we cast rays from all test views and sample 100K points at the ray-surface intersections for each object.
The Chamfer distance is computed using the points sampled from both the ground-truth and reconstructed mesh within a $[-1,1]^{3}$ bounding box, measured in $10^{-3}$ unit. All results are generated using four sparse-view input images.

### 4.2 Analysis and Ablation Study

Volume rendering (Stage 1) training strategies.
To verify the effectiveness of our training strategy that uses 256-res pretraining and 512-res fine-tuning (details in Sec. 3.2), we compare with a model having the same architecture but trained with high-resolution only (i.e., 512-res from scratch).
Tab. 1 shows the quantitative results on the GSO dataset with detailed training settings and timings used for the two training strategies.
As seen in the table, with the same total compute budget of 64 hours and 128 GPUs, our low-to-high-res training strategy achieves significantly better performance, with a 2.6dB increase in PSNR.
The key to enabling this is our fast low-res pretraining, which takes only 2.6 seconds per iteration, allowing for many more iterations to be trained within a shorter period and enabling much faster convergence.
Benefiting from effective pretraining, our high-res finetuning can be trained with a smaller batch size that reduces the iteration time.
In general, our 1st-stage training strategy significantly accelerates the LRM training, leading to high-quality NeRF reconstruction. This naturally improves the mesh reconstruction quality in the second stage.

**Table 1: Comparison between the model trained from scratch using 512 resolution images and the model trained by our pretraining + finetuning strategy. Note the two models used the same compute budget — 128 A100 (40G VRAM) GPUs for 64 hours.**
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | #Pts/Ray | BatchSize | $T_{\mathrm{iter}}$ | #Iter | $T_{\mathrm{total}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 512-res from scratch | 25.53 | 0.892 | 0.123 | 512 | 1024 | 7.2s | 32k | 64hrs |
| 256-res pretrain | 28.13 | 0.923 | 0.093 | 128 | 1024 | 2.6s | 60k | 44hrs |
| 512-res fine-tune | 512 | 256 | 3.6s | 20k | 20hrs |  |  |  |

Effectiveness of surface fine-tuning (Stage 2).
We justify the effectiveness and significance of our 2nd-stage surface fine-tuning by comparing our final meshes with
those directly extracted from 1st-stage model using marching cubes on GSO dataset.
The quantitative results are shown in Tab. 2, and the qualitative results
are presented in the Appendix.
Note that, while our 1st-stage NeRF reconstruction, i.e. MeshLRM (NeRF), can achieve high volume rendering quality, directly extracting meshes from it with Marching Cubes (MC), i.e. ‘MeshLRM (NeRF)+MC’, leads to a significant drop of rendering quality on all metrics.
On the other hand, our final MeshLRM model, fine-tuned with DiffMC-based mesh rendering, achieves significantly better mesh rendering quality and geometry quality than the MeshLRM (NeRF)+MC baseline, notably increasing PSNR by 2.5dB and lowering CD by 0.58.
This mesh rendering quality is even comparable — with a slight decrease in PSNR but improvements in SSIM and LPIPS — to our 1st-stage volume rendering results of MeshLRM (NeRF).

Table: Table 2: The effectiveness of our surface rendering fine-tuning stage. ‘MeshLRM (NeRF)’ is our first stage model. ‘MC’ is Marching Cube. Volume rendering for first row; surface rendering for last two rows.

|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CD$\downarrow$ |
| --- | --- | --- | --- | --- |
| MeshLRM (NeRF) | 28.13 | 0.923 | 0.093 | - |
| MeshLRM (NeRF)+MC | 25.48 | 0.903 | 0.097 | 3.26 |
| MeshLRM | 27.93 | 0.925 | 0.081 | 2.68 |

Table: Table 3: Ablation study for losses used in surface rendering fine-tuning on GSO dataset.

|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CD$\downarrow$ |
| --- | --- | --- | --- | --- |
| w/o Ray Opacity Loss | 18.46 | 0.836 | 0.218 | 51.98 |
| w/o Normal Loss | 27.93 | 0.925 | 0.080 | 2.87 |
| Ours (full model) | 27.93 | 0.925 | 0.081 | 2.68 |

Surface fine-tuning losses. We demonstrate the significance of the different losses applied during the mesh fine-tuning stage (details in Sec. 3.3) with an ablation study in Tab. 3 and Fig. 3.
We observe a significant performance drop in the model without our proposed ray opacity loss, leading to severe floater artifacts as shown in Fig. 3.
This is because, without supervision on empty space, the model may produce floaters to overfit to the training views rather than learning the correct geometry, as it does not receive a penalty for inaccuracies in empty space. The normal loss particularly helps to generate better geometry, i.e. a lower Chamfer distance.
While the normal loss does not lead to quantitative improvements in rendering (metrics are about the same) on the GSO dataset,
we still observe qualitative improvements with this loss, especially when using generated images, as shown in Fig. 3 (right).
Empirically, the normal loss leads to better robustness in handling inconsistent input views, a common challenge in text-to-3D or image-to-3D scenarios (e.g. the cactus).
Moreover, the geometric improvements from the normal loss are also crucial for real applications to use our meshes in 3D engines (like the one in Fig. 1), where accurate shapes are critical for high-fidelity physically-based rendering.

Table: Table 4: Comparison between using big MLP and tiny MLP as triplane decoder. Both are trained for 60k iterations. $T_{\mathrm{iter}}$ refers to the training time per iter. Results are volume rendering with Stage-1 256-res pretrained models.

|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | $T_{\mathrm{iter}}$ |
| --- | --- | --- | --- | --- |
| Triplane+Big MLP | 27.44 | 0.915 | 0.081 | 3.6s |
| Triplane+Tiny MLP | 27.47 | 0.915 | 0.080 | 2.7s |

Figure: Figure 3: Ablation study on losses used in Stage-2 surface rendering fine-tuning.

Tiny MLPs.
To justify our choices of using tiny MLPs for triplane coding, instead of a large one in previous LRMs, we compare with a version that replaces the two tiny MLPs with a 10-layer (i.e., 9 hidden layers) 64-width shared MLP decoder (as used in [, ]).
The quantitative results on the GSO dataset are in Tab. 4.
We find that the tiny MLP can achieve similar performance to the large MLP while offering a notable training speed advantage (2.7s / step vs 3.6s / step).
Moreover, we observe that the heavy shared MLP hardly converge during the Stage 2 DiffMC training while the tiny separate MLPs could.

### 4.3 Comparison Against Baselines

**Table 5: Comparison with feed-forward approaches on GSO dataset. $T_{\mathrm{infer}}$ is the wall-clock time from images to 3D representation (triplane for first two rows and mesh for last two rows), and $T_{\mathrm{train}}$ is the training budget; all reported using A100 GPUs. FPS is for rendering 512$\times$512 resolution images from the corresponding representations.**
|  | Render | FPS | $T_{\mathrm{infer}}$ | Params | $T_{\mathrm{train}}$ | Quality (GSO) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | (GPU$\times$day) | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CD$\downarrow$ |  |  |  |  |
| In3D-LRM [] | 128 pts/ray | 0.5 | 0.07s | 500M | 128$\times$7 | 26.54 | 0.893 | 0.064 | - |
| MeshLRM (NeRF) | 512 pts/ray | 2 | 0.06s | 300M | 128$\times$2.7 | 28.13 | 0.923 | 0.093 | - |
| In3D-LRM []+MC | Mesh | >60 | 1s | 500M | 128$\times$7 | 23.70 | 0.875 | 0.111 | 3.40 |
| MeshLRM | >60 | 0.8s | 300M | 128$\times$3 | 27.93 | 0.925 | 0.081 | 2.68 |  |

**Table 6: Comparison with optimization approaches on Open-Illumination and NeRF-Synthetic datasets. Inference time is wall-clock time with a single A100 GPU.**
|  | Inference | OpenIllumination | NeRF-Synthetic |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Time | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CD$\downarrow$ |
| FreeNeRF [] | 3hrs | 12.21 | 0.797 | 0.358 | 18.81 | 0.808 | 0.188 | - |
| ZeroRF [] | 25min | 24.42 | 0.930 | 0.098 | 21.94 | 0.856 | 0.139 | 6.01 |
| MeshLRM | 0.8s | 26.10 | 0.940 | 0.070 | 21.85 | 0.850 | 0.137 | 4.94 |

Comparisons with feed-forward methods.
To the best of our knowledge, the LRM model proposed in Instant3D [] is the only previous model that can also achieve 3D object reconstruction from four sparse views.
As reported by Instant3D, previous cost-volume-based feed-forward methods (like SparseNeus[]) cannot handle this challenging case. Therefore, we focus on comparing with the LRM model in Instant3D (labeled as In3D-LRM) on the GSO dataset, showing quantitative results with detailed training settings and timings in Tab. 5.
In the table, we show our results from both stages and compare them with both In3D-LRM and In3D-LRM + MC; for In3D-LRM + MC, Marching Cubes is directly applied to extract meshes from their NeRF reconstruction.

As shown in Tab. 5, our model, across both stages, achieves superior quality compared to Instant3D.
As expected, In3D-LRM + MC leads to much worse rendering quality than In3D-LRM, due to the quality drop caused by MC.
Thanks to our effective 2nd-Stage DiffMC-based training, MeshLRM can reconstruct significantly better meshes in terms of both rendering quality (with 4.2dB PSNR and 0.05 SSIM improvements) and geometry quality (lowering the CD by 0.72).
Our mesh rendering is even better than their volume rendering (In3D-LRM), notably increasing their PSNR by 1.4dB and SSIM by 0.03, while achieving substantially faster rendering speed because of outputting meshes.
Our superior quality is also reflected by the qualitative comparisons in Fig. 4, where our mesh renderings reproduce more texture and geometric details than the baselines.
From Tab. 5, we can also see that our high reconstruction quality is even achieved with a smaller model size and substantially less compute, less than half of the total GPUxDay compute cost required for Instant3D-LRM.
This is enabled by our simplified LRM architecture designs and efficient low-res-to-high-res 1st-stage training strategy, leading to significantly faster training speed and convergence.
Besides, our model also leads to a faster inference speed, enabling high-quality mesh reconstruction within 1 second.
Overall, our MeshLRM leads to state-of-the-art sparse-view mesh reconstruction quality and achieves high efficiency in parameter utilization, and training, inference, and rendering speed.
Meanwhile, our MeshLRM (NeRF) has good performance as well and is a state-of-the-art LRM model for sparse-view NeRF reconstruction.

Figure: Figure 4: Qualitative comparison between MeshLRM and other feed-forward methods. ‘In3D-LRM’ is the Triplane-LRM in Instant3D []; ‘MC’ is Marching Cube. ‘In3D-LRM’ uses volume rendering and others use surface rendering.

Comparisons with per-scene optimization methods.
We also compare our MeshLRM with recent per-scene optimization methods that are designed for sparse-view NeRF reconstruction, including ZeroRF [] and FreeNeRF [].
Because of using per-scene optimization, these methods are much slower, requiring tens of minutes and even several hours to reconstruct one single scene. As a result, it is not practical to evaluate them on the GSO dataset, comprising more than 1000 objects. Therefore, we conduct this experiment on the NeRF-Synthetic and OpenIllumination datasets, following the settings used in ZeroRF [] with four input views. Tab. 6 shows the quantitative novel view synthesis results, comparing our mesh rendering quality with their volume rendering quality. Since ZeroRF also offers a post-processing step to improve their mesh reconstruction from Marching Cubes, we also compare our mesh geometry with theirs using the CD metric on the NeRF-Synthetic dataset (where GT meshes are accessible).

As shown in Tab. 6, our NeRF-Synthetic results outperform FreeNeRF significantly; we also achieve comparable rendering quality and superior geometry quality to ZeroRF.
On the challenging OpenIllumination real dataset, our approach outperforms both FreeNeRF and ZeroRF by a large margin, consistently for all three metrics.
The OpenIllumination dataset, consisting of real captured images, demonstrates our model’s capability to generalize to real captures for practical 3D reconstruction applications, despite being trained only on rendered images.
In addition to our superior (or comparable) quality, our feed-forward mesh reconstruction approach is significantly faster than these optimization-based NeRF methods, in terms of both reconstruction (1000$\times$ to 10000$\times$ speed up) and rendering.

## 5 Applications

Our approach achieves efficient and high-quality mesh reconstruction in less than one second, thus naturally enabling many 3D generation applications, when combined with multi-view image generation techniques.

Text-to-3D Generation
We apply the multi-view diffusion models from Instant3D [] to generate 4-view images from text inputs, followed by our Mesh-LRM to achieve text-to-3D generation.
Our results and comparison with the original Instant3D pipeline with In3D-LRM (that is already quantitatively compared in Tab. 5 in terms of a reconstruction task) are shown in Fig. 5.
Our approach leads to better mesh quality and fewer rendering artifacts.

Image-to-3D Generation
We apply Zero123++ [] v1.2 to generate multi-view images from a single image input, followed by our MeshLRM to achieve image-to-3D generation. Our results and comparison with the One2345++ [], which uses a diffusion-based reconstruction model, are shown in Fig. 6. Our results are of much higher quality, containing sharper textures and more faithful geometric details than One2345++.

Figure: Figure 5: Text-to-3D results by applying Instant3D’s [] diffusion model to generate 4-view images from a text input. Our method can generate far more accurate and smoother geometry, along with sharp textures.

Figure: Figure 6: Image-to-3D results by applying Zero123++ [] to generate 6 multi-view images from a single image input. Our results outperform other reconstruction methods. Note that our model is trained on 4 views and can zero-shot generalize to 6 views.

## 6 Conclusion

In this paper, we have presented MeshLRM, a novel Large Reconstruction Model that can directly output high-quality meshes.
We achieved it by applying the Differentiable Marching Cubes (DiffMC) method and differentiable rasterization to fine-tune a pre-trained NeRF-based LRM trained with volume rendering.
As DiffMC requires backbone efficiency, we brought multiple improvements (tiny shared MLP and simplified image tokenization) to the LRM architecture, facilitating both NeRF and mesh training.
We also found that a low-to-high-res training strategy can significantly accelerate the training of the NeRF-based model.
Compared with existing methods, our method has both quality increase and speed improvement and is the only one that can output high-quality meshes.
In addition, we showed that our method can be directly applied to applications like text-to-3D and image-to-3D generation.
As meshes are the most widely accepted format for 3D assets in the industry, we believe our method takes a step towards automating 3D asset creation and potentially opens up new possibilities with novel 3D workflows.

## Acknowledgement

Thanks to Nathan Carr for his support on this project and to Linghao Chen for helping to run the One2345++ baseline.

## Appendix 0.A Effectiveness of surface fine-tuning (Stage 2).

We have discussed the effectiveness of our second-stage fine-tuning in Sec. 4.2 with quantitative results shown in Tab. 2 in the main paper.
We now show examples of qualitative results in Fig. 7, comparing our final meshes (MeshLRM) with the meshes (MeshLRM (NeRF) + MC) generated by directly applying Marching Cubes on the 1st-stage model’s NeRF results.
As shown in the figure, the MeshLRM (NeRF) + MC baseline can lead to severe artifacts with non-smooth surfaces and even holes, whereas our final model with the surface rendering fine-tuning can effectively address these issues and produce high-quality meshes and realistic renderings.
These qualitative results demonstrate the large visual improvements achieved by our final model for mesh reconstruction, reflecting the big quantitative improvements shown in the paper.

Figure: Figure 7: The model performance drops significantly when applying marching cubes to the volume rendering trained model (‘MeshLRM (NeRF) + MC’) without our mesh refinement fine-tuning (‘MeshLRM’).

## Appendix 0.B ABO and OpenIllumination datasets

**Table 7: Comparison with feed-forward approaches on ABO [] and Open-Illumination [] datasets.**
|  | ABO | OpenIllumination |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| In3D-LRM [] | 27.50 | 0.896 | 0.093 | 8.96 | 0.568 | 0.682 |
| MeshLRM (NeRF) | 28.31 | 0.906 | 0.108 | 20.53 | 0.772 | 0.290 |
| In3D-LRM []+MC | 22.11 | 0.850 | 0.144 | 8.92 | 0.507 | 0.691 |
| MeshLRM | 26.09 | 0.898 | 0.102 | 20.51 | 0.786 | 0.218 |

We compare our proposed method with In3D-LRM [] on 1000 randomly sampled examples in the ABO [] dataset with quantitative results shown in Tab. 7. In particular, ABO is a highly challenging synthetic dataset that contains many objects with complex glossy materials. This is a big challenge for our model, as well as In3D-LRM’s model, since both assume Lambertian appearance. In this case, the NeRF models are often better than the mesh models, since NeRF rendering can possibly fake glossy effects by using colors from multiple points along each pixel ray while mesh rendering cannot.
Nonetheless, as shown in the table, both our NeRF and final mesh model can outperform In3D-LRM’s NeRF and mesh variants respectively. Especially, thanks to our effective MC-based finetuning, our mesh rendering quality surpasses that of In3D-LRM + MC by a large margin (e.g. 3.92dB in PSNR and 0.48 in SSIM).

On the other hand, we also compare with In3D-LRM on the full OpenIllumination [] dataset in Tab. 7.
To clarify, this experiment setting is different from the setting used in Tab. 6 in the main paper, where we follow ZeroRF [] to test only 8 objects with combined masks in OpenIllumination; here, we evaluate all models on the full dataset with 100 objects with object masks and we also take square crops that tightly bounding the objects from the rendered/GT images to compute rendering metrics, avoiding large background regions.
As shown in the table, we observe that In3D-LRM cannot generalize to handle this challenging real dataset, leading to very low PSNRs, despite being trained on the same training data as ours. In contrast, our model can still work well on this out-of-domain dataset with high rendering quality.

## Appendix 0.C Additional Implementation Details

The transformer size follows the transformer-large config [].
It has $24$ layers and a model width of $1024$.
Each attention layer has a $16$ attention head and each head has a dim of $64$.
The intermediate dimension of the MLP layer is $4096$.
We use GeLU activation inside the MLP and use Pre-LN architecture.
The layer normalization is also applied after the input image tokenization, and also before the triplane’s unpachifying operator.

For the training of both stages, we use AdamW with $\beta_{2}=0.95$.
Other Adam-specific parameters are set as default and we empirically found that the model is not sensitive to them.
A weight decay of $0.05$ is applied to all parameters excluding the bias terms and layer normalization layers. The learning rate for stage 1 is $4e-4$.
We also use cosine learning rate decay and a linear warm-up in the first 2K steps. For stage 2, we use a learning rate of $1e-5$, combined with cosine learning rate decay and a linear warm-up during the first 500 steps. In total, there are 10k fine-tuning steps. The resolution of DiffMC is $256$ within a $[-1,1]^{3}$ bounding box, which is consistent with the triplane resolution.

## Appendix 0.D Limitations

Since our model employs surface rendering and assumes only Lambertian appearance without performing any inverse rendering, it is not sufficiently robust when the input images contain complex materials, e.g. some examples in the ABO dataset. The generated mesh may use white color to fit the specular areas. We believe that incorporating inverse rendering into the current pipeline could address this issue. However, this requires sufficient training data with accurate material annotations, which we leave for future exploration. On the other hand, while handling highly sparse input views, our model requires input camera poses. Although poses can be readily obtained from text- or image-to-multi-view models [, ] for 3D generation tasks, calibrating sparse-view poses for real captures can be a big challenge. In the future, it will be an interesting direction to explore combining our approach with recent pose estimation techniques [, ] for joint mesh reconstruction and camera calibration.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Is there evidence supporting DiffMC as a superior differentiable isosurface extraction method for mesh extraction compared to alternatives like DMTet or FlexiCubes?
- Is there comparative evidence showing that the proposed ray opacity loss formula outperforms existing density regularization techniques?
- How was the density threshold determined for DiffMC vs. standard Marching Cubes in the ablations? This choice could significantly impact surface quality, particularly in the non-finetuned case.
- Given the claimed speed improvements, how does MeshLRM's performance compare to other recently published approaches?
soundness: 3
strengths: The paper contains valuable insights that improve surface reconstruction quality and speed for LRMs. It makes a straightforward yet impactful contribution by integrating a differentiable isosurface extraction method — specifically DiffMC — into the training framework, enabling more precise mesh extraction. The proposed ray opacity loss demonstrates substantial qualitative improvements, effectively resolving the floater issue. Additionally, the insight about MLP over-parameterization leading to rendering speed slowdowns is important and directly applicable to other LRM-based approaches.

The paper is well-written and structured, clearly guiding the reader through model design, training strategy, and key components such as ray opacity loss and progressive resolution scaling, with their necessity well-supported by ablations.
summary: The paper presents a transformer-based framework for efficient 3D mesh reconstruction from sparse views. Unlike previous approaches relying solely on NeRF-based volumetric models with marching cubes as a post-hoc mesh extraction solution, MeshLRM incorporates differentiable mesh extraction and rendering directly within the LRM framework. The model undergoes sequential low-to-high-resolution training, followed by DiffMC-aware fine-tuning. Key contributions include DiffMC integration for improved mesh quality, a novel ray opacity loss to eliminate floaters, and training strategies like progressive resolution scaling for improved convergence.
weaknesses: The paper offers a specific training recipe for LRMs, but key components of this approach appear underexplored:

- The ray opacity loss is not entirely novel; ray regularization for zero density in empty space is a standard technique in NeRF approaches. While the formulation may be unique, it is unclear how it performs against simpler density-to-zero regularization or background mask losses.
- The inclusion of DiffMC, while valuable, also lacks thorough evaluation. Alternative isosurface extraction methods, such as DMTet, are available, yet the rationale for choosing DiffMC over these options is not discussed.
- The paper’s comparative analysis is limited and primarily focuses on a single baseline (Instant3D-LRM). Although MeshLRM demonstrates quantitative improvements, qualitative results are mixed, showing reduced high-frequency details in geometry and texture. For example, textures in most samples in Figure 5 lack sharpness compared to the baseline, and the geometry generally appears to be over-smoothed, e.g. frog’s legs merge with the stand in Sample 4, Fig. 5.
- The paper does not discuss recent works, such as InstantMesh or TripoSR, suggesting it may not have been updated since its initial publication on Arxiv six months before the submission. Additionally, the baselines used for comparison are approximately a year old, which dilutes the paper's relevance to the community given the rapid advancements in the field.

While the addition of isosurface extraction is important, it lacks thorough investigation. Most training strategy improvements are supported by ablations, but these are not evaluated against alternative options. The value of the paper's proposed recipe is further diluted by the existence of other potentially superior reconstruction methods, such as InstantMesh and TripoSR.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Regarding the second stage of training, I am curious if the authors could elaborate on which parts of the network require fine-tuning. Specifically, could the model be effective with fine-tuning only the tiny MLPs or merely the last few layers of the transformer blocks?
soundness: 4
strengths: The paper is clearly written, and the quality of the result is noticeably better than previous NeRF-based approach – Inst-LRM in terms of extracted mesh quality. From the perspective of engineering and practicality, the design choice proposed in this paper is sound and seems effective. The ablation in the paper is extensive and seems sound to me.
summary: The paper introduces MeshLRM that outputs a density and color field suitable for mesh extraction using marching cubes. Building on LRM, MeshLRM incorporates a second training stage where NeRF rendering is replaced with differentiable marching cubes (diffMC) rendering. The authors also present comprehensive ablation studies on additional losses, such as Ray Opacity Loss, which enhance the robustness of the second training stage.
weaknesses: First, I have some reservations about the title "MeshLRM" and certain descriptions in the introduction, which may be over-claimed.  For instance, in lines 67-68, the statement, "We propose to address this with MeshLRM, a novel transformer-based large reconstruction model, designed to directly output high-fidelity 3D meshes from sparse-view inputs," could be misleading. In my view, the network does not "directly" output a mesh; instead, it produces density and color fields similar to its baseline model, which still require marching cubes to extract the mesh. The phrase "direct output mesh" more accurately describes a different family of models, such as MeshGPT [1].

2nd, another concern is the novelty of the paper. While combining LRM with diffMC is an effective and practical engineering approach, the concept appears somewhat ad hoc, mainly merging existing components rather than presenting deep or non-trivial new insights. However, this does not constitute a significant reason for rejection, as the paper still meets the typical standards of previously accepted work.

[1] MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers, CVPR 24.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: My questions have been listed in the weakness part, the author's response on them will serve as the main basis for my final rating. Besides, the paper does not mention the future availability of code or pre-trained models, which is crucial for reproducibility and further research by the community. Providing such resources would greatly benefit the field.
soundness: 3
strengths: - The incorporation of differentiable mesh extraction and rendering within the LRM framework is reasonable, enabling end-to-end mesh reconstruction without post-optimization steps.
- The introduction of the ray opacity loss is a novel contribution that addresses the challenge of stabilizing training for mesh reconstruction.
- The paper is well-written and easy to follow. The mesh visualization in the teaser is impressive.
summary: This paper presents MeshLRM, a large reconstruction model that enables the generation of high-quality 3D meshes from posed sparse-view images. It incorporates differentiable mesh extraction and rendering within the LRM framework, simplifying the architecture and improving training strategies for faster and better quality reconstruction. The model introduces a ray opacity loss for stable DiffMC-based training and achieves state-of-the-art performance in sparse-view mesh reconstruction, supporting applications like text-to-3D and single-image-to-3D generation.
weaknesses: - Limited evaluation on real-World data.  
The paper primarily evaluates MeshLRM on synthetic datasets. While the performance on these datasets is impressive, the model's ability to generalize to real-world images with complex lighting and materials is less explored. Incorporating more experiments on real-world datasets such as OmniObject3d [1] could provide a clearer picture of the model's practical applicability.

- Incomplete discussion and comparison with newer baselines.  
The paper compares with Instant3D and LGM, the former is not open-sourced, while the latter is based on Gaussian Splatting and leverages a ConvNet instead of transformer. I think there are some newer methods should be discussed and compared in the paper, e.g., InstantMesh [2], GeoLRM [3] and MeshFormer [4]. Especially, I believe InstantMesh is a pefect baseline for comparison since it is open-sourced and also adopts the "LRM+differentiable mesh extraction" paradigm.

- Concerns on technical novelty and training stability.  
Pioneer works like Magic3D [5] and InstantMesh have explored the pipeline of two-stage 3D generation by combining the NeRF and mesh representations, so I think the usage of mesh representation is reasonable but not novel.  
Besides, I have another concern on the training stability of mesh representation. According to the paper, the training process of MeshLRM is splitted into two stages, using NeRF and mesh respectively, and "Once trained with volume rendering, our model already achieves high-fidelity sparse-view NeRF reconstruction, which can be rendered with ray marching to create realistic images" (Section 3.3, line 272). Why is stage 1 essential if the proposed ray opacity loss can indeed stablize the training process on the mesh representation? What if we skip stage 1 and directly train the model on the mesh representation from scratch? I think the authors should add some discussions on this issue in the paper.  

[1] Wu, Tong, et al. "Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.  
[2] Xu, Jiale, et al. "Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models." arXiv preprint arXiv:2404.07191 (2024).  
[3] Zhang, Chubin, et al. "Geolrm: Geometry-aware large reconstruction model for high-quality 3d gaussian generation." arXiv preprint arXiv:2406.15333 (2024).  
[4] Liu, Minghua, et al. "Meshformer: High-quality mesh generation with 3d-guided reconstruction model." arXiv preprint arXiv:2408.10198 (2024).  
[5] Lin, Chen-Hsuan, et al. "Magic3d: High-resolution text-to-3d content creation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Why are density fields considered more compatible with the Marching Cubes step in Line 73? In traditional multi-view reconstruction tasks, Signed Distance Functions (SDFs) have proven to be more effective than density fields for mesh reconstruction, as surfaces are clearly defined by the zero-level set. What makes density fields preferable in this context?

2. I  do not fully understand the challenges associated with the second-stage mesh-based LRM optimization. When using Flexicube in Instantmesh, there do not appear to be issues related to mesh floater. Could you elaborate on the advantages of choosing Differentiable Marching Cubes (DiffMC) over Flexicube?
soundness: 2
strengths: 1. Utilizing DiffMC in LRM is novel. Indeed, meshes offer superior 3D representation.
2. The reconstructed mesh appears to be of high quality, featuring smooth geometry.
3. The proposed ray-opacity loss is technically sound and well-founded.
summary: This manuscript introduces a two-stage feed-forward large reconstruction model that utilizes sparse views as input. In the first stage, NeRF-like volume rendering is employed for supervision, with progressive training regarding resolution. In the second stage, Differentiable Marching Cubes (DiffMC) is leveraged to convert the density field into a mesh, which supports fast rendering and geometry supervision. With the integration of a ray-opacity loss, MeshLRM is capable of generating high-quality meshes. Furthermore, by simplifying the encoder, MeshLRM achieves faster inference speeds.
weaknesses: 1. The definition of post-optimization in Line 56 is unclear. In traditional multi-view reconstruction tasks with SDF-based volume rendering, meshes can be directly extracted by Marching Cubes (MC) by identifying the zero-level set without any post-optimization.
2. A discussion is lacking on why converting NeRFs into meshes leads to a significant drop in rendering quality and geometric accuracy, as mentioned in Line 66.
3. The definitions of the post-processing steps, specifically Marching Cubes in NeRF and Differentiable Marching Cubes in MeshLRM, should be illustrated more clearly. If both processes are considered post-processing, then the disadvantages of NeRF should not include the need for additional post-optimization.
4. There are some grammatical issues. For example, a comma is missing in Line 76.
5. I recommend adding an illustration in Line 75 to depict why NeRF-based LRM pretraining is important.
6. Considering there are two-stage optimizations, the claim of end-to-end optimization in Line 83 is not precise enough.
7. The assertion in Line 205 is not convincing enough. A comparison with In3D-LRM does not substantiate the claim that DINO cannot handle high-resolution images. There should be an ablation study in MeshLRM to adequately discuss why DINO is not necessary. Since you claim a contradiction in this paragraph, it is crucial to address this.
8. There is no quantitative comparison between NeRF-based LRM and Mesh-based LRM. Additionally, it appears that NeRF-based LRM achieves better PSNR.
9. There is a lack of quantitative comparison in the progressive training regarding resolution.
10. There are too few comparison methods. Only LGM and In3D are compared.

Meta-review: The paper received mixed reviews, with one accept, two borderline accepts, and one borderline reject. While Reviewer apsR expressed a generally positive view of the paper, all reviewers raised concerns regarding the novelty of the work. The rebuttal emphasized the paper’s contribution as a simplified version of the LRM architecture. However, this argument was not sufficiently convincing to change the reviewers’ perspectives, particularly those of Reviewer ftqh and Reviewer 1jbs.

Reviewer apsR highlighted that the ray opacity loss introduced in the paper is not entirely novel, and ray regularization for zero density in empty space is a standard technique in NeRF approaches. Reviewer 6QBv criticized the method for simply combining LRM with diffMC, describing the approach as somewhat ad hoc (while Reviewer 6QBv did not take this as a significant reason for rejection, the Area Chair agrees with Reviewer 1jbs that 3D generation should not be held to a lower standard for acceptance). Reviewer ftqh noted that earlier works, such as Magic3D and InstantMesh, have already explored two-stage 3D generation pipelines by combining NeRF and mesh representations. While the usage of mesh representations is reasonable, it is not novel in this context. Reviewer ftqh also raised a significant concern regarding the lack of publicly available code, which is crucial for reproducibility and further research by the community. Reviewer 1jbs agreed with the other reviewers on the novelty issue and actively participated in the discussion, demonstrating a strong sense of responsibility in their role as a reviewer. Reviewer 1jbs also highlighted, alongside Reviewer ftqh, the lack of public code availability, noting that the authors only plan to provide a demo in the future. This limitation significantly undermines the impact of this work which is mainly an engineering contribution.

The area chair has thoroughly reviewed the paper, supplementary materials, and the discussions among reviewers. The paper’s lack of novelty significantly limits its contribution to the field, as the proposed method largely combines existing techniques without introducing substantial new insights. While providing open source code is also commendable for impactful engineering contributions, the paper falls short in this regard. The active participation of Reviewer 1jbs in the discussion is particularly valued, as it demonstrated a thoughtful and responsible evaluation of the work, ultimately leaning toward a weak reject. The concerns of Reviewer 1jbs, alongside those of other reviewers, highlight that the technical contributions do not meet the high standards of ICLR.

Decision: Reject