MeshFormer : High-Quality Mesh Generation with 3D-Guided Reconstruction Model
Authors: Minghua Liu
Chong Zeng
Xinyue Wei
Ruoxi Shi
Linghao Chen
Chao Xu
Mengqi Zhang
Zhaoning Wang
Xiaoshuai Zhang
Isabella Liu
Hongzhi Wu
Hao Su
Venue: NeurIPS 2024 oral (2024)
Keywords: sparse view 3D reconstruction
3D generation
3D AIGC
reconstruction model
Primary area: machine_vision
TL;DR: We introduce MeshFormer, a sparse-view reconstruction model that can deliver high-quality meshes and be trained efficiently.

Abstract: Open-world 3D reconstruction models have recently garnered significant attention. However, without sufficient 3D inductive bias, existing methods typically entail expensive training costs and struggle to extract high-quality 3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. Specifically, instead of using a triplane representation, we store features in 3D sparse voxels and combine transformers with 3D convolutions to leverage an explicit 3D structure and projective bias. In addition to sparse-view RGB input, we require the network to take input and generate corresponding normal maps. The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's learning. Moreover, by combining Signed Distance Function (SDF) supervision with surface rendering, we directly learn to generate high-quality meshes without the need for complex multi-stage training processes. By incorporating these explicit 3D biases, MeshFormer can be trained efficiently and deliver high-quality textured meshes with fine-grained geometric details. It can also be integrated with 2D diffusion models to enable fast single-image-to-3D and text-to-3D tasks. **Videos are available at https://meshformer3d.github.io/**

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model
Minghua L,u Chong Zeng
Xinyue Wei Ruoxi !
Linghao Chen 2.3 Chao Xu Mengqi Zhang² Zhaoning Wang
Xiaoshuai Zhang 1.2f Isabella Liu Hongzhi Wu Hao Su
UC San Diego 2 Hillbot Inc. 3 Zhejiang University
UCLA University of Central Florida
Project Website:
https://neshhormerr... github. io/
Figure 1: Given a sparse set (e.g.. 6) multi-view RGB images and their normal maps as input, MeshFormer reconstructs high-quality 3D textured meshes with ome-grained sharp geometric details feed-forward pass of just a few seconds. Here, ground truth multi-view RGB and normal images used as input.
Abstract
Open-world 3D reconstruction models have recently garnered significant attention. However, without sufficient 3D inductive bias, existing methods typically entail expensive training costs and struggle to extract high-quality 3D meshes. In this
Equal contribution. Work done during internship at Hillbot Inc. Work done during internship
San Diego
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

work, we introduce MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. Specifically, instead of using triplane representation, we store features in 3D sparse voxels and combine transformers with 3D convolutions to leverage an explicit 31D structure and projective bias. In addition to sparse-view RGB input, we require the network to take input and generate corresponding normal maps. The input normal maps can be predicted by 2D diffusion models, significantly aiding in the guidance and refinement of the geometry's 's learning. Moreover, by combining Signed Distance Function (SDF) supervision with surface rendering, we directly learn to generate high-quality meshes without the need for complex multi-stage training processes. By incorporating these explicit 3D biases, MeshFormer can be trained efficiently and deliver high-quality textured meshes with fine-grained geometric details. It can also be integrated with 2D diffusion models to enable fast single-image-to-31 and text-to-3D tasks.
Introduction
High-quality 3D meshes are essential for numerous applications, including rendering, simulation, and 3D printing. Traditional photogrammetry systems [57 [1] and recent neural approaches, such as NeRF 43). typically require a dense set of input views of the object and long processing times. Recently, open-world 3D object generation has made significant advancements, aiming to democratize 3D asset creation by reducing input requirements. There are several prevailing paradigms: training a native 3D generative model using only 3D data 13 95 or performing per-shape optimization with Score Distillation Sampling (SDS) losses 30,
Another promising direction is to first predict sparse set of multi-view images using 2D diffusion models [33, [59] and then lift these predicted
images into a 3D model by training a worward network [3 32) This strategy addresses the limited generalizability of models trained solely on 3D data and overcomes the long runtime and 3D methods.
inconsistency of per-shape-oppimizaaion
While many recent works explore utilizing priors from 2D diffusion models, such as generating
consistent multi-view images [59] 60) and predicting normal maps from RGB [2] 37 59]. the feed- forward model that converts multi-view images into 3D remains underexplored. One-2-3-45 [32) leverages a generalizable NeRF method for 3D reconstruction but suffers from limited quality and success rates. One-2-3-45- [31 improves on this by using two-stage 3D diffusion model, yet it still struggles to generate high-quality textures or fine- -eeiirriiedd geometry. Given that sparse-view reconstruction of open-world objects requires extensive priors, another family of works pioneered by the large reconstruction model (LRM) [66) combines large-scale transformer models with the triplane representation and trains the model primarily using rendering loss. Although straightforward, these methods typically require over a hundred GPUs to train. Moreover, due to their reliance on volume rendering, these methods have difficulty extracting high-quality meshes. For instance, some recent follow-up works [79 85 implement complex multi-stage "NeRF-to-mest training strategies, but the results still leave room for improvement.
In this work, we present MeshFormer, an open- open-world sparse-viee view reconstruction model that takes a sparse set of posed images of an arbitrary object as input and delivers high-quality 3D textured meshes with a single forward pass in few seconds. Instead of representing 3D data as "2D planes" and training "black box" transformer model optimizing only rendering loss, we find that by incorporating various types of 3D-native priors into the model design, including network architecture, supervision signals, and input guidance, our model can significantly improve both mesh quality and training efficiency. Specifically, we propose representing features in explicit 3D voxels and introduce a novel architecture that combines large-scale transformers with 3D (sparse) convolutions. Compared to triplanes and pure transformers models with little 3D-native design, McshFormer leverages the explicit 3D structure of voxel features and the precise projective correspondence between 3D voxels and 2D multi-view features, enabling faster and more effective learning.
Unlike previous works that rely on NeRF based representation in their pipeline, we utilize mesh representation throughout the process and train MeshFormer a unified, single-stage manner. Specif- ically, we propose combining surface rendering with additional explicit 3D supervision, requiring the model to learn a signed distance function (SDF) field. The network is trained with high-resolution SDF supervision. and efficient differentiable surface rendering is applied to the extracted meshes
2

---

for rendering losses. Due to the explicit 3D geometry supervision, MeshFormer enables faster training while eliminating the need for expensive volume rendering and learning an initial coarse NeRF. Furthermore, in addition to multi-view posed RGB images, we propose using corresponding normal maps as input, which can be captured through sensors and photometric techniques 4 [8] or directly estimated by recent 2D vision models [12 37 59). important clues for 3D reconstruction and fine-graine geometric details. We also task the model with learning normal texture in addition to the RGB texture, which can then be used to enhance the generated geometry through a traditional post-processing algorithm
These multi-view normal images provide
Thanks to the explicit 3D-native structure, supervision signal, and normal guidance that we have incorporated. MeshFormer can generate high-quality textured meshes with fine-graines geometric details, as shown in Figure [ Compared to concurrent methods that require over one hundred GPUs or complex multi-stage training, MeshFormer can be trained more efficiently and conveniently with just eight GPUs over two days, achieving on-par or even better performance. It can also seamlessly integrate with various 2D diffusion models to enable numerous tasks, such as single-image-tor and text-to-3D. In summary, our key contributions include:
We introduce MeshFormer, an open-world sparse-vier view reconstruction model capable of generating high-quality 3D textured meshes with fine- fine-grained geometric details in a few seconds. It can be trained with only 8 GPUs, outperforming baselines that require over one hundred GPUs.
We propose novel architecture that combines 3D (sparse) convolution and transformers. By explicitly leveraging 3D structure and projective bias, it facilitates better and faster learning.
We propose a unified single-stage training strategy for generating high-quality meshes by combin- ing surface rendering and explicit 3D geometric supervision.
We are the first introduce multi-view normal images input to the feed-forward reconstruction network, providing crucial geometric guidance. Additionally, we propose to predict extra 3D normal texture for geometric enhancement.
2 Related Work
Open-world 3D Object Generation Thanks to the emergence of large-scale 3D datasets g
9
and the extensive priors learned by 2D models [50 51 55 56). open-world 3D object generation
have recently made significant advancements. Exemplified by DreamFusion [4]. a line of work [5]
6, 10 26
70 [66 uses 2D models as guidance to generate 3D objects through per-shape optimization with SDS-like losses. Although these methods produce increasingly better results, they are still limited by lengthy runtimes and many other issues. Another line of
30
48
58, 60
62
65
84 (96] trains a feed-forward generative model solely on 3D data that consumes text prompts single-image inputs. While fast during inference, these methods struggle generalize to unseen object categories due to the scarcity of 3D data. More recently. works such as Zero123 (33 have shown that 2D diffusion models can be fine-tuned with 3D data for novel view synthesis. A
work 16 20
40
45
line of work 127 27 31 64 (77 (79 85), pioneered by One-2-3-45 (32,, proposes first predicting multi-view images through 2D diffusion models and then lifting them to 3D through a feed-forward network, effectively addressing the speed and generalizability issues. Many recent works have also explored better strategies to fine-tune 2D diffusion models for enhancing the 3D consistency of multi-
view images 14
89 9II. In addition to the feed-forwar models, the generated multi view images can also lifted to 3D through optimizations
17 23
36
59
60
69
72
80
81
Sparse-Viee View Feed-Forward Reconstruction Models When a small baseline between input images is assumed, existing generalizable NeRF methods [35
52
68 88) aim to find pixel correspondences
and learn generalizable priors across scenes by leveraging cost-volume-basee techniques
B 38 90)
or rransformer-based structures [19] /2 54 7 74). Some of methods have also incorporated a 2D
diffusion process into the pipeline I 21 66). However, these methods often struggle to handle large
baseline settings (e.g., only frontal-vis view reconstruction) or are limited by small training set and
fail to generalize to open- woold objects. Recently, many models (27 64 73 77 79
85 87 92 94
specifically aimed at open-world 3D object generation have been proposed. They typically build large networks and aim to learn extensive reconstruction priors by training on large-scale 3D datasets [9. For example, the triplane representation and transformer models are often used. By applying volume rendering or Gaussian splatting 66 86 92], they train the model with rendering losses. However, these methods typically require extensive GPUs to train and have difficulty extracting high-qualit,
3

---

singlee vieeew RGBBB text
Enhencement
Normel MLP
Sparse Vexel Former
interpataae
Voxel Former
color Texture
normal
Tiny Color MLP
Suracc
I069
occupency
sparse feeture volume (258*3)
color
volume
SDF vol
(64-))
mulb-view RGB normell
Thny SDF MLP
accupancy loss
SDF loss
textured mesh
20 Encoders
Transformer Leyers
multiview RGB pelch feal
Projecton Awara
Cross Attention
(Spanse)
Projecton Cross Attertion
Self Attention
(Spanse) 3D Conv
Projectlen-Auaa Awwa- Cross
multi view normal patch feat
256"3
learnall token
Figure 2: Pipeline Overview. MeshFormer takes a sparse set of multi-view RGB and normal images input, which can be estimated using existing 2D diffusion models. We utilize 3D feature volume representation, and submodules Voxel Former and Sparse Voxel Former share similar novel architecture, detailed in the gray region. We train MeshFormer mesh surface rendering and 512 SDF supervision. MeshFormer learns additional normal texture, which can used to further enhance the geometry and generate fine-grad pee-grained sharp geometric details.
unified single stage by combining
utilize multi-stage -NeRF-to-mesh training strategies improve the quality, the results still leave room for improvement.
meshes. While some recent (concurrent) works 779 85
Geometry Guidance for 3D Reconstruction Many recent works have shown that in addition multi- view RGB images, 2D diffusion models can be fine-tuned to generate other geometric modalities,
such depth maps 775). normal maps 12 37
coordinate maps (28
[7]. These additional
modalities can provide crucial guidance for 3D generation and reconstruction. While many recent
methods utilize these geometric cues inverse optimization guidance [5] 12
28
37
49 77], we
propose to take normal maps as input in a feed-forward reconstruction model and task the model with generating DD-coosistent normal texture for geometry enhancement of sharp details.
3D Native Representations and Network Architectures in 3D Generation The use of 3D voxel representations and 3D convolutions common in general 3D generation. However, most recent works focus on 3D-native diffusion 18 29
95), one of the key paradigms in 3D generation, which differs from the route taken by MeshFormer. These 3D-diffusion-based methods have some common limitations. For instance, they focus solely geometry generation and cannot directly predict high-qualig textures from the network (7 18 29 31
53 95 Due to the limited availability of 3D data, 3D-native diffusion methods also typically struggle with open-world world capabilities and are often constrained to
datasets (e.g., ShapeNet (2) in their experiments 7 29 95
In MeshFormer, our goal is achieve direct high-quality texture generation while handling arbitrary object categories. Therefore, we adopt different approach: sparse-view feed- forward reconstruction, opposed to 3D-native diffusion. In this specific task setting,
comparable works recent However, most of these methods rely on combination of triplane representation and large-scale transformers. In this paper, we demonstrate that 3D-native representations and networks can not only be used in 3D-native diffusion but can also be combined with differentiable rendering to train feed-forward sparse-view reconstruction model using rendering losses. In open-world sparse-view reconstruction, we are not limited to the triplane representation. Instead, 3D-native structures (c.g.. voxels), network architectures, and projective priors can facilitate more efficient training, significantly reducing the required training resources. While scalable networks are necessary learn extensive priors, scalability not exclusive to triplane- transformers. By integrating 3D convolutions with transformer layers, scalability can also be achieved.
LRM-style methods [64] 67
79 85
3 Method
As shown in Figure 2 MeshFormer takes a sparse set of posed multi-view RGB and normal images as input and generates high-quality textured mesh in single feed-forwart pass. In the following

---

sections, we will first introduce our choice of 3D representation and novel model architecture that combines large-scale transformers with 3D convolutions (Sec. 3.I). Then, we will describe our training objectives, which integrate surface rendering and explicit 3D SDF supervision (Sec.[3.2
Last but not least, we will present our normal guidance and geometry enhancement module, which plays a crucial role in generating high-quality meshes with fine-grained geometric details (Sec. 3.3
3.1 3D Representation and Model Architecture
Triplane vs. 3D Voxels Open-world sparse-view reconstruction requires extensive priors, which can be learned through large-scale transformer. Prior arts (27 67 77 79 85] typically utilize the triplane representation, which decomposes a 3D neural field into a set of 2D planes. While straightforward for processing by transformers, the triplane representation lacks explicit 3D spatial structures and makes it hard to enable precise interaction between each 3D location and its corresponding 2D projected pixels from multi-view images. For instance, these methods often simply apply self-attention across all triplane patch tokens and cross-attention between triplane tokens and all multi-view image tokens. This all-to-all attention is not only costly but also makes the methods cumbersome train. Moreover, the triplane representation often shows results with notable artifacts at the boundaries of patches and may suffer from limited expressiveness for complex structures. Consequently, we choose the 3D voxel representation instead, which explicitly preserves the 3D spatial structures.
Combining Transformer with 3D Convolution To leverage the explicit 3D structure and the power- ful expressiveness of large-scale transformer model while avoiding an explosion of computational costs, we propose VoxelFormer and Sparse VoxelFormer, which follow a 3D UNet architecture while integrating& a transformer at the bottleneck. The overall idea is that we use local 3D convolution to encode and decode a high-resolution 3D feature volume, while the global transformer layer handles reasoning and memorizing priors for the compressed low-resolution feature volume. Specifically, as shown in Figure 2 3D feature volume begins with learnable token shared by all 3D voxels. With the 3D voxel coordinates, we can leverage the projection matrix to enable each 3D voxel to aggregate 2D local features from multi-viww images via projection-awe -wwar -ross-attention layer. By iteratively performing projection-awe eogeettion-wware aware cross-attention and 3D (sparse) convolution, we can compress the 3D volume to a lower-resolution one. After compression, each 3D voxel feature then serves as latent token, and a deep transformer model is applied to a sequence of all 3D voxel features (position encoded) to enhance the model's expressiveness. Finally, we use the convolution- -based inverse upper branch with skip connection to decode a 3D feature volume with the initial high resolution.
Projection-Aware Cross Attention Regarding 3D-2D interaction, the input multi-view RGB and normal images are initially processed by 2D feature extractor, such as a trainable DINOv2 (46). to generate multi-view patch features. While previous cost www--oooluuesseeeeeeee methods B 38 typically use mean or max pooling to aggregate multi-view 2D features, these simple pooling operations might be suboptimal for addressing occlusion and visibility issues. Instead, we propose projection- aware Pross-attention mechanism to adaptively aggregate the multi-view features for each 3D voxel. Specifically, we project each 3D voxel onto the m views to interpolate m RGB and normal features. We then concatenate these local patch features with the projected RGB and normal values to form m 2D features. In the projection-awe -wwaee cross-attention module, we use the 3D voxel feature to calculate query and use both the 3D voxel feature and the m 2D features to calculate + keys and values. A cross-attention is then performed for each 3D voxel, enabling precise interaction between each 3D location and its corresponding 2D projected pixels, and allowing adaptive aggregation of 2D features, which can be formulated as:
CrossAAttentoon(Q = {o}, K = (p + [t), (p)) {w})
(I)
Where e denotes 3D voxel feature, and Di denotes its projected 2D pixel feature from view i, which is a concatenation of the RGB feature ff, the normal feature g¹, and the RGB and normal values of and nz, respectively.
Coarse-to-Fine Feature Generation As shown in Fig. 2 to generate a high-resolution 3D feature volume that captures the fine-gra tne-grained details of 3D shapes, we follow previous work [3] 95 by employing a coarse-to-fine strategy. Specifically, we first use VoxelFormer, which is equipped with full 3D convolution, to predict a low-resolution (e.g., 649), coarse 3D occupancy volume. Each voxel in this volume stores a binary value indicating whether it close to the surface. The predicted occupied voxels are then subdivided to create higher-resolution sparse voxels (e.g., 256*). Next, we utilize a second module, Sparse VoxelFormer, which features 3D sparse convolution [63] to predict features for these sparse voxels. After this, we trilinearly interpolate the 3D feature of any
5

---

near-surface 3D point, which encodes both geometric and color information, from the high- sighrresolution sparse feature volume. The features are then fed into various MLPs to learn the corresponding fields.
3.2 Unified Single-Stage Training: Surface Rendering with SDF Supervision
Existing works typically use NeRF [2 and volume rendering or 3D Gaussian splatting 22 since they come with a relatively easy and stable learning process. However, extracting high-quality meshes from their results is often non-trivial. For example, directly applying Marching Cubes [39 to density fields of learned NeRFs typically generates meshes with many artifacts. Recent methods 778 79 85 have designed complex, multi-stage "NeRF-to-mesh" training with differentiable surface rendering, but the generated meshes still leave room for improvement. On the other hand, skipping a good initialization and directly learning meshes from scratch using purely differentiable surface rendering losses is also infeasible, it is highly unstable to train and typically results in distorted geometry.
In this work, we propose leveraging explicit 3D supervision in addition to 2D rendering losses. As shown in Figure 2 we task MeshFormer with learning a signed distance function (SDF) field supervised by a high-resolution (e.g.. 512") ground truth SDF volume. The SDF loss provides explicit guidance for the underlying 3D geometry and facilitates faster learning. It also allows us to use mesh representation and differentiable surface rendering from the beginning without worrying about good geometry initialization or unstable training, as the SDF loss serves as a strong regularization for the underlying geometry. By combining surface rendering with explicit 3D SDF supervision, we train MeshFormer in a unified, single-stage training process. As shown in Figure 2
we employ three tiny MLPs that take as input the 3D feature interpolated from the 3D sparse feature volume to learn an SDF field, a 3D color texture, and a 3D normal texture. We extract meshes from the SDF volume using dual Marching Cubes [39 and employ NVDiffRast (25] for differentiable surface rendering. We render both the multi-view RGB and normal images and compute the rendering losses, which consist of both the MSE and perceptual loss terms. As result, our training loss can be expressed as:
(2) where Loce and LSDE are MSE losses for occupancy and SDF volumes, and A denotes the weight of each loss term. Note that do not use mesh geometry to derive normal maps; instead, we utilize the learned normal texture from the MLP, which will be detailed later.
L = λ1 AMSE + >2LLPIPS
Coolor
+ A Cuormal + Xa LLIPS
Cuormal + >s + Ce
3.3 Fine-Grained Geometric Details: Normal Guidance and Geometry Enhancement
Without dense-view correspondences, 3D reconstruction from sparse-view view RGB images typically struggles to capture geometric details and suffers from texture ambiguity. While many recent attempt to employ large-scale models to learn mappings from RGB geometric details, this typically requires significant computational resources. Additionally, these methods are primarily trained using 3D data, but it's still uncertain whether the scale of 3D datasets sufficient for learning such extensive priors. On the other hand, unlike RGB images, normal maps explicitly encode geometric information and can provide crucial guidance for 3D reconstruction. Notably, open-world normal map estimation has achieved great advancements. Many recent works 12 37 59
works 27 79 [55]
demonstrate that 2D diffusion models, trained on billions of natural images, embed extensive priors and can be fine-tuned to predict normal maps. Given the significant disparity in data scale between 2D and 3D datasets, it may be more effective to use 2D models first for generating geometric guidance.
addition to multi-view RGB images, MeshFormer also takes multi-view normal maps as input, which can be generated using recent open-world normal estimation models 12 37 59 In our experiments, we utilize Zero 13+++ v1.2 [59,, which trains an additional ControlNet 93 over the multi-view prediction model. The ControlNet takes multi-view RGB images, predicted by 1 as condition and produces corresponding multi-view normal maps, expressed in the camera coordinate frame. Given these maps, MeshFormer first converts them to a unified world coordinate frame, and then treats them similarly to the multi-view RGB images, using projection-awan (Sec. (S4c. the multi-view normal maps enable the networks to better capture geometry details, and thus greatly improve final mesh quality.
Input Normal Guidance As shown in Figure
cross-attention to guide 3D reconstruction. According to our experiments
Geometry Enhancement While the straightforward approach of deriving normal maps from the learned mesh and using a normal loss to guide geometry learning has been commonly used, we find that this approach makes our mesh learning less stable. Instead, we propose learning a 3D normal texture, similar to color texture, using a separate MI.P By computing the normal loss for MI.P queried normal maps instead of mesh-derived normal maps, we decouple normal texture learning

---

Imput
LGM
TripoSR
InstantMesh
Mesh RR
Ours
Figure 3: Qualitative Examples of Single Image to 3D (GSO dataset). Both the textured and textureless mesh renderings are shown. Please zoom in to examine details and mesh quality, and refer to the supplemental material for results of One-2-3-45+4 BI and CRM [77].
from underlying geometry learning. This makes the training more stable, as it is easier learn sharp 3D normal map than to directly learn sharp mesh geometry. The learned 3D normal texture can be exported with the mesh, similar to the color texture, to support various graphics rendering pipelines. In applications that require precise 3D geometry, such as 3D printing. the learned normal texture can also be used to refine the mesh geometry with traditional algorithms. Specifically, during inference, after extracting a 3D mesh from the SDF volume, we utilize post-processing algorithm 44 that takes input the 3D positions of the mesh vertices and the vertex normals estimated from the MLP. The algorithm adjusts the mesh vertices to align with the predicted normals a few seconds, further enhancing the geometry quality and generating sharp geometric details, shown in Figure 5
4 Experiments
4.1
Implementation Details and Evaluation Settings
Implementation Details trained MeshFormer on the Objaverse 9 dataset. The total number of network parameters is approximately 648 million. We trained the model using 8 H100 GPUs for about one week (350k iterations) with batch size of per GPU, although we also show that model can achieve similar results in just two days. Please refer to the supplementary for more details.

---

Table 1: Quantitative Results of Single Image 3D. Evaluated on the 1,030 and 1,038 3D shapes from the GSO [I] and the OmniObject3D [83] datasets, respectively. One-2-3-45++ [3]]
InstantMesh [85,, MeshL_M [9], and our method all take the same multi-view RGB images predicted by 7erol 23++ 59 as input. CD denotes Chamfer Distance.
GSO 11]
OmniObject3D
3
Method
F-Score f
CD 4
PSNR
LPIPS
F-Score t
CD 4
PSNR
LPIPS
One-2-3-45++ 31
0.936
0.039
20.97
0.21
0.871
0.054
17.08
0.31
TripoSR 66
0,896
0.047
19.85
0.26
0,895
0.048
17.68
0.28
CRM 777
0,886
0.051
19.99
0.27
0,821
0.065
16.01
0.34
LGM
0.776
0.074
18.52
0.35
0.635
0.114
14.75
0.45
InstantMesh
0.934
0.037
20.90
0.22
0.889
0.049
17.61
0.28
McshI.RM RM
79
0.956
0.033
21.31
0.19
0.910
0.045
18.10
0.26
Ours
0.963
0.031
21.47
0.20
0.914
0.043
18.14
0.27
Evaluation Settings We evaluate the methods on two datasets: GSO and OmniObject3D [8 Both datasets contain real-scanned 3D objects that were not seen during training. For the GSO dataset, we use all 1,030 3D shapes for evaluation. For the OmniObject3D dataset, we randomly sample up to 5 shapes from each category, resulting in 1,038 shapes for evaluation. We utilize both 2D and 3D metrics. For 3D metrics, we use both the F-score and Chamfer distance (CD), calculated between the predicted meshes and ground truth meshes, following [31 85). For 2D metrics, we compute both PSNR and LPIPS for the rendered color images. Since each baseline may use a different coordinate frame for generated results, carefully align the predicted meshes of all methods to the ground truth meshes before calculating the metrics. Please refer to the supplemental material for more details.
4.2 Comparison with Single/Sparse- View to 3D Methods
We compare McshFormer with recent open-world feed-forwart single/sparse-view to 3D meth- ods, including One-2-3-45++ [3]], TripoSR [67], CRM (77). LGM (64). InstantMesh [85), and McshLRM [79.. Many of these methods have been released recently and should be considered concurrent methods. For MeshLRM 79,, contacted the authors for the results. For the other methods, we utilized their official implementations. Please refer to the supplementary for details.
Since input settings differ among the baselines, evaluate all methods in unified single- to 3D setting. For the GSO dataset, we utilized the first thumbnail image as the single-view input. For the OmniObject3D dataset, we used rendered image with a random pose input. For One-2-3-
45++ (3I, InstantMesh [85], MeshL.M [79], and our MeshFormer, we first utilized 1ero123++ (59]
to convert the input single-view image into multi-view images before 3D reconstruction. Other baselines follow their original settings and take a single-view image directly as input. In addition to the RGB images, our MeshFormer also takes additional multi-view normal images as input, which are also predicted by Zero123++ [59]. Note that when comparing with baseline methods, we never use ground truth normal images to ensure fair comparison.
In Fig. 3 we showcase qualitative examples. Our MeshFormer produces the most accurate meshes with fine-grained, sharp geometric details. In contrast, baseline methods produce inferior mesh quality. For example, TripoSR directly extracts meshes from the learned NeRF representation, resulting in significant artifacts. While InstantMesh and Meshi.M, RM use mesh representation in their second stage, notable uneven artifacts are still observable upon zoom-in inspection. Additionally, all bascline methods incorrectly close the surface of the copper bell. We also provide quantitative results in Tab. I Although our baselines include four methods released just one or two months before the time of submission, our MeshFormer significantly outperforms many of them and achieves the best performance on most metrics across two datasets. For the color LPIPS metric, our performance is very similar to MeshLRM's 's despite perceptual loss being their main training loss term. We also highlight that many of the baselines require over one hundred GPUs for training, whereas our model can be efficiently trained with just 8 GPUs. Please refer to Sec. 4.4 for analysis on training efficiency.
4.3 Application: Text to 3D
In addition to the single image to 3D, MeshFormer can also be integrated with 2D diffusion models to enable various 3D object generation tasks. For example, we follow the framework proposed by (37) to finetune Stable Diffusion [56) and build text-to-multi-viee model. By integrating this
8

---

L; an erupting volcano, aerial view R: beautifully carved wooden knight chess piece
a DSLR photo human skull R: freshly baked leaf of sourdough bread cutting board
a golden goblet, low poly R: chimpanzee wearing headphones
L: pyramid shaped burrito with slice cut out of R: a rod panda
L; metal sculpture of lion's head, highly detuiled DSLR photo of delicious croissant
L: a rabbit, animated movie character, high detail 3d model
R: a DSLR photo of frog wearing sweater
Figure 4: Application: Text to 3D. Given a text prompt, a 2D diffusion model first predicts multi- view RGB and normal images, which then fed MeshFormer for 3D reconstruction. Please refer to the supplementary for comparisons with Instant3D [27]
Table We compare methods using limited training resources. Evaluated on the GSO | dataset.
Method
Training Resources F-Score
CD 4 PSNR-C1 LPIPS-C PSNR-N LPIPS-N +
MeshLRM 79
0.925
0.0397
21.09
0.26
21.69
0.22
Ours
8xH100 48h
0.960
0.0317
21.41
0.20
23.01
0.15
model, along with the normal prediction from 12++123++ task of text to 3D. 4shurs some interesting results, where convert single text prompt into a high-quality 3D mesh in just few seconds. Please refer to the supplemental materials for a qualitative comparison with one of the state-of-the-art text-to-3D methods, Instant3D [27).
[59] with MeshFormer, we can enable the
4.4 Analysis and Ablation Study
Explicit 3D structure vs. Triplane In Section 4.2 we demonstrated that MeshFormer outperforms baseline methods that primarily utilize the triplane representation. Here, we highlight two additional advantages of using the explicit 3D voxel structure: training efficiency and the avoidance of "triplane artifacts". Without leveraging explicit structure, existing triplane-based large reconstruction models require extensive computing resources for training. For example, TripoSR requires 176 A100 GPUs for five days of training. InstantMesh relies on OpenLRM (15). which requires 128 A100 GPUs for three days of training. MeshLRM also utilizes similar resources during training. By utilizing explicit 3D structure and projective bias, our MeshFormer can be trained much more efficiently using only 8 GPUs. To better understand the gap, trained both MeshLRM and our MeshFormer under very limited training resources, and the results are shown in Table 2 When using only 8 GPUs for two days, found that MeshLRM failed to converge and experienced significant performance degradation compared to the results shown in Table [ while our MeshFormer had already converged decent result, close the fully-trained version, demonstrating superior training efficiency.
We observe that the triplane typically generates results with axis-aligned artifacts, as shown Fig 3 (5th row, please zoom in). As demonstrated in the supplementary (Fig. 7 these artifacts also cause difficulties for MeshLRM [79] in capturing the words on objects. These limitations are likely caused by the limited number of triplane tokens (c.g.. 32 x 32 3), constrained by the global attention, which often leads to artifacts at the boundaries of the triplane patches. In contrast, MeshFormer leverages sparse voxels, supports higher feature resolution of 2563, and is free from such artifacts.
Normal Input and SDF supervision As shown Table (a), performance significantly drops when multi-view input normal maps are removed, indicating that the geometric guidance and clues provided by normal images are crucial for facilitating network training, particularly local geometric details. In (f), replace ground truth normal maps with normal predictions by Zero 123++ [59] and

---

Table 3: Ablation Study on the GSO [ dataset. -C denotes color renderings, and -N denotes normal renderings. CD stands for Chamfer distance. By default, ground truth multi-view images are used exclude the influence of errors from 2D diffusion models.
Setting
PSNR-C LPIPS-C
PSNR-N f LPIPS-N F-Score 1 CD 4
w/o normal input
24.82
0.129
24.85
0.107
0,964
0.024
b
w/o SDF supervision
20.72
0.244
20.42
0.257
0,940
0.035
w/o transformer layer
26.63
0.101
29.80
0.036
0,992
0.013
w/o projection -warro aware cross attention
25.48
0.155
29.01
0.045
0,991
0.013
w/o geametry enhancement
27.95
0.085
29.10
0,048
0,992
0.012
w/ pred normal
26.84
0.096
26.99
0.067
0.987
0.017
full
28.15
0.083
29.80
0.036
0.992
0.012
observe notable performance gap compared to (g). This indicates that although predicted multi-view normal images can beneficial, existing 2D diffusion models still have room for improvement in generating more accurate results. See supplementary for qualitative examples. As shown in (b), if we remove the SDF loss after the first epoch and train the network using only surface rendering losses, the geometry learning quickly deteriorates, resulting in poor geometry. This explains why existing methods [27 79 typically employ complex multi-stage training and use volume rendering to learn coarse NeRF in the initial stage. By leveraging explicit 3D SDF supervision as strong geometric regularization, we enable unified single-stage training, using mesh as the only representation.
Projection-Aware Cross-Attention and Transformer Layers We propose to utilize projection- aware cross-attention to precisely aggregate multi- view projected 2D features for each 3D voxel. con- ventional learning- based multi-view stereo (MVS) methods S 38 average or max pooling typically employed for feature aggregation. In Table (d), we replace the cross-attention with a simple average pool- ing and we observe significant performance drop. This verifies that projection-aware cross- attenntnttnnonn pro- vides more effective way for 3D-2D interaction while simple average pooling may fail to handle the occlusion and visibility issues. In the bottleneck of the UNet, we treat 3D (sparse) voxels as a se- quence of tokens and apply transformer layers to them. As shown in row (c), after removing these lay- ers, observe performance drop in metrics related to texture quality. This indicates that texture learning requires more extensive priors and benefits more from the transformer layers.
Before
Alter
Figure Geometry enhancement generates sharper details. Please zoom in see the details.
Geometry Enhancement We propose learn additional normal map texture and apply tra- ditional algorithm as post processin for geometry enhancement during inference. As shown in Figure the geometry enhancement aligns the mesh geometry with the learned normal texture and generates fine- ine-grained sharp details. In some cases (such as the wolf), the meshes output by the network already good enough, and the difference caused by the enhancement tends to be subtle. Row (e) also quantitatively verifies the effectiveness of the module.
Conclusion and Limitations
We present MeshFormer, open-world sparse-view reconstruction model that leverages explicit 3D native structure, supervision signals, and input guidance. MeshFormer can be conveniently trained in unified single-stage manner and efficiently with just 8 GPUs. It generates high-quality meshes with fine-grained geometric details and outperforms baselines trained with over one hundred GPUs.
MeshFormer relies on 2D models to generate multi-view RGB and normal images from a single input image text prompt. However, existing models still have limited capabilities to generate consistent multi-view images, which can cause performance drop. Strategies to improve model robustness against such imperfect predictions are worth further exploration, and we leave this future work.
10

---

References
[1] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalimi De Mello, Tero Karras, and Gordon Wetzstein. Geavs: Generative novel view synthesis with 3d-aware diffusion models, 2023.
[2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arxiv:/5/..000/1 2015.
[3] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 14124-14133, 2021.
[4] Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, and Kwan- See K. Wong. Seps-net: Self- calibrating deep photometric stereo networks. In CVPR, 2019.
[5] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). October 2023.
[6] Zilong Chen, Feng Wang, Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting, 2024.
[7] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang Yan Gui. Sdfusion: Multimodal 3d shape completion. reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[8] Maut Deitke, Ruoshi Liu, Matthe Wallingford, Huong Ngo. Oscar Michel. Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects, 2023.
[9] Matt Deitke, Dustin Schwenk, Jordi Salvador, Laca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe annotated 3d objects. 2022.
[10] Congyue Deng, Chiyu Jiang, Charles R Qi, Xinehen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20637- 20647, 2023.
[11] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[12] Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang. Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing diffusion priors for 3d geometry estimation from single image. arXiv preprint 2403::2403. 12013, 2024.
[13] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin. Daiging Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841-31854, 2022.
[14] Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole". Cat3d: Create anything in 3d with multi-view diffusion models. arxiv, 2024.
[15] Zexin He and Tengfei Wang. OpenIrm: Open-source large reconstruction models. https://github com/ wwDDDppap/ppellllliii 2023.
[16] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.044000 2023.
[17] Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, and Shubham Tulsiani, Mvd-fusion: Single-view 3d via depth-consistent multi-view generation. In CVPR, 2024.

---

[18] Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu. Hooman Shayani. and Chi-Wing Fu. Make-a-shape: ten-million-scale 3d shape model. In Forty-first International Conference on Machine Learning, 2024.
[19] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18365-18375, 2022.
[20] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions, 2023.
[21] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra. Holodiffusion: Training a 3d diffusion model using 2d images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18423 18433, 2023.
[22] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[23] Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew Davison. Eschernet: generative model for scalable view synthesis. arXiv preprint arXiv:2402.03908 2024.
[24] Jonáš Kulhánek, Erik Demer, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision, pages 198-216. Springer, 2022.
[25] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for nigh-peeformmnce differentiable rendering. ACM Transactions on Graphics, 39(6), 2020.
[26] Han-Hung Lee and Angel X. Chang. Understanding pure clip guidance for voxel grid nerf models, 2022.
[27] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprini vrXiv2231.002114 2023.
[28] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. 1axiv:23/00025000 2023.
[29] Yuhan Li, Yishun Dou, Xuanhong Chen, Bingbing Ni, Yilin Sun, Yutian Liu, and Fuzhen Wang. Gen- cralized deep 3d shape prior via part- distriiscrttized diffusion process. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16784-16794, 2023.
[30] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[31] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. 2-3-5-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv prepriut arXiv:231 1.00785, 2023.
[32] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[33] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298 9309, 2023.
[34] Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consistentt images from single-view image. arXiv preprint arXiv:2309.03453 2023.
[35] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Thcobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion- aware mmage-based rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7824-7833, 2022.
[36] Yuxin Liu, Minshan Xie, Hanyuan Liu, and Tien-Tsin Wong. Text-guided texturing by synchronized multi-view diffusion. arXiv preprint arXiv:2311. 2023.
[37] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using gross-domain diffusion. arXiv preprint vxXiv:2310.55000 2023.
12

---

[38] Xiaoxiao Long, Cheng Lin, Peng Wang. Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference Computer Vision, pages 210-227. Springer, 2022.
[39] William E. Lorensen and Harvey E. Cline. Marching cubes: a high resolution 3D surface construction algorithm, page 347-353. Association for Computing Machinery, New York, NY, USA, 1998.
[40] J. Lorraine, K. Xie, X. Zeng. C. Lin, T. Takikawa, N. Sharp. T. Lin, M. Liu, S. Fidler, and J. Lucas. Att3d: Amortized text-to-3d ohject synthesis. In 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pages 17900-17910, Los Alamitos, CA, USA, oct 2023, TEEE Computer Society.
[41] Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2.5: Diverse text-to-3d generation via multi-viww 2.5d diffusion, 2024.
[42] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[43] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[44] Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi. Efficiently combining positions and normals for precise 3d geometry. ACM Trans. Graph., 24(3):536-533, jul 2005.
[45] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts, 2022.
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michacl Rabbat, Vasu Sharma, Gabriel Synnacve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[47] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[48] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Skorokhadov, Peter Wonka, Sergey Tulyakov, and Bernadd Ghanem. Magic123: One image to high-quality object generation using both 2d and 3d diffusion priors. The Twelfih International Conference on Learning Reprexemtations (ICLR), 2024.
Hsin-Ying Ying Lee, Ivan
[49] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness text-to-3d. 3d, 2023.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[51] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. shro text-to-image generation, 2021.
[52] Konstantinos Rematas, Ricardo Martin-Brualle and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from a single view. arXiv preprint wr.Xiv:2102.88850 2021.
[53] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng. Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4209 2024.
[54] Yufan Ren, Tong Zhang, Marc Pollefeys, Sabine Süsstrunk, and Fangjinhua Wang. Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16685-16699 16695, 2023.
[55] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674- 10685, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society.
13

---

[56] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-lo-image diffusion models with deep language understanding, 2022.
[57] Johannes I. Schonberger and Jan-Michael Frahm. Structurefrrommmttion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[58] Junyoung Seo, Wooseok Jang. Min-Seop Kwak, Jachoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07990 2023.
[59] Ruoxi Shi, Hansheng Chen. Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. 2erol23++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15100 2023.
[60] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. arXiv:2308. 165/2, 2023.
[61] Robust Multiview Stereopsis. Accurate, dense, and robust multiview stereopsis. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 32(8), 2010.
[62] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang. Wen Liu, Zhenda Xie, and Yebin Liu. Dreameraft3d: Hierarchical 3d generation with bootstrapped diffusion prior, 2023.
[63] Haotian Tang. Shang Yang. Zhijian Liu, Hong. Zhongming Yu, Xiuyu Li, Guohao Dai, Yu Wang, and Song Han. Torchsparse*+: Efficient training and inference framework for sparse convolution on gpus. In IEED/ACM International Symposium on Microarchitecture (MICRO), 2023.
[64] Jiaxiang Tang. Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high nigh-resolution 3d content creation. arXiv preprint (xXi:22422.05044 2024.
[65] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023.
[66] Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Joshua B Tenenbaum, Frédo Durand, William T Freeman, and Vincent Sitzmann. Diffussio with forward models: Solving stochastic inverse problems without direct supervision. arXiv preprint vrXiv:2306.11719 2023.
[67] Dmitry Tochilkin, David Pankratz, Zexiang Liu. Zixuan Huang, Adam Letts. Yangguang Li, Ding Liang, Christian Laforte. Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image, 2024.
[68] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 15182 15192, 2021.
[69] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitrii Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from single image using latent video diffusion. arXiv, 2024.
[70] Haochen Wang, Xiaodan Du, Jiahao Li. Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for generation. arXiv preprint arXiv:2212.00774. 2022.
[71] Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, et al. attention all nerf needs? arXiv preprint arXiv:2207. 13299,,, 2022.
[72] Peng Wang and Yichun Shi. Imagedream: Image- prompt multi-viww diffusion for 3d generation. arXiv preprint or Xi::2112.022000 2023.
[73] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint 1rXiv22311.12024 2023.
[74] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo
Noah Snavely, and Thomas Funkhouser. Ibmet: Learning multi-view view image- based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 4699, 2021.
14

---

[75] Zhen Wang, Qiangeng Xu, Feitong Tan, Menglei Chai, Shichen Liu, Rohit Pandey, Sean Fanello, Achuta Kadambi, and Yinda Zhang. Mvdd: Multi-view depth diffusion models, 2023.
[76] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023.
[77] Zhengyi Wang. Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint 4rX1::2033.5034, 2024.
[78] Xinyue Wei, Fanbo Xiang, Sai Bi, Anpei Chen, Kalyan Sunkavalli, Zexiang Xu, and Hao Su. NeuManifold: Neural Watertight Manifold Reconstruction with Efficient and High- Quality Rendering Support. arXiv preprint, 2023.
[79] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshimm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
[80] Haohan Weng. Tianyu Yang, Jianan Wang, Yu Li, Tong Zhang, C. L. Philip Chen, and Lei Zhang. Consistent123: Improve consistency for one image to 3d object synthesis, 2023.
[81] Sangmin Woo, Byeongjun Park, Hyojun Go, Jin- Young Kim, and Changick Kim. Harmonyview: Harmo- nizing consistency and diversity in one-image-to-36 3d, 2023.
[82] Robert J. Woodham. Photometric method for determining surface arientation from multiple images, page 513-531. MIT Press, Cambridge, MA. USA, 1989.
[83] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 803 814, 2023.
[84] Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, and Xiaohui Zeng. Latte3d: Large-scale amortized text-to-enhanced3d synthesis. arXiv preprint arXiv:2403.1553 2024.
[85] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient mesh generation from single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.0719/ 2024.
[86] Yinghao Xu, Zifan Shi, Wang Yifan. Sida Peng, Ceyuan Yang, Yujun Shen, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arxiv: 2403. 14621, 2024.
[87] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein. Zexiang Xu, et al. Dmv3d: Demoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.00 190217,, 2023.
[88] Hao Yang, Lanqing Hong, Aoxue Li, Tianyang Hu, Zhenguo Li, Gim Hee Lee, and Liwei Wang. Contranerf: Generalizable neural radiance fields for synthetic- cecheetcc-t--reee ro-l novel view synthesis via contrastive learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16508- 16517, 2023.
[89] Jianglong Ye, Peng Wang. Kejie Li, Yichun Shi, and Heng Wang. Consistent- 1-t--3: Consistent image to 3d view synthesis via geometry- aware diffusion models. arXiv preprint arXiv:2310 03000, 2023.
[90] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578 4587, 2021.
[91] Wangbo Yu, Li Yuan, Yan-Pei Cao, Xiangjun Gao, Xiaoyu Li, Wenbo Hu, Long Quan, Ying Shan, and Yonghong Tian. Hifi-123: Towards high-fidelity one image to 3d content generation, 2024.
[92] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Namxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv, 2024.
[93] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusson models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3841 2023.
15

---

[94] Xin- Yang Zheng. Hao Pan. Yu-Xiao Guo. Xin Tong, and Yang Liu. Mvd?: Efficient multiview 3d reconstruction for multiview diffusion. In SIGGRAPH, 2024.
[95] Xin-Yang Zheng. Hao Pan, Peng-Shuai Wang. Xin Tong. Yang Liu, and Heung- Shum. Locally attentional sdf diffuusin for controllable 3d shape generation. arXiv preprint vrXiv:2305.044611 2023.
[96] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers, 2023.
16

---

A Appendix
A.1
Comparison with Instant3D
In Figure 6 showcase the comparison with Instant3D (27] the text-to-3D task. The results are obtained from the paper authors. While Instant3D [27] also generates 3D shapes that match the input text prompt, our method generates results with superior mesh quality and tne-grained, sharp geometric details.
Instant3D
Ours
crupting volcano, acrial view R:a beautifully carved wooden knight chess piece
L: DSLR photo of human skull R: a freshly baked loaf of sourdough bread on a cutting board
L: golden goblet, low poly R: chimpanzce wearing headphones
Instamma D
Ours
pyramid shaped burrito with slice R: red panda
L: metal sculpture lion's head, highly detailed R: DSLR photo of delicious croissant
L: rabbit, animated movie character, high detail 3d model R: DSLR photo of frog wearing sweater
Figure 6: Application: Text-to-3D. Comparison with Instant3D 27
A.2 Triplane Artifacts
MeshLRM
Ours
$98
PON CREATINE
PREMIUM
CREATINE
TYNLAS
TwINAAB PREMELUM FUEL
The Berr Eegiii
CREATIIA STATI FUEL
The Power Treatments
Figure 7: The triplane- triplane-based method MeshLRM [79 has difficulty capturing words on objects, even when ground truth multi-view RGB images are used as input.
17

---

normal
w/ GT normal w/ predicted normal
Figure 8: Ablation study on input normal maps. Evaluated on the GSO dataset "w/o normal" indicates that the model is trained with multi-view RGB images only. "w/ predicted normal" indicates that the model is trained with ground truth normal maps but evaluated with predicted normals by Zero 123++ [59]. "w/ GT normal" indicates that the model is trained and tested with ground truth normals.
Table 4: Normal consistency (angle error) between the mesh geometry (mesh vertex normals) and the predicted normal maps, both before and after the geometry enhancement post-processing. The ratio of mesh vertices below specific error threshold shown. Evaluated on the GSO dataset.
angle error threshold
before
after
1
8.83%
16.27%
2
26.39%
40.83%
5
60.55%
73.19%
10a
78.79%
86.43%
< 15a
86.46%
91.29%
As shown in Fig MeshLRM 79 has difficulty capturing words objects, even when ground truth multi-view RGB images are used as input. We speculate that this is due to the limited number of triplane patches (e.g., 32 x 32 restricted by global attention. In contrast, our method leverages sparse voxels and supports much higher feature resolution of 2563, making it free from such issues.
A.3 Ablation Study: Input Normal Maps
In Figure g we qualitatively demonstrate the effect of input normal maps. When the model is trained without multi-view normal maps, we find that the generated model can only capture the global 3D shape but fails to generate fine-grained geometric details. However, when the model is given predicted normal maps, the performance is significantly better, although there are still some small gaps when compared the results of ground truth normals (see the bread hole of the toaster and the wheel of the tram). This indicates errors inconsistencies from the 2D normal prediction models.
A.4 Ablation Study: Geometry Enhancement
We propose asking the network to predict an additional normal texture, which can used for further geometric enhancement by applying traditional algorithm post-processing. The geometric enhancement aims to align the mesh geometry with the predicted normal map by adjusting the vertex locations. However, the traditional algorithm used cannot guarantee that the mesh normals will
18

---

Table Analysis of our mesh generation quality over training time. Evaluated on the GSO
dataset.
Training Time
f LPIPS-C PSNR-C1
PSNR-N
LPIPS-N +
CD T
F-Score
8xH100 12h
21.28
0,2135
22.89
0.1536
0.0330
0,960
8xH100 24h
21.32
0.2076
22.96
0.1516
0,0320
0,960
8xH100 48h
21.41
0.2033
23.01
0.1484
0,0317
0,960
8xH100 120h
21.44
0.2029
23.04
0.1480
0,0314
0.961
8xH100 168h
21.47
0.2010
23.09
0.1466
0.0313
0.963
be fully aligned with the predicted normal maps after processing. This limitation arises because the algorithm operates in local space and avoids large vertex displacements. Moreover, the predicted normal maps may contain errors or inconsistencies, such as conflicting neighboring normals. The adopted algorithm is iterative numerical optimization method and does not compute an analytic solution.
However, have quantitatively verified that the post-processing module can significantly improve normal consistency with the predicted normal map. For example, before post- post-processing, only 26.4% of mesh vertices had normal angle error of less than 2 degrees. After post-processing, this number increased to 40.8%. For a 10-degree threshold, the ratio increases from 78.8% to 86.4%. For more details, please refer Table
A.5 Ablation Study: Training time
Our MeshFormer can be trained efficiently using only 8 GPUs, typically converging in approximately two days. [ablee[preeentts a quantitative analysis of our mesh generation quality over the training period. We observe that performance improves rapidly and nearly converges, with only marginal changes occurring after the two-day training period.
A.6 Training Details and Evaluation Metrics
Training Details: We trained the model using a subset of 395k 3D shapes filtered from the Obja- verse [9] dataset. These objects have distributable Creative Commons license and were obtained by the Objaverse team using Sketchfab's public API. For each filtered 3D shape, randomly rotated the mesh and generated 10 data samples. For each data sample, we compute a 5123 ground truth SDF volume using CUDA program and render multi-view RGB and normal images using BlenderProc. In our experiments, the resolutions of the occupancy volume and sparse feature volume 64 and 256, respectively. The resolution of the predicted and ground truth SDF volumes is 512. The model is trained with the Adam optimizer and a cosine learning rate scheduler. The loss weights \1, Ag are set to 80, 2, 16, 2, 8, and 8, respectively.
All data preparation, including image rendering and SDF computation, performed using an internal cluster. This process be completed using 4000 CPU cores in roughly one week. The generated data takes up approximately 30TB. All model training tasks are conducted in public cloud clusters. Our main model is trained using 8 H100 GPUs for week. All experiments listed in the paper can be completed in 15 days using 32 H100 GPUs (running multiple parallel experiments), excluding the preliminary exploration experiments.
For VoxelFormer, the UNet consists of four levels with resolutions of 643, 32 163 and 16 Each level includes ResNet module, projection-awann cross-attention module, and downsampling module, with channel sizes of 64, 128, 256, and 512. We added transformer layers at the bottleneck of the UNet, with each 3D voxel treated as token, and token channels set to 512.
Architecture Details:
For Sparse VoxelFormer, the sparse UNet consists of six levels with resolutions 2563, 1283, 643. 323, 163, and 16 Each level includes sparse ResNet module, projection-awann module, and downsampling module, with channel sizes of 16, 32, 64, 128, 512, and 2,048. We added 16 transformer layers at the bottleneck of the UNet, with each 3D sparse voxel treated as token, and token channels set to 1,024. The feature dimension of the output sparse feature volume (before the MLP) is 32.
cross-attention
19

---

Imput
One-2-3-45++
CRM
Ours
Figure 9: Qualitative Results of One-2-3-45++ [31 and CRM 77 on Single Image to 3D. Both
the textured and textureless mesh renderings are shown.
For both of them, skip connection is added to the UNet.
Evaluation Metries:
To account for the scale and pose ambiguity of the generated mesh from different baselines, we align the predicted mesh with the ground truth mesh prior to the evaluation metric calculation. This alignment process involves uniformly sampling rotations and scales for initialization and subsequently refining the alignment using the Iterative Closest Point (ICP) algorithm. We select the alignment that yields the highest inlier ratio. Both the ground truth and predicted meshes are then scaled to within a unit bounding box.
For 3D metrics, sample 100,000 points both the ground truth mesh and the predicted mesh and compute the F-score and Chamfer distance, setting the F-score threshold at 0.05, To evaluate texture quality, compute the PSNR and LPIPS between images rendered from the reconstructed mesh and those the ground truth. Following InstantMesh [85), we sample 24 camera poses, encompassing full 360-degree view around the object, and utilize BlenderProc for rendering RGB and normal images with a resolution 320x320. Since we use the VGG model for LPIPS loss calculation during training. employ the Alex model for LPIPS loss calculation during evaluation.
A.7 Training Details of MeshLRM
results MeshI.MM, except those Table 2 were reproduced by the MeshI.Ry authors at Hillbot following the original settings described the paper. For the results in Table we trained the model using the same training data as our method on 8x H100 GPUs for 48 hours. maintained the same batch size as reported in the paper and proportionally scaled down the original training time for each stage of MeshLRM based on total training time of hours. This included 5.8 seconds per iteration for 20,000 iterations in the 256-resolution pre-training, 12 seconds per iteration for 4,000 iterations in the 512-resolution fine-tuning, and 4.7 seconds per iteration for 4,000 iterations mesh refinement.
A.8 Qualitative Examples of One-2-3-45++ and CRM
Figure shows qualitative results of One-2-3-45++ and CRM on single image to 3D and method produces better results.
20

---

A.9 Broader Impact
We introduce an efficient approach for training open-wonee world sparse-view reconstruction models, which has the potential to significantly reduce energy consumption and carbon emissions, as baseline models typically require much more computing resources for training. Previously, the creation of 3D assets was reserved for specialized artists who spent hours or even days producing a single 3D model. Our proposed technique allows even novice individuals without specialized 3D modeling knowledge to create high-quality 3D assets in seconds. This democratization of 3D modeling has unleashed unprecedented creative potential and operational efficiency across various sectors.
However, like other generative Al models, it also carries the risk of misuse, such as spreading misinformation and creating pornography models. Therefore, it crucial to implement strict ethical guidelines to mitigate these risks.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe so. We provide overview of our contributions at the end of the introduction (seefsection 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly discuss limitations of our work in section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We do not present theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide extensive details about the implementation of our method, base- lines, and evaluation subsection 4.1 (subsection A.6 and ssubsection A.7 to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and model release have not yet passed our internal inspection, and the model also needs safety evaluation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aablic/guudees/Coeeebbbmsss cosssiiinplyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuiiaeeeaec
Stteeee//ooSSSSeesSSioeaa www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have included all details lubsectton 4.1 and subsection A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are reported because would be too computationally expensive to train the models multiple times. Additionally, the computation of evaluation metrics does not involve significant randomness or variation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include details about computing resource in subsection 4.1 and subsection A.7
subsection A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwaaaapcccpppaaaaaaaatat
Answer: [Yes]
Justification: The authors have read the NeurIPS Code of Ethics and ensured that research conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our method in subsection A.9
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: As discussed in ssbsecctioon A.9 our model carries the risk of misuse, including the spread of misinformation and the creation of porographic content. We will conduct an internal safety inspection and implement necessary safeguards before releasing the model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the original papers that produced the code and dataset used in paper, and have properly respected all licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "W4hA3ADSDk", "reviewer": "Reviewer_r7MY", "summary": "In this work, the authors propose MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. They leverage 3D sparse voxels as their representation and combine transformers with 3D (sparse) convolutions to inject 3D prior. Additionally, they propose to take the corresponding normal maps together with sparse-view RGBs as input and also generate them as output, which could be used for geometry enhancement. Extensive experiments show that MeshFormer can be trained efficiently and outperforms state-of-the-art methods in terms of generating high-quality textured meshes.", "questions": "- Do you use a shared backbone (trainable DINOv2) for both RGB and normal images? Do you use Plücker embedding here? \n\n- Could you provide a more detailed description for the Sparse VoxelFormer architecture? For example, how many sparse convolution layers are used in each resolution?\n\n- Instead of joint training, have you tried splitting the dense model and sparse model for two-stage training?\n\n- The output voxel resolution is $256^3$, while the SDF supervision is $512^3$. I notice that there is an interpolation step in Figure 2. It would be better to add a short text description for this.\n\n- Do you use GT multi-view normals for the teaser? If you use the GT normal images, please include that in the caption.\n\n- I suggest discussing XCube [a] in your literature review. XCube also utilizes sparse voxels as their 3D representation and leverages 3D sparse UNet with transformer layers. Additionally, they generate 3D shapes in a coarse-to-fine manner and use tiny MLPs to predict various attributes, such as normals, semantics, and SDF.\n\n[a] XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. CVPR 2024.", "limitations": "The authors already include limitations and broader impact in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- MeshFormer is able to generate high-quality textured meshes with fine-grained geometric details.\n\n- The authors find that using normal images together with RGB images greatly helps in predicting geometric details. Additionally, the model outputs a normal map, which can be used for geometry enhancement.\n\n- The proposed method explicitly leverages 3D native structure, input guidance, and training supervision, resulting in faster convergence speed and better geometric details.", "weaknesses": "- Pixel-based 2D methods (e.g., LGM) can preserve thin details, while 3D-based methods often smooth these details. How do you justify that? For example, in Figure 3 Column 4, the loose thread of the toy is captured by LGM, while MeshFormer ignores it.\n\n- The proposed name \"VoxelFormer\" seems improper to me. It seems more like a 3D UNet with a deep bottleneck composed of multiple transformer layers.\n\n- The projection-aware cross-attention layer projects 3D voxels onto the m views to interpolate m RGB and normal features. However, in the object case, one 3D voxel usually only corresponds to one view (due to occlusion). This cross-attention is projection-aware but not truly 3D-aware. Have you tried some occlusion-aware attention in your sparse model? Since you already have the coarse structure of the object, it could be used to filter out unneeded features.\n\n- According to Table 3 (d), you mention \"we replace the cross-attention with simple average pooling and observe a significant performance drop.\" Could you also try max-pooling? Additionally, do you concatenate the 3D feature voxel at every level of the network, as done in One-2-3-45++?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thin structures\nWe would like to clarify that the loose thread of the toy was not displayed due to a slight pose mismatch when visualizing the results. In fact, the loose thread is reconstructed by our MeshFormer. We have included additional views of our generated results (see Figure 1 of the rebuttal PDF), where the loose thread can be observed. You can check the video on our website.\n\nMeshFormer generates high-resolution (512) SDF volumes, which are sufficient to preserve very thin structures, as shown in Figure 2 of the rebuttal PDF.\n\n## The name \"VoxelFormer\"\nThe meaning of \"VoxelFormer\" is two-fold. On the one hand, it refers to combining voxel representation and transformer layers, in contrast to recent LRM-based methods that rely on triplane representation and transformers to achieve scalability. On the other hand, it can also be interpreted as Voxel-Form-er, meaning the module that generates the voxels. We are open and happy to consider other names if the reviewer has more suitable suggestions.\n\n## Occlusion-aware feature aggregation\nWe agree that occlusion-aware feature aggregation is very important. This is the primary reason we use a cross-attention layer to aggregate the projected multi-view features instead of using a simple average or max pooling method like previous approaches. We hope the cross-attention layer can implicitly utilize prior knowledge of coarse structure and visibility to focus on the visible views. Our experiments also verify its superiority over mean pooling aggregation (Table 3 (d) of the paper). While we could explicitly filter out some views according to the predicted occupancy from the first stage, as the reviewer suggested, we would like to point out that the occupancy band has some thickness, and accurately determining the visibility of each voxel can be quite challenging. This may require more advanced techniques, and we leave this as a promising future direction.\n\n## Table 3 (d)\nWe followed the suggestion to add an ablation variant using the max-pooling aggregation mechanism (please refer to Table 1 of the rebuttal PDF). We found that while the max-pooling aggregation performs slightly better than average pooling, it is still significantly inferior to our projection-aware cross-attention mechanism.\n\nYes, we follow the \"skip-connection\" scheme of the typical UNet to concatenate the voxel features before the bottleneck with the voxel features after the bottleneck. If this is not what you are asking, please let us know.\n\n## Shared backbone and Plücker embedding\nYes, we use a shared backbone (trainable DINOv2) for both RGB and normal images. We did not use Plücker embedding in our experiments. Instead, we leveraged the camera poses to unproject 2D image features into 3D feature volumes.\n\n## Detailed description of the sparse VoxelFormer architecture\nYes, we will follow your suggestion to include more details about the Sparse VoxelFormer architecture in our revision, such as the number of sparse convolution layers used at each resolution.\n\n## Joint training\nYes, we began our experiments by training the dense model and sparse model separately. However, we found that this approach leads to a domain gap for the occupancy field during inference, as the predicted occupancy by the dense model may be imperfect, while the sparse model is only trained with ground truth occupancy. Joint training can mitigate this gap and reduce artifacts during inference.\n\n## Interpolation from $256^3$ to $512^3$\nThank you for pointing this out. We will add the missing description in our revision. Specifically, we generate a sparse feature volume with a resolution of 256, and then trilinearly interpolate it to a sparse feature volume with a resolution of 512. This interpolated sparse features are then fed into the SDF decoder for predicting SDF values, which are subsequently used to compute the loss against the 512-resolution ground truth SDF.\n\n## Multi-view normals for the teaser\nYes, we use GT multi-view normals for the teaser. We will make this clearer in our revision.\n\n## XCube\nThank you for pointing this out. We will cite XCube and discuss it in our revision. We agree that XCube shares many high-level ideas with us, such as hierarchical sparse voxel representation and coarse-to-fine generation. However, we would like to point out that they follow the paradigm of the 3D native diffusion, which can only generate geometry but fails to directly predict texture from the model. In contrast, we follow the paradigm of the feed-forward sparse-view reconstruction and incorporate differentiable rendering into the pipeline, which enables the network to directly generate high-quality texture. Additionally, we combine 3D (sparse) convolution with the transformer layer to increase capacity and scalability of the network, while they mainly rely on 3D convolution only and may be limited in both capacity and scalability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dp7NSayvyW", "reviewer": "Reviewer_V11k", "summary": "This paper proposes an improved framework for feed-forward reconstruction models. The authors advocate a number of improvements over the initial design of Large Reconstruction Model, including model architecture and training schemes. Experiments show that the method reconstructs better geometry and texture on Google Scanned Objects and OmniObject3D datasets.", "questions": "Please see the questions in the weakness section.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is focused on ablating different components for feed-forward sparse-view reconstruction, and in-depth analyses are provided for each design choice. Although there are no complicated new method proposed, such analysis bring value for understanding how and why each component works.\n- The proposed method is evaluated on (preprocessed) real-world multi-view datasets, showing improvements over baselines on all metrics. Extensive ablative analyses are also provided to better understand the behaviors of the proposed method.", "weaknesses": "- Since this is more of an analysis paper, it would be good if the authors could also document the other components that were tried/ablated but did not see significant differences.\n- Since training resources was discussed and compared, it would be nice if there could be an analysis on the mesh generation/reconstruction quality over training time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Experiments tried/ablated but did not show significant differences\nWe are happy to follow the reviewer's suggestions to include more discussions about the experiments we have conducted in our revision, such as:\n- the difference between joint training and separate training of the dense model and sparse model;  \n- the difference between max-pooling, mean-pooling, and cross-attention in projection-aware feature aggregation;  \n- the comparison of inference time and memory consumption with baseline methods;  \n- mesh generation quality over training time;  \n- quantitative analysis of the geometry enhancement module;  \n- more qualitative examples on real-world images.  \n\nPlease let us know if there are any additional experiments you are interested in.\n\n\n## Mesh generation/reconstruction quality over training time\nOur MeshFormer can be trained efficiently with only 8 GPUs, generally converging in roughly two days. We have followed the suggestion to include a quantitative analysis of our mesh generation quality over training time. As shown in Table 2 of the rebuttal PDF, the performance quickly improves and nearly converges with marginal changes after the two-day training period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U9Seowt9eX", "reviewer": "Reviewer_mWQL", "summary": "In this work, the authors propose a sparse view reconstruction model that utilizes a set of images (with camera poses) and corresponding normal maps to produce a reconstructed textured mesh. The primary contribution lies in adopting voxel-based 3D representation and employing a network architecture that integrates both 3D convolution and attention layers. Moreover, direct geometry supervision (SDF loss) is applied during the training process, alongside rendering-based losses. Experimental results demonstrate that the generated 3D shapes achieve state-of-the-art performance when compared to existing works on the single-view to 3D task.\n\nHowever, as highlighted in the weakness section, there are potential misclaims regarding the technical contributions. It is highly recommended to revise the manuscript to cite and discuss these related works. Despite this, I am currently inclined towards accepting the paper and would be happy to champion it if the aforementioned issues are addressed in the final version.", "questions": "Some suggestions: \n- Considering the above existing and concurrent works (Weakness Section), it is difficult to be convinced that some of the proposed modules are novel. It is highly recommended to cite and discuss the differences with these prior works and adjust the claims accordingly.\n- Although it is acknowledged in the limitation section that the reconstruction performance will be affected by the errors of 2D models, it is recommended to include this as one of ablation case in Table 3 to better visualize this limitation.\n- Furthermore, as no real-world images have been tested within the proposed framework, it is advisable to avoid from using the term \"open-world\" (L384) to describe the current framework in order to prevent overclaims.", "limitations": "The main limitation is well described in Section 5.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The writing is clear and easy to follow.\n- The combination of SDF loss and rendering losses appears novel for training a feed-forward based network. Additionally, the ablation study in Table 3(b) clearly indicates that SDF supervision is crucial for achieving good geometry, as evidenced by the significant CD difference between (b) and (g).\n- Although [33] has explored using normal maps for the reconstruction task, it seems new to employ normal maps as inputs and supervision for a feed-forward reconstruction network.\n- Experimental results demonstrate state-of-the-art performance over existing baselines, as shown in Table 1 and Figure 3. Furthermore, it is illustrated that existing methods cannot achieve similar performance given the same computational resources (Table 2).\n- The ablation study confirms that various components are essential for the final performance, including considering normal input and SDF supervision.", "weaknesses": "Possibly Misclaimed Technical Novelties:\n\nHowever, the current manuscript may contain several misclaims regarding its technical novelties.\n\nOne claimed novelty is the adoption of a 3D voxel representation. However, the use of 3D voxel-like volumes in reconstruction is not a new idea and has been well-explored in various works, including:\n\nA. Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process, CVPR 2023\n\nB. SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation, CVPR 2023\n\nC. Locally Attentional SDF Diffusion for Controllable 3D Shape Generation, SIGGRAPH 2023\n\nD. One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion, CVPR 2024\n\nE. Make-A-Shape: a Ten-Million-scale 3D Shape Model, ICML 2024\n\nAdditionally, the use of convolution + transformer layers to process grid input seems to be standard procedure in 2D generation tasks, as seen in:\n\nDiffusion Models Beat GANs on Image Synthesis, NeurIPS 2021\n\nSimilar architectures have also been widely adopted in some of the aforementioned 3D reconstruction works, such as [A, C, D, E].\n\nRegarding image conditioning, the cross-attention with image patch features is also well-explored in various works mentioned above, such as [C, D, E].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# No real-world images tested?\nWe would like to clarify that one of our main testing datasets, OmniObject3D, is a real-world scanned 3D dataset. In addition, we also include some qualitative examples with real-world input in our rebuttal PDF (see Fig. 3), where MeshFormer performs quite well. \n\nThe term 'open-world' means that MeshFormer differs from many previous methods (such as A, B, C listed by the reviewer). Those methods are trained on datasets with a limited number of object categories (e.g., tens of categories in ShapeNet) and cannot generalize to novel categories. Unlike those methods (e.g., 3D native diffusion), MeshFormer takes as input sparse-view images and normal maps generated by 2D diffusion models, and demonstrates much stronger generalizability. MeshFormer is thus not limited to the training 3D dataset and can handle arbitrary object categories.\n\n# Ablation study of 2D model errors\nWe would like to clarify that, in Tab. 3 of the paper, we have analyzed the influence of errors from 2D normal models (rows f and g). Additionally, we provide some qualitative examples in Fig. 8.\n\nFor the effect of multi-view RGB, please compare Tab. 1, row ‘Ours’ (predicted RGB) and Tab. 3, row f (ground truth RGB) of the paper.\n\n# Claims about technical novelties\nThank you for pointing this out. We will cite these prior works and discuss the differences in our revision. We fully understand your concerns and would like to address the potential misunderstanding in detail.\n\n## Point 1\n\n The reviewer summarized that the \"primary contribution lies in adopting a voxel-based 3D representation and employing a network architecture ….\" **We respectfully disagree with this argument.** Our main claim is that by proposing a 3D-guided reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision, MeshFormer can significantly improve both mesh quality and training efficiency. Our main findings/contributions include:\n\n- (a) Using normal images as additional input in feed-forward reconstruction models greatly enhances the prediction of geometric details.\n   \n- (b) Proposing to learn and output a 3D normal map, which can be used for further geometric detail enhancement.\n   \n- (c) Combining SDF loss and rendering losses in training feed-forward reconstruction models enables a unified single-stage training process. In contrast, concurrent works rely on complex multi-stage \"NeRF-to-Mesh\" training strategies to export high-quality meshes (e.g., MeshLRM, InstantMesh) and struggle to generate high-quality geometry.\n   \n- (d) Explicitly leveraging 3D native voxel representation, network architecture, and projective priors fosters faster convergence speeds and significantly reduces the training requirement. \n\n**We would like to emphasize that all four points are crucial to MeshFormer.  We thus do not agree that our primary contribution lies solely or primarily in adopting 3D representation and network architecture.** For example, without points (a), (b), and (c), our mesh quality would be significantly compromised. \n\n## Point 2\nWe totally agree that the utilization of 3D voxel representation is common in general 3D generation. However, **all works listed by the reviewer (A-E) focus on 3D native diffusion, one of the paradigms in 3D generation, which differs from the route of MeshFormer.** There are some common limitations of this line of work. For instance, all of A-E focus on geometry generation only and cannot predict high-quality texture directly from the network. Also, due to the limited amount of 3D data, 3D native diffusion methods typically struggle with open-world capability and focus on closed-domain datasets (e.g., ShapeNet) in their experiments (A, B, C).\n\nIn MeshFormer, **we aim to achieve direct high-quality texture generation and handle arbitrary object categories**. We are thus following another route: sparse-view feed-forward reconstruction, instead of the 3D native diffusion. In this specific task setting, **many of the works provided by the reviewer are not suitable for comparison in our experiments**. More comparable works are recent LRM-style methods (e.g., InstantMesh, MeshLRM, LGM, TripoSR, etc). However, **most of them only utilize the combination of triplane representation and large-scale transformers**. \n\nIn our paper, we do not claim to be the first to use voxel representation in 3D generation. Instead, we would like to share our findings:\n\n- In this specific task setting (open-world sparse-view reconstruction with feed-forward textures), we are not limited to the triplane representation. 3D native structures (voxels), network architectures, and projective priors can facilitate more efficient training and significantly reduce the training resources required (from over one hundred GPUs to only 8 GPUs).\n- In this specific task setting, we require a scalable network to learn a lot of priors. However, not only triplane-based transformers can be scalable. When marrying the 3D convolution with transformer layers, it can also be scalable. \n- In addition to using 3D native representation and networks in 3D native diffusion, we can also combine them with differentiable rendering to train a feed-forward sparse-view reconstruction model with rendering losses.\n- For image conditioning, C and E only take a single image as a condition. D **first employs max pooling across multi-view features** and then uses cross-attention across the pooled multi-view features and the voxel feature. The max pooling is typically affected by occlusion (voxels are not visible in all views) and thus becomes less effective. Instead, we propose using cross-attention across all multi-view projected image features and voxel features, which can implicitly leverage structure and visibility priors to focus only on visible regions. We demonstrate that this strategy is more efficient than the pooling strategies used in D as shown in Tab. 3 of the paper and Tab. 1 of the rebuttal PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tt47afhkrP", "reviewer": "Reviewer_bHQc", "summary": "The paper proposes a high-quality feed-forward 3D object reconstruction method from sparse view RGB images. It uses an explicit voxel structure for better geometric inductive bias, auxiliary inputs such as 2D diffusion generated normal images and SDF representation for better geometric details, and an end-to-end trainable pipeline that eliminates the need for multi-stage refinement. The method gives high quality reconstruction results, especially in terms of fine-grained and smooth geometry.", "questions": "1. Table 3 row (a) shows the impact of normal input. When you remove the normal input, do you also remove the normal output and the normal loss? I ask this because in section 3.3 you say learning from RGB to geometric details directly can be difficult, so it makes more sense to just remove the normal input but preserve normal supervision to compare.", "limitations": "1. It requires 2D diffusion models to generate auxiliary inputs, which can drastically slow down the reconstruction speed.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although the network architecture and 3D representations are more complicated than previous methods, they are end-to-end differentiable and alleviate the training burden of multi-stage refinement.\n2. The idea of using 2D diffusion generated normal images as input to the reconstruction pipeline is interesting and insightful.\n3. It is more computationally efficient to train (Line 73).\n4. The qualitative results are impressive, especially the mesh normals.", "weaknesses": "1. In original LRM the only supervision signal needed is RGB images. The proposed method, however, needs access to the full 3D shape for supervising the occupancy. It is fine for hand-made 3D assets but might poses some difficulty when trying to scale to real datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Geometry supervision for real-world training datasets\nWe agree that image supervision is easier to add when extending to real-world training datasets. However, it is not impossible to obtain corresponding depth maps and even meshes for real-world RGB images, such as through depth sensors or Structure from Motion (SfM). Given the depth map (and even meshes), we can still apply direct 3D geometry supervision. If full 3D shapes are not captured, we can also apply partial supervision to the visible views (only supervising the visible points) while generating the full shape. We acknowledge that this may require more advanced techniques and designs, and we leave it as a promising avenue for future work.\n\n## Table 3, row (a):\nYes, for Table 3, row (a) of the paper, we remove the normal input but preserve the normal output and normal supervision.\n\n## 2D diffusion models drastically slow down the reconstruction speed\nWe currently use the normal ControlNet of Zero123++ to generate the multi-view normal inputs. It tiles the multi-view RGB images as a single input condition and generates the tiled normal maps in a single diffusion process, which takes approximately 4.1 seconds on an H100 GPU. For the application of single-image to 3D, generating tiled multi-view RGB images takes about 3.6 seconds. The total time on the 2D diffusion model side is only around 7.7 seconds, which is still acceptable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9U13nRPH0n", "reviewer": "Reviewer_3423", "summary": "This paper introduces MeshFormer, a sparse-view reconstruction model designed to generate high-quality 3D textured meshes from sparse RGB images and their corresponding normal maps. By leveraging voxel representation, 3D inductive biases, SDF loss, and normal information, the model shows comparable inference performance to concurrent methods, while the entire training process can be completed using only 8 GPUs within a week (concurrent methods typically require around 100 GPUs). Experimental results demonstrate the effectiveness of the design.", "questions": "1. Can the normal maps of the mesh be completely consistent with the normal maps predicted by the model after the post-processing algorithm?", "limitations": "Yes, the authors addressed limitations, potential negative societal impact, and mitigation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors provided a detailed explanation of the motivations behind the model designs (including the introduction of voxel representation, the introduction of 3D full (or sparse) convolution, and so on) and demonstrated the reasonableness of these choices.\n\n2. Compared to baseline methods, this model is simpler to train and demonstrates better qualitative and quantitative results. \n\n3. The ablation study demonstrates the effectiveness of normal input, SDF supervision, geometry enhancement, and other methods proposed in the paper.", "weaknesses": "1. Although the authors provide detailed textual descriptions in the method section, it would be better if more mathematical symbols and equations were used, which could explain the entire pipeline more clearly and unambiguously.\n\n2. For reproducibility, the authors should provide more implementation details, including a more detailed model architecture, the values of hyperparameters (e.g., \\lambda in the loss function), and other relevant information.\n\n3. The authors don’t report the comparison of inference time and memory usage between the proposed model and the baseline models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## More mathematical symbols and equations\nThank you for pointing this out. We will follow your suggestion to include more mathematical symbols and equations in our revision when introducing the method.\n\n## More implementation details\nWe will follow the suggestion to include more implementation details in our revision, such as the specifics of the network architecture and the weights of the loss terms.\n\n## Inference time and memory usage\nWe followed your suggestion to include a comparison of inference time and memory usage in the rebuttal PDF (please refer to Table 3). We found that while our method generates meshes at the highest resolution (512^3, 8x larger than 256^3), we still maintain a competitive speed and memory footprint.\n\n\n## Normal maps of the mesh (after post-processing) vs. normal maps predicted by the model\nUnfortunately, the post-processing we used cannot guarantee that the mesh normals will be completely aligned with the predicted normal maps after processing. This is because the algorithm operates in local space and avoids large vertex movements. Additionally, the predicted normal maps may contain errors or conflicts, such as inconsistent neighboring normals, which cannot be perfectly matched. The adopted algorithm is an iterative numerical optimization method and does not compute an analytic solution.\n\nHowever, we have quantitatively verified that the post-processing module can significantly improve mesh normal consistency with the predicted normal map. For example, before post-processing, only 26.4% of mesh vertices had a normal angle error of less than 2 degrees. After post-processing, this number increased to 40.8%. For a 10-degree threshold, the ratio increases from 78.8% to 86.4%. For more details, please refer to Table 4 in the rebuttal PDF. \n\nWe have also included qualitative examples to illustrate the importance of this post-processing module in recovering sharp geometric details and reducing noisy artifacts induced by the marching cubes algorithm. Please check out Figure 5 of the original paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful comments and valuable suggestions. We are pleased to note that all five reviewers were supportive of our work:

- They complimented the impressive mesh quality with fine-grained geometric details (r7MY, bHQc, 3423, mWQL, V11k).
- They praised our fast training speed and significantly reduced computational resources (r7MY, bHQc, 3423, mWQL).
- They noted that our method is well-motivated (3423) and the paper is well-written (mWQL).
- They highlighted our qualitative state-of-the-art performance (r7MY, mWQL, V11k, 3423) and insightful and extensive ablation study (mWQL, V11k, 3423).
- They acknowledged the novelty and benefits of our concrete findings/contributions:
  - Using normal images as additional input for feed-forward models greatly helps predict geometric details. (r7MY, mWQL, bHQc, 3423).
  - Proposing to output a normal map, which can be used for geometry enhancement (r7MY, 3423).
  - The combination of SDF loss and rendering losses enables unified single-stage training and achieves good geometry (r7MY, mWQL, bHQc, 3423).
  - Explicitly leveraging 3D native structures and projective priors fosters faster convergence speed (r7MY, bHQc, 3423).

We have also included a PDF with some figures and tables to address the specific concerns raised by the reviewers.

Meta-review: Post-rebuttal, the paper is a clear accept with all five reviewers scoring the paper as an Accept or higher. Reviewers praised the performance, novelty, and efficiency of the proposed end2end design.

In their rebuttal, authors have promised to include certain additional analyses, such as: 
- max-pooling vs. attention-based multi-view attention ablation (r7MY)
- additional analysis on which ideas did not work (V11k)
- the performance of the model as a function of the training time (V11k)
- better exposition using math with more implementation details (3423)
- speed & memory consumption (3423).
Authors are strongly encouraged to inlude the latter in the final camera-ready version.
Decision: Accept (oral)