MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers
Authors: Yiwen Chen
Tong He
Di Huang
Weicai Ye
Sijin Chen
Jiaxiang Tang
Zhongang Cai
Lei Yang
Gang Yu
Guosheng Lin
Chi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Generation
Transformers
Sequence Learning
Primary area: generative models
TL;DR: Generate Artist-Created Mesh aligned to given shapes

Abstract: Recently, 3D assets created via reconstruction and generation have matched the quality of manually crafted assets, highlighting their potential for replacement. However, this potential is largely unrealized because these assets always need to be converted to meshes for 3D industry applications, and the meshes produced by current mesh extraction methods are significantly inferior to Artist-Created Meshes (AMs), i.e., meshes created by human artists. 

Specifically, current mesh extraction methods rely on dense faces and ignore geometric features, leading to inefficiencies, complicated post-processing, and lower representation quality.
To address these issues, we introduce MeshAnything, a model that treats mesh extraction as a generation problem, producing AMs aligned with specified shapes.

By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods, thereby enhancing their application across the 3D industry.
The architecture of MeshAnything comprises a VQ-VAE and a shape-conditioned decoder-only transformer. We first learn a mesh vocabulary using the VQ-VAE, then train the shape-conditioned decoder-only transformer on this vocabulary for shape-conditioned autoregressive mesh generation. Our extensive experiments show that our method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies, while achieving precision comparable to previous methods.

Published as a conference paper at ICLR 2025
MESHANYTHING: ARTIST-CREATED MESH GENERA- TION WITH AUTOREGRESSIVE TRANSFORMERS
Yiwen Chen Tong He Di Huang', Weicai Ye?,
Sijin Chen';
Jiaxiang Tang"
Xin Chen', Zhongang Cai Lei Yang", Gang Yu',
Guosheng Lin't
Chi Zhang
SS-Lab, Nanyang Technological University
Shanghai AI Lab
Fudan University *Peking University Uniieessty of Chinese Academy of Sciences
TemsseTime Research "Stepfun
*Westlake University
https: htttps:/bbuacyye . github. io/mmesh-anything/
Dento Medl
Dana Mosk
Dane Maak
Chou
Figure MeshAnything converts different 3D representation into Artist-Creater Meshes (AMs), i.e., meshes created by human artists. It can be combined with various 3D asset pro- duction pipelines, such as 3D reconstruction and generation, to transform their results into AMs that be seamlessly applied in the 3D industry.
ABSTRACT
Recently, 3D assets created via reconstruction and generation have matched the quality manually crafted assets, highlighting their potential for replacement. However, this potential is largely unrealized because these assets always need to
WWrk done during a research internship at Shanghai Al Lab.
"ooresponning Authors.

---

Published as a conference paper at ICLR 2025
be converted to meshes for 3D industry applications, and the meshes produced by current mesh extraction methods are significantly inferior to Artist-Creand Create Meshes (AMs), i.., meshes created by human artists. Specifically, current mesh extraction methods rely on dense faces and ignore geometric features, leading to inefficiencies, complicated post-processing and lower representation quality. To address these issues, we introduce MeshAnything, a model that treats mesh ex- traction as generation problem, producing AMs aligned with specified shapes. By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods, thereby enhancing their application across the 3D industry. The architecture of MeshAnything comprises a VQ-VAE VAE and a shape-conditioned decoder-only transformer. We first learn a mesh vocabulary using the VQ- then train the shape-conditioned decoder- only transformer on this vocabulary for generation. Our extensive experiments show that method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies, while achieving precision comparable to previous meth- ods.
autoregressive mesh
INTRODUCTION
In recent years, the 3D community has experienced rapid advancements, with a variety of methods
developed for automatically producing high-quality 3D assets. These methods, including 3D recon-
struction (Mildenhall et al. 2020 Yu et al., 2021 Barron et al.) 2021; 2022 Kerbl et al. 2023b]
2023a Wang et al.) [023 Long Huang et al. (2024), 3D generation (Poole et al.) 2023 Liu et al.
et al. |023] Sun et al. 2023 Hong et al. (203)] Tang et al. [2024] Xu et al. [2024] Wei et al. 2024),
and scanning (Daneshmand et al.) 2018 Haleem & Javaid 2019 Haleem et al. 2002)) can produce
3D assets with shape and color quality comparable to manually created ones. The success of these methods reveals the potential replace manually created 3D models with automatically produced
ones in the 3D industry, including applications in games, movies, and the metaverse, significantly reducing time and labor costs.
However, this potential remains largely unrealized because the current 3D industry predominantly relies on mese- pipelines for their superior efficiency and controllability, while methods for producing 3D assets typically use alternative 3D representations to achieve optimal results across various scenarios. Therefore, substantial efforts (Lorensen & Cline 1987: Chernyaev 1995]
Lorensen & Cline, 1998 Shen et al.] 2021b Chen et al. 2022, Shen et al. (023) are devoted to converting other 3D representations into meshes and have achieved some success. Meshes produced by these methods approximate the shape quality of those created by human artists, which we refer to as Artist-Caa Creette Meshes (AMs), but they still fall short in addressing the aforementioned issues.
This is because all meshes produced by these methods (Lorensen & Cline 1987: Chernyaev 1995] Chen et al. |022 Shen et al.) (003) exhibit significantly poorer topology quality compared to AMs. As shown in Fig. these methods rely dense faces to reconstruct 3D) shapes, completely ignoring geometric characteristics. Using these meshes in the 31) industry leads three significant problems: First, converted meshes typically contain several orders of magnitude more faces compared to AMS, leading to significant inefficiencies in storage, render- ing, and simulation. Moreover, the converted meshes complicate post-processing and downstream tasks in the 3D pipeline. They significantly increase the challenge for human artists in optimizing these meshes due their chaotic and inefficient topologies. Finally, previous methods struggle to represent sharp edges and flat surfaces, resulting in oversmoothing and bumpy artifacts as shown in Fig. 2
Lorensen & Cline 1998| Shen et al.) 202Ib|Chen
In this work, we aim to solve the aforementioned issues to facilitate the application of automatically generated 3D assets in the 3D industry. As mentioned earlier, all previous methods (Lorensen &
Cline 1987 Chernyaev 1995 Lorensen & Cline 1998 Shen et al.) 202 Chen et al. 2022 Shen et al. 2 extract 3D meshes with excessively dense faces in a reconstruction manner, which in- herently cannot solve these issues. Therefore, we diverge from previous approaches by formulating mesh extraction as a generation problem for the first time: we teach models to generate Artist- Created Meshes (AMs) that are aligned with the given 3D assets. The meshes generated by our
2

---

Published as a conference paper at ICLR 2025
Faces: 90G
Fhoss 33k
Fros: 3.5k
Fooes: 1.1k
Faoas: 0.28k
Fares: 0.6dk Vertices: 0.31k
Vertioss 4.kk Martinng Cubes
Vertices: 53k Remnest-0.0l
Vertices: 6.7x
Vettioes 2.2k Ramesh-0.05
Vettioss 0.57% Remesb-0:10
Remesh 0.53
MeshA nythng
Faces: 2k
0.46k
Faces:
Vertices: 100k Marching Cubes
1322
Vertices 13k
Varlices Remesh-0.05
Verlioes
Vartices: 0,47k. MeslbAyythngg
Remesh-0.01
Remesb-0.00
Romesh-0.10
Figure 2: Comparison with Marching Cubes Lorensen & Cline (1987) and Remesh Blender Development Team (2024). We apply Marching Cubes and MeshAnything ground truth shapes and then apply remeshing to the Marching Cubes results with different voxel sizes. Existing meth- ods extract meshes reconstruction manner, ignoring the geometric features of the object and producing dense meshes with poor topology. These methods fundamentally fail to capture sharp edges and flat surfaces, as shown in the zoomed-in figure.
methods mimic the shape and topology quality of those created by human artists Consequently, our setting, namely Shape-Condition enabling seamless integration of the generated results into the 3D industry pipeline.
AM Generation, fundamentally free from all previous issues,
However, training such a model presents significant challenges. The first challenge constructing the dataset, as we need paired shape conditions and Artist-Created Meshes (AMs) for model train- ing. The shape condition must be efficiently derived from as many diverse 3D representations as possible to serve accurately represent 3D shapes and be efficiently processed into features that can be injected into the model. After weighing the trade-offs, representation, ease of derivation from most 3D representations, and the availability of mature point cloud encoders (Qi al. 2017a Zhao et al. 2024).
condition during inference. Additionally, it must have sufficient precision to
chose point clouds due to their explicit and continuous
We filter out high-quality AMS from Objaverse (Deitke et al. 2023b a and ShapeNet (Chang et al, 2015). When obtaining paired shape conditions, naive approach would be sample point clouds directly from AMs. However, this leads to poor results during inference because the sampled point clouds have excessive precision, while automatically produced 3D assets cannot provide point clouds of similar quality, causing domain gap between training and inference. To address this issue, we intentionally corrupt the shape quality of AMs. We first extract the signed distance func- tion from AMs (Wang et al. 2022), convert it into a coarser mesh using (Lorensen & Cline 1987), and then sample point clouds from this coarse mesh to narrow the domain gap shape conditions between inference and training.
Following (Siddiqui et al. 2023), we use a VQ-VAE (Van Den Oord et al., 2017) to learn a mesh vocabulary and train decoder-only transformer (Vaswani al. 2017) on this vocabulary for mesh generation. To inject shape condition, we draw inspiration from the recent success of multimodal large language models (MLLM) (Wu et al.) 2023 Liu et al. 2024b), where image features encoded by pre-traiie traineiied image encoders are projected into the token space of the large language models for efficient multimodal understanding. Similarly, we treat the mesh tokens obtained from the trained
3

---

Published as a conference paper at ICLR 2025
VQ- (Q-VAE the language token in LLMs and use pre-trained encoder (Zhao et al. 2024) encode the point clouds into shape features, which is later projected into the mesh token space. These shape tokens are placed at the beginning of the mesh token sequences, effectively serving the shape conditions for next-token predictions. After predictions, these predicted mesh tokens are decoded back meshes with the VQ- VQ-VAE decoder (Siddiqui et al. 2023
To further enhance the quality of mesh generation, we develop a novel noise-resistant decoder for robust mesh decoding. Our observation is that as the decoder in the VQ-VAE (Van Den Oord al. 2017) is only trained with ground truth token sequences from the encoder, it could potentially lead domain gap when decoding the generated token sequences. To mitigate this problem, we inject the shape condition into the VQ-VAE decoder as auxiliary information for robust decoding and fine- tune after the VQ-VAE training. This fine-tuning process involves adding noise to the mesh token sequences to simulate possible poor-quality token sequences from the decoder-only transformer, thus making the decoder robust to such poor-quality sequences.
Finally, we introduce our model, MeshAnything, trained based on the aforementioned techniques. As shown in Fig. [ MeshAnything can convert 3D assets across various 3D representations into AMs, thereby significantly facilitating their application. Furthermore, our extensive experiments demonstrate that our method generates AMs with significantly fewer faces and more refined topol- ogy, while achieving precision metrics that are close to comparable with previous methods.
In summary, our contributions are as follows:
We highlight important reason why current automatically produced 3D assets not replace those created by human artists: current methods cannot convert these 3D as- into Artist-Created Meshes (AMs). To solve this issue, we propose a novel solution called Shape-Conditioned AM Generation, which aims to generate AMs aligned with given shapes.
We introduce MeshAnything for Shape-Conditioned AM Generation. MeshAnything can be integrated with various 3D asset production methods, converting their results into AMs to facilitate their application in the 3D industry.
We develop novel resistaesittant decoder to enhance mesh generation quality. We in- ject the shape condition into the decoder as auxiliary information for robust decoding and fine-tune it using noised token sequences to narrow the domain gap between training and inference.
Extensive experiments demonstrate that Shape-Conditioned Mesh Generation is more suitable setting for mesh generation, and MeshAnything significantly surpasses previous mesh generation methods.
RELATED WORKS
2.1
MESH EXTRACTION
Methods for extracting meshes from 3D models are numerous and have been a subject of research for
decades. Following (Shen al. 2023), we categorize these methods into two main types: Isosurface
Extraction (Lorensen & Cline 1987
Bloomenthal
1988; CCernyyey 1995 Bloomenthal & Bajaj]
1997
Lorensen & Cline 1998 Chen al.
2022
and Gradient- Based Mesh Optimization (Chen
et al.
2019 et al. | HanocHa et al.
2020 Kato et al. 2018 Shen et al.
2021a a
Liao et al.)
2018
Shen et al., 2023).
Traditional isosurface extraction methods (Lorensen & Cline 1987
1998 C Chernyaev 1995 Doi
Koide 1991 Ju et al. 2002, [chhaffer al.) [207]] [Cen & Zhang. 2021 Chen et al., 2022) focus extracting a polygonal mesh that represents the level set of scalar function, area that has seen extensive study in various fields. The most popular method among them Marching
Cubes (Lorensen & Cline, 1987). It divides the space into cells, within which polygons created to approximate the surface.
Transitioning to more recent developments, the advent of machine learning has ushered in new techniques for generating 3D meshes (Chen et al. 2019; Gao et al.] 2020 Hanocka et al. 2020)
Kato al. 2018 Shen et al. 2021al Liao al. 2018 Shen al. 2023). This line of work explores
4

---

Published as a conference paper at ICLR 2025
using neural networks to generate 3D meshes, where the network parameters are optimized through gradient-basee basde methods under specific loss functions.
However, these approaches fundamentally differ from ours. They ignore the characteristics of the shape and inherently cannot produce meshes with efficient topology. In contrast, McshAnything formulates mesh extraction a generation problem for the first time, aiming mimic human artists in mesh extraction and thereby generating Artist-Created Meshes (AMs) with hundreds of times fewer faces.
2.2 3D MESH GENERATIONS
3D mesh generation can be mainly divided into two categories: generating dense meshes similar to those produced by previous mesh extraction methods, and generating Artist-Created Meshes (AMs).
The former category is currently the mainstream research focus. Methods such as (Gao et al. 2022 Wei et al. 2024 Xu et al.) 2024) directly generate meshes in
manner, but because they produce dense meshes with low-quality topology similar to previous mesh extraction methods, they still encounter the same issues when applied in the 3D industry.
2023 Tang et al. 2023b Wang et al. 2023 Chen et al. 2024b; Tang et al. 2023a Yang et al. 2023 Hong al. [023 Fang al. 2023 Chen
Notably, numerous 3D generation methods (Poole et al.
et al.)
2023a Liu et al. 2024c Shi et al. 20233 Li et al. [023]] Chen et al. 2023b 2024c Tang et al.,
2024 Wang et al. 2024 Tochilkin al., 2024 Liu al. [2024a]
2023c [2244][20230
Zhang et al. (024) can also
produce meshes. These methods first generate 3D assets and then convert them to dense meshes
using mesh extraction methods like (Lorensen Cline 1987). Consequently, they face challenges
when applied to the 3D industry due to their inefficient topology.
Recently, several works have focused on the second category: generating Artist-Created Meshes(AMs) (Nash et al., [2020] Alliegro et al.] 2023) (Siddiqui et al.) 2023) Chen et al. (024a). Although our approach also focuses on AM generation, it fundamentally differs from these meth- ods. Since they lack shape conditioning. these methods must simultaneously learn the complex 3D shape distribution which typically alone requires extensive training (Hong et al.) 2023 Tang et al.) 2024 and the topology distribution of AMs, leading to very challenging training processes. In contrast, our methods eliminate the challenge of learning the shape distribution, allowing the model to focus on learning the topology distribution. This not only significantly reduces training costs but also enhances the model's application value.
Among these methods, the most relevant ours is MeshGPT (Siddiqui al.. 2023 as we follow its architecture. (Siddiqui et al. (023) introduced a combination of a VQ-VAE Van Den Oord et al., 2017) and autoregressive transtormer architecture. It first learns mesh vocabulary with the VQ-VAB and then trains the transformer on the learned vocabulary for mesh generation. However, MeshGPT's s results are limited several categories in ShapeNet. MeshGPT requires a training GPU hours similar to ours, but our method can generalize to unlimited categories in Objaverse. As shown in Fig. 3 this is largely due to the difference in target complexity caused by MeshGPT needing to additionally learn the complex 3D shape distribution.
SHAPE-CONDITIONS AM GENERATION
In this section, we first introduce the formal formulation for compare it with previous mesh generation settings (Nash al. [020 Siddiqui et al.) 2023 Alliegro et al. 2023). We show that it can achieve better performance and broader range of applications compared to the settings previous mesh generation methods, with significantly less training effort.
AM Generation and
Shape-Conditioned AM Generation targets to estimate conditional distribution p(M[S). In this formula, M refers to the Artist-Created Mesh (AM), i.e., the mesh manually modeled by human artists. S refers to the 3D shape information that indicates the 3D shape to which M should align. The input form of S can be diverse, such as voxels or point clouds. Therefore, this versatility allows method to be integrated with any 3D pipeline that outputs S, such as 3D reconstruction (Milden- hall et al.) 2020 Kerbl et al.) 2023b), generation (Poole et al. 2023 Hong et al., 2023), and scanning, making these methods more efficient for the 3D industry.
5

---

Published as a conference paper at ICLR 2025
Training PPL os Triiring Itcrations
Volitation PR vs Train Iteraiiors
2.0
SrapecCandhhrre Image Unord (ona
Supacondiiona
mage Cordiandd
3.7
1.9
1.2
244
16R
508
100K
1200
100%
3806
1009
2008
2008
602
802
2004
2208
2408
2801
2018
2008
1206
Sarring
(a) Training Perplexity (PPI.)
(b) Validation Perplexity (PPL.
Figure 3: Training and validation perplexity (PPI.) for the mesh generation model under dif- ferent input conditions. All models are trained with the same settings as detailed in Section 5.2 The training and validation PPL of shape-conditioned mesh generation significantly lower than that of unconditional and mnnge-conditioned mesh generation. This indicates that the training bur- den shape-conditioned mesh generation is much lower since it avoids learning the complex 3D shape distribution.
Compared to existing AM generation work, they directly estimate the distribution p(M[C), where denotes conditions such as images, text or empty sets for unconditional generation. However, estimating p(M[C) requires an understanding of both the underlying shape, i.e., S, and complex topological structures M. Given this, we made the following approximation:
p(MIC) = S(M.
According to the chain rule, we have:
P(M.SIC) = p(M|S,C) p(S[c).
(2)
For distribution p(MSS C), given that S is much stronger and more direct condition than C, we can make the following approximation:
p(MIS,C)<P(M) R p(M[S).
(3)
Combining 1 and
M(MI) R ((MS)) P(SIC),
(4)
in which P(M(S) is the focus of our shape-conditionel mesh generation. As shown in Fig.
estimating p(M)S) is much more simpler than p(M[C), proving that our setting much easier [0 train than settings in privous methods.
As for P)S(C), In the 3D) community, numerous large models (Team 2024 Tang et al. 2024 Xu
jet al. 20224 Siddiqui et al.) 2023) aim to estimate using various 3D representations and demonstrate excellent results. Besides, some single scene 3D asset production methods (Mildenhall al. 2020
Kerbl et al. 2023b; Barron et al. 2021 2022 Poole et al. 2023) Liu et al. [023b] Sun et al. 2023) can also provide samples from this distribution. By integrating our framework with these existing methods, we can leverage their capabilities to enhance our mesh generation process. This integration allows for a more resource-efficien way to estimate p(M[C), significantly reducing the complexity and resources required compared to previous methods.
4 METHOD
In this section, detail our shape condition strategy in Section After that, we provide detailed description for MeshAnything, which consists of VQVAE with our newly proposed decoder (Section 4.2p and a shape-conditioned autoregressive transformer (Section 4.3

---

Published as a conference paper at ICLR 2025
Training
VC Enoasd
suuduun
Inference
Autoraatteee transttomer
Senortiie
OZ
Neah
Figure 4: Pipeline Overview. We introduce MeshAnything, an autoregressive transformer capable of generating Artist-Created Meshes that adhere to given 3D shapes. During training. we inject point clouds features into a decoder-only transformer and supervise it using token sequences derived from the Artist-Created meshes. After training, MeshAnything takes point clouds sampled from various 3D representations as input and generates aligned Artist-Created meshes.
4.1 SHAPE ENCODING FOR CONDITIONAL GENERATION
We begin by describing our shape condition strategy. MeshAnything targets learning p(M|S), so we need to pair each mesh M with a corresponding S, i.e., the shape condition. Choosing an appropriate 3D representation for non-trivial and should satisfy the following conditions:
1. It should be easily extracted from various 3D representations. This ensures that the trained models be integrated with a wide range of 3D asset production pipelines (Mildenhall
et al. 2020 Kerbl et al. 2023b, Hong et al. 2023 Poole et al. 2023 Tang et al. 2024).
2. It should be suitable for data augmentation to prevent overfitting. To ensure the effec- tiveness of S during training, any data augmentation applied to M must be equivalently applicable to S.
3. It should be efficiently and conveniently input into the model condition. To ensure the model comprehends the shape information and to maintain efficient training, S must be easily and effectively encoded into features.
Considering the first and second points, S should in an explicit representation. Further consid- ering the third point, the main explicit 3D representations that can be easily encoded as features are voxels and point clouds. Both representations are suitable, but voxels typically require a high resolution to accurately represent shapes, and processing high-resolution voxels into features is com- putationally expensive. Additionally, voxels, being a discrete representation, are less precise for data augmentation compared point clouds. Therefore, chose point clouds the representation for S. To enhance the expressive power of the point clouds, we also include normals into the point cloud representation.
To obtain point clouds from the ground truth mesh for training, we could simply sample point clouds directly from the surface of M. However, this would create problems during inference: the surfaces of automatically generated 3D assets are often rougher than those of AMs. For example, in AMs, we would sample a series of points on a flat plane, whereas automatically generated 3D assets would have uneven surfaces, causing a domain gap between training and inference.
Therefore, need ensure that S extracted from the ground truth M during training has a similar domain to the S extracted during inference. To bring their domains closer, we intentionally construct coarse meshes from AMs. We first extract the signed distance function from M with (Wang al. 2022], then convert it into a relatively coarse mesh using Marching Cubes (Lorensen & Cline 1987) to destroy the ground truth topology. Finally, we sample point cloud and its normals from the coarse mesh. This approach also helps avoid overfitting, as AMs typically have fewer faces, and each face can often sample multiple points. The network can easily recognize the ground truth topology by determining whether the points lie on the same plane.
Since almost all 3D representations can be converted into a coarse mesh using Marching Cubes (Lorensen & Cline 1987) sampled into point clouds, this ensures that the domain of S consistent during both traming and inference. We pair the point clouds extracted as with M to create data item ((M,, S.)), for training.
7

---

Published as a conference paper at ICLR 2025
4.2 VQ-VAE VAE WITH
DECODER
Following MeshGPT (Siddiqui et al. 2023). we first train VQ-VAE Van Den Oord et al. 2017) to learn vocabulary of geometric embeddings for better transformer (Vaswani al. 2017) learning. Different to MeshGPT, which uses graph convolutional networks (Wu et al. 2019) and ResNet (He)
al., 2016) as the encoder and decoder respectively, we employ transformers with identical struc- tures for both the encoder and decoder. When training VQ-VAE, meshes are discretized and input as a sequence of triangle faces:
M: (f. fNN),
(5)
where f; is the coordinates of the vertices of each face, and N is the number of faces in M. The encoder E then extracts feature vector for each face:
Z (31, 22 ... 2N) E(M),
(6)
where 2; is the feature vector for fi.
The extracted faces are then quantized into quantized features T with codebook B:
T = ();
Finally, the reconstructed mesh is decoded from T with decoder D by predicting the logits for each vertex's 's coordinates:
M D(2)
(8)
The VQ-VAE is trained end-to-end with cross-entropy loss on the predicted vertex coordinate logits and the commitment loss of vector quantization (Van Den Oord al. 2017). After the training of
VQ- the encoder-decoder of VQ-VAE is treated as tokenizer and detokenizer for autoregres- sive transformer training.
However, as shown in Fig. there are possible imperfections in the generation results. To address this issue, given our setting of Shape-Conditioned AM Generation, the VQ-VAE decoder can also take the shape condition input. Small imperfections in the token sequences generated by the transformer can potentially be corrected by a shape-aware decoder. Therefore, after completing the vanilla VQ-VAE training, we add additional decoder fine-tuning stage, where we inject the shape information into the transformer decoder. Then we add random Gumbel noise to the codebook sampling logits to simulate the potential imperfections in the token sequences generated by the transformer during inference. The decoder is then updated independently with the same cross- entropy loss to train it to produce refined meshes even when facing imperfect token sequences. Our experiments in Tab. 3 and Tab. 4 show that our method effectively enhances the decoder's 's noise resistance and mesh generation quality.
4.3 CONDIICCODD AUTOREGRESSIVE TRANSFORMER
To add shape condition to the transformer, inspired by the success of multimodal large language models (Wu et al., 2023; Liu et al. 2024bb Xu et al. 2023 Guo et al. 2023). we first encode the point cloud into lixed-length token sequence with point cloud encoder P and then concatenate it to the front of the embedding sequence from T VQ- VAE as the final input embedding sequence for the transformer:
T = concat(P(S),
where T' the training input for the transformer.
We borrow a pretrained point encoder from (Zhao et al. 2024 and add linear projection layer to project its output feature the same latent space as T. During training, the original point en- coder from (Zhao et al. 2024) is frozen; we only update the newly added projection layer and the autoregressive transformer with cross-entropy loss.
During inference, we input P(S) to the transformer and require it to generate the subsequent se- quence, +. i is then input to the noise-resistant decoder reconstruct meshes:
M= M=D(T)
(10)
where M is the final generated AM.
8

---

Published as a conference paper at ICLR 2025
Table I: Comparison of Mesh Generation Methods. As shown in the left table, compared to the Mesh Generation method, the meshes generated by MeshAny are better aligned with human preferences. In the right table, we compare MeshAnything with mesh extraction baselines, and it received the most votes. For detailed settings, please refer to Section 5.4
baseline
Method
Shape
Topologyt
Method
Shape
Topology?
PolyGen 12.7%
11.1%
MarchingCubes
38.1% 1%
10.2%
MeshGPT
24.1%
28.2%
Shape As Points
17.3%
6.2%
MeshAnything
63.2%
60.7%
MeshAnything
44.6%
83.6 %
We use the standard next-token prediction loss to train shape-conditioned transformers. For each sequence, we add <bos> token after the point cloud tokens and <eos> token after the mesh tokens to identify the end of a 3D mesh.
EXPERIMENTS
5.1
DATA PREPARATION
Data Selection. Existing AM generation works are limited to few categories. However, our method targets to operate on general shapes. MeshAnything is trained on a combined dataset of Objaverse (Deitke et al., 2023b) and ShapeNet (Chang et al., 2013), selected for their complementary chose Objaverse because it contains a large number of AMs without category limitations. On the other hand, ShapeNet offers higher data quality within limited categories.
characteristics.
We filter out meshes with more than 800 faces from both datasets. Additionally, we manually filtered out low quality meshes. Our final filtered dataset consists of 5 51k meshes from Objaverse and 5k meshes from ShapeNet. We randomly select 10% of this dataset as the evaluation dataset, with the remaining 90% used as the training set for all experiments.
Data Processing and Augmentation. Following the strategies of PolyGen (Nash et al., 2020 and MeshGPT (Siddiqui et al.) 2023), order faces by their lowest vertex index, then by the next lowest, and so on. Vertices are sorted in ascending order based on their z-y-x coordinates, where represents the vertical axis. Within each face, we permute the indices ensure the lowest index comes first. During training, we apply on-the-fly scaling, shifting, and rotation augmentations, normalizing each mesh unit bounding box from -0.5 0.5.
5.2 IMPLEMENTATION DETAILS
The encoder and decoder of VQ-VAE VAE both use the encoder of BERT (Devlin et al. 2018), while we choose OPT-350M (Zhang et al., 2022) as our autoregressive transformer architecture. The residual vector quantization Zeehhdoor al. 2022) depth set to 3, with codebook size of 8,192.
Our point encoder is based on the pretrained point encoder from (Zhao et al. 2024). which has been trained on Objaverse and thus can handle general shapes. This point encoder outputs fixed- length token sequence of 257 tokens, with 256 tokens primarily containing shape information and an additional head token containing semantic information about the shape. We sample 4096 points for each point cloud.
The training batch size for both the VQ-VAE and the transformer is set to 8 per GPU. The VQ-VAE trained on 8 A100 GPUs for 12 hours, after which we separately finetune the decoder part of the VQ- VAE into a noise-resistant decoder, as detailed in Section 4.2 Following this, the transformer trained on 8 A100 GPUs for 4 days.
5.3 QUALITATIVE EXPERIMENTS
As shown in Fig. MeshAnything effectively generates AMs from various 3D representations. In our experiments, we use Rodin (Team 2024) as the text-to-3D and image-to-33 3D method, and employ (Mildenhall et al. 2020) and (Kerbl al. 2023a the 3D reconstruction pipeline obtain

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative Comparisons with Prior Arts on Objaverse. MeshAnything significantly outperforms prior methods across all metrics. MMD, KID are scaled by 10
Method
COVT
MMD
1-NNA!
FID
KID
PolyCien
23.2
6.22
88.2
48.8
27.7
MeshGPT
41.7
3.83
67.3
25.1
6.11
MeshAnything
53.1
2.72
55.7
14.5
1.89
the corresponding NeRF and Gaussian Splatting models. For additional qualitative results, please refer to A.2 combined with other 3D asset production pipelines.
5.4 QUANTITATIVE EXPERIMENTS
From the generative model perspective, MeshAnything shape-conditioned mesh generation model. From the mesh extraction perspective, it extracts artist-created meshes from point clouds. Consequently, we compare MeshAnything with both types of methods. Additional experiments can be found in Appendix Section A.2
User Study. As shown in Tab. we conducted two user studies, comparing with mesh genera- tion baselines (Nash et al. [2020] Siddiqui et al., (023) and mesh extraction baselines (Lorensen & Cline 1987 Peng et al. 2021). respectively. The majority of participants in our user study were researchers from the 3D field, with smaller portion being 3D industry practitioners. The mesh generation baselines are trained on ShapeNet, and to ensure a fair comparison, we retrained them on Objaverse using the same transformer model MeshAnything, Since the mesh generation baselines are all unconditional mesh generation methods, whereas MeshAnything is a shape-conditioned mesh generation method, we sampled shapes randomly from the evaluation set of Objaverse as inputs for MeshAnything, while for the baseline methods, we performed random sampling directly.
In the mesh extraction baseline, since our method can also be viewed as point cloud to mesh approach, we included (Peng et al.] (02T). point cloud to mesh method, as baseline. Ad- ditionally, we optimized the results from the mesh extraction baseline using the Blender remesh method (Blender Development Team 2024) to simplify the topology.
We collected results from each method and asked users to vote for the best one terms of shape quality and topology quality. A total of 41 users participated, providing 1,230 valid comparisons. Both user studies demonstrated the superiority of our method. The only difference between the retrained MeshGPT and MeshAnything is whether they shape-conditioned, further proving the advantages of the shape-conditioned mesh generation setting.
Metrics. We follow the metric setting of (Chen et al. 2022: Siddiqui al. 2023 We detail this setting in Appendix Section. A.I
Comparison with Mesh Generation Pipelines. We the same retrained models from the user study for comparison. As shown Tab. 2 MeshAnything significantly outperforms prior meth- ods (Nash et al., 2020; (Sidiiqui al. 2023), indicating that it's superior in both the shape and topology quality. Since the only difference between the retrained MeshGPT and MeshAnything the inclusion of shape conditioning, the superior performance of MeshAnything further demon- strates that Shape-Conditioned Mesh Generation is more suitable setting for mesh generation.
CONCLUSION
In this work, we propose novel setting for improved mesh extraction and mesh generation, namely Chape-Condittione Artist-Created Mesh (AM) Generation. Following this setting, we introduce Me- shAnything, model capable of generating AMs that adhere to given 3D assets. MeshAnything can convert 3D assets in any 3D representation into AMs and thus can be integrated with diverse 3D asset production methods to facilitate their application in the 3D industry. Extensive experiments demonstrate the superior performance of our method, highlighting its potential to scale up for 3D industry application and its advantage previous methods.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This study is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well cash and in-kind contribution from the industry partner(s). This research is also supported by the MoE ACRF Tier 2 grant (MOE-T2EP20223- 0001).
REFERENCES
Antonio Alliegro, Yawar Siddiqui, Tatiana Tommasi, and Matthias Nießner. Polydiff: Generating 3d polygonal meshes with diffusion models. arXiv preprint arXiv:2312.11417 2023.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruuall,, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 5855-5864, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman, Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Blender Development Team. Blender (version 4.1.0) [computer software], 2024. Available from htmphttss://dcss. blender. www.eee..eoeeeameeeaeeeeeeeeeeee generate/remesht htmll
midiri flers
Jules Bloomenthal. Polygonization of implicit surfaces. Computer Aided Geometric Design, 5(4):
341-355, 1988.
Jules Bloomenthal and Chandrajit Bajaj. Introduction implicit surfaces. Morgan Kaufmann, 1997.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:/512.0353 020112, 2015.
Sijin Chen, Xin Chen, Anqi Pang. Xianfang Zeng, Wei Cheng, Yijun Fu, Fukun Yin, Yanru Wang, Zhibin Wang, Chi Zhang, et al. Meshxl: Neural coordinate field for generative 3d foundation models. arXiv preprint 2405:2224052085.5 2024a.
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lchtinen, Alee Jacobson, and Sanja Fidler. Learning to predict 3d objects with interpolation- based differentiable renderer. Ad- vances neural information processing systems, 32, 2019.
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. arXiv preprint arXiv:2311.14522 2023a.
Yiwen Chen, Chi Zhang, Xiaofeng Yang, Zhongang Cai, Gang Yu, Lei Yang, and Guosheng Lin. It3d: Improved text-to-3d generation with explicit view synthesis. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1237-1244, 2024b.
Zhiqin Chen and Hao Zhang. Neural marching cubes. ACM Transactions Graphics (TOG), 40 (6):1-15, 2021.
Zhiqin Chen, Andrea Tagliasacchi, Thomas Funkhouser, and Hao Zhang. Neural dual contouring. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.1658 16585, 2023b.
Zilong Chen, Yikai Wang. Feng Wang. Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06736 2024c.
Evgeni Chernyaev. Marching cubes 33: Construction of topologically correct isosurfaces. Technical report, 1995.
11

---

Published as a conference paper at ICLR 2025
Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL CoLhhtp::// www www.deee.d........ org
Morteza Daneshmand, Ahmed Helmi, Egils Avots, Fatemeh Noroozi, Fatih Alisinanoglu, Hasan Sait Arslan, Jelena Gorbova, Rain Eric Haamer, Cagri Ozcinar, and Gholamreza Anbarjafari. 3d scanning: A comprehensive survey. arXiv preprini arXiv:1801 00863,, 2018.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2337.056600 230660663,,, 2023a.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In CVPR, pp. 12142-13155 2023b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:/81O..00 /810004805 2018.
Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE TRANSACTIONS on Information and Systems, 74(1):214-224, 1991.
Jiemin Fang, Junjie Wang. Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: Editing 3d gaussians delicately with text instructions. arXiv preprint arXiv:2311.16037. 2023.
Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler.
Learning deformable tetrahedral meshes for 3d reconstruction. Advances in neural information processing systems, 33:9936-9920 2020.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. NeurlPS, 35:31841-31854, 2022.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309. 2.5.003,0,0001,, 2023.
Abid Haleem and Mohd Javaid. 3d scanning applications in medical field: a literature-based review. Clinical Epidemiology and Global Health, ((2):199-210) 2019.
Abid Haleem, Mohd Javaid, Ravi Pratap Singh, Shanay Rab, Rajiv Suman, Lalit Kumar, and Ibrahim Haleem Khan. Exploring the potential of 3d scanning in industry 4.0: An overview. International Journal of Cognitive Computing in Engineering, 3:161-171, 2022.
Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen Or. Point2mesh: A self-prior for de- formable meshes. arXiv preprint arXiv:2005. 1084, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint MrXiv:2311.044000 2023.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403. 17888, 2024.
Tao Ju, Frank Losasso, Scott Schaefer, and Joe Warren. Dual contouring of hermite data. In Pro- ceedings of the 29th annual conference Computer graphics and interactive techniques, pp- 339-346, 2002.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3907-3916, 2018.
12

---

Published as a conference paper at ICLR 2025
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4))1-14, 2023a.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), July 2023b.
URL /nttps://repo- sam. inria. wwwria.ffffugggeee ft/fungraph//ddd qassss an sp atting/
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong. Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. arXiv preprint arXiv:23 1.0223111.02144 2023.
Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recog- nition, pp. 2916-2925, 2018.
Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, and Wenping Wang. Part123: Part-aware 3d reconstruction from single-view image, 2024a. URL
pettps:/aarxha org/ wtw.g////////oo/oee 2405 . 16888
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024b.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One- 2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024c.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; shro-shot one image to 3d object. https://arriioorrab22223..../ 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; shro-shot one image to 3d object. arXiv preprint 2303. 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Syncdreamer: Generating multiview-consistenn images from a single-view image. arXiv preprint arXiv:2309.03455 2023c.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.1 150088 2023.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 1134)11 1987.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pp. 347-353. 1998.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference machine learning, pp. 7220-7229. PMLR, 2020.
Songyou Peng. Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Mare Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. Advances in Neural Information Processing Systems, 34:13032- 13044, 2021.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httttp:://peeneviiew... NNys
13

---

Published as a conference paper at ICLR 2025
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR 2017, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
Scout Schaefer, Tao Ju, and Joe Warren. Manifold dual contouring. IEEE Transactions on Visual- ization and Computer Graphics, ((33)660-619 2007.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for higl-uesolttion 3d shape synthesis. In Advances Neural Information Processing Systems (NeurIPS), 2021a.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6110 2021b.
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient- based mesh optimization. ACM Transactions on Gruphics (TOG), 42(4):1-16, 2023.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder- only transformers. arXiv preprint arXiv:2311.15475 2023.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310. 168188 2023.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023a.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05004, 2024.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint 1.X1::3003.14884, 2023b.
Deemos Team. Deemos rodin. https::/hyperhhumn. deemos.dddeemmmmmmmmm com wew.mmeccmmooommmooooicoo 2024.
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from single image. arXiv preprint vrXiv:2403.021555 2024.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Peng-Shuai Wang, Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape representations. ACM Transactions on Graphics (TOG), 41(4):1-15, 2022.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li. Hang Su, and Jun Zhu. Prolific- dreamer: High- fidelityy and diverse text-to-3d generation with variational score distillation. arXiv preprint a3355:2305. 162/3, 2023.
14

---

Published as a conference paper at ICLR 2025
Zhengyi Wang. Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu. Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2403.0m 05004,, 2024.
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference machine learning, pp- 6861-6877 PMLR, 2019.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Yu Philip. Multimodal large language models: survey. In 2023 IEEE International Conference Big Data (BigData), pp- 2247-2256. IEEE, 2023.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07199, 2024.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
Xiaofeng Yang. Yiwen Chen, Cheng Chen, Chi Zhang, Yi Xu, Xulei Yang, Fayao Liu, and Guosheng Lin. Learn to optimize denoising scores for 3d generation: A unified and improved diffusion prior on nerf and 3d gaussian splatting. arXiv preprint urXiv:2312.04820 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, pp- 4578-4587, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high high-quality 3d assets, 2024. URL//ttsss//axxiv.oooos
2406. 13897
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 2022.
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Clod
Figure 5: Additional qualitative results of MeshAnything. As shown. MeshAnything can be integrated with various 3D production pipelines to achieve highly controllable mesh generation.
Condtes
112
Dun
02
bmge
66
(3)
Figure 6: Qualitative Results. (a) further demonstrates our capability to achieve highly control- lable mesh generation when combined with 3D asset production pipelines. Besides, we compare our reseults with ground truth in (b) and (c). In (b), MeshAnything generates meshes with better topology and fewer faces than the ground truth. In (c), produce meshes with completely differ- ent topology while achieving a similar shape, proving that our method does not simply overfit understands how to construct meshes using efficient topology.
W.O. Muso Residant Dooolt
WOO. Náke Rositaat Dooder
Oas
Figure 7: Ablation on ResistRe Decoder. The decoder-only transformer may generate low- quality token sequences, and the decoder of VQ- VAE would typically produce flawed meshes based on these sequences. In contrast, our Noise-Resistan Decoder, aided by shape conditions, has the ability to resist these low-quality token sequences, producing higher-quality meshes.
A. METRICS
We follow the evaluation metric setting of (Siddiqui et al. 2023) in mesh generation experiments and the setting (Chen et al. 2022) mesh extraction experiments.
We quantitatively evaluate mesh quality by uniformly sampling 100K points from the faces of both the ground truth meshes and the predicted meshes, and then computing a set of metrics to assess various aspects of the reconstruction.
16

---

Published as a conference paper at ICLR 2025
Table 3:
Reconstruction Performance under Different Noise Levels with and without Noise- Resistant (NR) Decoder. Please refer to A. for metrics explanation.
CD(x10 3)
ECD(x10
NC
Noise Level
W/O NR
W/NR
W/O NR
W/ W/NR
W/O NR
W/ NR
0.0
0,011
0.007
0.035
0.023
0.987
0.993
0.1
0.187
0.028
0.613
0.138
0.973
0.991
0.5
1.167
0.639
2.538
1.329
0.964
0.981
1.0
2.131
1.798
4.317
2.316
0.952
0.969
Table 4: Ablation Noise-Resistaan (NR) Decoder for the Quality of Mesh Generation.
Method
CD (x10
ECD (x10
NC1
W/O NR
2.423 2.256
6,414 6.245
0.883
W/ NR
0.902
For mesh extraction, we report the following metries: Chamfer Distance (CD) to evaluate the overall quality of reconstructed mesh; Edge Chamfer Distance (ECD) to assess the preservation of sharp edges by sampling points near sharp edges and corners; and Normal Consistency (NC) to evaluate the quality of the surface normals. Additionally, we report the number of mesh vertices (#V) and the number of mesh faces (#F). We also provide the ratio of the estimated number of vertices the ground truth number of vertices (#V _R) and the same ratio for faces (#F_R)
For mesh generation, Coverage (COV) captures the diversity of generated meshes and is sensitive to mode dropping, but it does not reflect the quality of the results. Minimum Matching Distance (MMD) measures the average distance between the reference and their nearest neighbors in the generated set, though it lacks sensitivity to low-quality outputs. The I-Nearest Neighbor Accuracy (I-NNA) assesses both quality and diversity between the generated and reference sets. To evaluate topology quality, we render the ground truth meshes and generated meshes with their wireframes visualized. We then employ Frechet Inception Distance (FID) and Kernel Inception Distance (KID) on rendered images. MMD, and KID scores are scaled by factor of 10
A.2 EXPERIMENTS
Additional Qualitative Experiments We present more qualitative results of MeshAnything here. As shown in Fig. S and Fig. 66 MeshAnything effectively generates AMs from various 3D repre- sentations. When integrated with different 3D assets production pipelines, our method effectively achieves mesh generation with diverse conditions.
Next, Fig. demonsttaaees that MeshAnything does not simply overfit but understands how gener- ate meshes with efficient topology that conform to the given shape. To prove this, we use manually- created meshes as ground truth and use their shapes as conditions to test whether our model can generate meshes with comparable topology. To effectively use the ground truth as conditions, we first convert them into dense meshes using Marching Cubes (Lorensen & Cline 1987) to disrupt their face structure. Then, we sample point clouds with normals from the dense meshes to serve as shape conditions. The experimental results Fig. show that MeshAnything is capable of gener- ating meshes comparable to or even surpassing those modeled by human artists, exhibiting diverse and strong 3D modeling capabilities.
Comparison with mesh extraction baselines. Our method is related to various mesh extraction methods (Lorensen & Cline 1987 Chen & Zhang 2021 Chen et al. 2022 Shen et al. 2023 Peng et al. 2021 since we also convert other 3D representations into meshes. However, it is important to note that previous approaches are reconstruction- approach is generative, creating Artist- Crested Meshes (AMs) that are significantly more complex to produce than dense meshes. Therefore, strictly speaking, our method cannot be considered the same as these reconstruction-bbsed mesh extraction methods. The main purpose of this comparison
methods that produce dense meshes, while our
17

---

Published as a conference paper at ICLR 2025
Table 5: Quantitative evaluation with mesh extraction baselines. MC, FC, SAP refer Marching
Cubes
Lorensen & Cline [1877, FlexiCubes (Shen et al. (2023), and Shape As Points Peng et al.
2021
respectively. Please refer to A.1 for metrics explanation.
Method
CD
ECD
NCT
#V4
#F1
VRI
F F_R
(x10
(x10-
(x10*)
(x10*)
(a) Marching Cubes
1.532
6.733
0.954
73.22
146.0
440.2
462.2
(b) MC+Remesh (0,005)
2.174
7.813
0.912
127.8
167.9
748.1
534.6
(c) MC+Remesh (0.010)
2.083
7.578
0,929
39.01
41.78
225.4
132.3
(d) MC+Remesh (0.030)
2.915
8.329
0.863
5.848
4.410
34.38
14.05
(e) MC+Remesh (0.050)
4.179
8.138
0.814
2.299
1.538
13.64
4.920
(f) MC+Remesh (0.100)
7.312
10.771
0.748
0.625
0.359
3.735
1.149
(g) FC
1.190
6.121
0.967
59.12
121.1
378.2
391.1
(h) FC+Remesh (0.010)
1.861
6.940
0.933
37.98
40.19
205.5
124.2
(i) SAP
1,771
7.112
0.939
79.12
152.3
481.2
489.3
(i) SAP+Remesh (0.010)
2.367
7.862
0.925
39.17
42.87
239.1
136.6
(k) MeshAnything
2.256
6.245
0.902
0.172
0.318
0.888
0.871
to use these mesh extraction methods as a reference for evaluating the quality of the meshes generated by MeshAnything in terms of shape. We compare MeshAnything with Lorensen & Cline (1987): Shen al. (2023;; Peng et al. (202I). Among these, MarchingCubes is the most popular mesh extraction method, FlexiCubes represents the state-of-the-art in mesh extraction, and Shape as Points is the leading method for extracting mesh from point cloud.
We also combined these methods with the remesh technique to test whether they could significantly reduce the number faces while maintaining shape quality. We used Blender Remesh in voxel mode (Community 2018 Blender Development Team (024). specifically using Blender version 4.1, as the remesh method. Since our evaluation dataset includes non-watertight meshes, we first extract the signed distance fields (SDF) of all ground truth meshes using (Wang al. (022). which can handle non-watertight meshes. We then apply Marching Cubes with a resolution of 128 on these SDFs. Next, we apply Blender remesh (Blender Development Team, 2024) with different voxel sizes to the Marching Cubes results, as both the remesh method and our approach are capable of simplifying topology. Additionally, the Marching Cubes result used as the shape condition input to MeshAnything to obtain our results. The settings of (Shen et al. 2023) and (Peng et al.) 2021) follow their papers.
As shown in Tab. 5 we found that these methods require hundreds of times more faces to achieve results comparable our method. Comparing (a), (g), and (k), our method lags in Chamfer Distance (CD) and Normal Consistency (NC), mainly due to our method's inherent failure cases as a generative model, which makes less robust than these reconstruction-bat methods. When comparing with remesh methods, a face count similar to ours. Comparing (f) and (k), we find that even when remesh methods achieve comparable face count, the number of vertices is still several times higher than ours, indicating that the topology efficiency of remesh methods is far inferior to ours, as they completely ignore the shape characteristics of the 3D assets. It's important to note that the metries in mesh etraction can only indicate the quality of shape alignment, which do not effectively reflect the topological surprisingly find that our method can produce results
mesh extraction observe that they incur high cost achieve
advantages of method. Additionally, with fewer faces than the ground truth, demonstrating that MeshAnything is not overfitting to the data but instead learns an efficient topology representation, occasionally surpassing the ground truth meshes.
blations on Noise- Resistant Conditional Decoder. We perform ablation experiments verify the effectiveness of the Noise-Resistant Decoder. We begin with a VQ-VAE trained without any noise or conditioning. We then perform ablation between two settings: one where the decoder remains unchanged and unaware of the shape condition, and another where the shape condition is injected into the transformer, as described in Section 4.2 Next, we randomly sample a noise from gumbel distribution and add it to codebook sampling logits during the vector quantization process to simulate the potential low-quality token sequences generated by the transformer. We control the noise level by scaling the added noise.
18

---

Published as a conference paper at ICLR 2025
Table 6: Experiments on the Impact of Input Point Cloud Quality on Generated Results.
Method
CD1
ECD.
NCT
#V
#F
V.R.
F F_R.
(x10
(x10-3)
(x (x10²)
(x10²)
(a)
Noise scale 0.005
2.351
6.412
0.897
0.175
0.321
0.895
0.880
(b) Noise scale 0.020
2.980
6.970
0.881
0.180
0.330
0.901
0.910
(c) Noise scale 0.050
4.910
8.556
0.755
0.162
0.284
0.811
0.802
(d) Rodin
2.552
6.622
0.833
0.185
0.342
0.919
0.923
(c) MeshAnything
2.256
6.245
0.902
0.172
0.318
0.888
0.871
Table 7: The comparison experiment between MeshAnything and MeshGPT on the chair cat-
egory in ShapeNet. This experiment is conducted with the same settings in Table - except that both training and evaluation were performed exclusively on the chair category in ShapeNet.
Method
COV1
MMDI
1-NNAL
FID
KID
MeshGPT
49.2
2.98
61.0
16.4
2.04
MeshAnything
62.1
1.92
49.8
11.2
1.21
After training both models for enough epochs, we test their performance to the same level of noise. As shown in Tab. 3 as the intensity of the added noise increases, the Noise-Resistann Decoder with shape condition clearly achieves better reconstruction results. This indicates that the shape condition helps the decoder identify and correct imperfections in the input token sequences.
Next, we verify whether the Roise-Resistant Decoder indeed enhances the transformer's perfor- mance during inference. The test method used dense meshes derived from corrupted GT meshes as the condition for generating new meshes. The generated meshes were then assessed for shape alignment with the conditional shape. As shown in Tab. the model with Noise-Resistan Decoder achieved better results.
Experiments on the Impact of Input Point Cloud Quality on Generated Results. MeshAny- thing takes point clouds input, and its robustness to point cloud quality determines its versatility across various applications. We design two experiments to evaluate its tolerance to input point cloud quality: First, keeping the other evaluation settings unchanged, we apply Gaussian noise to the input point cloud coordinates and normals. Specifically, for each point, randomly sample Gaussian noise from a standard distribution, scale it by a noise factor, and add to the point's coordinates. The same approach is applied to the normals, but normalization is applied after adding the noise. Second, we use Rodin's generation result as the ground truth mesh, sample point clouds from this mesh input, and evaluate the deviation between the generated result and the ground truth.
As shown in Tab, 6, MeshAnything did not experience significant performance drop in (a) and (b), demonstrating resilience to noise in the point cloud, with a noticeable performance decrease only in (c). It is important to note that the input point cloud is normalized to the range [-1,1], and the noise scale in (c) is already quite large. The experiment (d) further demonstrates that MeshAnything can tolerate generated point clouds and effectively integrate with 3D generation models.
Experiments between MeshAnything and MeshGPT on the chair category in ShapeNet. To reduce the complexity of learning the shape distribution, we conducted the comparison experiment between MeshAnything and MeshGPT (Siddiqui et al., (023) exclusively on the chair category in ShapeNet. As shown in 7 the gap between MeshGPT and MeshAnything narrowed under this set- ting, though noticeable difference remains. This result supports our argument in Sec. that shape- conditioned mesh generation is considerably less challenging to learn than unconditional mesh gen- cration.
Sampling Strategy and Diversity of Generated Results All of our qualitative and quantitative experiments employed multinomial sampling, which equivalent to a beam search with a beam size of 1. We used top_k and top-p values set to 50 and 0.95, respectively. We provide the generation variants obtained using this sampling approach in [8 As shown in the figure, MeshAnything demonstrates the capability generate various topologies.
19

---

Published as a conference paper at ICLR 2025
Figure 8: Diversity of Generated Results. The generated results of MeshAnything obtained using multinomial sampling with different random seeds.
Qualitative Comparison with MeshGPT. We provide a quantitative comparison with MeshGPT (Siddiqui et al. 2023) and Polygen (Nash al. 2020). As shown in Fig- 9 it is chal- lenging for these two methods to produce high-quality meshes Objaverse. This advantage is primarily because MeshAnything avoids the need to learn the complex 3D shape distribution, mak- ing the training process significantly easier compared to unconditional mesh generation.
A.3 LIMITATIONS
Our method cannot generate meshes that exceed the maximum face count limit, which restricts its ability to convert large scenes and highly complex objects into meshes. Additionally, due to its generative nature, like (Lorensen & Cline 1987; Shen et al. 2023). After decades of development, reconstruction methods achieve near 100% success rates with reasonable inputs. However, generative ap- proach, our method inevitably produces occasional failure cases.
method is not stable reconstruction-ber based mesh extraction methods
A.4 FUTURE WORKS.
Autoregressive mesh generation remains inefficient. significantly limiting the practical application of this approach. Developing more efficient representations for mesh generation is essential. Fur-
20

---

Published as a conference paper at ICLR 2025
MeshGPT
PolyGen
Figure 9: Qualitative Comparison with MeshGPT (iddiqqii et al. 2023) and PolyGen (Nash et al,, 2020). Since MeshAnything is conditioned on point cloud input, a direct qualitative com- parison with these two methods is not feasible. Therefore, showcase unconditional generation results from MeshGPT and PolyGen. For a fair comparison, both methods use the same architecture and data approach. As shown, when scaled Objaverse, MeshGPT and PolyGen struggle consistently produce meshes with complex, general shapes.
thermore, improving the success rate of mesh generation is critical. Due to the cumulative error characteristic of autoregressive models, deviation in any one token prediction can cause the en- tire mesh generation process to fail. Future developments this line of research may benefit from techniques used in large language models (LLMs) to mitigate similar issues. Additionally, current mesh generation methods are limited to the object level. Extending these methods to the scene level equally important, as it would greatly expand the range of potential applications.
A.5
SOCIAL IMPACT
Our method points to promising approach for the automatically generation of Artist-Created Meshes, which has the potential significantly reduce labor costs the industry, thereby facil- itating advancements in industries such as gaming, film, and the metaverse. However, the reduced cost of obtaining 3D Artist-Creater meshes could also lead to potential criminal activities.
21


Reviews:
[{"review_id": "QiWagAuoZA", "reviewer": "Reviewer_7tHF", "summary": "This work introduces a method for conditional \"artist-created\" mesh generation. \nThe key idea is employing auto-regressive transformer on top of a VQ-VAE pre-trained latent space, with additional conditioning for VQ-VAE and noise tricks to improve generalization. Quantitative results on toy datasets suggest that the method outperforming recent baselines.", "questions": "- Can you provide details on the scalability of the method - i.e. what is the max number of faces/vertices the model can handle?\n- Is there anything specific about architecture why artist-generatedness of the meshes is critical?\n- How does this method stack against diffusion-based methods for mesh/point-cloud based generation (e.g. point-e)?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written and is easy to follow. \n+ The overall architecture of the method makes sense, and the problem of conditional mesh generation is important. \n+ There are several interesting techniques that could be useful for practitioners: e.g. conditioning VQ-VAE on the shape to improve reconstruction, and the gumbel noise trick to improve VQ-VAE's decoder robustness.\n+ Quantitative results (Table 2) suggest that the method outperforms competitors.", "weaknesses": "- The motivation for focus on artist-generated meshes is not very clear - would exactly the same method not work on a collection of reconstruction-based meshes? My assumption would be that the reason is that the method cannot scale to any realistic number of vertices. \n- The method is a combination of existing architectures (VQ-VAE based on BERT + OPT transformers), whereas the encoding scheme is the same as in polygen / meshgpt. \n- The scalability of the method is very questionable: both in terms of training and inference. If my undertstanding is correct, memory scales quadratically with the number of mesh faces, and authors explicitly mention that they filter out meshes with less than 800 faces. There is discussion of this in the appendix, but it is worth providing more information (memory and/or runtime numbers) which would be critical to get a complete impression of how usable the method actually is.\n- Comparison to diffusion-based methods (e.g. point-e) would be interesting (converting point clouds to meshes should be supported by publicly available code).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOd1Xl2y8c", "reviewer": "Reviewer_tUZi", "summary": "This paper introduces Meshanything for shape-conditioned AM generation. Meshanything first trains a VQ-VAE to obtain a set of mesh tokens and employs a noise-resistant decoder to enhance mesh generation quality. Subsequently, a shape-conditioned autoregressive transformer is trained to generate artist-created meshes. Experiments show that Methanything outperforms previous mesh generation methods with fewer faces.", "questions": "My suggestion is to change the description of mesh generation to mesh reconstruction/remeshing. \n\nI regard the proposed method as a novel learning-based remesher (AR applied), but I don't think the paper could be classified into a generation paper.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Mesh generation toward artist creation has been seldom explored in previous research. This work could serve as an improved remeshing tool, facilitating a more rational topology for the generated 3D assets.\n\n- The visualization is good. The final mesh reconstructions are nice looking, and the method outperforms previous methods in the experiments.\n\n- This work constructs coarse meshes for sampling point clouds as inputs, thereby narrowing the gap between training and inference. This approach is reasonable.", "weaknesses": "- This paper mainly focuses on the task of point cloud -> mesh, which is usually referred to as mesh reconstruction, rather than mesh generation. Though a transformer-based autoregressive architecture is applied, I still don't think it is appropriate to name the paper a mesh generation paper. \n\n- It is unfair to use meshes generated by Rodin as the shape conditioning for comparison with MeshGPT. From my understanding, the Rodin engine largely (or purely) provides the generation ability while MeshAnything only acts as a remesher component.\n\n- Another concern is that the number of AM mesh faces generated by MeshAnything is limited to a maximum of 800. Using such a limited number of faces makes it insufficient for representing meshes with more complex structures.\n\n- This work appears to have stringent data requirements, as the authors were only able to filter 56k high-quality artist-created meshes from the Objaverse and ShapeNet datasets for training. So it may be challenging to scale this method up to more data and modeling parameters (but it might be sufficient to train a point cloud reconstructor/mesh remesher).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "05N6vnZse4", "reviewer": "Reviewer_qFPE", "summary": "This paper tackles the important question of converting various 3D representations to explicit mesh representations similar to what artists create. For this, the authors adopt the idea of MeshGPT and inject shape-conditioned information to make the model focus on learning topology. To make the model agnostic to specific 3D representations, e.g., NeRF or Gaussian Splats, the authors choose to convert all representations to point clouds and generate the mesh based on the features from the point cloud. Further, the authors develop a fine-tuning strategy for the VQ-VAE's decoder to create a noise-resistant one to reduce the gap between the data in training and real usages. A user study demonstrates the effectiveness of the proposed approach.", "questions": "See \"weakness\".", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- originality-wise: the choice of converting all representations to a point cloud and using the point cloud to generate corresponding mesh is interesting;\n- quality-wise: qualitative results are of high quality;\n- clarify-wise: the paper is well-written and easy to follow;\n- significance: to produce a mesh similar to what artists create is of great importance to bridge the current 3D generation research into real-world usage.", "weaknesses": "## Concerns about the user study\n\nSince the goal of the project is to produce artist-created meshes, it is important to make sure that the model's output is aligned with real artists' preferences. However, there is no information about whether the 41 users who participated in the study (L517) are qualified especially Tab. 1 is the main results reported in the paper. For \"qualified\" users, I mean people who are artists with enough experience in the field of 3D creation or editing, etc.\n\nCan authors clarify? If the users are not qualified, I would not be convinced by the result.\n\n## Concerns about comparison with mesh generation pipeline\n\nIn L508, the authors state:\n> MeshAnything is a shape-conditioned mesh generation method, we sampled shapes randomly from the evaluation set of Objaverse as inputs for MeshAnything, while for the baseline methods, we performed random sampling directly.\n\nFurther, in L862, the authors state:\n> We quantitatively evaluate mesh quality by uniformly sampling 100K points from the faces of both the ground truth meshes and the predicted meshes, and then computing a set of metrics to assess various aspects of the reconstruction.\n\nIf I understand correctly:\n- the procedure for evaluating MeshAnything is: 1) sample a mesh and then sample points from the evaluation set; 2) run MeshAnything on the sampled point cloud to obtain a predicted mesh; and 3) sample points from the predicted mesh.\n- the procedure for the baselines are 1) directly run the generation; and 2) sample points from the generated mesh.\n\nSuch a setup is quite unfair in my opinion. Essentially, MeshAnything generates meshes aligned with the evaluation dataset. I think it will surely perform well in terms of those metrics for evaluating generation qualities in Tab. 2. Especially on Objaverse, which has tons of various categorical objects, the unfairness will be amplified.\n\nI also agree that it is not easy to compare with baselines as MeshAnything needs to have a point cloud conditioning. ShapeNet may be more suitable due to the constrained categories.\n\n## Questions about generation procedure\n\nCan the authors clarify how the generation is conducted? As a generative framework, should the model output various plausible meshes based on the input point cloud? However, in the paper, there is always one mesh corresponding to the input. Does this mean that the authors always use **max** logits during the sampling, i.e., greedy search? Can authors provide more generation variants, e.g., with beam search? I would like to know how diverse the mesh generation could be.\n\nFurther, a question related to the evaluation on Tab. 2 is whether the authors only use **max** sampling to evaluate. \n\n## Insufficient qualitative results\n\nThere are actually no qualitative comparisons to baselines, e.g., MeshGPT and PolyGen in the paper. Please provide. \n\n## Citation format\n\nPlease refer to the template instructions and use the correct citation commands. Currently, all citations are with `\\citet` instead of `\\citep`. They look so weird as those citations appearing without parenthesis break the sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qyEsyD6A5F", "reviewer": "Reviewer_3Tqd", "summary": "This paper presents MeshAnything, a model framing mesh extraction as a generative task, resulting in artist-crafted meshes that align with specified shapes. The approach first establishes a mesh vocabulary through VQ-VAE, followed by training a shape-conditioned, decoder-only transformer with a noise-resistant decoder design on the learned vocabulary, enabling shape-conditioned autoregressive mesh generation. This paper is well-crafted, and I anticipate that it will positively influence the field of 3D content generation though the pipeline appears somewhat straightforward. The performance of the method seems promising.", "questions": "1. Will the data and code be available for reproducibility?\n2. Can the limitations and potential future work be expanded in the main paper? Are there any failure cases the method cannot handle? Additionally, what would happen if watertightness is required during generation? Including these discussions could enhance the paper.\n3. \"Due to its generative nature, our method is not as stable as reconstruction-based mesh extraction methods.\" Could the authors clarify the meaning of \"stable\" in this context?\n4. How does the method perform on open surfaces?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is a well-written paper that clearly conveys the authors' contributions. The topic is highly relevant to the field, as it addresses the gap between generated shapes and practical applications - I expect the authors will consider open-sourcing their code to facilitate further research. The effectiveness of the method is demonstrated in the experimental sections.", "weaknesses": "I recommend that the authors showcase real-world application examples to highlight the significance of their research. For example, the authors could demonstrate how Artist-Created Meshes (AMs) improve shape manipulation for artist-driven shape modifications or offer enhanced rendering performance (e.g., efficiency) compared to conventional dense meshes.\n\n## Related Works\nAdditionally, recent advancements in 3D reconstruction and generation could be acknowledged, such as:\n- Part123: Part-aware 3D Reconstruction from a Single-view Image (SIGGRAPH 2024)\n- SyncDreamer: Generating Multiview-consistent Images from a Single-view Image ICLR 2024.\n- CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (SIGGRAPH 2024)\n\n\n## Exposition\n\n- \"MeshAnything converts any 3D representation into Artist-Created Meshes (AMs), i.e., meshes created by human artists.\" Consider rephrasing \"any 3D representation\" to \"different 3D representations.\"\n\n- \"This is because all meshes produced by these methods (Lorensen & Cline, 1987; Chernyaev, 1995; Lorensen & Cline, 1998; Shen et al., 2021b; Chen et al., 2022; Shen et al., 2023) exhibit significantly poorer topology quality compared to AMs. As shown in Fig. 2, these methods rely on dense faces to reconstruct 3D shapes, completely ignoring geometric characteristics.\" The logic here seems unclear. Generally, mesh topology pertains to the connectivity of vertices, while geometric features relate to vertex positions. Could the authors clarify how \"poorer topology quality (relying on dense faces)\" affects \"geometric characteristics\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)