MeshXL: Neural Coordinate Field for Generative 3D Foundation Models
Authors: Sijin Chen
Xin Chen
Anqi Pang
Xianfang Zeng
Wei Cheng
Yijun Fu
Fukun Yin
Zhibin Wang
Jingyi Yu
Gang Yu
BIN FU
Tao Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D
large generation models
Primary area: generative_models
TL;DR: We present MeshXL for large-scale auto-regressive 3D mesh generative pre-training.

Abstract: The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications. However, given its unstructured graph representation, the direct generation of high-fidelity 3D meshes is challenging. Fortunately, with a pre-defined ordering strategy, 3D meshes can be represented as sequences, and the generation process can be seamlessly treated as an auto-regressive problem. In this paper, we validate Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling. After that, we present MeshXL, a family of generative pre-trained auto-regressive models that addresses 3D mesh generation with modern large language model approaches. Extensive experiments show that MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.

MeshXL: Neural Coordinate Field for Generative 3D Foundation Models
Sijin Chen 1.,,, Xin Chen2.1, Anqi Pang", Xianfang Zeng", Wei Cheng', Yijun Fu?,
Fukun Yin Zhibin Wang", Jingyi Yu², Gang Yu², Bin Fu², Tao Chen
https: /i thub. com http///eeeaaaa/Meaaaaaa Fudan University Tencent PCG SShanghaiTede Tech University project lead
corresponding author
Figure 1: MeshXL can auto-regressively generate high-quality 3D meshes. We validate that Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is simple-yet-effecctive sequence representation for large-scale mesh modelling.
Abstract
The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications. However, given unstructured graph representation, the direct generation of high-fidelity 3D meshes is challenging. Fortunately, with pre-defined ordering strategy, 3D meshes can be represented as sequences, and the generation process can be seamlessly treated as an auto-regressive problem. In this paper, we validate Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling. After that, we present MeshXI,, family generative pre-trained auto-regressive models that addresses 3D mesh generation with modern large language model approaches. Extensive experiments show that MeshXL is able to generate high-quality 3D meshes, and can also serve foundation models for various down-stream applications.
Rescarch done when Sijin Chen was a rescarch intern at Tencent PCG.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The generation of high-quality 3D assets [61, 77, 29] is essential for various applications in video games, virtual reality, and robotics. Among existing 3D representations [51, 38, 57, 61]. 3D mesh represents 3D data graphs, which possesses the flexibility and accuracy representing sharp edges as well as both flat and curved surfaces. However, the direct generation of high-quality 3D meshes is challenging, given 1) the unstructured graph representation and 2) the demand for estimating accurate spatial locations and connectivity within vertices.
To generate 3D meshes, many works adopt an indirect way by first producing data in other 3D representations, such as point clouds [97, 49, 54], SDF [88, 94), and multi-view images [46, 82, 30j. After that, they adopt re-meshing methods [37] post-process the generated geometries. There are also attempts towards the direct generation of 3D polynomial meshes. PolyGen [53] adopts Iwo separate decoder-only transformers for vertices generation and vertices connectivity prediction. MeshGPT [65] first builds a mesh VQVAE to first turn meshes into tokens, and then learns to generate the token sequences with a GPT model [59]. Meanwhile, PolyDiff [2] directly adopts discrete denoising diffusion [4] on the discretized mesh coordinates.
Though these methods have achieved initial success in creating 3D assets, they suffer from certain limitations. To preserve sufficient high-frequency information, point clouds and voxels requires dense samplings on the object surfaces, which inevitably leads great redundancy while representing flat surfaces. The reconstruction- basse methods [82, 30, 67]. however, rely heavily on the accuracy of the multi-vew generation pipelines [46]. Additionally, the VQVAE-based 3D generation methods [88, 65] require sophisticated multi-stage training, which less favors learning from large scale data.
To tackle the above challenges and explore the potential of scaling up 3D generative pre-traee tre-training, way of 3D mesh representation, the Neural Coordinate Field
we introduce a (NeurCF). NeurCF represents the explicit 3D coordinates with implicit neural embeddings. We show that with a pre-defined ordering strategy, a 3D mesh can be represented by a one-and and--nld coordinate sequence, which further helps formulate 3D mesh generation an auto-regressive problem. After that, we present MeshXI., a family of generative pre-trained transformers [93, 59], for the direct generation of high-fidelity 3D meshes. Without resorting intermediate 3DD representations, NeurCF facilitates an end-to-end learning pipeline for the direct pre- pre-training on large-scale 3D mesh data.
By organizing high-quality 3D assets from ShapeNet [9], 3D-FUTURE [22]. Objaverse [17], and Objaverse- [16], we achieve collection of over 2.5 million 3D meshes to support large-scale generative pre-training. Extensive experiments demonstrate that the NeurCF representation facilitates MeshXL to generate higher-quality 3D meshes. By training on the collection of large-scale 3D mesh data, MeshXL can achieve better performance with larger numbers of parameters (Fig. 3 and Tab. 5), and surpass prior arts on multiple categories in the ShapeNet dataset [9] (Tab. 3).
In summary, our contributions can be summarized as follows:
We validate that Neural Coordinate Field is a sinple-and-effective representation of 3D mesh which is also friendly to large-scale auto-regressive pre-training.
We present a family of MeshXL.s that can be treated as strong base models for image-conditioned or text-conditioned 3D mesh generation tasks.
We show that MeshXL surpasses state-of-the-ard 3D mesh generation methods, and can produce delicate 3D meshes compatible with existing texturing methods.
2 Related Work
First, we present concise review of existing 3D representations. Subsequently, we discuss related works on 3D generation and recent efforts in developing 3D foundation models.
3D Representations. Researchers have long sought for accurate and efficient methods to represent 3D data. Point Cloud [54, 57, 58, 89] captures the spatial positions of discrete points in the Euclidean space, which is preferred by various 3D sensors [15, 87, 66, 3, 7]. Mesh [53, 2, 65, 12] represents the 3D structure with graphs. By connecting the vertices with edges, mesh can also be interpreted into a set of polygons the 3D space. Similar to point clouds, 3D Gaussians [38, 68] also record
the discrete Euclidean distribution in 3D space. However, each point is represented by 3D Gaussian

---

distribution function parameterized by its covariance matrix, color, and opacity. Given their fast convergence and rendering speed, 3D gaussians are often utilized for 3D reconstruction. Neural Radiance Field (NeRF) [51, 5] constructs a learnable volumetric function f using neural networks trained on multi-view images. Due to its derivability and flexibility, NeRF is also favored for 3D generative models [46, 99, 76, 56]. Additionally, there are other 3D representations such as multi-view images [74, 90, 100], voxel fields [61, 13, 45], and signed distance fields [94], among others [64, 88, 63J. In this paper, we consider the Neural Coordinate Field (NeurCF), an explicit spatial representation with implicit neural embeddings, and investigate its potential for scalable 3D asset generation.
3D Generation. With the exploration of various 3D representations and the collection of large-scale 3D datasets [17, 9, 16], researchers have also put much effort exploring the generation of high-fidelity 3D assets [42, 39]. The Generative Adversarial Network (GAN) [25, 80, 1,33 produces synthetic 3D data with generator G. and train a discriminator network to distinguish the generated and real data. Additionally, the potential of diffusion models [54, 28, 62] in the direct generation of 3D data is also widely explored [97, 2, 54, 50, 47]. The key idea behind diffusion is to transform the desired data distribution into a simpler distribution (e.g. gaussian) and learn desnoising model for the reverse process. Besides, researchers have also explored the potential of diffusion models in generating multi-view images [46, 16, 82, 43], and reconstruct them into 3D structures. In this paper, we mainly explore the auto-regressive methods for 3D) generation. AutoSDF [52] and MeshGPT [65] learn to generate discrete tokens and reconstruct them into 3D) representations with a VQVAE model [72]. PolyGen [53] adopts two decoder-only transformers that predict the location and connectivity of vertices, sequentially. In this paper, we explore the potential of explicit sequential modelling method for 3D meshes, and present a family of generative pre-trained transformers, MeshXL, for high-fidelity 3D mesh generation.
3D Foundation Models. The collection of large-scale high-quality 3D data [17, 16, 9, 81, 71, 21, 22] builds up the foundation for various 3D-related tasks [83, 27, 10, 41]. To explore the scaling effects in 3D learning, researchers have made great endeavors in building 3D foundation models for 3D understanding [96, 44, 98, 85, 86, 92, 100], reconstruction [30, 78, 67, 46, 16, 84, 73], and generation [61, 29, 65, 8J. With the introduction of large-scale 3D data in both variety and granularity [34, 41, 16], existing 3D foundation models are capable of generalizing to unseen
concepts [100, 86, 44], generating high-fidelity 3D assets [88, 36, 65], responding to complex instructions [31, 10, 32, 41], and generating actions that interacts with the 3D environments [20, 79, 95]. In this paper, we present fully end-to-end 3D mesh generation pipeline, explore the scaling effect for large-scale pre-training, and test whether our method can serve as a well-trained foundation model for various down-stream tasks.
3 Data
Data Sources. We provide details on the 3D data collections we use to train and evaluate our models. The whole data collection is built upon four widely-acknowledged 3D mesh datasets, i.e. ShapeNet v2 [9], 3D-FUTURE [22], Objaverse [17], and Objaverse-XL [16].
ShapeNet V2 [9] collects about 51k 3D CAD models for 55 categories. We split the data in 9:1 for training and validation by each category.
3D-FUTURE [22] present about 10k high-quality 3D mesh data for indoor furniture. However, because of the delicate design, the objects contain many faces. Therefore, only a small proportion of the data can be used to train our MeshXI. models.
Objaverse [17] is a large 3D data collection with more than 800k 3DD objects for about 21 cate- gories collected from Sketchfab. We split the data in 99:1 1 for training and validation, respectively.
Objaverse-XL [16] further expand Objaverse [17] into a dataset with more than 10M 3D objects with additional data collected from GitHub. Polycam, Thingiverse, and Smithsonian. We split the Github and Thingiverse part of the Objaverse-XX XL dataset into 99:1 1 for training and validation.
Data collection and filtering. To organize existing datasets, we build up filtering and pre-processing pipeline to ensure that the meshes meet our demand. We first collect meshes with fewer than 800 faces, and ensure that they have corresponding UV maps for rendering. After that, we render the 3D
3

---

Next Cooddrate Prediction
Coostinat Enitedding 6
(6.g) F00 (00))
Flac Ersbeskiing
foe ffon(("))- (1()))
Mash Einbooling
(a) Neural Coordinate Field
(b) Auto-regressive Mesh Generation
Figure 2: Mesh Representation. We present the Neural Coordinate Field (NeurCF) encode the discretized coordinates in the Euclidean space. Benefiting from NeurCF and pre-defined ordering strategy, our proposed MeshXL can directly generate the unstructured 3D mesh auto-regressively.
meshes, and discard those not center-aligned or occupying less than 10% of the rendered image. For those 3D objects with more than 800 but less than 20,000 faces, we use planar decimation to simplify the meshes. Finally, we achieve approximately 2.5 million 3D meshes (Tab. 1).
Planar Decimation Pipeline. To ensure the quality of the decimated 3D meshes, make sure either a lower Hausdorff distance Ohassdorff [65] or a similar rendered views [11].
Collecting text-mesh pairs. We render the 3D meshes with 12 different views and use CogVLM [75] to annotate 1) the front view and 2) the concatenated multi-view image. Then, we adopt the Mistral- 7B-Instruct model [35] with in-context examples generate a fused mesh caption.
Data Statistics. We present the data statistics of our large-scale 3D mesh collection in Tab. 1. After organizing and combing 3D assets from ShapeNet [9], 3D-FUTURE [22], Objaverse [17], and Objaverse-XL [16], could achieve total of 2.5 million 3D meshes.
Table Statistics for the Training Data and Validation Data. After combining four data sources, proposed MeshXL models trained approximately 2.5 million 3D meshes.
Pre-training
Text-to-3i 3D
Dataset
Train
Val
Train
Val
ShapeNet [9]
16,001
1,754
15,384
1,728
3D-Future [22]
1,603
Objaverse [17]
85,282
854
83,501
820
Objaverse-XL [16]
2,407,337
15,200
1,347,802
13,579
Total
2,510,223
17,808
1,446,678
16,127
Neural Coordinate Field
Neural Coordinate Field (NeurCF) is explicit representation with implicit neural embeddings. To be specific, for a Euclidean 3D coordinate system, we can partition the vertices coordinates into an N grid. Then, each discretized coordinate p -a,, y, z) can be encoded with the coordinate embedding layer ε, where F(p) (E(x), E()). Therefore, k-sided polynomial face f()
can be encoded with (f(() = (F(p) ( ... F(lp) ())). For simplicity, we share the learnable coordinate embeddings & among axes.
Ordering. Due to the graph representation, the order of the mesh vertices and the order of the edges among them are permutation-invariann. A pre-defined ordering strategy essential facilitate the sequence modelling in MeshXL. We employ the same ordering strategy as PolyGen [53] and MeshGPT [65]. The mesh coordinates are first normalized into unit cube based on the mesh's longest axis, and discretized into unsigned integers. Within each face, the vertices are cyclically permuted based their coordinates (z-y-z: order, from lower to higher), which helps to preserve the direction of normal vectors. Then, we order these faces based on the permuted coordinates (lower to high). To this end, we can represent each 3D mesh with a one-and-only coordinate sequence, aiding large-scale scale generative pre-training large collection of 3D mesh data. With the NeurCF

---

1.350
16
350M
125M
ShapeNet Validation PPL
1.325
350M
125M
1.325
350M
125M
Objaverse-XL Validation PPL
1.425
125M
350M
1.38
15
1.300
Objaverse Validation
1.300
1.400
Train PPL
1.4
1:275
1:275
1.375
1.250
1.250
1.350
1.225
1.225
1.325
1.2
1.200
1.200
1.300
1.175
1,175
1.1
100
120
1.150
1.150
50
100
1:250
100
Processed
Processed Tokens (Billions)
Processed Tokens (Billions)
Processed
Tokens (Billions)
Tokens (Billinns)
Figure 3: Training and Validation Perplexity (PPL) for MeshXI. Models. We train all the models from scratch on 150 billion tokens. We observe that the performance grows with model sizes.
representation, an n-faced 3D k-sided polynomial mesh can be represented as the coordinate sequence M E Z"xix3, and further be encoded into Emesh (E (f(1), [
Sequential Mesh Representation. One direct way to represent the 3D meshes to directly reshape M into vector with (n 33) tokens. As a special case, an n-faced triangular mesh can be represented by vector with 9n tokens. Mcanwhile, our representation can also be expanded to hybrid polynomial mesh representations with the proper introduction of separate tokens. For example, we can generate triangles within "<tri> ... </tri>" and quadrilaterals within "<quad> </quad>" also in one sequence. To identify the start and end of mesh sequence, we add a <bos> ("begin-of-xequencce) token before the mesh sequence and an <eos> l'end-of-sequencc" token after.
Comparisons. Since we represent each coordinate with learnable embeddings. NeurCF is end-to- end trainable representation for unstructured 3D meshes. Comparing to the decoupled vertex and polygon representation PolyGen [53], NeurCF only requires one coordinate sequence for each 3D mesh. Additionally, NeurCF is storage and computation efficient comparing to voxel fields (O(N3)) since it can easily scale up the resolution with a complexity of ((N).
Method
We first present the architecture and training objective for MeshXL models. Then, we show that MeshXL models can take additional modality as the condition for controllable 3D assets generation. After this, we investigate the effects of scaling.
Architecture. Sec, 4, we present sinple-yet-effecctive way to represent 3D mesh into a sequence. Therefore, the learning of 3D mesh generation can be formulated as auto-regressive problem, which can be seaminglessly addressed by modern Large Language Model (LLM) approaches. In our paper, adopt the decoder-only transformers using the OPT [93] codebase adapt the pre-trained OPT models to our next-coordinat. prediction setting, we fine-tune the whole model with wewly-iiitialized coordinate and position embeddings.
base models. To
Generative Pre-Traie Pre-Training. We train MeshXL models using the standard next-token prediction loss. Given the trainable weights and an |s|-length sequence s, the generation loss is calculated as:
|+|
MesthX (0) = log P (111)))
For each mesh sequence, we add a <bos> token before the mesh tokens, and an <cos> token after to identify the ending of 3D mesh. During inference, we adopt the top-k and top-p sampling strategy to produce diverse outputs.
X'-to-Mesh Generation. Here we mainly consider generating 3D meshes from images and texts. We first turn the extra conditions into tokens with pre-trained encoders [18, 19]. To align the additional
5

---

text/image feature with the mesh coordinate field, we adopt the Q-Former architecture [40] to compress the encoded feature into fixed-lengti of learnable tokens as the prefix of the MeshXL model. The overall training objective for the conditional mesh generation is shown in Eq. (2):
|s|
Lx to mesh (0) (0)=
Σ log P (111)111,,,,1111144
During inference, the model predicts the mesh tokens after the fixed-length prefix.
Scaling Up. We present MeshXL in various sizes, including 125M, 350M, and 1.3B. The detailed hyperparameters for training different models can be found in Tab. 2. To better analyze the scaling effects, we train all models from scratch on 150 billion tokens. We provide both training curve and validation perplexity for different models in Fig. 3. One can see that as the number of parameters grows, the model achieves a lower validation perplexity, indicating a higher probability to produce the validation data.
Table 2: Hyperparameters for different MeshXL Base Models. We present three MeshXL models with 125M, 350M, and 1.3B parameters, respectively.
Hyperparameters
MeshXL(125M)
MeshXL(350M)
MeshXL(1.3B)
# Layers
12
24
24
# Heads
12
16
32
dmudeel
768
1,024
2,048
deEN
3,072
4,096
8,192
Optimizer
Adam (33=0.9, 3g=0.99
Learning rate
1.0 10
1.0 10
1.0 x 10-4
LR scheduler
Cosine
Cosine
Cosine
Weight decay
0.1
0.1
0.1
Gradient Clip
1.0
1.0
1.0
Number of GPUs # GPU hrs (A100)
8
16
32
1,944
6,000
23,232
6
Experiments
We first briefly introduce the data, metries, and implementation details in Sec. 6.1. Then, we provide evaluations and comparisons on the generated meshes (cf. Sec. 6.2) and ablations (cf. Sec. 6.3). We also provide visualization results in Sec. 6.4.
6.1 Data, Metrics, and Implementation Details
Data. We pre-train the base model with 2.5 million 3D meshes collected from the combination of
ShapeNet [9], 3D-FUTURE [22], Objaverse [17], and Objaverse- [16]. We use planar decimation on meshes with more than 800 faces following MeshGPT [65] and Robustl RRobusLLowPolly [11]. For generative mesh pre-training, we randomly rotate these meshes with degrees from (0 90%, 180%, 270%), and adopt random scaling along each axis within range [0.9, 1.1] for data augmentation.
Metrics. We follow the standard evaluation protocols in MeshGPT [65] and PolyDiff [2] to measure the quality of the generated meshes with the following metrics. We use Coverage (COV) to quantify the diversity of the generated meshes, which is sensitive to mode dropping but cannot be used to assess the generation quality. Minimum Matching Distance (MMD) calculates the average distance between the reference set and their closest neighbors in the generated set, but is not sensitive to low-quality results. The --Nearest Neighbor Accuracy (1-NNA) quantifies the quality and diversity between the generation set and the reference set, whose optimal value is 50%. We also adopt the Jensen-Shannon Divergence (JSD) score. Among all the above metrics, we use Chamfer Distance to measure the similarity between two samples. We also render the generated meshes and adopt the Frechet Inception Distance (FID) and Kernel Inception Distance (KID) on the rendered images for feature-level evaluation. We multiply the MMD, JSD, and KID scores by 10
Implementation. We conduct all the experiments on a cluster consisting 128 A100 GPUs. We train our models under biloat 16 with the ZeRO-2 strategy [60] using the AdamW [48] optimizer with

---

Table 3: Quantitative Comparisons with Prior Arts on ShapeNet [9]. We scale MMD, JSD, KID by 10 MeshXL. can produce diverse and high-quality 3D meshes.
Category
Methods
COVt
MMDI
1-NNA
JSDL
FID..
KID,
PolyGien [53]]
7.79
16.00
99.16
228.80
63.49
43.73
GET3D [23]
11.70
15.92
99.75
155.25
67.84
42.10
MeshGPT [65]
42.00
4.75
69.50
55.16
39.52
8.97
Chair
MeshXI. (125M)
50.80
3.11
56.55
9.69
28.15
1.48
MeshXL (350M)
50.80
3.17
55.80
9.66
28.29
1.39
MeshXL (1.3B)
51.60
3.23
55.80
9.48
9.12
1.84
PolyGen [53]
44.00
3.36
67.20
25.06
54.08
14.96
GET3D [23]
16.80
10.39
91.90
226.97
67.65
34.62
MeshGPT [65]
34.30
6.51
75.05
92.88
53.75
7.75
Table
MeshXL(125MM
51.21
2.96
57.96
12.82
42.55
0.92
MeshXL
49.70
3.07
56.10
13.64
43.43
1.27
MeshXL (1.3B)
52.12
56.80
14.93
22.29
2.03
PolyGen [53]
31.15
4.01
83.23
55,25
70.53
12.1
MeshGPT [65]
34.92
2.22
68.65
57.32
52.47
6.49
Bench
MeshXI((125M)
54.37
1.65
43.75
16.43
35.31
0.82
MeshXL (350M)
53.37
1.65
42.96
15,41
36.35
0.96
MeshXL (1.3B)
56.55
1.62
39.78
15.51
35.50
1.60
PolyGee [53]
35.04
7.87
75.49
96.57
65.15
12.78
MeshGPT [65]
41.59
4.92
61.59
61.82
47.19
5.19
Lamp
MeshXL (125M)
55.86
48.24
43.41
34.61
0.84
MeshXL (350M)
53.52
4.18
49.41
34.87
25.94
1.92
MeshXL (1,3B)
51.95
4.89
47.27
41.89
31.66
0.99
learning rate decaying from 10-4 to 10-6 and weight decay of 0.1. The detailed hyperparameters for different models can be found in Tab. 2. To train our base models, we load the weights from the pre-trained OPT models [93] and initialize the word embeddings and positional embeddings from scratch. Without further specification, we generate 3D meshes with the top-k and top-p sampling strategy with & 50 and p 0.95.
6.2 Evaluations and Comparisons
We provide quantitative as well as qualitative comparisons on both unconditional and conditional 3D mesh generation on public benchmarks.
Unconditional Generation. We evaluate MeshXL as well as other baseline methods using the ShapeNet [9] data in Tab. 3. We split the data by 9:1 for training and validation by cach category. For evaluation, we fine-tunee our pre-trained base model and sample 1,000 meshes for each category. Among the listed methods, With a similar number of parameters, Mesh-XL. (350M) out out-performs MeshGPT by a large margin, showing higher COV score, lower MMD score, and closer 1-NNA score to 50%. This indicates that MeshXL can produce diverse and high-quality 3D meshes.
reproduce the MeshGPT [65] with a GPT2-medium model (355M) [59].
Table 4: User Study. Compared baseline methods, the meshes generated by MeshXL are better aligned with human preference terms of both geometry and designs.
Methods
Quality
Artistic
Triangulation"
PolyGen [53]
2.53
2.72
3.15
GET3D [23]
3.15
2.46
3.15
MeshXI.
3.96
3,45
3.72
Reals
4.08
3.33
3.75
User Study. To evaluate how well the generated 3D meshes align with human preference, we perform user studies on the chair category in Tab. with several baseline methods [53, 23]. We recruit and instruct the participants to score each mesh from 0 to 5 (higher is better) based on its 1) quality: the smoothness of object surfaces and completeness of the mesh, 2) artistic: how much do you believe this object is designed and created by artists, and 3) triangulation: how well do the connectivity among vertices aligns with the models created by professional designing software [14]. baseline evaluation, also ask the participants to score the ground truth 3D geometries sampled from the
7

---

Input
Completed Mesh
Ground Truth
Figure Evaluation of Partial Mesh Completion. Given some partial observation of the 3D mesh (white), MeshXL is able to produce diverse object completion results (blue).
ShapeNet data. We have collected a total of 434 valid responses. The results show that the 3D meshes created by MeshXI. are consistently preferred by human in all dimensions.
6.3 Ablation Studies
Necessity of Mesh VQVAE. Comparing to MeshGPT [65], MeshXL is an end-to-end trainable model that produces 3D meshes with next-coordinate prediction. We show Tab. that, MeshXL out-performs MeshGPT with similar numbers of parameters. We also show that MeshXL can produce high quality 3D meshes with both sharp edges and smooth surfaces in Fig. Furthermore, McshXL save the effort training vector quantized mesh tokenizer [65, 72]. which further facilitates generative pre-training on large scale datasets.
Effectiveness of Model Sizes. To analyze whether pre-training a larger model benefits 3D mesh generation, we evaluate MeshXL base models with different sizes on the Objaverse [17] dataset in Tab. We observe that as the model size grows, the generated samples exhibits a larger COV, smaller ISD score, and a closer 1-NNA to 50%, which indicates an improved diversity and quality.
Table 5: Effectiveness of Model Sizes on Objaverse. As the model size grows, MeshXL achieves closer 1-NNA 50%, larger COV and smaller JSD, indicating better diversity and quality.
Method
COVt
MMDJ
1-NNA
JSDI
FID
KID +
MeshXL. (125M)
39.76
5.21
67.34
26.03
17.32
4.48
MeshXL (350M)
40.79
5.20
65.68
23.71
15.14
3.33
MeshXL (1.3B)
42.86
4.16
61.56
20.99
12.49
2.94
Shape Completion. To analysis whether our method is capable of producing diverse outputs, we adopt MeshXI. (1.3B) model predict the whole object given some partial observations. In practice, we use 50% of the object mesh as input, and ask the model to predict the rest 50% in Fig. 4. One can that Mesh-XL is able produce diverse and reasonable outputs given the partial observation of the 3D mesh
X X-to-Mesh Generation. To adopt the MeshXL base models to the X-toommssh generation setting, we adopt the Q-Former [40] to encode the additional conditions prefixes. We showcases several conditional generation results in Fig. 5. We show that MeshXI. can generate high-quality 3D meshes given the corresponding image text the additional inputs.

---

Image
Generated
Ground Truth
Text
Generated
Ground Truth
basic with four leges and an open back."
"4 legs, solid seat and backing."
"A basic looking square wooden table."
Figure 5: Evaluation X'-to-mesh generation. We show that MeshXL can generate high-quality 3D meshes given the corresponding image or text as the additional inputs.
Texturing. We adopt Paint3D [91], coarse-to-fine texture generation pipeline, generate textures for the 3D meshes produced by MeshXL in Fig. 6. We show that 3D meshes produced by MeshXL can easily fit in existing texturing methods to produce high-quality 3D assets.
Generaled
Texture Mesh
UV Map
Gerurated
Texhued Mesh
Map
Figure 6: Texture Generation for the Generated 3D Meshes. We adopt Paint3D [91] to generate textures for 3D meshes produced by MeshXL.
6.4 Visualizations
We provide qualitative comparisons on the generated meshes in Fig. 7. MeshXL is able to produce high-quality 3D meshes with both sharp edges and smooth surfaces. We also visualize the normal vectors to compare the smoothness of object surfaces. The results show that 3D meshes generated by GET3D [23] have rough surfaces with tens of thousands of triangles, while MeshXI. depicts the shape with much smoother surfaces and less triangles.

---

PolyGen
GET3D
MeshGPT
MIsIXL
Figure 7:
Qualitative comparisons. We visualize the generated meshes and normal vectors. MeshXL is able to produce high-quality 3D meshes with both sharp edges and smooth surfaces.
Discussions
Difference with PolyGen [53]. PolyGen treats 3D mesh data as vertex sequence and face sequence. PolyGen first generates the ordered vertices with the vertex transformer, then predicts the connectivity PolyGen,
among vertices with the face transformer. Comparing a more straightforward approach that models the 3D mesh as a one-and-only coordinate sequence, which further supports the direct and end-to-end pre-training on a large collection of 3D data.
proposed MeshXL adopts
Difference with MeshGPT [65]. MeshGPT consists of a mesh VQVAE [72] and a decoder-one only transformer [59]. MeshGPT first learns mesh VQVAE quantize the 3D meshes into discrete tokens. After that, MeshGPT trains decoder-only transformer to generate the discrete tokens for 3D mesh reconstruction. In comparison, proposed MeshXI. is end-to-end method that learns the neural representation of coordinates and outputs 3D meshes directly.
Extensibility. Our method, MeshXL, built upon the concept of auto-regressive methods. Therefore, our method is not restricted to the decoder-only transformers [59, 93, 69, 70], and can also be extended to other causal language models (i.e. Mamba [26], RWKV [55], and xLSTM [6]).
8 Limitations, Future Work, and Conclusions
Limitations and Future Work. The main drawback of MeshXLs is the inference time. During sampling. MeshXL will generate 7,200 tokens for an 800-faced 3D mesh, which takes a relatively long time because of the auto-regressive process. As for future works, recent endeavors the RNN-related methods [6, 55, 26) and multiple tokens prediction for LLMs [24] might open up great opportunities in saving the inference cost.
Conclusion. We validate that NeurCF, an explicit coordinate representation with implicit neural embeddings, is sumple-an--effective representation of 3D meshes. By modelling the 3D mesh generation as an auto-regressive problem, we seek help from modern LLM approaches and present a family of generative pre-trained models, MeshXL, for high-fidelity 3D mesh generation. We show that MeshXL performs better given larger-scale training data and increased parameters. Extensive results show our proposed MeshXI. can only generate high-quality 3D meshes, but also exhibits great potential serving as base models for conditional 3D assets generation.
Acknowledgement
This work is supported by National Natural Science Foundation of China (No. 62071127), National Key Research and Development Program of China (No. 2022ZD0160101 Shanghai Natural Science Foundation (No. 23222 402900), Shanghai Municipal Science and Technology Major Project (No.20215H2ZDZX00033,
10

---

References
[1] Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference machine learning, pages 40-49. PMLR, 2018.
[2] Antonio Alliegro, Yawar Siddiqui, Tatiana Tommasi, and Matthias Nießner. Polydiff: Generating 3d polygonal meshes with diffusion models, arXiv preprint arXiv:2312.11417, 2023.
[3] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1534-1543, 2016.
[4] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neurad Information Processing Syxtems, 14:17981-17993 2021.
[5] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti- anti-aliasing neural radiance fields. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 5855-5864 2021.
[6] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp. Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04511 007517 2024.
[7] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1297-9307, 2019.
[8] Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Difftf++: 3d-aware diffusion transformer for large-vocabulle vocabullcaar 3d generation. arxiv preprint arXiv:2405.08003 2024.
[9] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, al. Shapenet: An information-rict 3d model repository. arXiv preprint arXiv:1512.0300 151.00302, 2015.
[10] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning. arXiv preprint 2ttdi:::3111166::: 2023.
[11] Zhen Chen, Zherong Pan, Kui Wu, Etienne Vouga, and Xifeng Gao. Robust low-poly meshing for general 3d models. ACM Transactions on Graphics (TOG), 42(4):1-20, 2023.
[12) Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 45 54, 2020.
[13] Christopher Choy, YonYoun Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pages 3075-30010 3084, 2019.
[14] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
[15] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[16] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
[17] Malt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt. Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13152-13153, 2023.
[18] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018
11

---

[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
[20] Danny Driess, Fei Xia, Mehdi SM Sajjadi. Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, al. Palm-e: An embodied multimodal language model. Xrviv preprint arXiv:2333.033777 2023.
[21] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933-10942, 2021.
[22] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision, pages 1-25, 2021.
[23] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin, Daiqing Li, Litany, Zan Gojeie, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31 141--11144, 2022.
[24] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404. 19737, 2024.
[25] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[26] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:3312.00752, 2023.
[27] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao. Xianzhi Li, Hongsheng Li, et al. Point- bind & point-llm: Aligning point cloud with multi modality for 3d understanding, generation, and instruction following, arXiv preprint 20Xiv:2339.006155 2023.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbcel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[29] Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint 2rXiv:2403.022:
2024.
[30] Yicong Hong. Kai Zhang, Jiuxiang Gu, Sai Bi. Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 2023.
[31] Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances Neural Information Processing Systems, 36:20482-20494, 2023.
[32] Jiangyong Huang, Silong Yong. Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[33] Moritz Ibing, Isaak Lim, and Leif Kobbelt. 3d shape generation with grid-based implicit functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13559- 13568, 2021.
[34] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. arXiv preprint wxXiv:200.009344 2024.
[35] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:23
2023.
[36] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[37] Michael Kazhdan. Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings the fourth Eurographics symposium on Geometry processing, volume 7, 2006.
12

---

[38] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühher, and George Drettakis, 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
[39] Chenghao Li, Chaoning Zhang, Atish Waghwase, Lik-Hang Lee, Francois Rameau, Yang Yang. Sung-Ho -o Bae, and Choong Seon Hong. Generative ai meets 3d: A survey on text-to-3d aige era. arXiv preprint arXiv:2305.0613 2023.
[40] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference machine learning, pages 19730-19742. PMLR, 2023.
[41] Mingsheng Li, Xin Chen, Chi Zhang. Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dhench: Let's instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312. 10763, 2023.
[42] Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingho Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3d generation: A survey. arXiv preprint 1787 2024.
[43] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:231 1xXxv:331100788, 1007855 2023.
[44] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han. Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open- understanding. Advances in Neural Information Processing Systems, 36, 2024.
[45] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[46] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov. Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298 1009899909,11 2023.
[47] Zhen Liu, Yao Feng. Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffu- sion: Score-based generative 3d mesh modeling, arXiv preprint arXiv:2303.0813: 2023.
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1111.05001 2017.
[49] Shitong Luo and Wei Hu. Diftusion probabilistic models for 3d point cloud generation. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[50] Zhaoyang Lyu. Jinyi Wang. Yuwei An, Ya Zhang, Dahua Lin, and Bo Dai. Controllable mesh generation through sparse latent point diffusion models. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 271 271-280, 2023.
[51] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[52] Paritosh Mittal, Yen-Chi Cheng. Maneesh Singh, and Shubham Tulsiani. Autosdf' Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 306-315, 2022.
[53] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference machine learning, pages 7220-7229. PMLR, 2020.
[54] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:22 12.0875,, 2022.
[55] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michacl Chung, Mattco Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnss for the transformer era. arXiv preprint arXiv:2305.1 13048, 2023.
[56] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.1490 2022.
13

---

[57] Charles R Qi. Hao Su, Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recogmition, pages 652-660, 2017.
[58] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019.
[60] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[61] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams, Xeube (X**); Large-scale 3d generative modeling using sparse voxel hicrarchics. arXiv preprint ar arXiv:2312.03866, 2023.
[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[63] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101, 2021.
[64] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20875-20886, 2023.
[65] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Niebner. Mcshgpt: Generating triangle meshes with decoder-only transformers. arXiv preprint 1xXiv2331.114/11 2023.
[66] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgh-d scene understanding benchmark suite. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 567-576, 2015.
[67] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint 40X5v:2402.05054, 2024.
[68] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 2023.
[69] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 208888888.888...... 2023.
[71] Mikacla Angelina Uy, Quang- Hieu Pham, Binh- Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark datasct and classification model on real-world data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1588-1597, 2019.
[72] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 7.
[73] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311. 12024, 2023.
[74] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi modal 3d perception suite towards embodied ai. arXiv preprint arXiv:2312. 16170, 2023.
14

---

[75] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi. Yan Wang, Junhui Ji. Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03073 2023.
[76] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances Neural Information Processing Systems, 36, 2024.
[77] Zhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, and Rynson W.H. Lau. Themestation: Generating theme-aware aware 3d assets from few exemplars. 2024.
[78] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshirm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
[79] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li. and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.131333 2023.
[80] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information procesing systems, 29, 2016.
[81] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912- 2015.
[82] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404. 071911 2024.
[83] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empow- ering large language models to understand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
[84] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403.1 11.3./4622) 14621, 2024.
[85] Le Xue, Mingfei Gao, Chen Xing, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Car- los Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1179 1 1189, 2023.
[86] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.0822 2023.
[87] Chandan Yeshwanth, Yuch-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-22, 2023.
[88] Fukun Yin, Xin Chen, Chi Zhang, Biao Jiang, Zibo Zhao, Jiayuan Fan, Gang Yu, Taihao Li, and Tao Chen. Shapegpt: 3d shape generation with a unified multi-modal language model. arXiv preprint 2023.
2rXiv:2311.176/11
[89] Wang Yu, Xuelin Qian, Jingyang Huo, Tiejun Huang, Bo Zhao, and Yanwei Fu. Pushing the limits of 3d shape generation at scale. arXiv preprint arXiv: 11500, 2023.
[90] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chen- ming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale datasct milti-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9150-91 1000-166111111 2023.
[91] Xianfang Zeng. Paint3d: Paint anything 3d with lighting-less texture diffusion models. arXiv preprint 1391:3, 2023.
[92] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552 8562, 2022.
15

---

[93] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 200000. 2022.
[94] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based Shape-image-tex aligned latent representation. Advances in Neural Information Processing Systems, 36, 2024.
[95] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d vision- langonllangggeeeee action generative world model. arXiv preprint 0rXiv:2403.09633 2024.
[96] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale, arXiv preprint arXiv:2310.0611 2023.
[97] Lingi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 5835, October 2021.
[98] Haoyi Zhu, Honghui Yang. Xiaoyang Wu, Di Huang, Sha Zhang, Xianglong He, Tong He, Hengshuang Zhao, Chunhua Shen, Yu Qiao, et al. Ponderv2: Pave the way for 3d foundataion model with universal pre-training paradigm. arXiv preprint arXiv:2310.08586 2023.
199] Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d 3d with advanced diffusion guidance. arXiv preprint arXiv:2305. 18766, 2023.
[100] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng. Zipeng Qin. Shanghang Zhang, and Peng Gao. Pointelip v2: Prompting clip and gpt for powerful 3d open-wont world learning. In Proceedings of the IEFE/CVE International Conference on Computer Vision, pages 2639-2650, 2023.
16

---

A Appendix
A.1 More Visualization Results
Unconditional Results ShapeNet. We visualize unconditional 3D mesh generation results for chair, table, lamp and bench in Fig. 8. One can see that MeshXL is able to produce diverse and high-quality 3D meshes.
Figure 8: Gallery results. Additional generation results for chair, table, lamp, and bench.
Unconditional Generation on Objaverse. We visualize 3D meshes randomly sampled from MeshXL base model in Fig. 9. After training on large-scale collection of 3D mesh data, MeshXI. is able to produce diverse and high-quality 3D meshes.
17

---

Figure 9: Gallery results. MeshXL is able to produce diverse 3D meshes with high quality.
18

---

A.2 Mesh Quality Assessment
How well the triangulation. We evaluate the aspect ratio, face area, and number faces for better evaluation in Tab. 6. Though the meshes generated by MeshXL. have higher average aspect ratio, we manage to achieve smaller variance with much less 3D faces. This indicates the stability of our generation ability and the efficiency of the direct mesh representation. Since train our MeshXLs only on triangular meshes, long-thin triangles inevitably exist in our training data. By co-training our MeshXLs on triangular meshes, 3D quads, and even hybrid representations, we could reduce the existence of long thin triangles for better generation quality.
Table 6: Mesh Quality Assessment. We evaluate the aspect ratio, face area and number faces for the generated 3D meshes.
Aspect Ratio
Face Area
Number of Faces
Method
mean
std.
mean
std.
mean
std.
GET3D [23]
6.27
116.03
0.000
0.000
27251.80 .80
11535.135
MeshXI. (125M)
10.47
16.88
0.031
0.096
327.34
174.53
MeshXL (350M)
10.25
16.09
0.032
0,099
342.24
193.97
MeshXI. (1.3B)
10.23
15.91
0.034
0.102
320.36
195.43
A.3 Inference Time Analysis
The inference cost of MeshXL is closely related to the numbers of generated faces and the model sizes. We perform inference cost analysis with a batch size of one using bfloat1 on a single RTX 3090. We carry out a analysis (Tab. 7) on 1) the average inference time of generating given number triangles, and 2) the average inference time of generating 3D meshes.
Table Inference cost of MeshXI. models. We carry inference cost analysis time duration and memory usage under 6 with single RTX 3090.
MeshXL (125M)
MeshXL (350M)
MeshXL (1.3B)
num faces
time (8)
GPU mem. (GB)
time (s)
GPU mem. (GB)
time (s)
GPU mem. (GB)
100
6.30
1.59
11.30
2.98
12.08
8.41
200
12.50
1.65
22.70
3,20
24.03
9.17
400
25.21
1.85
45.81
3.78
48.09
11.17
800
49.88
2.28
92.19
5.74
96.49
21.66
29.49
44.65
49.43
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of the paper clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work performed by the authors, providing a balanced view of the study's 's scope and potential areas for improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: All the theorems, formulas, and proofs the paper have been numbered and cross-referenced, and all assumptions have been clearly stated referenced in the statement of any theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper has fully disclosed all the information needed to reproduce the main experimental results of the paper the extent that affects the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
21

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will open access to the code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. ceips.cc/puut public/guides httpe/guidddeeodooubibiiiii SoooeeSumiissiooPooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports error bars and other appropriate information about the statistical significance of the experiments, ensuring the reliability and validity of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: For each experiment, the paper provides sufficient information on the computer resource.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttp///eerip//e//ee//eee/ hccpppllcc/tticccsiieeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics, ensuring ethical standards are upheld throughout the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed, providing comprehensive evaluation of the study's broader implications.
23

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
The paper describes safeguards that have been put in place for the responsible release of data or models that have high risk for misuse, ensuring that appropriate measures are taken to mitigate potential risks.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
24

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper are well documented, and the docu- mentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "rzBN9nXU3B", "reviewer": "Reviewer_XQPQ", "summary": "This paper addresses the challenge of generating high-fidelity 3D meshes by introducing Neural Coordinate Field (NeurCF), an effective representation for large-scale sequential mesh modeling. The authors present MeshXL, a family of generative pre-trained auto-regressive models, which applies modern large language model techniques to 3D mesh generation. Extensive experiments demonstrate that MeshXL produces high-quality 3D meshes and outperforms existing methods on various tasks. Key contributions include validating NeurCF as a viable representation for 3D meshes, presenting MeshXL as robust base models for conditioned 3D mesh generation, and showcasing MeshXL's superior performance in generating detailed 3D meshes compatible with current texturing methods.", "questions": "1. How does this method address common mesh surface artifacts in modeling, such as ensuring correct connectivity between different components, avoiding self-intersections, and preventing flipping?\n2. In Section 4, we generate triangles within “<tri> · · · </tri>” and quadrilaterals within “<quad> · · · </quad>”. However, what is the form of the output in the results presented in the paper? Should these sequences of triangles and quadrilaterals be generated separately or can they be combined in the final meshing result?", "limitations": "Please check weakness", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper trains a foundational mesh generation model using extensive datasets from ShapeNet, 3D-FUTURE, Objaverse, and Objaverse-XL, with the addition of data augmentation.\n2. It proposes a novel 3D mesh representation that can be encoded as a token sequence, effectively leveraging the capabilities of autoregressive large language model approaches.\n3. The paper establishes a fair evaluation metric, considering both the generation score (as shown in Table 2) and the 3D mesh quality from a graphics perspective (as shown in Table 3).", "weaknesses": "1. This method does not appear to incorporate domain knowledge from traditional remeshing techniques to ensure correct connectivity between different components, avoid self-intersections, and prevent flipping.\n2. In the user study, more objective metrics for measuring mesh quality should be considered. For instance, in downstream tasks like ray tracing, long thin triangles should be avoided, and aspect ratio can be used to measure how thin these triangles are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "📝 **W1: The absence of domain knowledge to prevent potential artifacts**.\n\n💡 **A:** In our work, we put emphasis on exploring a sequential way to model 3D meshes that better suits large-scale generative training on extensive 3D data. Therefore, the potential generation of surface artifacts is currently the limitation for our method, as well as PolyGen and MeshGPT. However, our MeshXL has the potential to incorporate with traditional methods by treating the domain knowledge as **filters** or even **reward models** to eliminate certain artifacts.\n\n📝 **W2: Accessing meshes with additional objective metrics**.\n\n💡 **A:** We will add more objective metrics in our main paper. We calculate the face area and the aspect ratio with respect to the definition: $\\text{Aspect Ratio} = \\frac{\\text{longest edge}}{\\text{shortest altitude}}$. Since PolyGen generates polygon meshes rather than triangle meshes, we could not calculate the aspect ratio for PolyGen. From the below table, though GET3D achieves a lower average aspect ratio, it suffers from a higher variance with tens of thousands of faces. Meanwhile, MeshXL achieves a much stable aspect ratio with a larger average face area, indicating that the our MeshXL has the stability to generate high-quality 3D meshes.\n\n| Method | Aspect Ratio | | | Face Area | | | Number of Faces| |\n|-|-|-|-|-|-|-|-|-|\n| | mean | std. | | mean | std. | | mean | std. |\n| GET3D | 6.27 | 116.03 | | 0.000 | 0.000 | | 27251.80 | 11535.135 |\n| MeshXL - 125m | 10.47 | 16.88 | | 0.031 | 0.096 | | 327.34 | 174.53 | \n| MeshXL - 350m | 10.25 | 16.09 | | 0.032 | 0.099 | | 342.24 | 193.97 |\n| MeshXL - 1.3b | 10.23 | 15.91 | | 0.034 | 0.102 | | 320.36 | 195.43 |\n\n**How to alleviate long thin triangles**. Currently, we train our MeshXL on 3D triangular mesh data. Therefore, long thin triangles inevitably exist in our ground truth training data, for example, the legs of tables or chairs. To alleviate the generation of long thin triangles, we will train our method on 3D quads in the future, which has been proved to be an efficient and common mesh representation in the 3D design industry, and could avoid the generation of long thin triangles.\n\n\n📝 **Q1: Addressing potential artifacts in the generated meshes.**\n\n💡 **A:** See Weakness 1.\n\n\n\n📝 **Q2: The sequential mesh representations.**\n\n💡 **A:** We will clarify the sequential mesh representations in the revision.\n\n1. **We train only on triangular meshes in our paper**. In order to accelerate the data processing and training procedure, we choose an easier setting by turning all 3D meshes into triangular meshes to validate that the **next-coordinate generation** is capable of high-quality 3D mesh generation. Thus, the output of our method is always represented with $\\text{<tri>} (x,y,z),(x,y,z),(x,y,z); ... \\text{</tri>}$.\n2. **We can easily extend MeshXL to hybrid representations**. In our paper, we present “<tri> · · · </tri>” and “<quad> · · · </quad>” as a potential that our method can be extended to both triangular meshes, 3D quads, and even the hybrid representation. We will further extend our work on the hybrid representation to alleviate the existence of long thin triangles as mentioned in weakness 2.\n3. If there are both triangles and quadrilaterals in a 3D mesh, our method also follows the ordering strategy introduced in line 111 - 114, which first permutes the vertices within each face cyclically based their coordinates (z-y-x order, from lower to higher), and then order the faces based on the permuted coordinates from lower to higher. Therefore, **a 3D mesh with both triangles and quadrilaterals can also be represented by one unique sequence**. Thus, even with the hybrid representations, our MeshXL can generate these 3D meshes in one sequence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6bcuD5TKa9", "reviewer": "Reviewer_e1fg", "summary": "This paper proposes a way to use LLM to generate polygon meshes. The key idea is to model mesh generation as the next coordinate prediction, using a strategy similar to that of prior work like PolyGen and MeshGPT. The paper has shown the capability to create a mesh with reasonable quality. \n\nI think the key contribution for the paper is that it shows the potential to use LLM style scaling to achieve better mesh generation, while the weakness is that it's unclear how much technical contribution given most techniques are already been explored in prior works like MeshGPT and PolyGen.", "questions": "* Question about the ordering. Is the ordering for creating the sequence unique? how does it related/differ from the ordering polygon?  I think that in L103 the paper should cite PolyGEN for the partition idea.\n\n* L29-35 I believe this positioning risks overclaiming. The auto-regressive model can also have cumulative error issues. It's not entirely clear why auto-regressive mesh generation does not have \"great redundancy when representing flat surfaces\".\n\n* L105 - what is \"polynomial face\"? I think L101-106 requires much more detailed descriptions for readers to find it reproduciable.", "limitations": "See the weakness section. Other limitations include restricted context length (i.e. mesh has very long context lengths, as many meshes have trillions of triangles).", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper shows the potential to use large-scale LLM models to produce better mesh generation. Specifically, with more compute and larger models, one can achieve stable training and better mesh generation quality.", "weaknesses": "My main concern about the paper is its potential lack of novelty. The key idea of using an Autoregressive model to generate a mesh has been explored in many prior works, such as PolyGen and MeshGPT. However, the writing and the results show that this paper differs technically from existing methods other than running on larger datasets with larger-scale models. At the same time, it's not clear from the writing how this paper handles many technical challenges of modeling mesh as a sequence in a new way, such as permutation invariance of the faces. \n\nAlso, I think the paper lacks a proper evaluation of mesh quality. The introduction is set to claim that the generative mesh has better quality and is potentially more suitable for downstream applications compared to modeling other representations, such as point clouds or voxels. However, few metrics are indicating that the generative mesh is of good quality that the artist can edit. For example, what's the ratio of the generative mesh being watertight? How well is the triangulation? Most generative metrics this paper reports concern the shape this mesh represents but not how well this mesh triangulates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "📝 **W1: The novelty of MeshXL.**\n\n💡 **A:** The motivation of our MeshXL is to extend the **mesh representations**, **architecture design**, and **training strategy** in existing auto-regressive methods, i.e. PolyGen and MeshGPT, to support **efficient large-scale training on extensive 3D mesh data**. Specifically, our method improves existing methods from the following aspects:\n\n1. **Training strategy.** Both MeshGPT and PolyGen adopt a two-stage training pipeline. This, however, leads to a complicated training procedure less favors large-scale generative pre-training. However, **MeshXL is a fully end-to-end pipeline, which naturally favors large-scale pre-training**.\n2. **Mesh representation and architecture**. By modeling 3D mesh data into **one unique coordinate sequence**, our MeshXL only consists of a decoder-only transformer for **next-coordinate prediction**. Meanwhile, PolyGen requires a vertex sequence and a vertex index sequence to turn vertices into polygons. Additionally, MeshGPT adopts a vector-quantized approach by learning a fixed-size codebook to represent the vertex feature before learning to generate token sequences auto-regressively. Though we have introduced a bit more redundancy in our mesh representation, our method enjoys a much simpler architecture design and better supports large-scale training.\n3. **Data Processing**. The data pre-processing in our MeshXL is also much simpler as we are only required to permute 3D faces into a coordinate sequence. Our simplified data pre-processing increases the total throughput to further support large-scale training. Meanwhile, MeshGPT requires to build a face graph with respect to face connectivity within the 3D meshes to extract face and vertices features.\n\n\n📝 **W2: Additional metrics for mesh quality assessment.**\n\n💡 **A:** We will add more metrics for better mesh quality assessment in the revision.\n1. **How well is the triangulation**. Following suggestions from reviewer XQPQ, we evaluate the aspect ratio, face area, and number of faces for a better evaluation in the following table. Though the meshes generated by our MeshXL have a higher average aspect ratio, we achieve a smaller variance with much less 3D faces. This indicates the **stability** of our generation ability and the **efficiency** of the direct mesh representation. Since we train our MeshXLs only on triangular meshes, long-thin triangles inevitably exist in our training data. In future works, we will co-train our MeshXLs on triangular meshes, 3D quads, and even hybrid representations to reduce the existence of long thin triangles for better generation quality.\n| Method | Aspect Ratio | | | Face Area | | | Number of Faces| |\n|-|-|-|-|-|-|-|-|-|\n| | mean | std. | | mean | std. | | mean | std. |\n| GET3D | 6.27 | 116.03 | | 0.000 | 0.000 | | 27251.80 | 11535.135 |\n| MeshXL - 125m | 10.47 | 16.88 | | 0.031 | 0.096 | | 327.34 | 174.53 | \n| MeshXL - 350m | 10.25 | 16.09 | | 0.032 | 0.099 | | 342.24 | 193.97 |\n| MeshXL - 1.3b | 10.23 | 15.91 | | 0.034 | 0.102 | | 320.36 | 195.43 |\n\n2. **Watertight meshes**. A watertight mesh does not have any boundary edges. Therefore, it is debatabe whether we should generate watertight meshes in 3D assets generation. Currently, watertightness is mainly required to perform physical simulation or turn 3D meshes into implicit field and distant functions. However, in 3D assets generation, many common 3D shapes including cloth, terrain, and leaves are not watertight. Furthermore, it is also challenging and inaccurate to specify the interior of a 3D mesh for an reliable evaluation. Therefore, our method mainly focus on establishing a more direct representation for 3D mesh generation.\n\n\n📝 **Q1: Question about the ordering.**\n\n💡 **A:** We will cite PolyGen in Line 103, and clarify the relation between our mesh representations and PolyGen's in section 3 .\n1. We adopt the same ordering system as PolyGen and MeshGPT, which first permutes the vertices within each face cyclically based their coordinates in z-y-x order (from lower to higher), then permutes the faces based on the permuted coordinates from lower to higher. Since there are no identical polygons in a 3D mesh, the ordering strategy will create **one unique sequence** for each 3D mesh.\n2. **Relation to the ordering in PolyGen**. We adopt the same vertex representation as PolyGen, but a **different mesh representation**. Our MeshXL and PolyGen both represent vertices with discrete coordinates. However, **MeshXL represents a 3D mesh only with an ordered coordinate sequence**, while PolyGen decouples the vertices and polygons in 3D meshes, and represents each 3D mesh with two sequences, i.e., the vertex sequence and an index sequence to connect the generated vertices. Therefore, our coordinate sequence representation enables us to train an end-to-end model directly on large-scale 3D data.\n\n\n\n📝 **Q2: Clarifying the claims.**\n\n💡 **A:** We will clarify our claims and motivations in our revision.\n1. We will modify the claim into \"*comparing to the indirect way of mesh generation, our end-to-end pipeline better supports large-scale pre-training, especially considering the training and data processing efficiency*\". We will also do further study to explore better auto-regressive pipelines.\n2. In our main paper, our intended idea is that the 3D mesh representation has great flexibility that can represent flat surfaces with much less data (i.e. two triangles for a rectangle surface vs lots of points and voxels) and preserve details with more faces in curved surfaces. We will clarify this in our revision.\n\n\n📝 **Q3: The definition of \"polynomial face\".**\n\n💡 **A:** In our paper, a k-sided polynomial face is a polygon with k vertices and k lines. For example, triangles and quadrilaterals are special cases with three and four sides, respectively. We will clarify this in our revision. We have also open-sourced our code to help readers better understand our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6WqNeIYLAm", "reviewer": "Reviewer_CF59", "summary": "The paper proposes MeshXL, a mesh generation model based on the Neural Coordinate Fields(NeurCF), which encodes discretized coordinates of mesh vertices into a sequence of tokens. \nThen a decoder only transformer is trained to generate meshes unconditionally/conditioned on modality. \nThe model is trained on multiple datasets for better performance. \nIn terms of quantitative and qualitative results, the method outperforms other baselines.", "questions": "1. Since the MeshGPT is closely related, I think comparisons should be included. \nhttps://github.com/lucidrains/meshgpt-pytorch\n\n2. As mentioned in the weakness part, I am really wondering why end-to-end training can generate better results compared to two-stage method like PolyGen and MeshGPT, especially considering 2D SOTA generation model like the latent diffusion model is two-stage.\n\n3. I am not sure the comparisons between MeshXL and PolyGen/GET3D are fair in Table. 2 since it uses pre-trained MeshXL and fine-tunes it for a specific category. I believe using the same dataset for training is more convincing.\n\n4. Why does the scaling law seem not to apply to the lamp as shown in Table 2?\n\n5. Normals in Fig. 7 seem worse (Columns 1, 3, 4) compared to GET3D.\n\n6. will you release the code and pre-trained models since training MeshXL takes a lot of resources?", "limitations": "Limitation is discussed, I do not see any issues.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method proposes an end-to-end transformer model for mesh generation based on its neural coordinate field. \nOn the performance side, it produces 3D meshes with better quality.", "weaknesses": "The weaknesses of the paper mainly lies in the technical part and the comparison part. \nTo my knowledge, the main difference between previous discrete mesh generation methods like MeshGPT is that MeshGPT first encodes the mesh with VQVAE and then trains a generation model, while the paper does it in a single stage. \nThe paper does produce great results but seems miss a reason why single stage can give performance. \nIn the introduction part (lines 21–28), the paper discusses previous generative methods but does not explicitly point out why they are missing. \nI think adding more analysis can help readers understand the paper better. \n\nI am not sure the comparisons between MeshXL and PolyGen/GET3D are fair in Table. 2 since it uses pre-trained MeshXL and fine-tunes it on a specific category. I believe using the same dataset for training is more convincing.\n\nFor reference, I think template-based deformation methods and command-based methods can be added to the discussion:\nTM-NET: Deep Generative Networks for Textured Meshes\nShare With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency\n\nDeepSVG: A Hierarchical Generative Network for Vector Graphics Animation\nDeepCAD: A Deep Generative Network for Computer-Aided Design Models\nComputer-Aided Design as Language", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "📝 **W1: Reasons why single stage method works, and analysis on previous works.**\n\n💡 **A:** We thank the reviewer for helping us improve our paper.\n1. **The coordinate sequence representation and auto-regressive generation make our method come true**. With a well-defined ordering system, each 3D mesh can be represented by **one unique coordinate sequence**. Additionally, decoder-only transformers have long been proven to excel in auto-regressive sequence generation. Therefore, by establishing a decoder-only transformer trained on the collection of coordinate sequences (3D meshes), our one-stage pipeline is able to generate high-quality 3D meshes. \n2. **Analysis of existing works**. Though previous methods have achieved initial success in 3D assets generation, they suffer from certain limitations. \n    1. To preserve high-frequency information, the point and voxel representations require dense sampling on object surfaces, which inevitably lead to great redundancy when it comes to flat surfaces.\n    2. The reconstruction-based methods rely heavily on the quality of the generated multi-vew images. \n    3. The VQVAE-based 3D generation methods require a two-stage training strategy, which poses extra challenges in large-scale training.\n\n\n📝 **W2: Fair comparison with previous methods.**\n\n💡 **A:** We reproduce MeshGPT with `gpt2-medium` (355m) and re-implement MeshXL (marked as MeshXL$^{ShapeNet}$) by first training the model on all ShapeNet categories before fine-tuning it to a specified category. 📊 Please refer to **the global rebuttal** for exact results. With a similar amount of parameters (350m), our method could achieve better generation results with a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%. After pre-trained on larger datasets, our method can achieve even better generation results.\n\n\n📝 **W3: Comparison with deformation-based and command-based methods.**\n\n💡 **A:** The training of MeshXL enjoys a better flexibility comparing to both deformation-based or command-based methods.\n1. **MeshXL vs. deformation-based methods**. The deformation-based methods build on prior geometry knowledge for great initialization. However, The deformation-based method requires external expert knowledge for good template initialization, which **is difficult to generalize to complex 3D meshes**. However, our MeshXL direct learns from large-scale collection of diverse 3D mesh data.\n2. **MeshXL vs. command-based methods**. The command-based methods adopt command sequences to represent the creation process of the visual data. However, collecting commands is much harder than collecting the generated results (3D meshes). Additionally, the command space requires careful designs and is often limited (`arc`, `circle`, and `extrude` in DeepCAD) to create only simple objects, while our MeshXL learns directly on large-scale 3D mesh data with great diversity.\n\n📝 **Q1: Comparison with MeshGPT**\n\n💡 **A:** See weakness 2.\n\n📝 **Q2: Reason why end-to-end training leads to better mesh generation results.**\n\n💡 **A:** The main motivation of our work is to explore an simple and effective mesh representation to support large-scale training. Therefore, \n1. **We can hardly say whether two-stage methods are better or worse than end-to-end learning**. However, training a good **vector-quantized** tokenizer is challenging [R1]. In latent diffusion, instead of adopting a VQVAE, the denoising diffusion process learns to generate latent codes predicted by a VAE, which learns the feature distribution of the latent codes.\n2. **Our end-to-end design better supports large-scale training**.\n    1. By representing a 3D mesh with **one unique coordinate sequence**, MeshXL can directly learn from large scale 3D data.\nMeanwhile, MeshGPT requires to train a **vector-quantized** representation of vertex feature. Additionally, PolyGen requires to generate a vertex sequence and then connect the generated vertices into polygons. Therefore, learning a good alignment between different modules is challenging and requires careful supervision.\n    2. Based on the quantitative results on ShapeNet categories in Table 2, MeshXL can produce high-quality 3D meshes with higher COV scores, lower MMD scores, and closer 1-NNA scores to 50% than previous methods.\n\n[R1] Li, Tianhong, et al. \"Autoregressive Image Generation without Vector Quantization.\" arXiv preprint arXiv:2406.11838 (2024).\n\n📝 **Q3: Fair comparison with existing methods**\n\n💡 **A:** See weakness 2.\n\n📝 **Q4: Scaling law does not apply to the `lamp` category.**\n\n💡 **A:** The limited training data leads to overfitting. After pre-processing the ShapeNet dataset, the lamp subset contains **only 565 samples** for training. Therefore, larger models will easily overfit. Instead, we show in Figure 3 of the main paper that when pre-training on extensive 3D data, MeshXL achieves both lower training and validation loss. Additionally, by evaluating on Objaverse, we also notice that as the model size grows, we achieve better generation results.\n\n| Model | COV$\\uparrow$ | MMD$\\downarrow$ | 1-NNA | JSD$\\downarrow$ | FID$\\downarrow$ | KID$\\downarrow$ |\n|-|-|-|-|-|-|-|\n|MeshXL - 125m| 39.76 | 5.21 | 67.34 | 26.03 | 17.32 | 4.48 |\n|MeshXL - 350m| 40.79 | 5.20 | 65.68 | 23.71 | 15.14 | 3.33 |\n|MeshXL - 1.3b| **42.86** | **4.16** | **61.56** | **20.99** | **12.49** | **2.94** |\n\n\n\n📝 **Q5: Normal vector comparison with GET3D**\n\n💡 **A:** In Figure 7, we show the normals to compare **the smoothness of object surfaces**. The results show that 3D meshes generated by GET3D have rough surfaces with tens of thousands of triangles, while ours depict the 3D shape with much smoother surfaces and less triangles.\n\n📝 **Q6: Open-sourcing.**\n\n💡 **A:** We have released our code and pre-trained weights to the community. We will also keep updating for additional features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NzmURJHoqz", "reviewer": "Reviewer_AHoX", "summary": "The paper is about 3D mesh generative models. The generative method is done by generating triangles one-by-one (auto-regressively). Each face is represented by 3 vertices (9 coordinates).", "questions": "How is resolution defined? The number N (in L103) seems to be missing in the context. In L111, the authors mentioned it should be an unsigned integer. But it is still not clear what the number should be. In my experience, the number should be large (or the classification will be very difficult when training the autoregressive transformer). If the number is too small, then the resolution is limited.\n\nI didn't find the sampling time and hardware. Sampling 7200 tokens takes lots of time. I would like to see a time analysis (and maybe memory consumption).\n\nThe training is done on Objaverse-XL but the main results are mainly about ShapeNet. The comparison is not fair. Also I can only find Fig 6 is about general objects instead of Shapenet objects. The quality seems to be very limited.\n\nEven the authors call the method neural coordinate fields, but it is nothing about field. A field should be like a function with continuous domain (like nerf and signed distance fields). The method is similar to PolyGen. All vertices are discretized.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea to generate faces sounds interesting than previous methods. Unlike PolyGen which needs to generate connections (faces) between vertices, the method seems to be more elegant and easier for programming. The ShapeNet results are also convincing.", "weaknesses": "The major limitation of the method is the sequence length. The method is only able to handle 800 faces with the current resources. This makes the method is nearly impossible to generate complicated meshes.\n\nThe method also looks a bit brutal. Some faces share vertices. But the method needs to generate the shared vertices multiple times. The method can also generate intersecting faces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "📝 **W1: The limitation of sequence length.**\n\n💡**A:** The main focus of our paper is to establish end-to-end methods with less inductive bias and pave the way for scaling up learning from large-scale 3D data. We restrict the number of faces to 800 for better alignment with PolyGen and MeshGPT. However, by equipping MeshXL with modern LLM techniques, such as RoPE and ring attention, we are ready to extend our method to more complicate 3D meshes. Additionally, we will also be able to train larger MeshXL networks.\n\n\n📝 **W2: Potential redundancy in the mesh representation and intersecting faces.**\n\n💡**A:** **The redundancy in mesh representation paves the way for large-scale pre-training.** We acknowledge that the NeurCF representation does generate shared vertices multiple times. However, this enables us to represent each 3D mesh with **only one coordinate sequence** and design an **end-to-end** pipeline on large scale 3D data.\n\n**Analysis of potential surface artifacts.** The potential occurrence of surface artifacts could be the limitation for our MeshXL, and all existing auto-regressive mesh generation methods (PolyGen and MeshGPT) for now. However, after training directly on large-scale 3D meshes, MeshXL can already generate high-quality 3D mesh data under a high success rate. It is also possible to further improve MeshXL by treating traditional methods as reward models or filters to eliminate the surface artifacts.\n\n\n📝 **Q1: The resolution of MeshXL.**\n\n💡**A:** To align with MeshGPT, we choose the resolution of $N=128$ in all our experiments. However, our MeshXL adopts an architecture close to large language models, whereas we can easily extend MeshXL to a larger $N$ with a time and space complexity of $O(N)$ by simply learning a $N$-sized coordinate embeddings and an output classifier with $N$ output channels. Comparing to the $O(N^3)$ complexity in traditional hard coding algorithms, it is easier for us to train models with higher resolution. We will also release the models with larger resolution in the future.\n\n\n📝 **Q2: Inference time analysis**\n\n💡**A:** The inference time of our MeshXL is closely related to the **numbers of generated faces** and the **model sizes**. We perform inference time analysis with **a batch size of one using BFloat16 on a single RTX 3090**. We carry out a an analysis on 1) the average inference time of generating a given number of triangles. \n| | MeshXL - 125m | | | MeshXL - 350m | | | MeshXL - 1.3b | |\n|-|-|-|-|-|-|-|-|-|\n| #faces | time (s) | GPU Mem (GB) |   | time (s) | GPU Mem (GB) |   | time (s) | GPU Mem (GB) |\n| 100 | 6.30   | 1.59 | | 11.30 | 2.98 | | 12.08 | 8.41   |\n| 200 | 12.50 | 1.65 | | 22.70 | 3.20 | | 24.03 | 9.17   |\n| 400 | 25.21 | 1.85 | | 45.81 | 3.78 | | 48.09 | 11.17 |\n| 800 | 49.88 | 2.28 | | 92.19 | 5.74 | | 96.49 | 21.66 |\n\nWe also show 2) the average inference time of generating 3D meshes.\n| | MeshXL - 125m | MeshXL - 350m | MeshXL - 1.3b |\n|-|-|-|-|\n|avg. time (s) | 29.49 | 44.65 | 49.43|\n\n📝 **Q3: Fair comparisons and evaluations on larger datasets.**\n\n💡 **A:** We reproduce MeshGPT with `gpt2-medium` (355m) and re-implement MeshXL (marked as MeshXL$^{ShapeNet}$) by first training the model on all ShapeNet categories before fine-tuning it to a specified category. 📊 Please refer to **the global rebuttal** for exact results. With a similar amount of parameters (350m), our method could consistently achieve better generation results with a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%.\n\nAdditionally, to study the effectiveness of large-scale pre-training, we conduct evaluations on Objaverse. We show that as the model size grows, MeshXL exhibits better 3D mesh generation quality, with a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%.\n| Model | COV $\\uparrow$ | MMD $\\downarrow$ | 1-NNA | JSD $\\downarrow$ | FID $\\downarrow$ | KID $\\downarrow$ |\n|-|-|-|-|-|-|-|\n| MeshXL - 125m | 39.76 | 5.21 | 67.34 | 26.03 | 17.32 | 4.48 |\n| MeshXL - 350m | 40.79 | 5.20 | 65.68 | 23.71 | 15.14 | 3.33 |\n| MeshXL - 1.3b   | **42.86** | **4.16** | **61.56** | **20.99** | **12.49** | **2.94** |\n\nAdditionally, we have provided additional generation results in the **uploaded PDF file**. We have also open-sourced our code and pre-trained weights to the community.\n\n\n\n📝 **Q4: Justification of Neural Coordinate Field.**\n\n💡**A:** We thank the reviewer for pointing out this constructive suggestion.\n1. **Justification of Neural Coordinate Field**. Neural coordinate is a representation that treats vertex coordinates as coordinate embeddings. Meanwhile, field should be a function within continuous domain. As we currently learn coordinate embeddings that uniformly spread along each axis, we will try learning interpolation functions (e.g. linear interpolation, gaussian interpolation, or B-spine basis functions) to extend our embeddings to the continuous domain. We will also try extending our method to the continuous domain by replacing the coordinate embeddings with sinusoid function. We will perform detailed experiments in our revision. \n2. **Relation to the ordering in PolyGen**. We adopt the same vertex representation as PolyGen, but a **different mesh representation**. Our MeshXL and PolyGen both represent vertices with discrete coordinates. However, **MeshXL represents a 3D mesh only with an ordered sequence of coordinate**, while PolyGen decouples the vertices generation and polygon generation, and adopts two sequences for each mesh, i.e., the vertex sequence and the face index sequence to connect the generated vertices. Therefore, the potential redundancy in our representation in turn supports end-to-end training and better suits learning from large-scale 3D data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for approval: 1. a **novel and elegant 3D mesh representation** (R1, R4) that 2) **effectively leverages LLM approaches** (R1, R2) for **end-to-end large-scale** training (R1, R3), and 3) a **stable training and better mesh generation quality by scaling up** (R2, R3) supported by 4) a **fair and convincing evaluation** (R1, R4) covering both generation score and mesh quality from a graphics perspective (R1). (R1 - Reviewer XQPQ, R2 - Reviewer e1fg, R3 - Reviewer CF59, R4 - Reviewer AHoX)

We also thank all the reviewers for their valuable suggestions to help us improve our paper. We will address your concerns and revise the paper carefully. We have provided **additional visualization results in the attached pdf file**. Please find our item-to-item responses to your concerns below.

**Motivation and Novelty**. To better support large-scale training, we verify that we can represent a 3D mesh into **one unique coordinate sequence** based on a well-defined ordering strategy. With this simple mesh representation, our MeshXL only requires a single decoder-only transformer for sequence modeling. Comparing to prior two-stage works, **our MeshXL is an end-to-end single-stage pipeline** does not require sophisticated data pre-processing, careful model design, or complex training strategy. Therefore, our MeshXL better suits learning from large-scale 3D data.

**Additional baseline and evaluations**. In the following table, we re-produce the MeshGPT with `gpt2-medium` (355m) using the third-party implementation. We also follow the setting from previous works by pre-training MeshXL (350m) on ShapeNet before fine-tuning to specified categories, which is marked as (MeshXL$^{ShapeNet}$) in the following table. One can see that our method consistently achieves better results than all previous methods. Comparing to MeshGPT with a similar amount of parameters (~350m), our method achieves a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%.

| Category | Method | COV$\uparrow$ | MMD$\downarrow$ | 1-NNA | JSD$\downarrow$ | FID$\downarrow$ | KID$\downarrow$ |
|-|-|-|-|-|-|-|-|
| **Chair** | PolyGen | 7.79 | 16.00 | 99.16 | 228.80 | 63.49 | 43.73 |
| | GET3D | 11.70 | 15.92 | 99.75 | 155.25 | 67.84 | 42.10 |
| | MeshGPT | 42.00 | 4.75 | 69.50 | 55.16 | 39.52 | 8.97 |
| | MeshXL - 125m | 50.80 | **3.11** | 56.55 | 9.69 | 28.15 | 1.48 |
| | MeshXL$^{ShapeNet}$ - 350m | 47.94 | 3.26 | 57.54 | 13.42 | 29.14 | 1.79 |
| | MeshXL - 350m | 50.80 | 3.17 | **55.80** | 9.66 | 28.29 | **1.39** |
| | MeshXL - 1.3b   | **51.60** | 3.23 | **55.80** | **9.48** |  **9.12** | 1.84 |
| **Table** | PolyGen | 44.00 | 3.36 | 67.20 | 25.06 | 54.08 | 14.96 |
| | GET3D | 16.80 | 10.39 | 91.90 | 226.97 | 67.65 | 34.62 |
| | MeshGPT | 34.30 | 6.51 | 75.05 | 92.88 | 53.75 | 7.75 |
| | MeshXL - 125m | 51.21 | 2.96 | 57.96 | **12.82** | 42.55 | **0.92** |
| | MeshXL$^{ShapeNet}$ - 350m | 49.75 | **2.90** | **54.72** | 13.75 | 44.92 | 1.80 |
| | MeshXL - 350m | 49.70 | 3.07 | 56.10 | 13.64 | 43.43 | 1.27 |
| | MeshXL - 1.3b   | **52.12** | 2.92 | 56.80 | 14.93 | **22.29** | 2.03 |
| **Bench** | PolyGen | 31.15 | 4.01 | 83.23 | 55.25 | 70.53 | 12.10 |
| | MeshGPT | 34.92 | 2.22 | 68.65 | 57.32 | 52.47 | 6.49 |
| | MeshXL - 125m | 54.37 | 1.65 | 43.75 | 16.43 | **35.31** | **0.82** |
| | MeshXL$^{ShapeNet}$ - 350m | 55.75 | **1.46** | **44.64** | **10.66** | 36.81 | 1.48 |
| | MeshXL - 350m | 53.37 | 1.65 | 42.96 | 15.41 | 36.35 | 0.96 |
| | MeshXL - 1.3b   | **56.55** | 1.62 | 39.78 | 15.51 | 35.50 | 1.60 |
| **Lamp**  | PolyGen | 35.04 | 7.87 | 75.49 | 96.57 | 65.15 | 12.78 |
| | MeshGPT | 41.59 | 4.92 | 61.59 | 61.82 | 47.19 | 5.19 |
| | MeshXL - 125m | **55.86** | 5.06 | 48.24 | 43.41 | 34.61 | **0.84** |
| | MeshXL$^{ShapeNet}$ - 350m | 52.74 | **3.39** | 41.15 | **25.03** | 31.18 | 1.06 |
| | MeshXL - 350m | 53.52 | 4.18 | **49.41** | 34.87 | **25.94** | 1.92 |
| | MeshXL - 1.3b   | 51.95 | 4.89 | 47.27 | 41.89 | 31.66 | 0.99 |

Meta-review: The paper received three positive reviews before rebuttal. After rebuttal, all reviewers became positive about the paper. Therefore, the decision is to accept the paper. The paper introduces an explicit coordinate representation with implicit neural embeddings, enabling large-scale sequential mesh modelling. The approach is justified through extensive experiments and its usefulness is also verified through various down-stream applications. In the final version, it is recommended that to take the reviewers comments into consideration and revise the paper based on promise in the proposal. In particular, please articulate the difference between MeshXL's ordering method and PolyGen's.

Although this is not required to be discussed in the paper. One question for the mesh representation is that a single geometric shape can have many mesh discretizations. How would the proposed approach generate the same geometry but different meshes. We do not have this issue in language models or part-based representations of 3D shapes.
Decision: Accept (poster)