MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

Authors: sijin  chen
xin  chen
anqi  pang
xianfang  zeng
wei  cheng
yijun  fu
fukun  yin
zhibin  wang
jingyi  yu
yu  gang
bin  fu
tao  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 03:07:30

Abstract: The polygon mesh representation of 3D data exhibits great flexibility, fast rendering speed, and storage efficiency, which is widely preferred in various applications. However, given its unstructured graph representation, the direct generation of high-fidelity 3D meshes is challenging. Fortunately, with a pre-defined ordering strategy, 3D meshes can be represented as sequences, and the generation process can be seamlessly treated as an auto-regressive problem. In this paper, we validate Neural Coordinate Field (NeurCF), an explicit coordinate representation with implicit neural embeddings, is a simple-yet-effective representation for large-scale sequential mesh modeling. After that, we present MeshXL, a family of generative pre-trained auto-regressive models that addresses 3D mesh generation with modern large language model approaches. Extensive experiments show that MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.

Main paper: 
 introduction: 
 The generation of high-quality 3D assets [61, 77, 29] is essential for various applications in video games, virtual reality, and robotics. Among existing 3D representations [51, 38, 57, 61] , 3D mesh represents 3D data as graphs, which possesses the flexibility and accuracy representing sharp edges as well as both flat and curved surfaces. However, the direct generation of high-quality 3D meshes is challenging, given 1) the unstructured graph representation and 2) the demand for estimating accurate spatial locations and connectivity within vertices.To generate 3D meshes, many works adopt an indirect way by first producing data in other 3D representations, such as point clouds [97, 49, 54] , SDF [88, 94] , and multi-view images [46, 82, 30] . After that, they adopt re-meshing methods [37] to post-process the generated geometries. There are also attempts towards the direct generation of 3D polynomial meshes. PolyGen [53] adopts two separate decoder-only transformers for vertices generation and vertices connectivity prediction. MeshGPT [65] first builds a mesh VQVAE to first turn meshes into tokens, and then learns to generate the token sequences with a GPT model [59] . Meanwhile, PolyDiff [2] directly adopts discrete denoising diffusion [4] on the discretized mesh coordinates.Though these methods have achieved initial success in creating 3D assets, they suffer from certain limitations. To preserve sufficient high-frequency information, point clouds and voxels requires dense samplings on the object surfaces, which inevitably leads to great redundancy while representing flat surfaces. The reconstruction-based methods [82, 30, 67] , however, rely heavily on the accuracy of the multi-vew generation pipelines [46] . Additionally, the VQVAE-based 3D generation methods [88, 65] require sophisticated multi-stage training, which less favors learning from large scale data.To tackle the above challenges and explore the potential of scaling up 3D generative pre-training, we introduce a simple-yet-effective way of 3D mesh representation, the Neural Coordinate Field (NeurCF). NeurCF represents the explicit 3D coordinates with implicit neural embeddings. We show that with a pre-defined ordering strategy, a 3D mesh can be represented by a one-and-only coordinate sequence, which further helps us formulate 3D mesh generation as an auto-regressive problem. After that, we present MeshXL, a family of generative pre-trained transformers [93, 59] , for the direct generation of high-fidelity 3D meshes. Without resorting to intermediate 3D representations, NeurCF facilitates an end-to-end learning pipeline for the direct pre-training on large-scale 3D mesh data.By organizing high-quality 3D assets from ShapeNet [9] , 3D-FUTURE [22] , Objaverse [17] , and Objaverse-XL [16] , we achieve a collection of over 2.5 million 3D meshes to support large-scale generative pre-training. Extensive experiments demonstrate that the NeurCF representation facilitates MeshXL to generate higher-quality 3D meshes. By training on the collection of large-scale 3D mesh data, MeshXL can achieve better performance with larger numbers of parameters (Fig. 3 and Tab. 5), and surpass prior arts on multiple categories in the ShapeNet dataset [9] (Tab. 3).In summary, our contributions can be summarized as follows:• We validate that Neural Coordinate Field is a simple-and-effective representation of 3D mesh, which is also friendly to large-scale auto-regressive pre-training. • We present a family of MeshXLs that can be treated as strong base models for image-conditioned or text-conditioned 3D mesh generation tasks. • We show that MeshXL surpasses state-of-the-art 3D mesh generation methods, and can produce delicate 3D meshes compatible with existing texturing methods. 
related work: 
 First, we present a concise review of existing 3D representations. Subsequently, we discuss related works on 3D generation and recent efforts in developing 3D foundation models.3D Representations. Researchers have long sought for accurate and efficient methods to represent 3D data. Point Cloud [54, 57, 58, 89] captures the spatial positions of discrete points in the Euclidean space, which is preferred by various 3D sensors [15, 87, 66, 3, 7] . Mesh [53, 2, 65, 12] represents the 3D structure with graphs. By connecting the vertices with edges, mesh can also be interpreted into a set of polygons in the 3D space. Similar to point clouds, 3D Gaussians [38, 68] also record the discrete Euclidean distribution in 3D space. However, each point is represented by a 3D Gaussian distribution function parameterized by its covariance matrix, color, and opacity. Given their fast convergence and rendering speed, 3D gaussians are often utilized for 3D reconstruction. Neural Radiance Field (NeRF) [51, 5] constructs a learnable volumetric function f using neural networks trained on multi-view images. Due to its derivability and flexibility, NeRF is also favored for 3D generative models [46, 99, 76, 56] . Additionally, there are other 3D representations such as multi-view images [74, 90, 100] , voxel fields [61, 13, 45] , and signed distance fields [94] , among others [64, 88, 63] . In this paper, we consider the Neural Coordinate Field (NeurCF), an explicit spatial representation with implicit neural embeddings, and investigate its potential for scalable 3D asset generation.3D Generation. With the exploration of various 3D representations and the collection of large-scale 3D datasets [17, 9, 16] , researchers have also put much effort exploring the generation of high-fidelity 3D assets [42, 39] . The Generative Adversarial Network (GAN) [25, 80, 1, 33] produces synthetic 3D data with a generator G, and train a discriminator network D to distinguish the generated and real data. Additionally, the potential of diffusion models [54, 28, 62] in the direct generation of 3D data is also widely explored [97, 2, 54, 50, 47] . The key idea behind diffusion is to transform the desired data distribution into a simpler distribution (e.g. gaussian) and learn a desnoising model for the reverse process. Besides, researchers have also explored the potential of diffusion models in generating multi-view images [46, 16, 82, 43] , and reconstruct them into 3D structures. In this paper, we mainly explore the auto-regressive methods for 3D generation. AutoSDF [52] and MeshGPT [65] learn to generate discrete tokens and reconstruct them into 3D representations with a VQVAE model [72] .PolyGen [53] adopts two decoder-only transformers that predict the location and connectivity of vertices, sequentially. In this paper, we explore the potential of an explicit sequential modelling method for 3D meshes, and present a family of generative pre-trained transformers, MeshXL, for high-fidelity 3D mesh generation.3D Foundation Models. The collection of large-scale high-quality 3D data [17, 16, 9, 81, 71, 21, 22] builds up the foundation for various 3D-related tasks [83, 27, 10, 41] . To explore the scaling effects in 3D learning, researchers have made great endeavors in building 3D foundation models for 3D understanding [96, 44, 98, 85, 86, 92, 100] , reconstruction [30, 78, 67, 46, 16, 84, 73] , and generation [61, 29, 65, 8] . With the introduction of large-scale 3D data in both variety and granularity [34, 41, 16] , existing 3D foundation models are capable of generalizing to unseen concepts [100, 86, 44] , generating high-fidelity 3D assets [88, 36, 65] , responding to complex instructions [31, 10, 32, 41] , and generating actions that interacts with the 3D environments [20, 79, 95] . In this paper, we present a fully end-to-end 3D mesh generation pipeline, explore the scaling effect for large-scale pre-training, and test whether our method can serve as a well-trained foundation model for various down-stream tasks.3 Data Data Sources. We provide details on the 3D data collections we use to train and evaluate our models. The whole data collection is built upon four widely-acknowledged 3D mesh datasets, i.e. ShapeNet V2 [9] , 3D-FUTURE [22] , Objaverse [17] , and Objaverse-XL [16] .• ShapeNet V2 [9] collects about 51k 3D CAD models for 55 categories. We split the data in 9:1 for training and validation by each category.• 3D-FUTURE [22] present about 10k high-quality 3D mesh data for indoor furniture. However, because of the delicate design, the objects contain many faces. Therefore, only a small proportion of the data can be used to train our MeshXL models.• Objaverse [17] is a large 3D data collection with more than 800k 3D objects for about 21k categories collected from Sketchfab. We split the data in 99:1 for training and validation, respectively.• Objaverse-XL [16] further expand Objaverse [17] into a dataset with more than 10M 3D objects with additional data collected from GitHub, Polycam, Thingiverse, and Smithsonian. We split the Github and Thingiverse part of the Objaverse-XL dataset into 99:1 for training and validation.Data collection and filtering. To organize existing datasets, we build up a filtering and pre-processing pipeline to ensure that the meshes meet our demand. We first collect meshes with fewer than 800 faces, and ensure that they have corresponding UV maps for rendering. After that, we render the 3D ℰ mesh ℳ = ℰ face 𝑓𝑓 (1) , ⋯ , ℰ face 𝑓𝑓 (𝑛𝑛) Coordinate Embeddings: ℰ Mesh Embedding: meshes, and discard those not center-aligned or occupying less than 10% of the rendered image. For those 3D objects with more than 800 but less than 20,000 faces, we use planar decimation to simplify the meshes. Finally, we achieve approximately 2.5 million 3D meshes (Tab. 1).ℰ face 𝑓𝑓 𝑖𝑖 = ℱ 𝑝𝑝 1 (𝑖𝑖) , ⋯ , ℱ 𝑝𝑝 𝑘𝑘 (𝑖𝑖)Planar Decimation Pipeline. To ensure the quality of the decimated 3D meshes, we make sure either a lower Hausdorff distance δ hausdorff [65] or a similar rendered views [11] .Collecting text-mesh pairs. We render the 3D meshes with 12 different views and use CogVLM [75] to annotate 1) the front view and 2) the concatenated multi-view image. Then, we adopt the Mistral-7B-Instruct model [35] with in-context examples to generate a fused mesh caption.Data Statistics. We present the data statistics of our large-scale 3D mesh collection in Tab. 1.After organizing and combing 3D assets from ShapeNet [9] , 3D-FUTURE [22] , Objaverse [17] , and Objaverse-XL [16] , we could achieve a total of 2.5 million 3D meshes. (f (i) ) = (F(p (i) 1 ), • • • , F(p (i) k )).For simplicity, we share the learnable coordinate embeddings E among axes.Ordering. Due to the graph representation, the order of the mesh vertices and the order of the edges among them are permutation-invariant. A pre-defined ordering strategy is essential to facilitate the sequence modelling in MeshXL. We employ the same ordering strategy as PolyGen [53] and MeshGPT [65] . The mesh coordinates are first normalized into a unit cube based on the mesh's longest axis, and discretized into unsigned integers. Within each face, the vertices are cyclically permuted based their coordinates (z-y-x order, from lower to higher), which helps to preserve the direction of normal vectors. Then, we order these faces based on the permuted coordinates (lower to high). To this end, we can represent each 3D mesh with a one-and-only coordinate sequence, aiding large-scale generative pre-training on a large collection of 3D mesh data. With the NeurCF We train all the models from scratch on 150 billion tokens. We observe that the performance grows with model sizes.representation, an n-faced 3D k-sided polynomial mesh can be represented as the coordinate sequence M ∈ Z n×k×3 , and further be encoded intoE mesh = (E face (f (1) ), • • • , E face (f (n) )).A Sequential Mesh Representation. One direct way to represent the 3D meshes is to directly reshape M into a vector with (n • k • 3) tokens. As a special case, an n-faced triangular mesh can be represented by a vector with 9n tokens. Meanwhile, our representation can also be expanded to hybrid polynomial mesh representations with the proper introduction of separate tokens. For example, we can generate triangles within "<tri> • • • </tri>" and quadrilaterals within "<quad> • • • </quad>" also in one sequence. To identify the start and end of a mesh sequence, we add a <bos> ("begin-of-sequence") token before the mesh sequence and an <eos> ("end-of-sequence") token after.Comparisons. Since we represent each coordinate with learnable embeddings, NeurCF is an end-toend trainable representation for unstructured 3D meshes. Comparing to the decoupled vertex and polygon representation in PolyGen [53] , NeurCF only requires one coordinate sequence for each 3D mesh. Additionally, NeurCF is storage and computation efficient comparing to voxel fields (O(N 3 )), since it can easily scale up the resolution with a complexity of O(N ).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper discusses the limitations of the work performed by the authors, providing a balanced view of the study's scope and potential areas for improvement.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
method: 
 We first present the architecture and training objective for MeshXL models. Then, we show that MeshXL models can take an additional modality as the condition for controllable 3D assets generation. After this, we investigate the effects of scaling.Architecture. In Sec. 4, we present a simple-yet-effective way to represent a 3D mesh into a sequence. Therefore, the learning of 3D mesh generation can be formulated as an auto-regressive problem, which can be seaminglessly addressed by modern Large Language Model (LLM) approaches. In our paper, we adopt the decoder-only transformers using the OPT [93] codebase as our base models. To adapt the pre-trained OPT models to our next-coordinate prediction setting, we fine-tune the whole model with newly-initialized coordinate and position embeddings.Generative Pre-Training. We train MeshXL models using the standard next-token prediction loss. Given the trainable weights θ and an |s|-length sequence s, the generation loss is calculated as:EQUATIONFor each mesh sequence, we add a <bos> token before the mesh tokens, and an <eos> token after to identify the ending of a 3D mesh. During inference, we adopt the top-k and top-p sampling strategy to produce diverse outputs.X -to-Mesh Generation. Here we mainly consider generating 3D meshes from images and texts. We first turn the extra conditions into tokens with pre-trained encoders [18, 19] . To align the additional text/image feature with the mesh coordinate field, we adopt the Q-Former architecture [40] to compress the encoded feature into a fixed-length of 32 learnable tokens as the prefix of the MeshXL model. The overall training objective for the conditional mesh generation is shown in Eq. ( 2):EQUATIONDuring inference, the model predicts the mesh tokens after the fixed-length prefix.Scaling Up. We present MeshXL in various sizes, including 125M, 350M, and 1.3B. The detailed hyperparameters for training different models can be found in Tab. 2. To better analyze the scaling effects, we train all models from scratch on 150 billion tokens. We provide both training curve and validation perplexity for different models in Fig. 3 . One can see that as the number of parameters grows, the model achieves a lower validation perplexity, indicating a higher probability to produce the validation data. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We will open access to the code after the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We first briefly introduce the data, metrics, and implementation details in Sec. 6.1. Then, we provide evaluations and comparisons on the generated meshes (cf. Sec. 6.2) and ablations (cf. Sec. 6.3). We also provide visualization results in Sec. 6.4.Data. We pre-train the base model with 2.5 million 3D meshes collected from the combination of ShapeNet [9] , 3D-FUTURE [22] , Objaverse [17] , and Objaverse-XL [16] . We use planar decimation on meshes with more than 800 faces following MeshGPT [65] and RobustLowPoly [11] . For generative mesh pre-training, we randomly rotate these meshes with degrees from (0 • , 90 • , 180 • , 270 • ), and adopt random scaling along each axis within range [0.9, 1.1] for data augmentation.Metrics. We follow the standard evaluation protocols in MeshGPT [65] and PolyDiff [2] to measure the quality of the generated meshes with the following metrics. We use Coverage (COV) to quantify the diversity of the generated meshes, which is sensitive to mode dropping but cannot be used to assess the generation quality. Minimum Matching Distance (MMD) calculates the average distance between the reference set and their closest neighbors in the generated set, but is not sensitive to low-quality results. The 1-Nearest Neighbor Accuracy (1-NNA) quantifies the quality and diversity between the generation set and the reference set, whose optimal value is 50%. We also adopt the Jensen-Shannon Divergence (JSD) score. Among all the above metrics, we use Chamfer Distance to measure the similarity between two samples. We also render the generated meshes and adopt the Frechet Inception Distance (FID) and Kernel Inception Distance (KID) on the rendered images for feature-level evaluation. We multiply the MMD, JSD, and KID scores by 10 3 .Implementation. We conduct all the experiments on a cluster consisting 128 A100 GPUs. We train our models under bfloat16 with the ZeRO-2 strategy [60] using the AdamW [48] optimizer with a learning rate decaying from 10 -4 to 10 -6 and a weight decay of 0.1. The detailed hyperparameters for different models can be found in Tab. 2. To train our base models, we load the weights from the pre-trained OPT models [93] and initialize the word embeddings and positional embeddings from scratch. Without further specification, we generate 3D meshes with the top-k and top-p sampling strategy with k = 50 and p = 0.95.We provide quantitative as well as qualitative comparisons on both unconditional and conditional 3D mesh generation on public benchmarks.Unconditional Generation. We evaluate MeshXL as well as other baseline methods using the ShapeNet [9] data in Tab. 3. We split the data by 9:1 for training and validation by each category.For evaluation, we fine-tune our pre-trained base model and sample 1,000 meshes for each category. Among the listed methods, we reproduce the MeshGPT [65] with a GPT2-medium model (355M) [59] .With a similar number of parameters, Mesh-XL (350M) out-performs MeshGPT by a large margin, showing a higher COV score, a lower MMD score, and a closer 1-NNA score to 50%. This indicates that MeshXL can produce diverse and high-quality 3D meshes. User Study. To evaluate how well the generated 3D meshes align with human preference, we perform user studies on the chair category in Tab. 4 with several baseline methods [53, 23] . We recruit and instruct the participants to score each mesh from 0 to 5 (higher is better) based on its 1) quality: the smoothness of object surfaces and completeness of the mesh, 2) artistic: how much do you believe this object is designed and created by artists, and 3) triangulation: how well do the connectivity among vertices aligns with the models created by professional designing software [14] . As a baseline evaluation, we also ask the participants to score the ground truth 3D geometries sampled from the ShapeNet data. We have collected a total of 434 valid responses. The results show that the 3D meshes created by MeshXL are consistently preferred by human in all dimensions.Necessity of Mesh VQVAE. Comparing to MeshGPT [65] , MeshXL is an end-to-end trainable model that produces 3D meshes with next-coordinate prediction. We show in Tab. 3 that, MeshXL out-performs MeshGPT with similar numbers of parameters. We also show that MeshXL can produce high quality 3D meshes with both sharp edges and smooth surfaces in Fig. 7 . Furthermore, MeshXL can save the effort training a vector quantized mesh tokenizer [65, 72] , which further facilitates generative pre-training on large scale datasets.Effectiveness of Model Sizes. To analyze whether pre-training a larger model benefits 3D mesh generation, we evaluate MeshXL base models with different sizes on the Objaverse [17] dataset in Tab. 5. We observe that as the model size grows, the generated samples exhibits a larger COV, smaller JSD score, and a closer 1-NNA to 50%, which indicates an improved diversity and quality. Shape Completion. To analysis whether our method is capable of producing diverse outputs, we adopt MeshXL (1.3B) model to predict the whole object given some partial observations. In practice, we use 50% of the object mesh as input, and ask the model to predict the rest 50% in Fig. 4 . One can see that Mesh-XL is able to produce diverse and reasonable outputs given the partial observation of the 3D mesh.X -to-Mesh Generation. To adopt the MeshXL base models to the X -to-mesh generation setting, we adopt the Q-Former [40] to encode the additional conditions as prefixes. We showcases several conditional generation results in Fig. 5 . We show that MeshXL can generate high-quality 3D meshes given the corresponding image or text as the additional inputs. Texturing. We adopt Paint3D [91] , a coarse-to-fine texture generation pipeline, to generate textures for the 3D meshes produced by MeshXL in Fig. 6 . We show that 3D meshes produced by MeshXL can easily fit in existing texturing methods to produce high-quality 3D assets.Textured Mesh UV Map Generated Mesh Textured Mesh UV MapFigure 6 : Texture Generation for the Generated 3D Meshes. We adopt Paint3D [91] to generate textures for 3D meshes produced by MeshXL.We provide qualitative comparisons on the generated meshes in Fig. 7 . MeshXL is able to produce high-quality 3D meshes with both sharp edges and smooth surfaces. We also visualize the normal vectors to compare the smoothness of object surfaces. The results show that 3D meshes generated by GET3D [23] have rough surfaces with tens of thousands of triangles, while MeshXL depicts the 3D shape with much smoother surfaces and less triangles. MeshXL is able to produce high-quality 3D meshes with both sharp edges and smooth surfaces.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have specified all the training and test details necessary to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussions: 
 Difference with PolyGen [53] . PolyGen treats 3D mesh data as a vertex sequence and a face sequence.PolyGen first generates the ordered vertices with the vertex transformer, then predicts the connectivity among vertices with the face transformer. Comparing to PolyGen, our proposed MeshXL adopts a more straightforward approach that models the 3D mesh as a one-and-only coordinate sequence, which further supports the direct and end-to-end pre-training on a large collection of 3D data.Difference with MeshGPT [65] . MeshGPT consists of a mesh VQVAE [72] and a decoder-only transformer [59] . MeshGPT first learns a mesh VQVAE to quantize the 3D meshes into discrete tokens. After that, MeshGPT trains a decoder-only transformer to generate the discrete tokens for 3D mesh reconstruction. In comparison, our proposed MeshXL is an end-to-end method that learns the neural representation of coordinates and outputs 3D meshes directly.Extensibility. Our method, MeshXL, is built upon the concept of auto-regressive methods. Therefore, our method is not restricted to the decoder-only transformers [59, 93, 69, 70] , and can also be extended to other causal language models (i.e. Mamba [26] , RWKV [55] , and xLSTM [6] ).Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper reports error bars and other appropriate information about the statistical significance of the experiments, ensuring the reliability and validity of the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
limitations, future work, and conclusions: 
 Limitations and Future Work. The main drawback of MeshXLs is the inference time. During sampling, MeshXL will generate 7,200 tokens for an 800-faced 3D mesh, which takes a relatively long time because of the auto-regressive process. As for future works, recent endeavors on the RNN-related methods [6, 55, 26] and multiple tokens prediction for LLMs [24] might open up great opportunities in saving the inference cost.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: For each experiment, the paper provides sufficient information on the computer resource.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed, providing a comprehensive evaluation of the study's broader implications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: The paper describes safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse, ensuring that appropriate measures are taken to mitigate potential risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new assets introduced in the paper are well documented, and the documentation is provided alongside the assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We validate that NeurCF, an explicit coordinate representation with implicit neural embeddings, is a simple-and-effective representation of 3D meshes. By modelling the 3D mesh generation as an auto-regressive problem, we seek help from modern LLM approaches and present a family of generative pre-trained models, MeshXL, for high-fidelity 3D mesh generation. We show that MeshXL performs better given larger-scale training data and increased parameters. Extensive results show our proposed MeshXL can not only generate high-quality 3D meshes, but also exhibits great potential serving as base models for conditional 3D assets generation.Unconditional Results on ShapeNet. We visualize unconditional 3D mesh generation results for chair, table, lamp and bench in Fig. 8 . One can see that MeshXL is able to produce diverse and high-quality 3D meshes.Figure 8 : Gallery results. Additional generation results for chair, table, lamp, and bench.Unconditional Generation on Objaverse. We visualize 3D meshes randomly sampled from MeshXL base model in Fig. 9 . After training on a large-scale collection of 3D mesh data, MeshXL is able to produce diverse and high-quality 3D meshes. How well is the triangulation. We evaluate the aspect ratio, face area, and number of faces for a better evaluation in Tab. 6. Though the meshes generated by our MeshXL have a higher average aspect ratio, we manage to achieve a smaller variance with much less 3D faces. This indicates the stability of our generation ability and the efficiency of the direct mesh representation. Since we train our MeshXLs only on triangular meshes, long-thin triangles inevitably exist in our training data. By co-training our MeshXLs on triangular meshes, 3D quads, and even hybrid representations, we could reduce the existence of long thin triangles for better generation quality. The inference cost of MeshXL is closely related to the numbers of generated faces and the model sizes. We perform inference cost analysis with a batch size of one using bfloat16 on a single RTX 3090. We carry out a an analysis (Tab. 7) on 1) the average inference time of generating a given number of triangles, and 2) the average inference time of generating 3D meshes. Justification: The abstract and introduction of the paper clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics, ensuring ethical standards are upheld throughout the study.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper received three positive reviews before rebuttal. After rebuttal, all reviewers became positive about the paper. Therefore, the decision is to accept the paper. The paper introduces an explicit coordinate representation with implicit neural embeddings, enabling large-scale sequential mesh modelling. The approach is justified through extensive experiments and its usefulness is also verified through various down-stream applications. In the final version, it is recommended that to take the reviewers comments into consideration and revise the paper based on promise in the proposal. In particular, please articulate the difference between MeshXL's ordering method and PolyGen's.

Although this is not required to be discussed in the paper. One question for the mesh representation is that a single geometric shape can have many mesh discretizations. How would the proposed approach generate the same geometry but different meshes. We do not have this issue in language models or part-based representations of 3D shapes.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of generating high-fidelity 3D meshes by introducing Neural Coordinate Field (NeurCF), an effective representation for large-scale sequential mesh modeling. The authors present MeshXL, a family of generative pre-trained auto-regressive models, which applies modern large language model techniques to 3D mesh generation. Extensive experiments demonstrate that MeshXL produces high-quality 3D meshes and outperforms existing methods on various tasks. Key contributions include validating NeurCF as a viable representation for 3D meshes, presenting MeshXL as robust base models for conditioned 3D mesh generation, and showcasing MeshXL's superior performance in generating detailed 3D meshes compatible with current texturing methods.
strength_weakness: strength_weakness: 1. This method does not appear to incorporate domain knowledge from traditional remeshing techniques to ensure correct connectivity between different components, avoid self-intersections, and prevent flipping.
2. In the user study, more objective metrics for measuring mesh quality should be considered. For instance, in downstream tasks like ray tracing, long thin triangles should be avoided, and aspect ratio can be used to measure how thin these triangles are.
questions: questions: 1. How does this method address common mesh surface artifacts in modeling, such as ensuring correct connectivity between different components, avoiding self-intersections, and preventing flipping?
2. In Section 4, we generate triangles within “<tri> · · · </tri>” and quadrilaterals within “<quad> · · · </quad>”. However, what is the form of the output in the results presented in the paper? Should these sequences of triangles and quadrilaterals be generated separately or can they be combined in the final meshing result?
limitations: limitations: Please check weakness
paper_summary: paper_summary: This paper proposes a way to use LLM to generate polygon meshes. The key idea is to model mesh generation as the next coordinate prediction, using a strategy similar to that of prior work like PolyGen and MeshGPT. The paper has shown the capability to create a mesh with reasonable quality. 

I think the key contribution for the paper is that it shows the potential to use LLM style scaling to achieve better mesh generation, while the weakness is that it's unclear how much technical contribution given most techniques are already been explored in prior works like MeshGPT and PolyGen.
strength_weakness: strength_weakness: My main concern about the paper is its potential lack of novelty. The key idea of using an Autoregressive model to generate a mesh has been explored in many prior works, such as PolyGen and MeshGPT. However, the writing and the results show that this paper differs technically from existing methods other than running on larger datasets with larger-scale models. At the same time, it's not clear from the writing how this paper handles many technical challenges of modeling mesh as a sequence in a new way, such as permutation invariance of the faces. 

Also, I think the paper lacks a proper evaluation of mesh quality. The introduction is set to claim that the generative mesh has better quality and is potentially more suitable for downstream applications compared to modeling other representations, such as point clouds or voxels. However, few metrics are indicating that the generative mesh is of good quality that the artist can edit. For example, what's the ratio of the generative mesh being watertight? How well is the triangulation? Most generative metrics this paper reports concern the shape this mesh represents but not how well this mesh triangulates.
questions: questions: * Question about the ordering. Is the ordering for creating the sequence unique? how does it related/differ from the ordering polygon?  I think that in L103 the paper should cite PolyGEN for the partition idea.

* L29-35 I believe this positioning risks overclaiming. The auto-regressive model can also have cumulative error issues. It's not entirely clear why auto-regressive mesh generation does not have "great redundancy when representing flat surfaces".

* L105 - what is "polynomial face"? I think L101-106 requires much more detailed descriptions for readers to find it reproduciable.
limitations: limitations: See the weakness section. Other limitations include restricted context length (i.e. mesh has very long context lengths, as many meshes have trillions of triangles).
paper_summary: paper_summary: The paper proposes MeshXL, a mesh generation model based on the Neural Coordinate Fields(NeurCF), which encodes discretized coordinates of mesh vertices into a sequence of tokens. 
Then a decoder only transformer is trained to generate meshes unconditionally/conditioned on modality. 
The model is trained on multiple datasets for better performance. 
In terms of quantitative and qualitative results, the method outperforms other baselines.
strength_weakness: strength_weakness: The weaknesses of the paper mainly lies in the technical part and the comparison part. 
To my knowledge, the main difference between previous discrete mesh generation methods like MeshGPT is that MeshGPT first encodes the mesh with VQVAE and then trains a generation model, while the paper does it in a single stage. 
The paper does produce great results but seems miss a reason why single stage can give performance. 
In the introduction part (lines 21–28), the paper discusses previous generative methods but does not explicitly point out why they are missing. 
I think adding more analysis can help readers understand the paper better. 

I am not sure the comparisons between MeshXL and PolyGen/GET3D are fair in Table. 2 since it uses pre-trained MeshXL and fine-tunes it on a specific category. I believe using the same dataset for training is more convincing.

For reference, I think template-based deformation methods and command-based methods can be added to the discussion:
TM-NET: Deep Generative Networks for Textured Meshes
Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency

DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation
DeepCAD: A Deep Generative Network for Computer-Aided Design Models
Computer-Aided Design as Language
questions: questions: 1. Since the MeshGPT is closely related, I think comparisons should be included. 
https://github.com/lucidrains/meshgpt-pytorch

2. As mentioned in the weakness part, I am really wondering why end-to-end training can generate better results compared to two-stage method like PolyGen and MeshGPT, especially considering 2D SOTA generation model like the latent diffusion model is two-stage.

3. I am not sure the comparisons between MeshXL and PolyGen/GET3D are fair in Table. 2 since it uses pre-trained MeshXL and fine-tunes it for a specific category. I believe using the same dataset for training is more convincing.

4. Why does the scaling law seem not to apply to the lamp as shown in Table 2?

5. Normals in Fig. 7 seem worse (Columns 1, 3, 4) compared to GET3D.

6. will you release the code and pre-trained models since training MeshXL takes a lot of resources?
limitations: limitations: Limitation is discussed, I do not see any issues.
paper_summary: paper_summary: The paper is about 3D mesh generative models. The generative method is done by generating triangles one-by-one (auto-regressively). Each face is represented by 3 vertices (9 coordinates).
strength_weakness: strength_weakness: The major limitation of the method is the sequence length. The method is only able to handle 800 faces with the current resources. This makes the method is nearly impossible to generate complicated meshes.

The method also looks a bit brutal. Some faces share vertices. But the method needs to generate the shared vertices multiple times. The method can also generate intersecting faces.
questions: questions: How is resolution defined? The number N (in L103) seems to be missing in the context. In L111, the authors mentioned it should be an unsigned integer. But it is still not clear what the number should be. In my experience, the number should be large (or the classification will be very difficult when training the autoregressive transformer). If the number is too small, then the resolution is limited.

I didn't find the sampling time and hardware. Sampling 7200 tokens takes lots of time. I would like to see a time analysis (and maybe memory consumption).

The training is done on Objaverse-XL but the main results are mainly about ShapeNet. The comparison is not fair. Also I can only find Fig 6 is about general objects instead of Shapenet objects. The quality seems to be very limited.

Even the authors call the method neural coordinate fields, but it is nothing about field. A field should be like a function with continuous domain (like nerf and signed distance fields). The method is similar to PolyGen. All vertices are discretized.
limitations: limitations: Yes

Comments: title: Appreciate the recognition
Thanks for the response
Reply to the rebuttal
comment: We sincerely appreciate your recognition of our work and the valuable feedback to help us improve our paper.

In our revision, we will highlight that our work **paves the way for scaling up training on large-scale 3D mesh data**. Our **mesh representation** turns a 3D mesh into one unique coordinate sequence, which enables us to simplify our **architecture design** into a decoder-only transformer model, facilitating an **end-to-end training pipeline** that does not require sophisticated data pre-processing, careful model design, or complex training strategy and better suits large-scale 3D mesh data.

To further improve our paper, we will include a detailed discussion between MeshXL's mesh representation and that of PolyGen. Additionally, we will also incorporate objective evaluations on the triangulations for mesh quality assessment.

We acknowledge that the potential generation of certain artifacts is a limitation for now. To alleviate the potential occurrence of artifacts, we will keep exploring methods to integrate domain knowledge as filters or reward models. We will also work on co-training MeshXL on triangle meshes, 3D quads, and even hybrid representations to reduce the occurrence of long thin triangles as also mentioned by reviewer XQPQ.

Once again, we thank you very much for your recognition and valuable suggestions from all reviewers to help us improve our work.
Thanks for the detailed response. After reading other reviews together with the rebuttal, I lean to believe that the paper does carry some technical novelties to make mesh generation more scalable - some of which help simplify the pipeline, which is under-appreciated by the community. With that being said, I am willing to change my score toward acceptance.

I strongly encourage the authors to revise the paper to bring out the technical contribution needed to simplify the pipeline and make mesh generation more scalable. I would appreciate more detailed discussion on the difference between MeshXL's ordering method and PolyGen's. 

The evaluation on mesh quality is still unconvincing since the presented metrics can be biased by the fact that the generated meshes are outputted by a model which only sees short meshes. This means that the model can sacrifice other aspects of the mesh qualities, such as being watertight or without self-intersecting faces, just to generate a subset of faces well. I believe this is still the most significant limitation of the current method and would appreciate it if the authors make appropriate acknowledgment of that.
Dear Reviewers,

We sincerely appreciate your constructive feedback on our paper, "MeshXL: Neural Coordinate Field for Generative 3D Foundation Models." As the discussion session is about to end, we apologize if we did not fully address your concerns in our discussions. If there are additional concerns or anything you would like us to address, please let us know. 

We deeply appreciate your valuable time and effort.

Best regards,

The Authors
We sincerely appreciate your recognition of our work! We will incorporate the valuable feedback from all reviewers to enhance our main paper with additional analysis, comparisons, and evaluations. Thank you once again for your time and effort in reviewing our paper!
Thanks for your great efforts! After reading the response, some major issues have been addressed well, so I still lean towards positive for the submission. I encourage the author to add these clarifications to the main paper. Thanks!
Dear Reviewers,

Thank you for your valuable feedback on our paper "MeshXL: Neural Coordinate Field for Generative 3D Foundation Models". We hope our responses have addressed your concerns to your satisfactory. If you have any further concerns, please let us know during the discussion session.

Thank you again for your valuable time and effort!

Best regards,

All authors