EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation

## Abstract

Abstract Current auto-regressive mesh generation methods suffer from issues such as incompleteness, insufficient detail, and poor generalization.
In this paper, we propose an Auto-regressive Auto-encoder (ArAE) model capable of generating high-quality 3D meshes with up to 4,000 faces at a spatial resolution of 512 3 superscript 512 3 512^{3} 512 start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT .
We introduce a novel mesh tokenization algorithm that efficiently compresses triangular meshes into 1D token sequences, significantly enhancing training efficiency.
Furthermore, our model compresses variable-length triangular meshes into a fixed-length latent space, enabling training latent diffusion models for better generalization.
Extensive experiments demonstrate the superior quality, diversity, and generalization capabilities of our model in both point cloud and image-conditioned mesh generation tasks.

## 1 Introduction

Automatic 3D content generation, particularly the generation of widely used polygonal meshes, holds the potential to revolutionize industries such as digital gaming, virtual reality, and filmmaking.
Generative models can make 3D asset creation more accessible to non-experts by drastically reducing the time and effort involved.
This democratization opens up opportunities for a wider range of individuals to contribute to and innovate within the 3D content space, fostering greater creativity and efficiency across these sectors.

Previous research on 3D generation has explored a variety of approaches.
For example, optimization-based methods, such as using score distillation sampling (SDS) (Poole et al., ; Lin et al., ; Liu et al., ; Tang et al., ), lift 2D diffusion priors into 3D without requiring any 3D data.
In contrast, large reconstruction models (LRM) (Hong et al., ; Wang et al., ; Xu et al., ; Li et al., ; Weng et al., ) directly train feed-forward models to predict neural radiance fields (NeRF) or Gaussian Splatting from single or multi-view image inputs.
Lastly, 3D-native latent diffusion models (Zhang et al., ; Wu et al., ; Li et al., ) encode 3D assets into latent spaces and generate diverse contents by performing diffusion steps in the latent space.
However, all these approaches rely on continuous 3D representations, such as NeRF or SDF grids, which lose the discrete face indices in triangular meshes during conversion. Consequently, they require post-processing, such as marching cubes (Lorensen & Cline, ) and re-meshing algorithms, to extract triangular meshes. These meshes differ significantly from artist-created ones, which are more concise, symmetric, and aesthetically structured. Additionally, these methods are limited to generating watertight meshes and cannot produce single-layered surfaces.

Recently, several approaches (Siddiqui et al., ; Chen et al., ; ; Weng et al., ; Chen et al., ) have attempted to tokenize meshes into 1D sequences and leverage auto-regressive models for direct mesh generation.
Specifically, MeshGPT (Siddiqui et al., ) proposes to empirically sort the triangular faces and apply a vector-quantization variational auto-encoder (VQ-VAE) to tokenzie the mesh.
MeshXL (Chen et al., ) directly flattens the vertex coordinates and does not use any compression other than vertex discretization.
Since these methods directly learn from mesh vertices and faces, they can preserve the topology information and generate artistic meshes.
However, these auto-regressive mesh generation approaches still face several challenges.
(1) Generation of a large number of faces: due to the inefficient face tokenization algorithms, most prior methods can only generate meshes with fewer than 1,600 faces, which is insufficient for representing complex objects.
(2) Generation of high-resolution surface: previous works quantize mesh vertices to a discrete grid of only $128^{3}$ resolution, which results in significant accuracy loss and unsmooth surfaces.
(3) Model generalization: training auto-regressive models with difficult input modalities is challenging. Previous approaches often struggle to generalize beyond the training domain when conditioning on single-view images.

In this paper, we present a novel approach named EdgeRunner to address the aforementioned challenges.
Firstly, we introduce a mesh tokenization method that compresses sequence length by 50% and reduces long-range dependency between tokens, significantly improving the training efficiency.
Secondly, we propose an Auto-regressive Auto-encoder (ArAE) that compresses variable-length triangular meshes into fixed-length latent codes.
This latent space can be used to train latent diffusion models conditioned on other modalities, offering better generalization capabilities.
We also enhance the training pipeline to support higher quantization resolution.
These improvements enable EdgeRunner to generate diverse, high-quality artistic meshes with up to 4,000 faces and vertices discretized at a resolution of $512^{3}$ — resulting in sequences that are twice as long and four times higher in resolution compared to previous methods.

In summary, our contributions are as follows:

- 1.
We introduce a novel mesh tokenization algorithm, adapted from EdgeBreaker, which supports lossless face compression, prevents flipped faces, and reduces long-range dependencies to facilitate learning.
- 2.
We propose an Auto-regressive Auto-encoder (ArAE), comprising a lightweight encoder and an auto-regressive decoder, capable of compressing variable-length triangular meshes into fixed-length latent codes.
- 3.
We demonstrate that the latent space of ArAE can be leveraged to train latent diffusion models for better generalization, enabling conditioning of different input modalities such as single-view images.
- 4.
Extensive experiments show that our method generates high-quality and diverse artistic meshes from point clouds or single-view images, exhibiting improved generalization and robustness compared to previous methods.

## 2 Related Work

### 2.1 Optimization-based 3D Generation

Early 3D generation methods relied on SDS-based optimization techniques (Jain et al., ; Poole et al., ; Wang et al., ; Mohammad Khalid et al., ; Michel et al., ) due to limited 3D data.
Subsequent works advanced in generation quality (Lin et al., ; Wang et al., ; Chen et al., ; ; Sun et al., ; Qiu et al., ),
reducing generation time (Tang et al., ; Yi et al., ; Lorraine et al., ; Xu et al., ),
enabling 3D editing (Zhuang et al., ; Singer et al., ; Raj et al., ; Chen et al., ),
and conditioning on images (Xu et al., ; Tang et al., ; Melas-Kyriazi et al., ; Liu et al., ; Qian et al., ; Shi et al., ).
Other approaches first predict multi-view images, then apply reconstruction algorithms to generate the final 3D models (Long et al., ; Li et al., ; Pang et al., ; Tang et al., ).
Recently, Unique3D (Wu et al., ) introduced a method that combines high-resolution multi-view diffusion models with an efficient mesh reconstruction algorithm, achieving both high quality and fast image-to-3D generation.

### 2.2 Feed-forward 3D Generation

With the introduction of large-scale datasets (Deitke et al., ; ), more recent works propose to use feed-forward 3D models. The Large Reconstruction Model (LRM)(Hong et al., ) demonstrated that end-to-end training of a triplane-NeRF regression model scales effectively to large datasets and generates 3D assets within seconds. While LRM significantly accelerates generation speed, the resulting meshes often exhibit lower quality and a lack of diversity. Subsequent research has sought to improve generation quality by incorporating multi-view images as inputs(Xu et al., ; Li et al., ; Wang et al., ; He & Wang, ; Siddiqui et al., ; Xie et al., ; Wang et al., ) and by adopting more efficient 3D representations (Zhang et al., ; Li et al., ; Wei et al., ; Zou et al., ; Tang et al., ; Xu et al., ; Zhang et al., ; Chen et al., ; Yi et al., ).

### 2.3 Diffusion-based 3D Generation

Analogous to 2D diffusion models for image generation, significant efforts have been made to develop 3D-native diffusion models capable of conditional 3D generation. Early approaches typically rely on uncompressed 3D representations, such as point clouds, NeRFs, tetrahedral grids, and volumes (Nichol et al., ; Jun & Nichol, ; Gupta et al., ; Cheng et al., ; Ntavelis et al., ; Zheng et al., ; Zhang et al., ; Liu et al., ; Müller et al., ; Chen et al., ; Cao et al., ; Chen et al., ; Wang et al., ; Yariv et al., ; Liu et al., ; Xu et al., ; Yan et al., ) to train diffusion models. However, these methods are often limited by small-scale datasets and struggle with generalization or producing high-quality assets.
More recent approaches have focused on adapting latent diffusion models to 3D (Zhao et al., ; Zhang et al., ; Wu et al., ; Li et al., ; Lan et al., ; Hong et al., ; Tang et al., ; Chen et al., ). These methods first train a VAE to compress 3D representations into a more compact form, which enables more efficient diffusion model training. Unlike the straightforward image representations in 2D, 3D latent diffusion models involve numerous design choices, leading to varied performance outcomes. For example, CLAY (Zhang et al., ) has demonstrated that a transformer-based 3D latent diffusion model can scale to large datasets and generalize well across diverse input conditions.

### 2.4 Auto-regressive Mesh Generation

The above works require additional post-processing steps to extract triangular meshes and fail to model the mesh topology. Recenlty, approaches using auto-regressive models to directly generate meshes have emerged. MeshGPT (Siddiqui et al., ) pioneered this approach by tokenizing a mesh through face sorting and compression with a VQ-VAE, followed by using an auto-regressive transformer to predict the token sequence. This method allows for the generation of meshes with varying face counts and incorporates direct supervision from topology information, which is often overlooked in other approaches.
Subsequent works (Chen et al., ; Weng et al., ; Chen et al., ) have explored different model architectures and extended this approach to conditional generation tasks, such as point cloud generation. However, these methods are limited to meshes with fewer than 1,000 faces due to the computational cost of mesh tokenization and exhibit limited generalization capabilities. A concurrent work, MeshAnythingV2 (Chen et al., ), introduces an improved mesh tokenization technique, increasing the maximum number of faces to 1,600.
Our approach also falls under the category of auto-regressive mesh generation but aims to further extend the maximum face count and provide control over the target face number during inference.

Figure: Figure 2: Pipeline of our method. Our ArAE model compresses variable-length mesh into fixed-length latent code, which can be further used to train latent diffusion models conditioned on other input modalities, such as single-view images.
Refer to caption: x2.png

## 3 EdgeRunner

### 3.1 Compact Mesh Tokenization

Auto-regressive models process information in the form of discrete token sequences.
Thus, compact tokenization is crucial as it allows information to be represented with fewer tokens accurately.
For example, text tokenizers have been a central research direction for large language models (LLMs).
The GPT and Llama series (Touvron et al., ; Brown et al., ) utilize the byte-pair encoding (BPE) tokenizer, which combines sub-word units into single tokens for highly compact and lossless compression.

In contrast, tokenization techniques used in prior auto-regressive mesh generation works mainly suffer from two issues.
Some prior works use lossy VQ-VAEs (Siddiqui et al., ; Chen et al., ; Weng et al., ), which sacrifices the mesh generation quality.
Others opt for zero-compression by not using face tokenizers (Chen et al., ), which poses training challenges due to the inefficiency.

In this paper, we introduce a tokenization scheme that allows us to represent a mesh compactly and efficiently, which is based on the well-established triangular mesh compression algorithm EdgeBreaker (Rossignac, ).
The key insight for mesh compression is to maximize edge sharing between adjacent triangles.
By sharing an edge with the previous triangle, the next triangle requires only one additional vertex instead of three.
We illustrate our mesh tokenization process below and provide the algorithm details in the appendix.

Figure: Figure 3: Illustration of our mesh tokenizer. Our tokenizer traverses the 3D mesh triangle-by-triangle and converts it into a 1D token sequence. Through edge sharing, we reach a compression rate of 50% (4 or 5 tokens per face on average) compared to naïve tokenization of 9 tokens per face.
Refer to caption: x3.png

Figure: Figure 4: Half-edge representation for triangular faces.
Refer to caption: x4.png

Half-edge.
We use the half-edge data structure (Weiler, ) to represent the input mesh for triangular face traversal.
An illustration is provided in Figure 4.
We use $\mathcal{H}_{\cdot\cdot}^{\cdot}$ to denote a half-edge.
For example, $\mathcal{H}_{41}^{3}$ is the half-edge pointing from vertex 4 to 1, with vertex 3 across the face.
Starting from $\mathcal{H}_{41}^{3}$, we can traverse to the next half-edge $\mathcal{H}_{13}^{4}$ and the next twin half-edge $\mathcal{H}_{31}^{2}$.
Reversely, the previous half-edge is $\mathcal{H}_{34}^{1}$ and the previous twin half-edge is $\mathcal{H}_{43}^{5}$.

Vertex Tokenization.
To tokenize a mesh into a discrete sequence, vertex coordinates require discretization. Following previous works (Siddiqui et al., ), we normalize the mesh to a unit cube and quantize the continuous vertex coordinates into integers according to a quantization resolution, which is 512 in this work. Each vertex is therefore represented by three integer coordinates, which are then flattened in XYZ order as tokens.
With some abuse of notion, we denote the XYZ tokens as a single vertex token using 1.

Face Tokenization.
We traverse through all faces following the half-edges.
To illustrate the process, we use the mesh example in Figure 3.
The process starts with one half-edge, where $\mathcal{H}_{23}^{1}$ is picked as the beginning of the current traversal.
We signify the start of a traversal as B.
We then append the vertex across the half-edge 1 as the first vertex token.
Within the same triangular face, the two vertices 2 3 are also appended following the direction of $\mathcal{H}_{23}^{1}$.

During traversal, we visit the next twin half-edge whenever possible, and only reverse the half-edge direction to the previous twin half-edge when we exhaust all triangles in the current traversal.
Returning to the example in Figure 3, we follow $\mathcal{H}_{23}^{1}$ and reach $\mathcal{H}_{13}^{4}$.
Thus, we append N to signify the next twin traversal direction and we only need to append 4 as 1 3 are shared.
The same process is repeated for $\mathcal{H}_{43}^{5}$ with N 5 added to the current sub-sequence.

We have completed the current traversal as no adjacent faces can be found for $\mathcal{H}_{43}^{5}$.
The sub-sequence for the current traversal is thus B 1 2 3 N 4 N 5.

To begin a new sub-sequence, we reversely retrieve the last-added half-edges to traverse in the opposite directions.
As the last-added half-edge $\mathcal{H}_{43}^{5}$ doesn’t have any adjacent faces, we skip it and instead consider $\mathcal{H}_{13}^{4}$.
We go opposite to its previous twin half-edge $\mathcal{H}_{14}^{6}$.
As this is a new sub-sequence, B 6 1 4 are added.

We continue finding the un-visited faces in the neighborhood of $\mathcal{H}_{14}^{6}$ and arriving at its previous twin half-edge $\mathcal{H}_{16}^{7}$.
Thus, we add P 7 to the current sub-mesh sequence as 6 1 are shared.
The process is repeated and P 8 P 2 N 9 are added. As all triangular faces have been visited, the face tokenization process of the mesh is complete.

Auxiliary Tokens.
Similar to prior work in LLMs, we prepend a BOS at the beginning of a mesh sequence and append a EOS at the end of a mesh sequence.

Detokenization.
It is straightforward to reconstruct the original mesh from a mesh token sequence.
We iterate over the tokens while maintaining a state machine.
Each B of a sub-sequence is always followed by three vertex tokens.
Each N or P is followed by a single vertex token, and we retrieve two previous vertex tokens based on the traversal direction to reconstruct the triangle.
Finally, we merge duplicate vertices, as they may appear multiple times from different sub-sequences, and output the reconstructed mesh.

Advantages.
Our tokenizer benefits model training in several ways:
(1) Each face requires an average of 4 to 5 tokens, achieving approximately 50% compression compared to the 9 tokens used in previous works (Chen et al., ; ). This increased efficiency enables the model to generate more faces with the same number of tokens and facilitates training on datasets containing a higher number of faces.
(2) Our traversal is designed to avoid long-range dependency between tokens. Each token only relies on a short context of previous tokens, which further mitigate the difficulty of learning.
(3) The traversal ensures that each face’s orientation remains consistent within each sub-mesh. Consequently, the generated mesh can be accurately rendered using back face culling, a feature not consistently achieved in prior methods.
We will further illustrate these benefits in Section 4.

### 3.2 Auto-regressive Auto-encoder

Although our decoder is auto-regressive and generates variable-length token sequences, we observe limitations in both generation diversity and its ability to follow conditioning. In contrast, diffusion models, which have been extensively studied (Rombach et al., ; Zhang et al., ), demonstrate promising results in addressing these challenges.
To apply diffusion models, a key challenge is the requirement of fixed-length data, as mesh generation has a variable-length data structure and the number of triangle faces can vary significantly due to different complexities.
Therefore, we propose an Auto-regressive Auto-encoder (ArAE) to encode the variable-length mesh into a fixed-length latent space, similar to the role of variational auto-encoders (VAEs) in latent diffusion models.
To train our ArAE model, we choose point clouds as the encoder input for geometric information and our tokenized mesh sequences as output.
Thus the ArAE model itself also works as a point cloud conditioned mesh generator.

Architecture.
The architecture of our ArAE model is illustrated in Figure 2.
ArAE consists of a lightweight encoder and an auto-regressive decoder.
To extract geometric information from the mesh surface, we follow prior works to sample a point cloud and apply a transformer encoder (Chen et al., ; ; Zhang et al., ; ).
Specifically, we sample $N$ random points, $\mathbf{X}\in\mathbb{R}^{N\times 3}$, from the surface of the input mesh and use a cross-attention layer to extract the latent code:

$$ $\mathbf{Z}=\text{CrossAtt}\big{(}\mathbf{Q},\text{PosEmbed}(\mathbf{X})\big{)}$ (1) $$

where $\mathbf{Q}\in\mathbb{R}^{M\times C}$ represents the trainable query embedding with a hidden dimension of $C$, $\text{PosEmbed}(\cdot)$ is a frequency embedding function for 3D points (Zhang et al., ), and $\mathbf{Z}\in\mathbb{R}^{M\times L}$ represents the latent code, $M<N$ and $L<C$ denote the latent size and dimension, respectively.
The decoder is an auto-regressive transformer, designed to generate a variable-length mesh token sequence.
For simplicity, we adopt the OPT architecture (Zhang et al., ), as used in prior works (Siddiqui et al., ; Chen et al., ; ).
A learnable embedding converts discrete tokens into continuous features, and a linear head maps predicted features back to classification logits.
Stacked causal self-attention layers are employed to predict the next token based on previous tokens.
The latent code $\mathbf{Z}$ is prepended to the input before the BOS token, allowing the decoder to learn how to generate a mesh token sequence conditioned on the latent code.

Face Count Condition.
Given a point cloud or a single-view image as input, multiple plausible meshes with varying numbers of faces and topologies can be generated. The number of faces is particularly crucial as it directly affects the mesh’s complexity (low-poly v.s. high-poly) and the generation speed.
To manage meshes with a broad range of face counts, we aim to provide some level of explicit control over the targeted number of faces. This control facilitates the estimation of generation time and the complexity of the generated mesh during inference.
We propose a simple face count conditioning method for coarse-grained control. Specifically, we append a learnable face count token after the latent code condition tokens.
We bucket face count into different ranges and assign different tokens to each range.
For instance, we use four distinct tokens to represent face counts in the following ranges: less than or equal to 1000, between 1000 and 2000, between 2000 and 4000, and greater than 4000.
Additionally, during training, we randomly replace these tokens with a fifth unconditional token. This approach ensures that the model still learns to generate meshes without specifying a targeted face count.

Loss Function.
The ArAE is trained using the standard cross-entropy loss on the predicted next tokens:

$$ $\mathcal{L}_{\text{ce}}=\text{CrossEntropy}(\mathbf{\hat{S}}[:-1],\mathbf{S}[1 :])$ (2) $$

where $\mathbf{S}$ denotes the one-hot ground truth token sequence, and $\mathbf{\hat{S}}$ represents the predicted classification logits sequence.
Additionally, to constrain the range of the latent space for easier training of subsequent diffusion models, we apply an L2 norm penalty to the latent code:

$$ $\mathcal{L}_{\text{reg}}=||\mathbf{Z}||^{2}_{2}$ (3) $$

The final loss is a weighted combination of the cross-entropy loss and the regularization term.

### 3.3 Image-conditioned Latent Diffusion

With the fixed-length latent space provided by our ArAE architecture, it is now feasible to train mesh generation models conditioned on different inputs, akin to how 2D image generation models are trained.
Among various input modalities, we showcase the capabilities of our model using single-view images, which are among the most commonly used conditions for mesh generation.

We follow the approach of previous methods (Chen et al., ) and utilize a diffusion transformer (DiT) as the backbone.
Specifically, we employ the image encoder from CLIP (Ilharco et al., ; Mohammad Khalid et al., ) to extract image features for conditioning.
Cross-attention layers are used to integrate the image condition into the denoising features, while AdaLN layers incorporate timestep information.
We use the DDPM framework (Ho et al., ) and mean square error (MSE) loss to train the DiT model.
At each training step, we randomly sample a timestep $t$ and a Gaussian noise $\epsilon\in\mathbb{R}^{M\times L}$.
The loss is calculated between the predicted noise and the random noise.

## 4 Experiments

Figure: Figure 5: Comparison on point cloud conditioned generation. We show the reference dense mesh and generated meshes conditioned on randomly sampled point cloud.
Refer to caption: x5.png

Figure: Figure 6: Comparison on image conditioned generation. We show the input image and generated meshes. Since Unique3D outputs dense meshes, we only visualize the surface without wireframe.
Refer to caption: x6.png

### 4.1 Qualitative Results

Point Cloud Conditioned Generation.
We first compare the generated meshes conditioned on point clouds in Figure 5.
For the test samples, we use other image-to-3D methods (Zhang et al., ; Li et al., ) to generate dense meshes, ensuring that these samples have never been seen in the training dataset.
Notably, MeshAnything (Chen et al., ; ) uses both point coordinates and normal vectors as input to a pretrained encoder (Zhao et al., ).
In contrast, our encoder takes only point coordinates and is trained from scratch, while still achieving better performance.
Our model demonstrates greater stability on challenging test cases, producing more aesthetically pleasing meshes. Additionally, the proposed tokenization algorithm enables the generation of longer sequences, facilitating the modeling of complex shapes and improving the closure of loops for watertight surfaces.

Image Conditioned Generation.
In Figure 6, we compare the meshes generated from single-view images.
Since other auto-regressive methods cannot directly condition on images, we compare our results with a recent optimization-based approach (Wu et al., ).
Our end-to-end image-to-mesh generation produces artistic meshes that better capture the semantics of the input images.
Additionally, our model exhibits strong generalization on challenging 2D-style or realistic-lighting images, despite being trained exclusively on images rendered from 3D meshes with simple shading.

Face Count Control.
In practice, users may have specific requirements for the face count or level of detail in the generated mesh.
Our design enables coarse-grained control over the face count by allowing users to specify the face count condition.
As demonstrated in Figure 7, our model is capable of generating meshes with different ranges of face count from the same input point cloud.

Figure: Figure 7: Coarse-grained face count control. Our model supports controlling the targeted number of face, which allows to generate meshes with different levels of detail.
Refer to caption: x7.png

Figure: Figure 8: Ablation Study on different quantization resolutions and image conditioning strategies.
Refer to caption: x8.png

### 4.2 Quantitative Results

Tokenization Algorithm.
In Table 1, we compare our mesh tokenization algorithm with AMT from MeshAnythingV2 (Chen et al., ).
Both algorithms aim to traverse all triangular faces.
While AMT can only move to a fixed side that shares the last two vertices in the sequence, our algorithm allows movement to both the left and right sides.
This flexibility reduces the number of sub-sequences, making training easier and improving model robustness, as demonstrated in Figure 5.
Although AMT requires only 3 coordinate tokens for a subsequent face and our method requires 4 tokens (with an additional face type token), the amortized compression ratio remains similar.

Inference Speed.
Since we are using next-token prediction, the inference time for each generation is dependent on the length of the sequence.
Our model is basically a large transformer, and the major compute is used for attention calculation.
On a A100 GPU with flash-attention (Dao, ) and KV cache enabled, our model runs at about 100 tokens per second.
Specifically, it takes about 45 seconds to generate a mesh with 1,000 faces, about 90 seconds for 2,000 faces, and about 3 minutes for 4,000 faces.

**Table 1: Comparison of the tokenizer algorithm. Our tokenizer has a comparable compression ratio as AMT (Chen et al., ) but uses much fewer sub-sequences per mesh, which makes model training easier. Moreover, our implementation is three times faster (mesh per second).**
|  | Compression Ratio $\downarrow$ | Sub-sequence Count $\downarrow$ | Tokenization Speed $\uparrow$ |
| --- | --- | --- | --- |
| AMT | 46.2% | 199.5 | 8.6 |
| Ours | 47.4% | 54.7 | 25.2 |

### 4.3 Ablation Studies

Quantization Resolution.
We initially experimented with a quantization resolution of $128^{3}$ as used in previous works.
While this lower resolution made it easier to train the classification head, we observed noticeable quality degradation, as shown in the left part of Figure 8.
To address this problem, we increased the quantization resolution to $512^{3}$ for our final model, which resulted in more accurate vertex positions and smoother surfaces.

Direct Image Conditioning.
We also attempted to directly train an image-conditioned auto-regressive mesh decoder.
The image features from the CLIP vision encoder were prepended as condition tokens, and the model directly predicted the mesh token sequence.
However, we found that this approach was more difficult to converge and generally exhibited poorer generalization on unseen test cases, as shown in the right part of Figure 8.
Consequently, we designed the ArAE model to learn a fixed-length latent space using a simpler point-cloud condition, and we trained latent diffusion models to map complex condition modalities to this latent space. This approach led to better generalization and more efficient training.

## 5 Conclusion

In this work, we propose a novel Auto-regressive Auto-encoder that compresses variable-length triangular meshes into fixed-length latent codes, and we further demonstrate the potential of this latent space for image-to-mesh generation.
Additionally, we introduce an efficient mesh tokenization algorithm that enables scaling up the model, resulting in longer context lengths and higher resolution.
Our approach highlights the potential for improving the scalability of auto-regressive models for practical 3D mesh generation, offering a promising direction for future research and applications.

#### Acknowledgments

This work is supported by the Sichuan Science and Technology Program (2023YFSY0008), National Natural Science Foundation of China (61632003, 61375022, 61403005), Grant SCITLAB-20017 of Intelligent Terminal Key Laboratory of SiChuan Province, Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS11), and PEK-SenseTime Joint Laboratory of Machine Vision.

## Appendix A Appendix

### A.1 Implementation Details

#### A.1.1 Mesh Tokenizer

Figure: Algorithm 1 Tokenization

Tokenization Algorithm.
To formally define the algorithm, we first describe how to use the half-edge notation to refer the mesh data in Figure 9.
For a vertex $\mathbf{v}$, its three coordinates are denoted as $(\mathbf{v}.x,\mathbf{v}.y,\mathbf{v}.z)$. For a face $\mathbf{t}$, we store its three half-edges in an array $\mathbf{t}.\text{halfedges}$. Additionally, we use $\mathbf{v}.vis$ and $\mathbf{t}.vis$ to indicate whether this vertex or face has already been visited in the algorithm.

Figure: Figure 9: Mesh notation starting from half-edge $\mathbf{c}$.
Refer to caption: x9.png

Comparison.
We compare the original EdgeBreaker algorithm with our modified version and highlight the key differences.
For clarity, we use N and P tokens in the main paper to represent moving to the right or left triangles (the next twin or previous twin half-edges). To remain consistent with the original algorithm, we now revert to using L and R.

(1) In the original EdgeBreaker algorithm, newly added vertices are stored in a residual form relative to the previous triangle, following the parallelogram rule (Rossignac, ). This method can potentially reduce the number of bits when compressed further in binary format. However, since our objective is to convert the mesh into a discrete token sequence rather than binary bits, relative vertex encoding is unnecessary. Moreover, our discretization reduces the accuracy of relative coordinates, so we simply use the absolute coordinates for each vertex.

(2) As illustrated in Figure 10, the original algorithm employs five face type tokens: C, L, E, R, and S.
The special S token is used to indicate a bifurcation in the traversal tree. However, since the binary tree must be flattened into a 1D sequence, both branches dependent on the S-type triangle can introduce long-range dependencies, particularly if the first-visited branch is long, as shown in Figure 11. To avoid such dependencies, we explicitly duplicate the necessary vertices before visiting the second branch, which can be interpreted as restarting a sub-mesh sequence. While this approach slightly worsens the compression rate, it effectively eliminates long-range dependencies, which can be difficult for the transformer model to learn.

(3) During tokenization, the C token must be distinguished from the others, but it behaves similarly to the L token during de-tokenization. Since tokenization is unnecessary during inference, we replace all C tokens with L for training. Additionally, we merge the E token into B, as E always follows B after removing the S token.
For simplicity, we don’t consider special cases such as holes or handles, since they can still be represented at the cost of several degenerated faces, which can be easily cleaned during post-processing.
As a result, our modified algorithm uses only three face types: L, R, and B.

#### A.1.2 Auto-regressive Auto-encoder

Datasets.
For the ArAE model, we use meshes from the Objaverse and Objaverse-XL datasets (Deitke et al., ; ). Given that the original datasets include many low-quality 3D assets, we filter the meshes empirically based on metadata captions or file names (Tang et al., ).
Our training set comprises approximately 112K meshes with fewer than 4,000 faces from both Objaverse and Objaverse-XL.
For finetuning, we select only the higher-quality meshes from Objaverse, totaling around 44K.
Each mesh undergoes a preprocessing pipeline where close vertices are merged, collapsed faces are removed, and the mesh is normalized to fit within a unit cube.
The vertices are quantized at a resolution of $512^{3}$.
Our tokenizer’s vocabulary thus includes 512 coordinate tokens, 3 face type tokens ( N, P, B) and 3 special tokens ( BOS, EOS, PAD), resulting in a total vocabulary size of 518.

Architecture.
We illustrate the detailed architecture of our network in Figure 12.
The encoder of our ArAE model employs a cross-attention layer as outlined by (Zhang et al., ).
We sample 8,192 points uniformly from the mesh surface and use a $4\times$ down-sampled embedding to query the latent code.
Consequently, the latent code has a shape of $\mathbf{Z}\in\mathbb{R}^{2048\times 64}$.
The decoder transformer is comprised of 24 self-attention layers, each with a hidden dimension of 1,536 and 16 attention heads.
Overall, our ArAE model contains approximately 0.7 billion trainable parameters.

Training and Inference.
We train the ArAE model on 64 A100 (80GB) GPUs for approximately one week. The batch size is 4 per GPU, leading to an effective batch size of 256.
We use the AdamW optimizer (Loshchilov & Hutter, ) with a cosine-decayed learning rate that ranges from $5\times 10^{-5}$ to $5\times 10^{-6}$, a weight decay of 0.1, and betas of $(0.9,0.95)$. Gradient clipping is applied with a maximum norm of 1.0.
Given that each point cloud input can correspond to multiple plausible meshes but only one ground truth mesh is available, we apply robust data augmentation techniques to address data insufficiency during training:
(1) The input mesh is randomly scaled by a factor between $[0.75,0.95]$.
(2) The mesh is randomly rotated along the vertical axis by up to 30 degrees.
(3) Random quadric edge collapse decimation (Garland & Heckbert, ) is applied to perturb the number of faces. There is a 50% chance at each iteration that the mesh will be decimated to no more than one-quarter of its original number of faces.
During inference, we employ a multinomial sampling strategy with a top-k of 10.
To ensure that the generated sequence is valid for decoding by the tokenizer, we apply the following rules to post-process the logits of the next token based on the prefix token sequence:
(1) The token following BOS should be B.
(2) After B, there should be 9 coordinate tokens.
(3) After N or P, there should be 3 coordinate tokens.
(4) Otherwise, the next token should be one of N, P, B, or EOS.

Figure: Figure 10: Comparison of the original EdgeBreaker traversal order with our modified version.
Refer to caption: x10.png

Figure: Figure 11: Our tokenization reduces potential long range dependency in the token sequence.
Refer to caption: x11.png

#### A.1.3 Latent Diffusion

Datasets.
For the image-conditioned diffusion model, we use meshes from the Objaverse dataset along with rendered images from the G-Objaverse dataset (Qiu et al., ).
The final training dataset comprises approximately 75K 3D meshes, each accompanied by rendered RGB images from 38 different camera poses.
We select only 21 camera poses that align with the front face of most objects in Objaverse empirically.

Architecture.
Our DiT model consists of 24 layers of self-attention and cross-attention, as detailed in Section 3.
Each layer can be expressed as follows:

$$ $\displaystyle\gamma_{1},$ $\displaystyle\ \beta_{1},\alpha_{1},\gamma_{2},\beta_{2},\alpha_{2}=\text{ Chunk}(\textbf{T}+\textbf{t}_{e})$ (4) x $\displaystyle=\textbf{x}+\alpha_{1}\times\text{SelfAttention}(\text{LayerNorm} _{1}(\textbf{x})\times(1+\beta_{1})+\gamma_{1})$ (5) x $\displaystyle=\textbf{x}+\text{CrossAttention}(\textbf{x},\textbf{c})$ (6) x $\displaystyle=\textbf{x}+\alpha_{2}\times\text{FeedForward}(\text{LayerNorm}_{ 2}(\textbf{x})\times(1+\beta_{2})+\gamma_{2})$ (7) $$

where T is a layer-wise learnable scale shift table, $\textbf{t}_{e}$ denotes the timestep features, and c denotes the condition features.
All attention layers have a hidden dimension of 1,024 and use 16 attention heads.
We utilize the pretrained CLIP-H model from OpenCLIP (Ilharco et al., ) as the image feature encoder.
Features from the last layer of the vision transformer are employed as conditions for the cross-attention layers.
The DiT model has approximately 0.5 billion trainable parameters in total.

Training and Inference.
We train the DiT model on 16 A800 (40GB) GPUs for approximately one week.
The lightweight ArAE encoder allows for on-the-fly calculation of the latent code during training, so we can also apply random down-sampling and rotation as data augmentation techniques for the point cloud input.
The batch size is set to 32 per GPU, resulting in an effective batch size of 512.
We use the same optimizer as for the ArAE model.
Training employs the min-SNR strategy (Hang et al., ) with a weight of 5.0.
The DDPM is configured with 1,000 timesteps and utilizes a scaled linear beta schedule.
For classifier-free guidance during inference, 10% of the image conditions are randomly set to zero for unconditional training.
During inference, we use the DDIM scheduler (Song et al., ) with 100 denoising steps and a Classifier-Free Guidance (CFG) scale of 7.5.

Figure: Figure 12: Network Architecture of our ArAE and DiT models. Our ArAE contains an encoder that takes a point cloud as input and encodes it to a fixed-length latent code, and an auto-regressive decoder that predicts variable-length mesh token sequence. The latent space can be used to train diffusion models conditioned on other more difficult conditions, such as single-view images.
Refer to caption: x12.png

### A.2 More Results

#### A.2.1 Face Orientation

In Figure 13, we show the face orientation and rendering with back-face culling.
The tokenization algorithm of our method and MeshAnythingV2 (Chen et al., ) ensures that the face orientation in each mesh sequence is correct, while MeshAnything (Chen et al., ) fails to do this.

Figure: Figure 13: Face orientation. Our model ensures correct face orientation of the generated mesh.
Refer to caption: x13.png

#### A.2.2 Generation Progress

In Figure 14, we show the generation progress by simultaneously de-tokenizing the sequence. Our model successfully learns how to generate a valid sequence for the modified EdgeBreaker algorithm to de-tokenize.

Figure: Figure 14: Visualization of the generation progress. Please visit our project page for video results.
Refer to caption: x14.png

#### A.2.3 User Study

**Table 2: User Study on point-conditioned mesh generation. The rating is of scale 1-5, the higher the better.**
|  | Input Consistency | Triangle Aesthetic | Overall Quality |
| --- | --- | --- | --- |
| MeshAnything | 2.93 | 2.43 | 2.43 |
| MeshAnythingV2 | 2.64 | 2.36 | 2.14 |
| Ours | 4.83 | 4.54 | 4.58 |

Automatically evaluating the aesthetic quality of mesh topology is challenging, so we rely on a user study.
We selected 8 test cases and provided renderings of both the mesh geometry and wireframe.
Volunteers were asked to evaluate the mesh quality on a scale from 1 to 5.
As shown in Table 2, our method is preferred across all evaluated aspects.

#### A.2.4 Limitations

Figure: Figure 15: Limitations. We show some failure cases of our method to illustrate our limitations.
Refer to caption: x15.png

Despite the promising results, our approach has several limitations:
(1) The compression ratio of our tokenization algorithm remains insufficient. Complex meshes, such as those representing game characters, typically require a much higher number of faces, exceeding the current capacity of our model.
(2) Although our model can generate meshes with up to 4,000 faces, the success rate decreases as the sequence length increases, and the generation time becomes progressively longer.
(3) While the auto-regressive next-token prediction allows for diverse outputs from the same input, it also leads to potentially unsatisfactory results. Since there are infinite plausible face topologies for the same surface, precise control over the generation is difficult to achieve.
We illustrate some failure cases in Figure 15.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: I have a few questions for the authors to consider:

The paper presents an improvement in the number of faces that can be generated compared to previous methods, reaching up to 4,000 faces. However, for many practical applications in industries such as gaming, film, etc, this number is still limited. How to increase the face count further? Are there any technical barriers within the current model that restrict the generation of meshes with even more faces?

Comparison with Unique3D is not fair. How does the method perform when compared with CLAY? How does EdgeRunner compare in terms of quality and diversity of generated meshes? While the code CLAY is not publicly available, its geometric rendering effects can be observed on Hugging Face and its corresponding products are available for comparison.
soundness: 2
strengths: EdgeRunner demonstrates strengths in the mesh tokenization algorithm and the Auto-regressive Auto-encoder framework, which address previous challenges in mesh generation.

It offers a mesh tokenization algorithm for efficient compression into 1D token sequences, enabling mesh generation with up to 4,000 faces at a 512^3 resolution. The model's ability to compress variable-length meshes into a fixed-length latent space facilitates the training of latent diffusion models for enhanced generalization.
summary: The paper introduces EdgeRunner, an Auto-regressive Auto-encoder (ArAE) model for artistic mesh generation. It proposes a mesh tokenization algorithm to address incompleteness and poor generalization. 

The model can generate 3D meshes with up to 4,000 faces at a spatial resolution 512^3. It also compresses variable-length meshes into a fixed-length latent space, enabling the training of latent diffusion models for better generalization. 

The main contributions include a mesh tokenization algorithm for lossless face compression, an ArAE model for fixed-length latent code generation, the use of this latent space for training latent diffusion models with better generalization.
weaknesses: A. While the paper presents a compelling case for EdgeRunner's capabilities, a notable weakness is that it does not fundamentally differentiate itself from existing works like MeshGPT and the MeshAnything series in terms of the core generative approach. 

B. Despite the improvements in tokenization and the auto-regressive framework, the generated 3D geometries may not adhere to the modeling and wiring conventions that human artists typically follow. This could potentially limit the acceptance and practical application of the generated meshes within professional 3D content creation pipelines, where adherence to standard modeling practices is often a requirement.

C. The paper also has a limitation in terms of polycount, as the capability of generating meshes with up to 4,000 faces, while an improvement over previous methods, still falls short for many real-world scenarios. The constraint on the number of faces limits the model's ability to capture the complexity and detail required for high-fidelity 3D representations, which further limits the application. 

D. Another limitation of the paper is the potentially unfair comparison made with Unique3D, as the latter employs a multi-view approach to 3D generation. Unique3D's methodology is inherently challenged by inconsistencies or deformities that can arise from generating a 3D model from multiview images, which is a different set of problems compared to EdgeRunner. This discrepancy means that the comparison may not accurately reflect the strengths and weaknesses of each method, as they are optimized for different conditions and face distinct sets of challenges. Consequently, the paper's claims about EdgeRunner's superiority might not be fully substantiated.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What does the dashed line mean in Figure 2?
2. Does the output number of faces strictly adhere to the face count control? For instance, when using a control token of 1000 to 2000, will the model generate only mesh sequences within that range?
soundness: 3
strengths: 1. The proposed method can generate meshes with up to 4000 faces, surpassing the capabilities of previous baselines. 
2. The trained model shows strong generalization on novel inputs.
3. Extensive experiments highlight the advantages of the proposed method in achieving high-quality mesh generation.
summary: This paper focuses on generating artistic meshes with up to 4000 faces and a resolution of 512³. It introduces a novel mesh tokenization algorithm based on EdgeBreaker to facilitate lossless face compression. To produce fixed-length latent codes for arbitrary mesh sequences, the authors propose an encoder alongside an auto-regressive decoder. Additionally, they investigate the incorporation of image conditions to guide the generation of the meshes.
weaknesses: 1. The proposed method and the baselines are trained on different datasets (for example, MeshAnything does not have access to Objaverse-XL and reserves 10% of Objaverse for evaluation). As a result, the comparisons can be unfair.
2. Is the training sequence unique for each mesh? How did you define the start of the sequence?
3. Although the authors report the inference speed, there is no comparison of the inference speed against other methods when generating similar number of faces.
4. The details of user study are missing. How many users are there in the test? What is the details of the task or questions that volunteers are asked to do or answer? Are 8 test cases randomly selected? It is better to have more test cases rather than only 8 ones.
5. To demonstrate the robustness across all test cases, it is better to involve quantitative evaluations like those in *MeshAnything*, where the authors sample points from both the generated mesh and the ground-truth mesh, calculating the CD or ECD.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: I have some questions primarily about the implementation and public availability of this work:

- Would it be possible to share the mesh tokenization script on an anonymous GitHub? This would help in understanding how the "traversal" process works and can be implemented.
- I didn’t find any mention in the paper regarding the public availability of the code and pre-trained model. Will these resources be released?
soundness: 4
strengths: The paper proposes a novel method for mesh tokenization which allows to handle mesh with more faces and in higher resolution.  It also introduces a latent space with a unified token length, enhancing the model's generalization ability. These two contributions represent the technical novelties of the paper. 

The paper includes a comprehensive comparison with several state-of-the-art mesh generative models, such as MeshAnything, MeshAnythingV2, and Unique3D. This thorough evaluation makes the study well-rounded and complete.

The method is clearly described and easy to follow. The figures and renderings are thoughtfully designed, giving the paper a polished visual appeal. Additionally, the accompanying website provides extensive results, further illustrating the method's effectiveness and aiding in comprehension of its impact.
summary: This paper introduces EdgeRunner, an auto-regressive mesh generative model that presents a novel and efficient mesh tokenization algorithm capable of handling a higher number of mesh faces and higher resolutions. By encoding variable-length meshes into a unified latent space, the proposed method can generate meshes from point clouds or single-view images. The paper demonstrates a well-motivated and effective approach, creating a cohesive narrative.

However, key information, such as training data and experimental settings, is missing from the main text, which limits readability. Given the novelty and effectiveness of the method, I suggest accepting the paper, but I strongly recommend that the authors restructure the content to improve readability and accessibility.
weaknesses: While I find no weaknesses in the method itself, I believe the presentation of the paper needs restructured. Important information, such as experimental settings and training datasets, currently appears only in the supplementary materials, which makes the paper difficult to follow.

Presentation:
- Line 102: The term "EdgeBreaker" appears for the first time without a reference, making this part challenging to understand.
- Line 200: It seems that the mesh tokenization approach is heavily inspired by EdgeBreaker, proposed by Rossignac in 1999. However, there is no mention or citation of this foundational paper. Although the authors provide a comparison in the supplementary material on Line 920, omitting a discussion of this algorithm in the main methods section does not adequately acknowledge its influence.
- Line 358: The experiments section begins directly with results, without any mention of training data, settings, or other critical experimental details. These details appear only in the supplementary material, which is insufficient, as they are essential for understanding the experiments.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Is it possible to generate unconditional samples using your approach?
soundness: 4
strengths: The proposed mesh tokenization based on maximizing edge sharing between adjacent triangles is sensible, well referenced and very useful in practice.  It supports lossless compression at 50% rate designed to reduce long range dependencies between tokens which is beneficial for learning.  

The proposed pipeline solves several issues of 3d mesh generation such as the ability to generate variable length outputs through the AR decoder combined with a fixed length latent code which enables the use of diffusion models for conditional generation

Generated examples are shown to be of great quality.

Literature review is comprehensive.
summary: The paper proposes a method for 3d mesh generation. They propose a mesh serialization copression algorithm to tokenize 3d meshes. An encoder-auto regressive decoder network operating in this representation is trained to compress this representation into a fixed length latent space. Additionally the authors propose a latent diffusion approach for generating 3d mesh representations in this latent space conditioned on images and 3d point clouds.
weaknesses: There is no discussion about pretraining the model, loss curves showing training progression, or explaining how unconditional generation works compared to the conditional method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer to the questions in weakness section.
soundness: 3
strengths: 1) The method of tokenization is based on the half edge data structure and is able to achieve 50% more compression than the existing method. This allows for smoother training and avoiding long range dependencies.

2) The paper shows generalization using single-image to mesh examples. The examples shown in the paper are of high quality.

3) The paper shows that the fixed latent space learned by the autoregressive model can be used by the diffusion training to learn many modalities including single-image to mesh capabilities.
summary: The paper introduces a novel mesh tokenization algorithm The tokenization supports lossless face compression and reduces long-range dependencies. This helps in smooth and efficient training of an auto regressive encoder termed as (ArAE). This encoder is capable of compressing point clouds to into fixed-length latent codes representing meshes. ArAE's latent space can be leveraged to train latent diffusion models for better generalization, enabling conditioning of different input modalities such as single-view images. Hence, the method can generate meshes from point clouds or single-view images, showing generalization and multimodal capabilities .
weaknesses: 1) The paper talks about the point cloud sampler but not enough information is provided on it. How is the current method sensitive to the number of points , regions of missing points, and irregularities in the point cloud positions?

2) How is the Siddique et.al's method compared with the output quality? The paper mentions the method in context of the tokenization is lossy reconstruction but does not provide the evidence of the final output quality.

3) The quality of the single-image to meshes would be more exciting to see if the poses of the input images are varied. For example, dog and frog examples in Figure 6, and compared with the comparing method.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: I am curious about the engineering tricks the authors used to save GPU memory and scale up the training datasets.
soundness: 4
strengths: (i) The idea is very novel. Adapting the EdgeBreaker algorithm for mesh token compression is super interesting with the key insight "maximize edge sharing between adjacent triangles". It is a kind of Renaissance. I really like this style. This solution avoids the use of some lossy VQ-VAE models and preserves the geometry information during tokenization. Meanwhile, the customized traversal can also avoid the computation of long-range dependencies and ensure the face's orientation consistency in each sub-mesh. In addition, the core idea of using ArAE to handle variable-length mesh into fixed-length latent tokens is awesome. This can support the multi-modality inputs and address the variable-length issue in mesh generation.

(ii) The presentation is well-dressed. The writing is clear and easy to follow. The pipeline in Figure 2 is simple and clear to show the workflow of the proposed method. The visual comparison of point cloud conditioned mesh generation in Figure 5 and image conditioned mesh generation in Figure 6 are very clear to demonstrate the advantages of the proposed method over state-of-the-art algorithms.

(iii) The performance is very solid. The proposed method not only achieves more visually pleasant generation results with the highest user study score but also yields a much higher compression ratio to boost the model's efficiency. The experiments are sufficient. Especially the user study in Figures 7 and 8. It seems like more faces lead to finer-grained generated meshes. I also like the style of the visual ablation in Figure 8, studying the effect of resolution and image conditioning strategies.

(iv) The implementation details are all provided in the appendix. I believe other researchers can easily reproduce the results. I trust the authors for the reproducibility.
summary: This paper proposes an Auto-regressive Auto-Encoder (ArAE) model for multi-modality mesh generation supporting point clouds and single-view image inputs. Besides, an efficient mesh tokenization inspired by the EdgeBreaker algorithm is designed for more edge sharing during compression.
weaknesses: Just some minor issues:

(i) It would be better to remove the ablation study table from the supplementary to the main table as there are no other quantitative evaluation results. Although the visual comparison is good, I think the main paper should have some numerical comparison.

(ii) Maybe Figure 4 and the left part of Figure 3 should be in the same figure according to the description in Section 3.1. I think Figure 4 , instead of the right part of Figure 3, is more related to the left part of Figure 3.

(iii) For better understanding, in Line 320, the authors should add some description where to introduce the image condition input since in Line 298 - 302, the ArAE is introduced as a point clouds conditioned mesh generator.

(iv) A small typo: (low-poly *v.s.* high-poly) $\rightarrow$ (low-poly *vs.* high-poly)

Meta-review: The paper introduces EdgeRunner, an auto-regressive auto-encoder with a novel mesh tokenization algorithm for high-quality artistic mesh generation, demonstrating significant advancements in mesh compression and fixed-length latent code learning. Reviewers praised the paper's technical contributions, including the innovative use of EdgeBreaker-inspired tokenization, strong experimental results, and the ability to generalize to point clouds and single-view image inputs, producing visually appealing meshes. However, concerns were raised about limited quantitative evaluations, lack of comparisons to alternative methods like CLAY, the relatively low face count (4,000 faces), and the need for clearer explanations of training settings and dataset usage in the main text.

Decision: Accept (Poster)