PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance

## Abstract

Abstract Generating compact and sharply detailed 3D meshes poses a significant challenge for current 3D generative models. Different from extracting dense meshes from neural representation, some recent works try to model the native mesh distribution (i.e., a set of triangles), which generates more compact results as humans crafted. However, due to the complexity and variety of mesh topology, these methods are typically limited to small datasets with specific categories and are hard to extend. In this paper, we introduce a generic and scalable mesh generation framework PivotMesh, which makes an initial attempt to extend the native mesh generation to large-scale datasets.
We employ a transformer-based auto-encoder to encode meshes into discrete tokens and decode them from face level to vertex level hierarchically. Subsequently, to model the complex typology, we first learn to generate pivot vertices as coarse mesh representation and then generate the complete mesh tokens with the same auto-regressive Transformer. This reduces the difficulty compared with directly modeling the mesh distribution and further improves the model controllability. PivotMesh demonstrates its versatility by effectively learning from both small datasets like Shapenet, and large-scale datasets like Objaverse and Objaverse-xl. Extensive experiments indicate that PivotMesh can generate compact and sharp 3D meshes across various categories, highlighting its great potential for native mesh modeling.

## 1 Introduction

The field of 3D generation has witnessed remarkable advancements in recent years [, , ].
Meshes, the predominant representation for 3D geometry, are widely adopted across various applications from video games and movies to architectural modeling.
Despite the promising performance of current methods, they mostly rely on neural 3D representation like triplanes [, ] and FlexiCubes [].
Post-processed meshes extracted from these representations tend to be dense and over-smoothed, which are unfriendly for modern rendering pipelines as shown in Figure 1 (bottom).
In contrast, meshes crafted by humans are typically more compact with fewer faces, reusing geometric primitives to efficiently represent real-world objects.

Figure: Figure 1: Different from 3D generation methods based on neural representations like InstantMesh [], our methods can generate compact and sharp meshes with much fewer faces when producing similar shapes.
Refer to caption: x1.png

To avoid extracting dense meshes through post-processing, another promising direction is emerging that focuses on explicitly modeling the mesh distribution (i.e., native mesh generation).
This line of works [, , ] generates meshes by predicting the 3D coordinates of faces, thus producing compact meshes as humans.
However, due to the complexity and variety of topological structures in meshes, these methods are typically confined to small datasets like Shapenet with single or narrow object categories, hindering the generalizability across diverse types of objects.
Therefore, it still remains a challenge to establish a generic generative model for native mesh generation at scale.

In this paper, we propose PivotMesh, a generic and scalable framework to extend mesh generation to large-scale datasets across various categories.
PivotMesh consists of two parts: a mesh auto-encoder and a pivot-guided mesh generator.
First, the auto-encoder is based on the Transformer to encode meshes into discrete tokens.
We also adopt a two-stage decoding strategy to decode mesh tokens from face level to vertex level hierarchically, which further improves the reconstruction performance and mesh surface continuity.
Second, we employ an auto-regressive Transformer to learn the joint distribution of pivot vertices and complete mesh tokens, where the pivot vertices serve as the coarse representation to guide the following mesh generation.
Specifically, pivot vertices are selected based on vertex degree and dropped randomly to prevent overfitting.
As shown in Figure 1 (top), once the model is trained, it can produce meshes from scratch, starting with the generation of pivot vertices followed by the complete mesh token sequence.
Furthermore, it can perform conditional generation given the pivot vertices from the reference mesh and supports downstream applications.

PivotMesh is designed to be scalable and extensible.
We initially evaluate its effectiveness on small dataset ShapeNet [] as previous settings [].
Next, we carefully curate and train our model on the existing largest 3D datasets Objavese [] and Objaverse-xl [].
By leveraging large datasets, our model can generate generic meshes across various categories to accelerate the mesh creation process.
Both the qualitative and quantitative experiments show that the proposed PivotMesh beats previous mesh generation methods like PolyGen [] and MeshGPT [] by a large margin.

The contributions of this paper can be summarized as follows:

- •
We propose a generic and scalable mesh generation framework PivotMesh, which makes an initial attempt to extend the native mesh generation to large-scale datasets.
- •
We present a Transformer-based auto-encoder to preserve the geometry details and surface continuity in meshes by efficiently decoding from face level to vertex level hierarchically.
- •
We introduce pivot vertices guidance for complex mesh geometry modeling, which serves as the coarse representation to guide the complete mesh generation in a coarse-to-fine manner.
- •
PivotMesh achieves promising performance in various applications like mesh generation, variation, and refinement, accelerating the mesh creation process.

## 2 Related Works

#### Neural 3D Shape Generation.

Most previous attempts learn 3D shape with various representations, e.g., SDF grids [, , , ] and neural fields [, , , , , , ].
To improve the generalization ability, researchers start to leverage pretrained 2D diffusion models [, , ] with score distillation loss [, , ] in a per-shape optimization manner.
Multi-view diffusion models [, , , , , ] are used to further enhance the quality and alleviate the Janus problem.
Recently, Large Reconstruction Models (LRM) [, , , , , , ] train the Transformer backbone on large scale dataset [] to effectively generates generic neural 3D representation and shows the great performance of scaling.
However, these neural 3D shape generation methods require post conversion [, ] for downstream applications, which is non-trivial and easy to produce dense and over-smooth meshes.

#### Native Mesh Generation.

Compared with the well-developed generative models of neural shape representations, the generation of the mesh remains under-explored.
Some pioneering works try to tackle this problem by formulating the mesh representation as surface patches [], deformed ellipsoid [], mesh graph [] and binary space partitioning [].
PolyGen [] uses two separated auto-regressive Transformers to learn vertex and face distribution respectively.
Polydiff [] learns the triangle soups of mesh with a diffusion model.
MeshGPT [] is most relevant to our work, which first tokenizes the mesh representation with a GNN-based encoder and learns the mesh tokens with a GPT-style Transformer.
Despite its promising results on small datasets, the capability of MeshGPT is limited due to the design of its tokenizer and sequence formulation.
Furthermore, the performance of these works is constrained by the 3D data scale and model generalizability, constraining the downstream applications of mesh generation.
Unlike the above works, PivotMesh makes an initial attempt to build a generic generative model for native mesh generation within large-scale datasets.

## 3 Method

In this section, we will introduce the details of the proposed PivotMesh as shown in Figure 2. The challenges to scale up native mesh generation are analyzed in Section 3.1.
Meshes formulated as triangle face sequences are first encoded into discrete tokens by the proposed mesh auto-encoder (Section 3.2). Then, we use an auto-regressive Transformer to learn the joint distribution of pivot vertices and mesh tokens (Section 3.3).

### 3.1 Challenges for Native Mesh Generation on Large Datasets

There are two main challenges for scaling up native mesh generation to large datasets.

Mesh Reconstruction.
It is challenging to tokenize meshes into tokens due to the high requirement on reconstruction performance to preserve the mesh surface continuity.
Previous works like MeshGPT [] formulate meshes as face graphs for reconstruction, which only focuses on face-level relationships and neglects the connection and interaction among vertices.
Furthermore, the limited network capability of auto-encoder (i.e., GNN and CNN) also hinders its scalability on large-scale datasets.
To this end, we propose a Transformer-based auto-encoder to preserve the geometry details and surface continuity by decoding from face level to vertex level hierarchically.

Complex Typology Modeling.
Due to the complexity and variety of mesh topology, directly modeling the mesh sequence on large-scale datasets makes it easy to produce trivial meshes with simple geometry like cubes.
It remains challenging to model complex typology in mesh structure.
For this purpose, we propose to first model a coarse representation of meshes (i.e., pivot vertices), then generate the full meshes in a coarse-to-fine manner.

### 3.2 Encode Meshes into Discrete Tokens

A triangle mesh $\mathcal{M}$ with $n$ faces can be formulated as the following sequence:

$$ $\mathcal{M}:=(f_{1},f_{2},...,f_{n})=(v_{11},v_{12},v_{13},v_{21},v_{22},v_{23 },...,v_{n1},v_{n2},v_{n3}),$ (1) $$

where each face $f_{i}$ consists of 3 vertices and each vertex $v_{i}$ contains 3D coordinates discretized with a 7-bit uniform quantization.
To effectively learn the mesh distribution, we first tokenize the sequence into discrete tokens with the proposed transformer-based Auto-encoder.

#### Attention-based Tokenizer.

We use a Transformer-based architecture as the backbone for the encoder, capturing the long-range relationship between faces.
Furthermore, we replace the vanilla positional encoding in the Transformer with a single-layer GNN to capture the local topology of meshes.
This preserves the permutation invariance of faces with higher scalability, yielding more effective and robust token representation for meshes.
Specifically, the features of each face $f_{i}$ (e.g., vertices embedding, normal embedding) are concatenated as one face embedding $F_{i}$ and sent to the Transformer-based face encoder:

$$ $F_{i}=\text{FaceEnc}(\text{FaceEmbed}(f_{i})),$ (2) $$

In the vector quantization module, each face embedding $F_{i}$ is converted back to vertices embedding $V_{i1:i3}$, aggregated by shared vertex indices, and quantized with residual vector quantization [].
Each vertex embedding is quantized with $r$ codes, thus the total length of the token sequence $T$ is $|T|=3nr$.
Then, the quantized vertex tokens are reorganized to face embedding $F^{\prime}_{i}$ after the quantization module.
Similar to [], we empirically found that such vertex quantization is much easier for generative modeling due to the repeated appearance of vertex tokens.

#### Hierarchical Decoding.

To further improve the reconstruction performance and mesh surface contiguity, we design a hierarchical decoder from face level to vertex level.
The face embedding $F^{\prime}_{i}$ from the vector quantization module is first passed to a face-level decoder. Then, the decoded face embedding is converted to vertex embedding $V^{\prime}_{i}$ by a simple multi-layer perception (MLP).
The vertex embedding is then decoded by a vertex-level decoder, whose architecture is similar to the face decoder except that its input sequence is 3 times longer.

$$ $\displaystyle F^{\prime}_{i}$ $\displaystyle=\text{FaceDec}(F^{\prime}_{i}),$ (3) $\displaystyle V^{\prime}_{i}$ $\displaystyle=\text{VertexDec}(\text{MLP}_{n\rightarrow 3n}(F^{\prime}_{i})),$ $$

The final decoded vertex embedding $V_{i}$ is then converted to the quantized 3D coordinate logits $\in(1,2,...,2^{7})$ for each axis ($x$, $y$ and $z$) and computes the cross entropy with the input mesh sequence.
Such hierarchical architecture allows the connection and interaction among both face and vertex level, thus improving the reconstruction accuracy and surface continuity.

Figure: Figure 2: The overall method of PivotMesh. (a) Triangle mesh sequences are tokenized into mesh tokens and hierarchically decoded from face level to vertex level via our mesh auto-encoder. (b) The auto-regressive Transformer first learns to generate pivot vertices as coarse mesh representation and then generates the complete mesh tokens in a coarse-to-fine manner.
Refer to caption: x2.png

### 3.3 Guide Mesh Generation with Pivot Vertices

#### Pivot Vertices Selection.

The way pivot vertices are selected greatly affects the performance of the following mesh generation.
For this purpose, we design a degree-based selection strategy for informative pivot vertices. Specifically, a mesh can be regarded as a graph, where each vertex $v_{i}$ represents a node and the connection between vertices represents the edges. Then, we compute the vertex degree $deg(v_{i})$ in mesh graphs and select the pivot vertices set $P$ with the top-degree vertices.
The size of the pivot vertices set is proportional to the number of vertices with a fixed ratio $\eta_{select}$.
Furthermore, to prevent overfitting in pivot-to-mesh modeling, we randomly drop some pivot vertices with the ratio $\eta_{drop}$ of all vertices for each training iteration.
In our experiments, the select ratio $\eta_{select}=15\%$ and the dropping ratio $\eta_{drop}=5\%$, yielding the final pivot vertex ratio $\eta=10\%$.
The benefits of our pivot selecting strategy fall into two aspects.
First, it leverages frequently occurring vertices (with higher degree), enabling the Transformer to utilize these as conditional tokens mesh sequence generation efficiently.
Second, it tends to preserve intricate mesh details, as regions with finer geometry typically necessitate more faces and thus larger vertex degrees.

#### Coarse-to-fine Mesh Modeling.

We employ a standard auto-regressive Transformer decoder to learn the joint distribution of the pivot vertex tokens $p_{i}\in P$ and the complete mesh tokens $t_{i}\in T$.
A learnable start and end token are used to identify the beginning and end of the token sequence, while a pad token is used to separate the pivot vertex tokens and mesh tokens.
The order of both pivot vertices tokens and full mesh tokens is sorted by $z$-$y$-$x$ coordinates from lowest to highest.
Different from the Transformer in Section 3.2, we add absolute positional encoding here to indicate the position in the token sequence.
The token sequences are modeled with a Transformer with parameter $\theta$ by maximizing the log probability:

$$ $\prod_{i=1}^{|T|}p(t_{i}|t_{1:i-1},P;\theta)\prod_{j=1}^{|P|}p(p_{j}|p_{j:j-1} ;\theta),$ (4) $$

With such formulation, the auto-regressive Transformer first learns to generate pivot vertex tokens $P$ as coarse mesh representation and then generates the complete mesh tokens $T$ in a coarse-to-fine manner.
In such a coarse-to-fine manner, our model can effectively learn the complex mesh typology, which can be easily extended to large-scale datasets.
Once the model is trained, it can produce meshes from scratch, starting with the generation of pivot vertices followed by the full mesh sequence.
Furthermore, different from previous mesh generation methods [, ], our model can perform conditional generation given the pivot vertices from the reference mesh, and also supports downstream applications like mesh variation and refinement.

## 4 Experiment

### 4.1 Experiment Settings

#### Datasets.

Our model is trained on various classes and scales datasets, including ShapeNetV2 [], Objaverse [], Objaverse-xl [].
For ShapeNet, follow previous settings [, ], we use 4 subsets (chair, table, bench, lamp) and filter the faces less than 800 after plannar decimation (with a fixed angle tolerance $\alpha=10^{\circ}$).
For Objaverse and Objaverse-xl, we apply the data curation and filter the objects whose faces are less than 500 without decimation to preserve the mesh quality.
The final dataset size after filtering of Shapenet, Objaverse, and Objaverse-xl is around 10k, 40k, and 400k respectively.
For each dataset, we split 1k samples for testing, and leave the rest as the training data.
For data augmentation, we use random scaling on each axis (from 0.95 to 1.05) and random shifts ($\pm 0.01$) to enhance the data diversity.

#### Baselines.

We benchmark our approach against leading mesh generation methods: PolyGen [], which generates polygonal meshes by first generating vertices followed by faces conditioned on the vertices with two separate Transformer; MeshGPT [], which tokenizes the mesh sequence into mesh tokens with a GNN-ResNet based Auto-encoder and learns the mesh tokens directly with an Auto-regressive Transformer.

#### Metrics.

Following the evaluation settings of previous mesh generation methods [, ],
we evaluate the reconstruction quality by two metrics, triangle accuracy and l2 distance.
We use the following metrics for mesh quality assessment: Minimum Matching Distance (MMD), Coverage (COV), and 1-Nearest-Neighbor Accuracy (1-NNA).
For MMD, lower is better; for COV, higher is better; for 1-NNA, 50% is the optimal.
We use a Chamfer Distance (CD) distance measure for computing these metrics on 1024-dim point clouds uniformly sampled from meshes.

#### Implementation Details.

For the auto-encoder, the face encoder has 12 layers with a hidden size of 512, the face decoder has 6 layers with a hidden size of 512, and the vertex decoder has 6 layers with a hidden size of 256.
For vector quantization, the number of residual quantizers $r=2$, and the codebook is dynamically updated by exponential moving averaging with codebook size 16384 and codebook dimension 256.
It is trained on an 8×A100-80GB machine for around 1 day with a batch size of 64 for each GPU.
For auto-regressive Transformer, it has 24 layers with a hidden size of 1024.
It is trained on an 8×A100-80GB machine for around 3 days with batch size 12 for each GPU.
The temperature used for sampling is set to 0.5 to balance the quality and diversity.
We use flash attention for all Transformer architecture and fp16 mixed precision to speed up the training process.
We use AdamW [] as the optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.99$ with a learning rate of $10^{-4}$ for all the experiments.

Figure: Figure 3: Qualitative comparison of unconditional generation on ShapeNet. Each line represents a subset of ShapeNet (bench, chair, lamp, table).
Refer to caption: x3.png

**Table 1: The unconditional generation results on Shapenet dataset. The proposed PivotMesh significantly improves the mesh quality compared with baselines by a large margin.**
| Model | PolyGen | MeshGPT | PivotMesh |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Subset | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ |
| Chair | 29.47 | 13.34 | 81.45 | 41.31 | 10.30 | 61.84 | 52.89 | 9.77 | 56.71 |
| Table | 38.67 | 15.84 | 66.27 | 43.00 | 9.77 | 62.83 | 51.68 | 9.28 | 56.55 |
| Bench | 37.50 | 10.8 | 79.69 | 46.87 | 9.48 | 67.19 | 51.56 | 8.50 | 53.91 |
| Lamp | 31.76 | 33.87 | 81.76 | 45.88 | 23.43 | 57.06 | 50.58 | 22.65 | 54.71 |
| Mixed | 33.91 | 13.47 | 74.63 | 43.89 | 11.48 | 63.24 | 50.42 | 11.03 | 60.89 |

### 4.2 Mesh Generation From Scratch

#### Comparison with baselines on various-scale benchmarks.

We first evaluate the proposed PivotMesh on the commonly used benchmark Shapenet with four selected categories, chair, table, bench, and lamp. Following the previous setting [, ], we first pretrain our model on the mixture dataset of four selected categories and then finetune each category separately.
We report the generation results both on the mixed dataset and each subset in Table 1.
Furthermore, we train our model on the larger scale datasets Objaverse and Objaverse-xl and report the performance in Table 2.
For all these experiments, our method can achieve state-of-the-art performance on all evaluation metrics.
As shown in Figure 3 and Figure 4, our model can generate meshes with the best visual quality and geometry complexity.
PolyGen often produces incomplete meshes due to the accumulation error by the separate training of the vertex model and face model.
MeshGPT can produce complete meshes but it is trapped in simple geometry due to its network capability and the complex mesh sequence.
With the hierarchical auto-encoder and pivot vertices guidance, our model can produce compact meshes with sharp details and complex geometry.

Figure: Figure 4: Qualitative comparison of unconditional generation on Objaverse.
Refer to caption: x4.png

**Table 2: The unconditional generation results on the Objaverse and Objaverse-xl dataset.**
| Model | Objaverse | Objaverse-xl |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ | COV(%)$\uparrow$ | MMD($10^{-3}$)$\downarrow$ | 1-NNA(%)$\downarrow$ |
| PolyGen | 23.86 | 24.01 | 84.07 | 21.79 | 22.68 | 83.40 |
| MeshGPT | 35.03 | 17.30 | 63.86 | 41.50 | 14.76 | 64.25 |
| PivotMesh | 46.48 | 16.66 | 58.55 | 45.30 | 14.33 | 57.75 |

#### Shape Novelty Analysis.

To show that our model can create novel shapes instead of memorizing the training set, we conduct a shape novelty analysis similar to [, , ].
We generate 500 shapes and search the 3 closest neighbors from the training set measured in Chamfer Distance (CD), shown in Figure 5 (left). We also show top-1 CD distribution between the generated shapes and the training set in Figure 5 (right).
The CD results show that our method not only covers shapes in the training set (low CD values) but also creates novel and realistic shapes (high CD values).

Figure: Figure 5: Shape novelty analysis on Objaverse dataset. We show the 3 nearest neighbors measured in Chamfer Distance (CD) for generated shapes (left). We plot the distribution of 500 generated shapes from our method and their minimum CD to the training set (right). Shapes at the 50th percentile look different from the closest train shape. It shows that our method not only covers shapes in the training set (low CD values) but also creates novel and realistic shapes (high CD values).
Refer to caption: x5.png

**Table 3: Ablation study of auto-encoder and auto-regressive Transformer on Objaverse.**
| Method | Triangle Accuracy(%)$\uparrow$ | L2 Distance$\downarrow$ |
| --- | --- | --- |
| w/o Transformer | 86.89 | 10.34 |
| w/o hier. decode | 92.30 | 5.37 |
| Ours | 97.89 | 1.05 |

### 4.3 Ablation Study

#### The effectiveness of Transformer-based hierarchical auto-encoder.

For mesh auto-encoder, previous works like MeshGPT [] employ GNN as the encoder and CNN as the decoder.
Table 3(b) (w/o Transformer) shows the reconstruction performance of such architecture, which is significantly lower than our Transformer-based auto-encoder.
The performance gain is mainly from the high capability and scalability of the Transformer compared with the GNNs and CNNs.
Table 3(b) (w/o hier. decode) shows the performance of the auto-encoder when removing the hierarchical decoding mechanism.
It shows that such a hierarchical network design endows the decoder with better alignment in both face and vertex levels, thus improving the final reconstruction results.

#### The effectiveness of pivot vertices guidance and their selection.

Table 3(b) (w/o pivot guidance) shows the results of directly employing an auto-regressive Transformer to model the mesh tokens from our auto-encoder.
Without the pivot vertex guidance, the model fails to produce meshes with complex geometry.
In Table 3(b) (w/o pivot selection), we employ random pivot vertex selection instead of the proposed degree-based strategy.
Some metrics from its results are even worse than that without using pivot vertices guidance.
The strong performance degradation is because the randomly selected vertices make it hard to summarize the whole geometry of the meshes and the Transformer is incapable of learning such pivot-mesh joint distribution, showing the importance of pivot vertices selection strategy.

Figure: Figure 6: Conditional generation results with pivot vertices guidance from Objaverse test set. The generated meshes are marked in blue and the ground truth meshes are marked in gray. The diversity of generation results is also shown on the bottom line.
Refer to caption: x6.png

### 4.4 Pivot Vertices Guidance Analysis

To trade-off between the generalization and the visual quality for pivot-guided mesh generation, we first pretrain our model in the mixture of objaverse and objaverse-xl datasets. Then, we finetune the pretrained model on the well-curated objaverse to further improve the generated mesh quality.

#### Pivot-guided Mesh Generation.

Given a reference mesh, we first encode it into mesh tokens and then select the pivot vertices.
Our model can generate the corresponding meshes as shown in Figure 6.
It shows that our model can generate diverse and high-quality meshes while maintaining the high-level structure corresponding to the pivot vertices.

#### Downstream Applications.

PivotMesh can serve as a generic mesh generative model to support various applications in Figure 7.
For mesh variation, our model can generate diverse variants with the user-given meshes.
For mesh refinement, it can be regarded as a special case of mesh variation, but with coarse meshes as input. Our model can refine the coarse meshes to fine meshes with detailed geometry to accelerate the mesh creation process.

Figure: (a) Mesh Variation
Refer to caption: x7.png

Figure: Figure 8: Some failure cases for the pivot-guided mesh generation. Since the same pivot vertices may indicate diverse meshes, our method may sometimes produce undesired geometry. This can be alleviated by adding more control conditions like images and texts.
Refer to caption: x9.png

### 4.5 Limitations and Future Work

Although PivotMesh can produce compact meshes with high quality, it still has some limitations.
First, the controlling ability of PivotMesh is still not enough.
As shown in Figure 8, since the same pivot vertices may indicate diverse meshes, our method may sometimes produce undesired geometry. This can be alleviated by adding more control conditions like images and texts and we leave it for our future work.
Second, the scale of data and the number of model parameters is still limited.
Due to the constrained computation resources, we only train PivotMesh on meshes less than 500 for both Objaverse and Objaverse-xl, and the number of model parameters is much less than recent advances in large language models []. Given the increased data and model scale, the performance and capabilities of PivotMesh could be further boosted.

## 5 Conclusion

In this paper, we introduce PivotMesh, a scalable framework to generate generic meshes with compact and sharp geometry. By employing pivot vertices as a coarse representation to guide the mesh generation process and leveraging a Transformer-based hierarchical auto-encoder, PivotMesh demonstrates its capability to generate high-quality meshes on both small and large-scale datasets.
The proposed model significantly outperforms existing methods and sets a new benchmark for native mesh generation tasks, showcasing its potential for creating novel shapes and supporting downstream applications.

## Appendix A Additional Results

### A.1 Comparison with InstantMesh

To produce similar shapes for a more straightforward visual comparison, we use the rendering images of our generated mesh as the image condition for InstantMesh to generate the following instances in Figure 1 and Figure 9.

Figure: Figure 9: Additional results for comparison with InstantMesh.
Refer to caption: x10.png

### A.2 Unconditional Generation on ShapeNet

Figure: Figure 10: Additional results for unconditional generation on Shapenet.
Refer to caption: x11.png

### A.3 Unconditional Generation on Objaverse

Figure: Figure 11: Additional results for unconditional generation on Objaverse.
Refer to caption: x12.png

### A.4 Pivot-guided Mesh Generation

Figure: Figure 12: Additional results for pivot-guided mesh generation on Objaverse.
Refer to caption: x13.png

### A.5 Shape Novelty Analysis

Figure: Figure 13: Additional results for shape novelty analysis on Objaverse.
Refer to caption: x14.png

### A.6 PivotMesh Applications

Figure: Figure 14: Additional results for the applications of PivotMesh.
Refer to caption: x15.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: The paper is well-written and presents a contribution to the field of 3D mesh generation. I am impressed with the quality of the generated results, but there are some minor issues: the comparison with other similar baselines and more diverse generated shapes. With the above suggestions addressed, the paper would be a strong candidate for publication. I recommend accepting this paper with minor revisions.
Detailed questions please refer to weaknesses.
soundness: 3
strengths: 1. PivotMesh addresses a significant challenge in 3D mesh generation by proposing a scalable framework that can handle large-scale datasets with simplified triangles. The use of pivot vertices as a coarse representation for guiding mesh generation is innovative and effectively handles complex topologies.
2. The paper provides a thorough evaluation of PivotMesh across various datasets and applications, including mesh generation, variation, and refinement. The generated meshes are of high quality, with sharp details and complex geometries, outperforming existing methods.
summary: The paper introduces PivotMesh, a framework for generating compact and detailed 3D meshes at scale. The approach leverages a transformer-based auto-encoder to encode meshes into discrete tokens, which are then decoded hierarchically from face to vertex level. A key contribution is the use of pivot vertices as a coarse representation to guide the generation of complete mesh tokens, allowing for complex topology modeling. The method is evaluated on diverse datasets, including ShapeNet, Objaverse, and Objaverse-xl, demonstrating its versatility and effectiveness in generating high-quality 3D meshes.
weaknesses: 1. Diversity of Generated Meshes: While the paper shows diverse mesh generation, a more systematic analysis or quantification of diversity could strengthen the results. The evaluations on more diverse shapes with complex topologies are encouraged to be conducted, such as some shapes with lots of holes, the thin structures (ficus in nerf-synthetic data), and so on.
2. More direct comparisons with the current state-of-the-art methods, especially in terms of computational efficiency and mesh quality, could be beneficial. There is more relative work, such as meshanything(2), EdgeRunner, the comparison is very essential to demonstrate the superiority of the proposed method.
3. The limitations and failure cases should be discussed comprehensively. And the paper acknowledges that the controlling ability of PivotMesh is not sufficient, and sometimes undesired geometries are produced. Enhancing the control mechanisms could be a valuable addition.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - Including MeshXL's quantitative results would be highly beneficial, despite its use of a different training dataset. While this is concurrent work and the results would be for reference only, they could serve as a valuable benchmark, helping readers better understand the effectiveness of the proposed method.
- The paper claims that the proposed pivot mesh formulation can generate shorter sequences, providing estimated sequence lengths in Table 1. However, measuring these lengths on a real dataset would offer more clarity and insight into the method's efficiency.

Justification of Rating:
Overall, this work provides a novel direction for compact mesh generation and demonstrates its effectiveness in various aspects. The quantitative and qualitative comparisons show that it achieves good performance compared to existing and concurrent works. Additionally, various applications have outlined its potential beyond unconditional generative tasks. Despite some minor suggestions, I am currently leaning towards acceptance of this work.
soundness: 4
strengths: - The exploration of generating pivot vertices before faces appears to be a novel approach. Ablation studies validate its effectiveness, particularly without pivot guidance and degrees selection.
- Extensive experiments compare unconditional generation quality across both small and large datasets, such as ShapeNet and ObjaverseXL. The proposed framework outperforms existing methods in all settings.
- The framework demonstrates versatility through various applications, including point cloud-guided generation, mesh variations, and coarse mesh refinement.
- A novel analysis ensures the diversity and originality of the generated objects.
summary: In this work, the authors propose a novel framework for explicit mesh generation that can be trained on large-scale datasets. They employ a transformer-based autoencoder to obtain discretized tokens representing the mesh, which are later decoded hierarchically. Additionally, the generative networks are designed to first produce a sequence of pivot mesh vertices as a coarse representation, then use these as conditions to generate the actual mesh tokens. Experimental results demonstrate that the proposed framework achieves comparable and often superior performance to concurrent works. The authors also showcase various downstream applications of their method.
weaknesses: Although concurrent works are mentioned, several studies have begun exploring the generation of compact meshes in large datasets, demonstrating some success to a certain extent. It is recommended to revise the introduction and abstract to avoid claiming that existing work is difficult to extend to large datasets.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - It is confusing how the discrete mesh tokens from auto-encoder and pivot vertices are combined for conditional mesh generation.  
- In lines 303-306 of page 6, the authors mentioned about face filtering in the datasets. Is there a distribution map to summarize the number of vertices and faces in the training and test meshes?
soundness: 3
strengths: - A transformer-based auto-encoder is proposed to encode the triangle meshes into discrete tokens.
- An auto-regressive transformer is proposed to generate the complete mesh tokens from pivot vertices.
- By conditioning on pivot vertices, the proposed method outperforms PolyGen and MeshGPT. 
- Downstream applications of the method are shown for mesh variation and coarse mesh refinement.
summary: This paper focuses on conditional triangle mesh generation based on the pivot vertices of the provided mesh.  The authors proposed to tokenize the mesh with transformer, rather than the GCN architecture used by MeshGPT. The pivot-guided method is shown to outperform PolyGen and MeshGPT.
weaknesses: - Both the auto-encoder and auto-regressive transformer use 24 layers of transformers, resulting the method to be memory taken and less efficient.

Meta-review: This paper introduces a novel generative model for 3D meshes designed to train on a large-scale dataset. To effectively train on such a dataset, the authors first generate a sequence of pivot meshes, a coarser representation of the mesh, and then use these as a condition to generate the actual fine mesh. The authors also employ a transformer-based hierarchical autoencoder to represent a mesh with a sequence of discretized tokens. The experimental results demonstrate the superior performance of the proposed method compared to concurrent work and the effectiveness of generating pivot vertices as an intermediate representation.

All reviewers agreed that the idea of using pivot vertices is novel and innovative, and its effectiveness has been demonstrated through the experimental results. The versatility of the proposed method in point-cloud-guided generation, mesh variations, and coarse mesh refinement was also appreciated. The discussion after the rebuttal quickly converged to acceptance.

Decision: Accept (Poster)