Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale

Authors: Zekun Hao, David W. Romero, Tsung-Yi Lin, Ming-Yu Liu
Primary area: generative models
Keywords: Mesh generation, 3D Generation, Hourglass Transformer, Autoregressive mesh generation

Abstract:
Meshes are a fundamental representation of 3D surfaces. However, creating high-quality meshes is a labor-intensive task that requires significant time and expertise in 3D modelling. While a delicate object often requires over $10^4$ faces to be accurately modeled, recent attempts at generating artist-like meshes are limited to $1.6$K faces and heavy discretization of vertex coordinates. Hence, scaling both the maximum face count and vertex coordinate resolution is crucial to producing high-quality meshes of realistic, complex 3D objects. We present Meshtron, a novel autoregressive mesh generation model able to generate meshes with up to 64K faces at 1024-level coordinate resolution --over an order of magnitude higher face count and $8{\times}$ higher coordinate resolution than current state-of-the-art methods. Meshtron's scalability is driven by four key components: 
(i) an hourglass neural architecture, 
(ii) truncated sequence training, 
(iii) sliding window inference, 
and (iv) a robust sampling strategy that enforces the order of mesh sequences.
This results in over $50\%$ less training memory, $2.5{\times}$ faster throughput, and better consistency than existing works. Meshtron generates meshes of detailed, complex 3D objects at unprecedented levels of resolution and fidelity, closely resembling those created by professional artists, and opening the door to more realistic generation of detailed 3D assets for animation, gaming, and virtual environments.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730021004225

thread_type:
official_review

<thread_object>
creator: reviewer_qyfu

<content>
summary:
This paper introduces a transformer-based approach to mesh generation. Given an input point cloud, the hourglass architecture (MESHTRON), like an Unet or an AE, compresses and reconstructs the mesh as output. The intermediate states (or latent) have an explicit meaning, corresponding to vertex and face representation. During training the model is trained by exploiting the adjacency of the triangles combined with a sliding window approach. Such a choice reduces memory consumption and allows the model to scale to complex and longer mesh sequences.

strengths:
The proposed approach shows potential. The intermediate latent representation makes use of the geometric construction of meshes. The training procedure allows the architecture to scale to longer and more complex meshes.

weaknesses:
Although the paper is very promising, several details are unclear. I would appreciate it if the authors clarify them:
1. several details are obscured in the manuscript, or not appropriately presented:
   - the meaning of the intermediate representation is not well presented. It can be inferred from Figure 4, but I the authors should to discuss it in the text for the sake of the reader - please expand Section 3.1;
   - it is not clear how the sliding window is used during inference. For example, how is the sliding window decided? And given that the sequence is truncated during training, how is a larger mesh generated? Please, provide more details on the sliding window inference process in Section 3.2.
2. the authors claim the length of the mesh sequence can be 64K in faces, L158-159. However, I cannot find such examples (even in the supplementary material). Is this a theoretical limit or do authors have examples and did not include them in the material?
3. the authors point out "MESHTRON does not work from time to time", would you please elaborate? This is not an appropriate sentence for a scientific paper.
4. how is the global conditioning obtained? Is the input point cloud feed to the model and then the sequence used for cross attention?
5. would it be possible to condition from an image latent representation (say encoded by DinoViT) and project it onto the latent space of MESHTRON? (it is fine if it is future work, but worth mentioning in the paper)
6. a comparison with MeshGPT would be appreciated even if qualitative. As the code is not available, the authors could maybe reach out to MeshGPT's authors to get some examples.

questions:
Please see the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730412346710

thread_type:
official_review

<thread_object>
creator: reviewer_tbnl

<content>
summary:
This paper presents a method for efficiently generating meshes from point clouds. The authors propose an hourglass transformer network architecture and a truncated sequence training scheme, which enable the generation of meshes with a large number of faces and diverse types. Compared to existing methods, this approach requires less training memory and achieves faster throughput.

strengths:
The proposed method addresses an important challenge in 3D mesh generation: the number of faces significantly affects the quality and fidelity of the results, making a higher face count crucial. Experimental results indicate that this method can produce highly detailed meshes. 

Additionally, the authors present the motivation for their algorithm design, based on observed issues in real-world data, offering valuable insights for the academic community.

weaknesses:
The author mentions that the tokens of the last vertex have high perplexity, which led to the design of the Hourglass Transformer to address this issue. However, the causal relationship here does not seem entirely clear. Within the architecture of the Hourglass Transformer, there is no obvious special treatment for the last vertex tokens. While the Hourglass Transformer is an effective structure, linking it directly to the perplexity of the last vertex tokens feels somewhat tenuous.


In the experiments, the authors used high-quality point clouds as input. However, in practical applications, point clouds often contain noise. This paper does not discuss or experiment on this issue, which inevitably raises concerns about its practicality.  

The author only compares their method with one other approach and does not include a related work section. This makes it difficult for those not working in mesh generation to follow the advancements in the field and compare the proposed method. For instance, how does it differ from existing mesh generation methods such as CLAY[1]?

[1] Zhang L, Wang Z, Zhang Q, et al. CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets[J]. ACM Transactions on Graphics (TOG), 2024, 43(4): 1-20.

questions:
I believe that the problem this paper aims to address (generating detailed meshes) is meaningful. However, there are many areas for improvement in the experimental design and writing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677946503

thread_type:
official_review

<thread_object>
creator: reviewer_rux9

<content>
summary:
The author presents a methodology for generating high-quality mesh generation from point clouds. The author proposes an autoregressive model that has been well evaluated and chosen regarding the explicit need for mesh generation. The authors claim well-analysed contributions in the chosen neural architecture and the training methodology and reconstruct the mesh by limiting the scope of possible solutions to enforce mesh sequences during training and inference.

strengths:
The work is well ablated, and the choice in architectural design taken by the authors is reasonable and easy to understand. Overall, the work is well written and well structured, making it easy to understand why and what was chosen and how it was implemented. The appendix is additionally very useful to reimplement this work. Overall, the work seems to have well-reasoned analysis for training mesh generations from autoregressive models and additionally adds limitations to the mesh generation process that seemingly seem to be very impactful in this process.

weaknesses:
I am not an expert in this research area, but for me, the main weakness is the claim that previous work "recent attempts at generating artist-like meshes are limited to 1.6K faces and heavy discretization of vertex coordinates". I want to refer the authors to works such as [1, 2] that show mesh reconstruction with more that 1.6K faces.
Both of the mentioned works [1, 2] show high-quality reconstructions and are not compared by the authors.

[1] Shen, Tianchang, et al. "Flexible Isosurface Extraction for Gradient-Based Mesh Optimization." ACM Trans. Graph. 42.4 (2023): 37-1.
[2]Shen, Tianchang, et al. "Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis." Advances in Neural Information Processing Systems 34 (2021): 6087-6101.

questions:
- How much does the number of points in the point cloud impact the final mesh generation process
- How does the model handle partial point clouds?  
- Above all, I would like the point regarding the comparision with [1, 2] to be addressed and would change my rating if it was a misunderstanding on my side.


[1] Shen, Tianchang, et al. "Flexible Isosurface Extraction for Gradient-Based Mesh Optimization." ACM Trans. Graph. 42.4 (2023): 37-1.
[2]Shen, Tianchang, et al. "Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis." Advances in Neural Information Processing Systems 34 (2021): 6087-6101.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731801010666

thread_type:
official_review

<thread_object>
creator: reviewer_hx86

<content>
summary:
The authors propose an autoregressive approach for generating high-quality meshes from input point clouds. Compared to existing methods, this approach achieves over an order of magnitude increase in face count. The introduction of the Hourglass Transformer and truncated sequence training effectively reduces training memory requirements and enhances computational efficiency.

strengths:
The paper addresses the challenge of 3D mesh generation by achieving high-quality outputs with up to 64K faces at 1024-level coordinate resolution, representing a significant advancement over existing methods. The intuition and motivation behind the approach are clearly articulated, supported by a thoughtful discussion of real-world data. The authors further validate their method by conditioning on a variety of mesh types, including artist-created meshes and text-to-3D generated meshes, demonstrating its versatility.

weaknesses:
First, I find it unclear why the Hourglass Transformer is considered the appropriate solution to address the challenge of generating the latter tokens of a triangle. While the authors explain both the difficulty and the Hourglass Transformer reasonably well, the connection between the two could be more explicitly justified. Second, the model’s reliance on curated, high-quality datasets raises concerns about scalability and applicability to domains with limited or noisy data. In real-world scenarios, point clouds are often noisy, yet the paper does not evaluate the robustness of the proposed approach under such conditions. Finally, although the authors claim significant advantages over state-of-the-art methods, they only compare their approach to one such method. Broader validation and comparisons with a wider range of methods are necessary to substantiate their claims.

questions:
While the 64K-face mesh generation represents a notable improvement over existing methods, I question whether this represents the theoretical limit of the proposed approach. In real-world scenarios involving complex scenes, mesh face counts often exceed several million, raising concerns about the scalability of the method to handle such high levels of complexity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an autoregressive method to generate meshes from point clouds. The method is claimed to produce meshed of up to 64K faces at higher resolution than prior work and resembling meshes created by artists in terms of detail and quality. Technical contributions are an hourglass network architecture, a training protocol, a sliding-window based inference procedure, and a robust sampling strategy.

The reviewers found the paper well motivated, the technical contributions well described, and the architecture design choices well explained.
The paper provides quantitative results comparing architecture variants and for training on truncated sequences and qualitative comparisons to prior work.

Concerns and requests raised during the discussion period were sufficiently addressed to a large extent. The main weakness of the paper however, is the limited comparison to prior work. Initially, a related work section was missing completely, but was added to the appendix in the revised version after requests were made by the reviewers. The requests for additional experimental validation and comparisons to prior work was only met through qualitative results. Although the qualitative results indeed show improvements over prior work, a more thorough, quantitative comparison on a larger set of samples would be necessary to sufficiently validate the proposed method (Even the revised version does not contain a single quantitative comparison against prior work.)