Title: ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Text-to-3D Generation

LLM Agents

3D Modeling
Abstract: 3D generation from natural language offers significant potential to reduce expert manual modeling efforts and enhance accessibility to 3D assets. However, existing methods often yield unstructured meshes and exhibit poor interactivity, making them impractical for artistic workflows. To address these limitations, we represent 3D assets as shape programs and introduce ShapeCraft, a novel multi-agent framework for text-to-3D generation. At its core, we propose a Graph-based Procedural Shape (GPS) representation that decomposes complex natural language into a structured graph of sub-tasks, thereby facilitating accurate LLM comprehension and interpretation of spatial relationships and semantic shape details. Specifically, LLM agents hierarchically parse user input to initialize GPS, then iteratively refine procedural modeling and painting to produce structured, textured, and interactive 3D assets. Qualitative and quantitative experiments demonstrate ShapeCraft's superior performance in generating geometrically accurate and semantically rich 3D assets compared to existing LLM-based agents. We further show the versatility of ShapeCraft through examples of animated and user-customized editing, highlighting its potential for broader interactive applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method of using LLM to generate 3D shape and texture. The main contribution is proposing a hierarchical structure of the target 3D shape, Graph-based procedural shape representation (GPS) to improve the quality of the generated 3D shapes. Meanwhile, the use of procedural modeling is helpful for generating better results compared with previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The proposed method generate nice shape and the use of hierarchical structure is sound.\n- The proposed method can also generate texture beside the geometry.\n\nWeakness:\n- Meanwhile, I think the paper tends to propose too many content at a time, and therefore fail to describe the method thoroughly in the main paper. Although some details are provided in the supplemental material, it is still makes the paper hard to follow and understand.\n- I personally would argue that the paper should focus on the shape generation solely.\n- The generated texture quality is not good and not better than previous methods shown in Figure 3. The texture is quite rough and do not match to the generated shape and prompt accurately. This again makes me think this paper should focus on the shape generation.\n\nQUESTIONS:\nFollowings are some suggestions to improve the expositions of the paper:\n- I would recommend to mention the underlying program is based on Blender API. For example, when the GPS representation is introduced.\n- The description of the \"Hierarchical shape parsing via CoT reasoning\" and \"Representation bootstrapping.\" are both very unclear and confusing. I think those function notations are not helping readers to understand what these processes are. I understand that there is a page limit of a neurips submission, and I think this is why I strongly recommend the authors to focus on just shape generation. Therefore, there will be space to describe these processes in more details.\n    - For example, how \"DecomposeHierarchy\" is done? also how \"BuildGraph\" is done?\n- Although it is not ncessary to do a comparison, I am wondering why there is no discussion between the proposed method and BlenderLLM [Du et al. 2024]?\n\nLIMITATIONS:\nIt is a bit unclear what is the limitation of the proposed method.\nEven though there is a limitation paragraph in Sec. 5, I feel that they are very vague and confusing. \nTherefore, I think the limitations are not well discussed, and strongly recommend the authors to provide examples and explanations on this part so that readers can understand the proposed method better.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ShapeCraft, a novel system that leverages Large Language Model (LLM) agents to autonomously generate structured, textured, and interactive 3D models from natural language instructions. The core innovation is the Graph-based Procedural Shape (GPS) representation, which decomposes 3D shapes into hierarchical components represented as nodes in a directed acyclic graph (DAG). Each node contains a natural language description, bounding volume, and code snippet, enabling flexible programmatic updates and post-modeling edits. Experiments demonstrate that ShapeCraft outperforms existing LLM-based and optimization-based methods in geometric accuracy, semantic richness, and editability. The system also supports post-modeling interactions like animation and customization, making it practical for real-world applications.\n\nSTRENGTHS AND WEAKNESSES:\nShapeCraft represents a notable advancement in text-to-3D generation by combining LLM agents with a structured procedural representation. The strengths and weaknesses are listed as follows:\n\n** Strength\n(1) The GPS representation bridges the gap between natural language instructions and procedural 3D modeling, enabling structured, editable outputs compatible with industry tools like Blender. Besides, hierarchical decomposition and bounding volumes simplify spatial reasoning for LLMs, improving accuracy.\n(2) Multi-path sampling and VLM feedback loops enhance robustness, allowing the system to explore diverse solutions and correct errors iteratively.\n(3) The system supports post-modeling edits and animations, making it valuable for artists and developers. It also demonstrates the superiority over existing methods in quantitative metrics (IoGT, CLIP score) and qualitative results (e.g., fewer artifacts).\n\n** Weakness\n(1) The system's performance hinges on the reasoning and coding capabilities of underlying LLMs, which may introduce inconsistencies or failures for complex or ambiguous prompts.\n(2) While the GPS representation is flexible, its effectiveness for highly abstract or novel shapes (e.g., fantastical designs) is not thoroughly evaluated. The paper focuses on static objects; dynamic or deformable shapes (e.g., animated characters) are not addressed.\n(3) The iterative refinement process, though efficient, may still require significant computational resources for high-quality outputs. The paper lacks detailed analysis of computational costs compared to baseline methods.\n\nQUESTIONS:\nOverall, I think this is a nice try of how to combine VLM agent with text-to-3D task. Please check for my concerns as listed below:\n(1) The iterative refinement depends on VLM feedback for error correction. What happens if the VLM fails to detect geometric flaws (e.g., intersecting meshes, incorrect proportions)? Is there a risk of error propagation?\n(2) The paper claims efficiency gains from parallel sampling, but no concrete runtime comparisons are provided. How does the computational cost scale with shape complexity compared to end-to-end diffusion-based methods?\n(3) The BRDF-based texturing is conditioned on rendered images, but does it handle complex material interactions (e.g., transparency, subsurface scattering) or only simple reflectance properties?\n(4) The paper shows successful examples but does not analyze where the method fails (e.g., ambiguous prompts, complex topology). A thorough discussion of limitations would strengthen the work.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nShapeCraft proposes a procedural 3D generation pipeline using a multi-agent framework. It presents Graph-based Procedural Shape Representation (GPS) for expressing 3D shapes as programs. The method decomposes objects into components as individual nodes with description, bounding volume and an attached code snippet. When generating an objects, the LLM parses the instruction hierarchically into various components to build the complete object. It further applies bootsrapping using a VLM as a judge to correct for errors in the generated shape. The authors also propose to iteratively sample multiple possible creative paths in parallel to capture the full diversity. To add texture to the generated shape, the method follows score distillation sampling and focuses on it at component level as well as global view. The method is comapred against 4 recent baselines as well as an ablation is conducted to show the need for iterative updates which allows better instruction following by correcting errors.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The paper is mostly well-written and all components are explained. \n* The results show good gains in generation quality compared to baselines (both qualitative and quantitative)\n\n## Weaknesses\n* Limited novelty-- Decomposition using LLM and graph representation has already been studied quite extensively. The related work also fails to mention many papers doing very similar things- Holodeck, LayoutGPT, FlairGPT, etc. FlairGPT in particular (and many others) decomposes a scene, builds a graph, calls code snippets to build the scene, which is quite similar and hence should be mentioned in related work. \n* The iterative refinement based on visual feedback is expensive and also explored before (3D-GPT, SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code [different from this paper although they have the same name], BlenderAlchemy). These references should also be added. \n* The texture is generated separately using a SDS based pipeline. I find the author's naming of component-aware SDS a bit gimmicky since it's a simple gathering of node points and there is no real novelty here. \n* While the method sees stronger results compared to baselines, the underlying LLM used in these earlier works are weaker. ShapeCraft uses the newly released Qwen3-235B and it becomes very tricky to understand if the performance gains come from the underlying LLM/prompts or the method in question.\n\nQUESTIONS:\n* Can the authors compare the average token cost of ShapeCraft against other baselines?\n* The method says it uses a pre-defined chain of thought and the authors choose to disable the thinking module. Given the advances in thinking models, is an explicit chain of thought model needed and is the improvement substantial?\n* How does the LLM understand the proportion ratios?\n\nLIMITATIONS:\nLimitations can be expanded such as token cost and generation fidelity compared to generative methods such as Trellis, reliance on SDS for texture", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ShapeCraft, an LLM agent system that generates structured, textured, and interactive 3D models from text. Its core innovation is a Graph-based Procedural Shape (GPS) representation, where an LLM decomposes an object into a graph of components, each defined by a description, bounding box, and a procedural code snippet. An agentic workflow with Vision-Language Model (VLM) feedback iteratively generates and refines the geometry for each part. The resulting models are well-aligned with the prompt, structured, and editable, a significant advantage over existing methods. Experiments show ShapeCraft outperforms a range of text-to-3D approaches.\n\nSTRENGTHS AND WEAKNESSES:\nPaper strengths:\n\n- The Graph-based Procedural Shape (GPS) representation design allows for the generation of structured and editable assets. \n- The experimental evaluation is strong. ShapeCraft shows superiority over a wide range of state-of-the-art methods in both raw mesh and textured mesh generation. \n\nPaper weaknesses:\n\n- Lack of deeper analysis: The system's success is heavily reliant on the capabilities of powerful, and likely proprietary, LLMs and VLMs. The paper would benefit from a more detailed analysis of failure modes related to these models (e.g., API hallucination, misinterpretation of geometric instructions). Furthermore, more quantitative ablation studies of each design are needed, such as the Blender Wrapper Library and different hyperparameters of multi-path sampling.\n- Scalability to complex topologies: The examples presented (e.g., furniture) are primarily compositions of relatively simple geometric primitives. It is unclear how well the approach would scale to generating objects with highly organic or intricate topologies, such as animals, human characters, or complex mechanical parts, where procedural generation from primitives might be insufficient.\n- Complexity and computational cost: The overall pipeline is quite complex, involving multiple feedback loops and parallel sampling paths. A more thorough discussion of the computational cost (in terms of runtime and, more importantly, the number of API calls to LLMs/VLMs) would be valuable; understanding its cost relative to other agentic systems is important.\n\nQUESTIONS:\n- During the initial GraphInit phase, how are the bounding boxes (Bi) for each component determined? Does the LLM generate numerical coordinates and dimensions, and if so, how does it reason about the spatial coherence of the overall layout before the VLM-based bootstrapping step?\n- Could you elaborate on the design of the Blender Wrapper Library? For instance, what level of abstraction does it provide? How significant is the role of this library in constraining the search space and simplifying the task for the LLM?\n- Regarding the multi-path sampling in Algorithm 1, what were the typical values used for the number of paths (P) and steps (T) in your experiments? How much diversity did you observe across the generated paths? Did they explore genuinely different modeling strategies or tend to converge to similar final shapes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}