言語処理学会 第30回年次大会 発表論文集（2024年3月）

Out-of-distribution Shape Generationusing Large Language Models and Geometry Nodes
Yutaro Yamada1 , Machel Reid2
1 Yale University
2 Google DeepMindyutaro.yamada@yale.edu

概要
Recent advancements in text-to-image models have been
impressive. Yet, they still struggle with accurately following prompts, especially when asked to create out-ofdistribution objects like a “chair with three legs.” In
this paper, we introduce a novel method using “GeometryNodes”–a procedural generation tool for creating shapes inBlender, which is a 3D modeling software. This tool allows
for the generation of shapes based on general parameterslike ’radius’ and ’height.’ We use a large language model(LLM) to interpret text prompts and convert them into specific instructions for these shape generators. Our method
has proven effective in creating unconventional examples
in two categories: Chairs and Tables.

1 はじめにModels like DALL-E-3 [1] and Stable Diffusion [2] havemade significant advancements in text-to-image generation, producing images nearly indistinguishable from thosecreated by humans. However, they still face challenges inaccurately following prompts, particularly in representing
spatial relationships and object-attribute binding. Various
solutions have been suggested to overcome these issues.
For example, [3] propose to enhance pre-trained diffusion
models with additional inputs like scribbles or depth maps.
[4] uses compositional linguistic structures during the diffusion guidance.
While these methods show promise, they still struggle
with generating images of objects that do not typically exist, like a “a chair with five legs.” This difficulty partly
arises because such unusual objects are rarely included in
the training data. Additionally, it is likely that these models
do not separate and understand concepts the way humansdo, such as distinguishing the number of legs from the gen-

Our approach

DALL-E-3

図 1 Our approach vs. DALL-E-3, one of the most advanced text-to-image models for the task of generating out-ofdistribution samples like “a chair with three legs”. For DALL-E3, we use “A chair with three legs. Make sure the background isblack, and the object generated is a gray-colored 3D shape.” forour prompt. For our approach, our prompt is “a chair with threelegs.”

eral concept of a chair.
Large Language Models (LLMs) are known for their exceptional reasoning abilities [5]. This raises the question:
can we use the reasoning capabilities of LLMs to solve theabove problem? Our approach involves using Blender’s
Geometry Nodes, a tool to create parametric shape generator, in combination with the function-calling ability ofLLMs. By feeding specific arguments to the shape generator, we can partially address the above problem.
Geometry Nodes is a tool within Blender, an opensource 3D modeling software, that enables the creation of
3D meshes programmatically. It allows users to build acomputational graph of 3D operations and set certain parameters as inputs to this graph. For instance, using a set of
Geometry Nodes, one can create various designs of objects
by altering the input parameters. An example illustration ofGeometry Nodes setup is shown in Figure 3. This flexible
system is especially popular among programmer-oriented
3D designers, who use Geometry Nodes to instantly generate diverse shapes by tweaking specific parameters.

― 2074 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

In our work, we extend this concept. We have devel-3D vision community. For instance, InfiniGen [14] em-

oped two parametric shape generators for chairs and ta-ploys Geometry Node-based methods to create 3D scenes,

bles. Uniquely, these generators allow for the modification

providing an endless source of synthetic data for 2D and 3Dof certain attributes, like the number of legs. Alongsidevision tasks. There is also a work [15] that combines Infini-these generators, we provide docstrings - natural language

Gen with Large Language Models (LLMs), using LLMs asdescriptions explaining how to use each shape generator
effectively.

a language interface to control world creation. Our method
is distinct; we introduce new shape generators based on Ge-2

ometry Nodes, focusing specifically on creating samples

Related Workthat are outside the usual training distribution.

Alignment with human intentions for text-toimage models

The topic of aligning text-to-image mod-3Methodels with human intentions has been gaining significant in-

Our proposed text-to-3D approach involves LLM’s func-terest. Recently, there have been various methods to en-tion calling ability and shape generator based on Geometry

sure that image generation aligns more closely with whatnodes. The overall approach is shown in Figure 2. We de-humans intend. One effective approach involves using ad-scribe each step below.

ditional inputs like layout [6], bounding boxes [7], and user

3.1

sketches [3] to steer the direction of image generation.

LLM function calling

Text-guided image editing is another method that incor-

The first step in our process is to translate a user’s nat-porates human intentions. Recent advancements include

ural language description into a set of parameters for theusing pre-trained diffusion models along with a spatial

shape generator. To do this, we utilize the function call-mask to focus edits in specific areas, as explored by [8].

ing capability of Large Language Models (LLMs). This

Another approach uses cross-attention layers for text-onlyfeature, originally developed by OpenAI, enhances LLMs’

guidance in image editing, as demonstrated in [9]. Imagic,

chat completion abilities, enabling them to connect seam-

introduced by [10], stands out for its ability to make com-

lessly with external tools and applications. Many externalplex, non-rigid semantic edits through text-based optimiza-applications offer their functionalities through APIs (Ap-tion. The concept of model customization , as exploredplication Programming Interfaces). The ability of LLMs toby [11, 12], is particularly noteworthy. These techniques

execute these APIs in response to user text inputs is a nat-allow diffusion models to create images of a new concept

ural extension of their capabilities. While it is possible tofrom just a handful examples. Once the model is fine-tuned

prompt LLMs to generate strings in a specific format, it isto understand the concept, it can generate various images

more efficient for them to directly produce outputs in JSONof that subject in different contexts.
Prompt following There is ongoing work to improveformat. To meet this requirement, LLMs can be fine-tuned
to output in these specific formats. This approach has been

how diffusion models interpret and accurately follow text

gaining popularity, and there are efforts within the open-prompts. Early versions of Stable Diffusion faced chal-source community, like the one led by [16], to replicate

lenges with correctly associating attributes and generating

OpenAI’s function calling feature. To enhance the inter-compositions. For instance, they often misassigned col-action between LLMs and these applications, we provide

ors in images with two concepts or struggled to generatedetailed natural language descriptions of what each argu-images with multiple concepts. [4] tackles this problem

ment represents.

cess. ’Attend-and-excite,’ proposed by [13], employs an

Parametric shape generator based onGeometry Nodes

attention-based semantic guidance method. DALL-E-3 [1]The second phase in our text-to-3D process involves alargely resolves these issues by using better, more detailed

parametric shape generator using Geometry Nodes. In

captions on a larger scale.

Blender, each Geometry Node corresponds to a specificby integrating linguistic structures into the diffusion pro-Procedural generation

3.2

In the realm of procedural

operation, like creating a mesh, extruding mesh surfaces,generation, Geometry Nodes have gained traction in theand selecting certain mesh faces through various selection― 2075 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

User Input:"Build a chair with sixlegs"

LLM Function Calling

図2

Parameters:
Num_legs: 3,
Leg_height: 10,
Chair_seat_radius: 4,.
.
.

Shape Generator via
Geometry NodeOur text-to-3D pipeline using Large Language Models and Geometry Nodes.

mechanisms. In our project, we have developed two spe-the input prompts. SDXL successfully generates “a tablecialized Geometry Node groups specifically for chairs andwith three legs” and “a chair with five legs” (if we considertables. These groups are designed to be flexible, allowing

one of the overlapping legs as the fifth leg.) However, for

input parameters to adjust features such as the number oflegs, height, width, and curvature of the furniture. An ex-

the rest of cases, we see that SDXL struggle to follow thetext prompts. On the other hand, our approach success-ample of our Geometry Node setup are displayed in Figure

fully generates desired shapes in all cases. This shows that

3. The arguments to the chair generator are summarized

the reasoning ability of LLMs along with parametric shape

as follows: “number of legs”, “leg height”, “leg radius”,

generator as a structured prior helps in the case of generat-“chair seat height”, “chair seat radius”, “chair seat cornering out-of-distribution samples.

radius”, “Distance to legs from chair seat edge”, “backrest

5height”. The arguments for the table generator are sum-Limitations

marized as follows: “number of legs”, “leg height”, “legOur approach requires parametric shape generators,radius”, “tabletop height”, “tabletop radius”, “Distance towhich limits the applicability of our method. Future work

legs from table edge”. We then convert our Geometry

should explore if it is possible to learn to generate these

Node groups into Python scripts by using a converter 1）.

parametric shape generators, which can be written as aThis allows us to take the arguments we receive from thecode.

function calling and pass them directly to the scripted ver-6

sion of the Geometry Nodes.

4Conclusion
We tackle the issue of prompt following of text-to-image

Results

and text-to-3D models. We first show that even the mostHere we showcase how our text-to-3D pipeline works

advanced text-to-image models still struggle with generat-

for generating regular chairs and tables via user’s text input.

ing out-of-distribution samples. We then introduce our ap-We then show how we can also generate out-of-distribution

proach that combines Large Language Models and Geom-chairs and tables, and compare results from DALL-E-3 andetry Nodes. Large Language Models deal with reasoning

StableDiffusion-XL.

and translating user prompts into concrete API calls, which

4.1

then be used to execute Geometry Nodes to create 3D

Text-guided generation

mesh. We show that our approach can generate both regu-We first test our text-to-3D pipeline to generate regularlar four-legged chairs as well as out-of-distribution chairs

chairs and tables. As shown in Figure 4, we can see thatthe height attribute in text prompts is correctly reflected inlike “a chair with three legs” and “a chair with six legs”.
Since our approach requires preparing Geomtry Node se-generated 3D mesh.

tups prior to generation, this limits the applicability of our4.2

Out-of-distribution sample generation

method to pre-defined object categories. However, our approach shows greater flexibility and control over the range

Next we compare our approach with some of the mostof characteristics that are beyond commonly seen attributes

advanced text-to-image models: DALL-E-3 and Stable

in training data, thanks to the reasoning ability of Large

Diffusion XL. The results are shown in Figure 5 and Fig-Language Models.

ure 6. We see that DALL-E-3 fail to accurately follow1） https://github.com/BrendanParmer/NodeToPython― 2076 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

図3An example geometry node setup. Each box represents a mesh operation such as “Extrude mesh”, “Create a curve circle” etc.

``A tall table''
図4``A short table''

An example 3D mesh result for tables.

図 6 Comparing DALL-E-3, Stable Diffusion-XL (SDXL),
and our approach for generating out-of-distribution chairs. The
prompts we use are: “a chair with three legs/five legs/ six legs”,respectively.

参考文献

図 5 Comparing DALL-E-3, Stable Diffusion-XL (SDXL),
and our approach for generating out-of-distribution tables. The
prompts we use are: “a table with three legs/five legs/ six legs”,respectively.

[1] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang,Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving Image Generation with Better Captions.
[2] Dustin Podell, Zion English, Kyle Lacey, AndreasBlattmann, Tim Dockhorn, Jonas Müller, Joe Penna, andRobin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, July 2023.
[3] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding
Conditional Control to Text-to-Image Diffusion Models. In― 2077 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

[4]

[5]

[6]

[7]

[8]
[9]

[10]

[11]

[12]

[13]

[14]

Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847, 2023.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu,Xin Eric Wang, and William Yang Wang. Training-Free
Structured Diffusion Guidance for Compositional Textto-Image Synthesis. In The Eleventh InternationalConference on Learning Representations, September
2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak
Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct:
Synergizing Reasoning and Acting in Language Models.
In The Eleventh International Conference on Learning Representations, September 2022.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li,Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu,
Michael Zeng, and Lijuan Wang.
ReCo: RegionControlled Text-to-Image Generation. In Proceedings ofthe IEEE/CVF Conference on Computer Vision andPattern Recognition, pp. 14246–14255, 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu,Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae
Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pp.
22511–22521, 2023.
Omri Avrahami, Ohad Fried, and Dani Lischinski.
Blended Latent Diffusion. ACM Transactions onGraphics, Vol. 42, No. 4, pp. 149:1–149:11, July 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-toPrompt Image Editing with Cross-Attention Control. InThe Eleventh International Conference on Learning Representations, September 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.
Imagic: Text-Based Real Image Editing With Diffusion
Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 6007–6017, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch,
Michael Rubinstein, and Kfir Aberman. DreamBooth:Fine Tuning Text-to-Image Diffusion Models for SubjectDriven Generation. In Proceedings of the IEEE/CVF
Conference on Computer Vision and PatternRecognition, pp. 22500–22510, 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli
Shechtman, and Jun-Yan Zhu. Multi-Concept Customization of Text-to-Image Diffusion. In Proceedings of theIEEE/CVF Conference on Computer Vision andPattern Recognition, pp. 1931–1941, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, andDaniel Cohen-Or. Attend-and-Excite: Attention-Based
Semantic Guidance for Text-to-Image Diffusion Models.
ACM Transactions on Graphics, Vol. 42, No. 4, pp.
148:1–148:10, July 2023.
Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei,Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu
Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei[15][16]― 2078 ―Law, Ankit Goyal, Kaiyu Yang, and Jia Deng. Infinite Photorealistic Worlds Using Procedural Generation.
In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp.
12630–12641, 2023.
3D-GPT: Procedural 3D Modeling with Large Language
Models. In The Twelfth International Conference on
Learning Representations, October 2023.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E.
Gonzalez. Gorilla: Large Language Model Connectedwith Massive APIs, May 2023.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).