DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
Authors: Abhay Zala
Han Lin
Jaemin Cho
Mohit Bansal
Venue: COLM (2024)
Keywords: Text-to-Diagram Generation, Large Language Models, Vector Graphic Generation, Iterative Refinement, Human-in-the-Loop
TL;DR: We present DiagrammerGPT, a novel two-stage text-to-diagram generation framework leveraging the layout guidance capabilities of LLMs to generate more accurate diagrams.

Abstract: Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. A diagram is a symbolic/schematic representation that explains information using structurally rich and spatially complex visualizations (e.g., a dense combination of related objects, text labels, directional arrows/lines, etc.). Existing state-of-the-art T2I models often fail at diagram generation because they lack fine-grained object layout control when many objects are densely connected via complex relations such as arrows/lines, and also often fail to render comprehensible text labels. To address this gap, we present DiagrammerGPT, a novel two-stage text-to-diagram generation framework leveraging the layout guidance capabilities of LLMs to generate more accurate diagrams. In the first stage, we use LLMs to generate and iteratively refine ‘diagram plans’ (in a planner-auditor feedback loop). In the second stage, we use a diagram generator, DiagramGLIGEN, and a text label rendering module to generate diagrams (with clear text labels) following the diagram plans. To benchmark the text-to-diagram generation task, we introduce AI2D-Caption, a densely annotated diagram dataset built on top of the AI2D dataset. We show that our DiagrammerGPT framework produces more accurate diagrams, outperforming existing T2I models. We also provide comprehensive analysis, including open-domain diagram generation, multi-platform vector graphic diagram generation, human-in-the-loop editing, and multimodal planner/auditor LLMs.

Published as a conference paper at COLM 2024
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
Abhay Zala
Han Lin
Jaemin Cho
Mohit Bansal
UNC Chapel Hill
[aszala, hanlincs, jmincho, mbansal 0cs êcs. unc. unc.edu
https: //ppgg///dagrrmerrPT..... github.io
Abstract
Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. diagram is a symbolic/ /scooiicccc representation that explains information using structurally rich and spatially complex visual- izations (e.g,, a dense combination of related objects, text labels, directional arrows/lines etc.). Existing state-of-the-art T2T models often fail at diagram generation because they lack fine-grained object layout control when many objects are densely connected via complex relations such as arrows / /lines, and also often fail to render comprehensible text labels. To address this gap, we present DiagrammerGPT, novel two-stage text-to-diagram gen- eration framework leveraging the layout guidance capabilities of LLMs to generate more accurate diagrams. In the first stage, we use LLMs to gen- erate and iteratively refine 'diagram plans' (in a planner-auditor feedback loop). In the second stage, we use a diagram generator, DiagramGLIGEN, and text label rendering module to generate diagrams (with clear text labels) following the diagram plans. To benchmark the text-to-diagra generation task, introduce AI2D-Caption, densely annotated diagram dataset built on top of the AI2D dataset. We show that our DiagrammerGPT framework produces more accurate diagrams, outperforming existing T2I models. We also provide comprehensive analysis, including open-domain diagram generation, multi-platform vector graphic diagram generation, human-in-the-loopp editing, and multimodal Ghanerraauiite LLMs.
1
Introduction
Over the past few years, text-to-image (T2I) generation models (Rombach et al., 2021; Ramesh et al., 2022; Yu et al., 2022; Chang et al., 2023; Dai et al., 2023) have shown impressive advancements in image generation quality. Large language models (LLMs)) have also recently shown strong capabilities and usefulness in broad language understanding and generation tasks (Touvron al., 2023a;b; OpenAl, 2023b; Chung al., 2022; Brown al., 2020; Chowdhery et al., 2022). Recent works also have demonstrated that it is possible to leverage LLMs to control layouts for the downstream T2I models, for better semantic alignment with input text prompts (Cho et al., 2023b; Feng et al., 2023; Lian et al., 2023).
However, it has not been explored use the combination of LLM and T2I generation frameworks for creating diagrams. A diagram is a scbboliicsscemmatic representation that explains information using rich and spatially complex visualizations (e.g., a dense combination of objects, text labels, arrows, lines, etc.). A system that helps to create accurate diagrams would be useful for preparing many educational and academic resources (e.g., creating diagram that explains new concepts in books, presentations, and papers). While the existing T2I generation models are good at generating realistic images, they often fail at diagram generation because they lack fine-grained object layout control when many objects are densely connected via complex relations such as arrows / /ines and also often fail to render comprehensible text labels. In diagram generation, is more important to

---

Published as a conference paper at COLM 2024
Sagon showing the Earth revobre sround mt-sur four fines,
Diagram Planning
Diagram Generation
Diagram Plan GPT-4
DiagramGLIGEN
with Text Label Renderlng
Visualization
After Foodback
atagee [wart (16), eores (13), ced]
fntity Locatie::
crew
Figure 1: An overview of DiagrammerGiT, our two-stage framework for open-domain diagram generation. In the first diagram planning stage (Sec. 3.1), given a prompt, our LLM (GPT-4 (OpenAI, 2023b)) generates a diagram pla, which consists of dense entities, fine-grained relationships, and precise layouts. Then, the LLM iteratively refines the plan to correct mistakes. In the second diagram generation stage (Sec. 3.2), our DiagramGLIGEN outputs the diagram given the diagram plan, then, we render the text labels the diagram.
convey correct information (with correct object relationships) than generate photorealistic objects. In our experiments, existing T2T generation models usually generate diagrams where objects and arrows/line /lnee have incorrect relationships and rendered text labels are incomprehensible. Even the recent state-of-the-art (SOTA) model DALL-E 3 (OpenAI, 2023a) struggles to render accurate diagrams (mention in their system card and shown Fig. 6).
To address the issues in text-to-diagram generation, introduce DiagrammerGPT. novel two-stage framework capable of generating more accurate open-domain diagrams for multiple platforms by following LLM-generated diagram plans. As shown in Fig. 1, our DiagrammerGPT splits the text-to-diagram generation task into two stages: diagram plan- ning and diagram generation. For the first stage (Fig. left), we employ an LLM to create and refine 'diagram plans'. In the second stage (Fig. 1 right), we introduce DiagramGLiGa a layout-guided diagram generation module, to generate diagrams based on diagram plans, then explicitly render text labels, ensuring their readability.
In the first stage, diagram planning (Sec. 3.1), we employ an LLM (egg, GPT-4 (OpenAI, 2023b)) to act as planner and generate diagram plans given text prompts. The diagram plans consist of (1) a list of entities (i.e., objects and text labels), (2) relationships between the entities (i.e., arrows or lines between entities), and (3) entity layouts (i.e., 2D bounding box coordinates). After the initial generation of diagram plans, inspired by recent works on LLMs that self-refine previously generated contents (Chen et al., 2023c; Miao et al., 2023; Madaan et al., 2023), we use another LLM to act as an auditor and find potential errors such incorrect object positions relationships between objects. Then, the planner LLM takes feedback from the auditor LLM to update the diagram plan (e.g., in Fig. 1, the sun is smaller than the earths, the plan is adjusted to fix the relative scales). We find that our LLM-generated diagram plans are quite accurate and that refinement can effectively help correct some small errors (see Sec. 5.2 and feedback example in appendix for more details).
In the second stage, diagram generation (Sec. 3.2), we introduce DiagramGLIGEN, a layout- guided diagram generation module, to generate diagrams based on diagram plans, then explicitly render text labels, ensuring their readability. We implement DiagramGLIGEN based on GLIGEN (Li et al., 2023b) architecture, which adds gated self-attention layers to the Stable Diffusion v1.4 (Rombach et al., 2021) model. While the original GLIGEN model is only trained on natural images and take only objects for layout grounding, DiagramGL.IGEN is more specialized in the diagram domain, by being trained on our new A12D-Caption diagram dataset (see the following paragraph and Sec. 4.1 for more details) and taking the text labels and arrows as additional layout grounding inputs. As the SOTA diffusion models still struggle in rendering text (Liu et al., 2023b; Chen et al., 2023a; Cho et al., 2023b), we explicitly render text labels on top of the generated diagrams to ensure they are readable. Compared to the T2I baselines, our diagram generation stage allows more accurate object layouts and relationships between the objects (e.g., arrows /lines,, and clear text labels.
2

---

Published as a conference paper at COLM 2024
Initial Diagram Plan Generation
Diagram Plan Refinement via Iterative Feedback
the 1ith catle of flex, Ellutratiie the siffecent stagis
Planner (GPT-4)
Hha''s weong sctt the aureet disgra?
Auditor (GPT-4)
arreed stage West
Planner (GPT-4)
puppe (sa 129) zara (3)) ool Bussts Lectien: to tubrts 10,
uusese the dlag alas
(GPT-4)
dlggran
Update diegram up N
Illustration of the first stage of DiagrammerGPT:: diagram planning (Sec. 3.1). We use a planner LLM (e.g., GPT-4 (OpenAI, 2023b)) to create the fine-grained layouts of diagrams, which we call diagram plans. We first generate an initial diagram from the input text prompt with an LLM (left). Then we iteratively refine diagram plans in a feedback loop of the planner and auditor LLMs.
Figure 2:
Since there are no diagram datasets with detailed captions and fine-grained layout an- notations, we construct AI2D-Caption, a new dataset for the text-to-diagram generation task built top of the A12D (Kembhavi et al., 2016) dataset (Sec. 4.1). We employ LLaVA 1.5 (Liu et al., 2023a), a SOTA multimodal LLM, to create annotations of captions and object descriptions AI2D diagrams.
We comprehensively compare our DiagrammerGPT to recent sest-to-image/diaaeee diagram gener- ation methods, including Stable Diffusion v1.4 (Rombach et al., 2021), VPGen (Cho et al., 2023b), and AutomaTikk TikZ (Belouadi et al., 2023), in both zeroshot and fine-tuned settings (see Sec. 4.2 and Sec. 5.1). In our quantitative and qualitative analysis, our DiagrammerGP7 demonstrates more accurate diagram generation performance than the baseline models. Our method also outperforms Stable Diffusion vl.4, the closest and strongest baseline, in human preference study on both image-text alignment and object relationship criteria. In our error analysis, we show that diagram plans generated by our LLM (GPT-4) are quite accurate (before and after refinement), while DiagramGLIGEN sometimes makes mistakes when generating diagrams. This indicates that our DiagrammerGPT can benefit from future layout-guided image generation backbones stronger than GLIGEN (see Sec. 5.2).
We also conduct additional analysis, including open-domain diagram generation, multiple platforms vector graphic diagram generation, human-in-the-logg editing, and text-only us. multimodal LLM (e.g,, GPT-4Vision (OpenAI, 2023c); see Sec. 5.4 and appendix). First, experiment with generating diagrams in unseen domains (e.g., geology and plants) that are not covered in the LLM in-context learning domains (astronomy, biology, engineering), where our DiagrammerGPT could often generate semantically accurate diagram plans and diagrams from these unseen prompts. Second, we experiment with rendering our diagram plans in multiple platforms: Microsoft PowerPoint, Inkscape, and Adobe Illustrator (see Fig. 7). Third, we show that once a diagram plan is exported to another platform, end-users can also manually edit plans to their liking and send them back to our DiagramGLIGEN to generate images using their manually refined plan (i.e., human-in-the-loop refinement of diagram plan). Lastly, we experiment with using the recently released GPT-4Vision (OpenAI, 2023c) instead of text-only GPT-4 for our planner and audítor LLMs (in appendix).
2 Related Works
2.1 Text-to-Image Generation
In the text-to-image (T2I) generation task, models generate an image from a given text prompt. Recently multimodal language models (e.g., P'rrti (Yu et al., 2022) and MUSE (Chang et al., 2023)) and diffusion models (e.g., Stable Diffusion (Rombach et al., 2021), DALL-E 2 (Ramesh et al., 2022), and Imagen (Saharia et al., 2022)) have gained popularity for this task. These recent T2I generation models have demonstrated impressive photorealism in their zeroshot image generation capabilities. However, using these models directly for diagram
languagge
3

---

Published as a conference paper at COLM 2024
Generated diagram with text labels
Diagram plan
DiagramGLIGEN
Generated diagram
Visual
Self-Atttn
Text rendering Pillow)
Descriptions
Gated Self-Atth
Cross-Attn
Caption
Figure Illustration of the second stage of DiagrammerGiPT: diagram generation (Sec. 3.2). We first generate the objects from the diagram plan with DiagramGLIGEN, our layout- guided diagram generation model. Then, we Pillow to render clear text labels.
generation is challenging due to the significant distribution gap between diagrams and their training images. Additionally, these models often lack precise control of fine-grained layouts when many objects are densely connected via complex relations and frequently produce illegible text labels, which are essential for the diagram generation task.
2.2 Text-to-Image Generation with LLM-Guided Layouts
Large language models (LLMs) have demonstrated their usefulness in various language generation tasks (Touvron et al., 2023a;b; OpenAI, 2023b; Chung et al., 2022; Brown et al., 2020; Chowdhery et al., 2022). Recent works also leverage LLMs to control layouts for the downstream image generation models, for better semantic alignment with input text prompts (Cho al., 2023b; Feng al., 2023; Lian et al., 2023). However, these works focus on generating natural images and do not have capabilities that are crucial in diagram gener- ations, e.g,, rendering clear text labels or the ability to precisely control fine-grained layouts of many objects that are densely connected via complex relations such as arrows/ arrows/lines, as shown in our experiments (Sec. 5). A concurrent work, AutomaTike (Belouadi et al., 2023) uses LLMs to generate TikZ (Tantau, 2007)² code to produce scientific vector graphics. While TikZ can be used to draw specific types of diagrams, such 2D bar plots directional acyclic graphs,
difficult to generate diagrams including entities not supported by the TikZ primitives, such animals. In contrast, DiggrammerGPT allows for generating diagrams including diverse entities (e.g., different animals life cycle diagrams or different planets/ stars in astronomy diagrams).
3 DiagrammerGPT: Method Details
We introduce DiagrammerGPT,, novel two-stage framework for generating open-domain diagrams from text prompts, where an LLM first generates the overall plan, and a visual generator renders an actual diagram following the plan. In the first stage, Diagram Planning (Sec. 3.1), a planner LLM takes text description of a diagram as input and generates diagram plan, overall diagram layout that guides the downstream diagram generation module. The planner LLM generates the initial diagram plan and iteratively refines the diagram plan via feedback from auditor LLM. In the second Diagram Generation stage our new ayyutt-guddd diagram generation module, takes the
(Sec. 3.2), DiagramGLIIGNN diagram plan and generates the diagram. Finally, we render the text labels (which are from the diagram plan) onto the diagram to ensure clear and easy-to-read labels for each entity.
3.1 Stage Diagram Planning
As illustrated in the left part of Fig. 2, we use an LLM (e.g., GPT-4 (OpenAI, 2023b)) to create the overall layouts for diagrams, which we call diagram plans (see appendix for detailed plan configuration and example).
1A TeX package for generating graphics by composing primitive objects with basic polygons.
4

---

Published as a conference paper at COLM 2024
Initial diagram plan generation. We generate diagram plans with GPT-4 via 10 in-context learning examples. Plans consist of three components: (1) entities a dense list of objects; (2) relationships complex relationships between entities; and (3) layouts - 2D bounding boxes of the entities. See appendix for diagram plan generation prompts/ extra details.
Diagram plan refinement via iterative feedback. Although the planner (GPT-4) generates fairly accurate initial diagram plans, we can further refine it to account for potentially missing or improperly arranged entities. To address this issue, we introduce an auditor LLM that checks for any mismatch between the current diagram plan and the input prompt. It then provides feedback, enabling the planner LLM refine the diagram plans. Our auditor and planner LLMs form feedback loop to iteratively refine the diagram plans. For this auditor LLM, we employ GPT-4 again but with a different preamble and in-context examples designed to give useful feedback (see appendix for the prompts to initialize the auditor LLM). We repeat the feedback loop for up to iterations, using N= 4 in our experiments. The right part of Fig. exemplifies the feedback process.
3.2 Stage Diagram Generation
As shown in Fig. 3, we first generate the diagram images following the diagram plan with DiagramGLIGEN, then render text labels on the diagram.
Layout-guided diagram generation. Conveying factual information is more crucial than drawing photorealistic objects in diagram generation. In our experiments, we observe that existing T2I models often omit important objects, and generate incorrect object relationships and unreadable text labels (see Sec. 5 and Fig. 5). Therefore, we introduce DiagramGLIGEN, a layout-guided text-to-diagram generation model to tackle these issues. Inspired by GLIGEN (Li et al., 2023b), we implement DiagramGLIGEN by incorporating gated self-attention layers, which take layout grounding inputs, into Stable Diffusion v1. .... Furthermore, we enhance layout control by incorporating text labels and relationships as part of the layout grounding inputs during training, which can reduce the generation of unreadable text and redundant arrows/lines during inference (by not giving text labels during inference). We employ CLIP (Radford et al., 2021) text encoder to encode object descriptions and their relationships. We use the CLIP image encoder represent bounding box regions of the text labels in the ground truth diagrams. See the appendix for additional GLIGEN details and training setup.
DiagramGLIGEN:
Text label rendering. Text labels (egg, "Sun" labeling the sun object in Fig. 3) in diagrams can effectively assist readers in understanding new concepts (Johnson al., 2014). However, as shown in Cho et al. (Cho et al., 2023b), existing T2I generation models (including Stable Diffusion v1.4) still struggle to generate high-quality text labels. Therefore, we explicitly render clear text labels on the diagrams with the Pillow Python package (Clark, 2015).
Experimental Setup
In the following subsections, we introduce our AI2D-Caption dataset for the text-to-diagran generation task (Sec. 4.1), baseline models (Sec. 4.2), evaluation metrics (Sec. 4.3), and human evaluation setups (Sec. 4.4).
4.1 AIDD-Caption Dataset
We introduce the AI2D-Caption dataset for the lext-to-diagram generation task. AI2D- Caption is built on top of A12 Diagrams (AI2D) dataset (Kembhavi et al., 2016), which provides annotations of around 4.9K diagrams covering diverse scientific domains, from Grade 1-6 science textbooks. The original AI2D dataset's annotations are very short / are missing some aspects (i.e., each object bounding box is labeled simply as 'blob'), so we employ LLaVA 1.5 (Liu et al., 2023a), SOTA multimodal language model, to generate detailed captions for the diagrams and region descriptions of each bounding box (see Fig, for comparison of AI2D and AIDD-Caption and the appendix for implementation details).
5

---

Published as a conference paper at COLM 2024
A12D
AI2D-Caption
electical dECUR!
circuit!!
Mectkalccccmm
2
circul
Lipk
Baltery
diagram showing an electrical circuit with a light bulb, a battery, and switch.
(No Caption)
Figure 4: Example diagram annotation from the AI2D dataset (Kembhavi al., 2016) (left) and our Al2D-Caption (right). Al2D-Caption additionally provides annotations of the diagram caption and bounding box region descriptions.
We use all of the LLaVA 1.5-aided caption and region annotations for DiagramGLIGEN training and baseline fine-tuning.
To ensure that the LLM in-context examples and evaluation are accurate, we manually annotate diagram plans (i.e,, captions, object/text label bounding boxes, object descriptions, arrows) for randomly selected 105 diagrams (held out from the training set described above). Among the 105 diagrams, we use 30 diagrams that cover diverse scientific domains (10 for astronomy, 10 for biology, and 10 for engineering) as in-context examples, and 75 diagrams (25 for astronomy, 25 for biology, and 25 for engineering) test split.
4.2 Baseline Models
We compare our DiagrammerGPT to several baseline models, including Stable Diffusion v1.4 (Rombach et al., 2021), VPGen (Vicuna13B (Chiang et al., 2023) + GLIGEN) (Cho et al., 2023b), and Automa TikZ (Belouadi et al., 2023) (CLIMA-13B) For the baselines, we experiment with zero-shot and fine-tuned (on AI2D-Caption dataset where applicable) diagram generation. For VPGen, we fine-tune both the Vicunal3B and GLIGEN.
4.3 Evaluation Metrics
VPEval (objects, counts, relationships, texts). The VPEval metric (Cho et al., 2023b) evalu- ates diagrams in terms of the presence of objects (object evaluation), the number of objects (count evaluation), the correctness of spatial and connection relationships (relationship evaluation), and the presence of correct text labels (text evaluation). While the original VPEval uses BLIP-2 (Li al., 2023a) as its visual reasoning model, we employ the recently released LLaVA VA 1.5 (Liu et al., 2023a) model as our visual reasoning model, as we find it is more faithful in diagram question answering in our initial experiments and is an overall stronger model than BLIP-2 (Liu et al., 2023a). See appendix for setup details.
Captioning. In line with previous works (Hong et al., 2018; Hinz et al., 2020; Cho al., 2023a), we also use captioning We use LLaVA 1.5 (Liu et al., 2023a) caption each generated diagram and then compare this generated caption with the ground-truth caption using CIDEr (Vedantam et al., 2014) and BERTScore (Zhang et al., 2020).
way to determine the accuracy of the generated diagrams.
CLIPScore. Following previous works (Cho et al., 2023a; Saharia et al., 2022; Belouadi et al., 2023), we use CLIPScore (Hessel et al., 2021) to measure the similarity between the generated diagram and the original caption/ poption/grrunde grrund--uuuut -ttth diagram. Concretely, calculate two types of CLIPPcoree (1) CL Img Txt, similarity between the generated diagram and

---

Published as a conference paper at COLM 2024
VPEval (%)
Captioning 1
CLIPSeore
Methods
Object
Count
Text
Relationships
Overall
CIDEr
BERTScore
Img-Txt
Img-Img
Zerosioot
Stable Diffusion v1.4
63.1
27.8
0.0
79.3
39.0
7.7
87.5
27.3
65.3
VPGen
55.8
32.9
0.0
72.8
39.3
6.1
87.2
25.6
61.7
NatomaTikz
24.9
34.2
5.5
67.7
31.0
12.2
86.9
24.7
64.5
Fine-fumed
Stable Diffusion v1.4
69.8
35.4
0.0
81.9
45.1
18.2
88.5
30.1
68.1
VPGen
62.8
27.8
0.0
76.3
39.7
4.2
86.9
26.4
61.9
DiagrammerGPT (Ours)
47.5
89.4
Table Comparison of DiagrammerGPT to existing text-to-image generation baseline models. On all metrics, DiagrammerGPPT outperforms the baselines, indicating that our method more effective for generating accurate diagrams.
the ground-truth caption. (2) CLIPScore/mg-Iigg similarity between the generated diagram and the ground-trume truth diagram. We use OpenAI CLIP ViT-L/14 /14 (Radford et al., 2021).
4.4 Human Evaluation
Human error analysis of the two stages. As our DiagrammerGPT pipeline consists of two stages, it is important to understand where any errors in our pipeline come from. In both stages, we assess (1) Object Presence: whether all required objects for the diagrams are present, and (2) Object Relationships: if the objects have proper relationships to each other (e.g,, for a lunar eclipse, the earth should be between the sun and the moon). We have an expert rate both stages on Likert scale from to for 25 layouts 'images.
Pairwise preference: Image-text alignment & object relationships. We conduct a hu- man analysis comparing our DiagrammerGPT framework Stable Diffusion v1.4 on 50 diagrams. We choose Stable Diffusion v1.4 fine-tuned on AI2D-Caption for comparison because it is the closest baseline to our DiagrammerGPT and also shows the strongest results among the baselines (see Sec. 5.1). We ask crowd-sourced annotators (20 unique annotators) from Amazon Mechanical Turk? to evaluate the diagrams generated for each prompt (see appendix for setup details).
5 Results and Discussion
We show primary quantitative results (Sec. 5.1), human evaluation pairwise prefer- ence study and error analysis (Sec. 5.2), qualitative analysis (Sec. 5.3), additional analysis about open-domain generation, multi-platform vector graphic diagram generation, human- in-the-loop diagram plan editing, (Sec. 5.4), and an ablation of different ablations for diagram plan generation (Sec. 5.5). See appendix for more analysis DiagramGLIGEN setups and multimodal planner/ auditor LLMs.
5.1 Quantitative Results
VPEval. Table 1 left block shows the VPEval results. For both Stable Diffusion vl .4 and VPGen baselines, fine-tuning improves the score for object skill (eg., 63.1 + 69.8 for Stable Diffusion v1.4, and 55,8 - 62.8 for VPGen), but for VPGen count, it decreases scores (32.9
+ 27.8). For relationships, both models improve (79.3 81.9 for Stable Diffusion v1.4, and 72.8 + 76.3 for VPGen). For text, both models achieve 0 scores before and after fine-tuning. Our DiagrammerGPT outperforms both zeroshot and fine-tuned baselines on both overall and skill-specific VPEval scores, showcasing the strong layout control, object relationship representation, and accurate text rendering capability of our diagram generation framework.
Captioning and CLIPScore. Table 1 middle block shows captioning scores (with LLaVA 1.5), and the right block shows CLIPScore (with CLIP-VIT L/14). Our DiggrammerGPT
Phttpa://wwi
http::/wwwwww.ttcoommmm mturk.com
7

---

Published as a conference paper at COLM 2024
Stage Diagram Planning GPT-4)
Slage Diagram Generation (with DiagramGLLGGEN))
diagram plan
Diagram plan after refinement
Diagram
Objects Presence (7)
Object Relations (*)
Objects Presence (?) Object Relations (T)
Objects Presenoe (f)
Object Redations (T)
4.96
4.56
4%
4.72
2.96
3.36
Table 2: Step-wise error analysis of DiagrammerGPP on 25 Al2D-Caption test prompts. We use a Likert scale (1-5) to evaluate object presence and object relations of the diagram plan (before and after refinement) and the final generated diagram.
Input Prompt
Fine-tuned
AutomaTIkZ
Diagram (Ours)
Diagrammer (Ours)
A diagram showing the Earth's position four phases itrvoovess around the sun.
Figure 5: Example diagram generation results from baselines (fine-tuned Stable Diffusion v1.4 and AutomaTikZ) and DiagrammerGPT the AI2D-Caption test split. Our DiagrammerGPT correctly follows the caption while the baselines make several errors.
outperforms both the zeroshot and fine-tuned baselines, indicating our generated diagrams have more relevant information to the input prompt (which is critical aspect of diagrams), have better image-text alignment, and more closely resemble the ground-truth diagrams (image-image) than the baselines. Our DiagrammerGPT significantly outperforms both fine-tuned VPGen (26.4 us. 4.2) and fine-tuned Stable Diffusion v1.4 (26.4 us. 18.2) for CIDEr and also achieves a few higher points on BERTScore. For both CLIPSCore's, s, DiagrammerGPT has improvement over fine-tuned Stable Diffusion v1.4 (32.1 us. 30.1 and 73.9 us. 68.1).
5.2 Human Evaluation
Step-by-step error analysis. Table shows that our diagram plans exhibit high scores on both object presence (4.96) and object relationship (4.56) even before the refinement, and refinement increases the object relationship scores even further (4.56 + 4.72) by adjusting the entity layouts from the initial diagram plan (see appendix for refinement examples). During the diagram generation stage, the object presence and relationship scores decrease. This indicates that our DiagrammerGPT could generate more accurate diagrams, once we have access to layout-guided image generation backbone model stronger than Stable Diffusion v1.4 architecture.
Human preference. We conduct hu- man preference study, comparing our DiagrammerGI'T and fine-tuned Sta- ble Diffusion v1.4 (SD v1.4) in image- text alignment and object relation- ships. As shown in Table 3, our Di- agrammerGPT achieves higher pref- erence than Stable Diffusion v1.4 in both image-text alignment (68% vs 24%) and object relationships (62% 34%) criteria.
Human Preference (%)
Evaluation category
DiagrammerGPT
SD v1.4
Tie
Image-Text Text Alignment Object Relationships
68
24
8
62
34
4
Table 3: Human preference study on generated
diagrams: DiagrammerGPT us. SD v1.4.
5.3 Qualitative Analysis
Comparison with baselines. Fig. 5 shows example diagrams generated by the baselines (Stable Diffusion v1.4 and AutomaTikZ) and our DiagrammerGPT (both diagram plan and final generation diagram) on the AI2D-Caption test split (see appendix for additional example). Our diagram plan strongly reflects the prompt, and the final diagram is more aligned with the input prompt than the baselines. In the Fig. 5 example, diagram
8

---

Published as a conference paper at COLM 2024
Input Prompt
DALL-E 3
Diagram Plan (Ours)
DiagrammerGPT (Ours)
diagram showing the layers of the earth. it includes inner and outer cores, the mantle, and the crust
Figure 6: Open-domain diagram generation examples with DALL-E and DiagrammerGrI.
Input Prompt
Rendered Inkscape
Illustrator
diggamm showing food
starting from
bottom goss from cher. slug, icad, ts shand. is eagle tignt
It
botton, statting
stall, cryytsh, augatt
Figure 7: Examples of vector graphic diagrams generated from diagram plans.
correctly shows the earth in four phases revolving around the sun. Stable Diffusion v1.4 either over-generates objects in the image (egg,, too many earths), and AutomalikZ fails to generate proper diagram. Although our generated diagram plans are generally correct, sometimes DiagramGLIGEN can fail to properly follow all aspects. As noted in Sec. 5.2, our DiagramGLIGEN can improve once better backbone becomes available.
Diagram plan refinement. We analyze how our diagram refinement step (see Sec. 3.1) improves the diagram plans. The refinement can fix minor mistakes like missing connections in circuit or objects being placed in the wrong location. We show examples in the appendix.
5.4 Additional Analysis: Open-Domain, Open-Platfone and Human-in-the-Loop
Open-domain diagram generation. We demonstrate that our planner L.M can extend its capabilities to unseen domains beyond the three areas (astronomy/brle haanny//bboogyy/ /engineeri)))))) given in the in-context examples (30 total examples, 10 from each of astron- omy omy/biology/ domain, our planner LLM generates fairly accurate layouts, and our DiagramGLIGEN can generate diagram following the layouts. We also compare the recently released DALL-E (OpenAI, 2023a) model and find that it generally produces images with good aesthetic style but tends to generate diagrams with redundant objects (eg., excessive text descriptions or objects). It also struggles with creating diagrams that adhere to prompt (e.g,, generating incorrect layers in the earth example). The DALL-E system card (OpenAl, 2023a) also notes that DALL-E 3 tends to generate inaccurate information in diagrams. See appendix for more examples.
htty//oooogy//ngneeeeeeeee /engineering). As shown in Fig. 6, from input prompt in the earth science
Vector graphic diagram generation in different platforms. Although our primary focus is on pixel-level diagram generation pipeline with DiagramGLIGEN, our diagram plans can also be used for vector graphic diagrams, with multiple platforms like Microsoft PowerPoint, Inkscape, and Adobe Illustrator. Fig. 7 presents an example of vector graphic diagrams. The example delivers promising results by effectively conveying the crucial information and layouts described in the input text prompts (see additional example in the appendix). However, it also exhibits certain imitations: (1) inconsistency in icon styles; (2) limited icon retrieval capability. As the diagrams are editable via these same platforms, these limitations can be addressed by end-users editing the diagrams to their liking (see below).
Human-in-the-loop diagram plan editing. With the diagram plans being rendered in vector graphic platforms, as mentioned above, DiagrammerGPT provide editable dia-

---

Published as a conference paper at COLM 2024
the andrass Bue je
diagram showing the earth, moon, and sun with lext labels.
LLM Diagram Planner
Input Prompt
Inital Plan Generation
Initial Diagram
User Edits
User Refined Plan
New Diagram
Illustration of human-in-the-loop diagram plan editing. DiagrammerGPPT first provides an initial diagram plan with the corresponding generated diagram, users can then review the generated layouts/ /dagg//m and make adjustments based on their needs.
Figure 8:
gram plan, allowing for human-in-the-lopp editing. As illustrated in Fig. 8, DiagrammerGPT first generates an initial diagram plan along with the rendered image. Users can then review the generated layouts/ tooouu//dagramss and make adjustments based on their needs /wnnss (e.g,, move the objects, add/ remove objects, adjust object sizes, etc.). With the human-refined diagram plan, users can either keep it in vector format and use icons (as mentioned in the previous paragraph) or give back to DiagramGLIGEN and then create pixel-level diagrams, resulting in diagrams/ layouts that are better suited to end-users requirements.
5.5 Different LLMs for diagram plan generation
We also experiment with different LLMs such as GPT-3.5 Turbo (Ope- nAI, 2023d), GPT-4 (OpenAI, 2023b), LLaMA (8B) (AI@Meta, 2024), and LLaMA2-Chat (13B) (Touvron et al., 2023b). We compare them with 'layout recall' - how often each ground-truth object/ text label can be found in the predicted diagram plan. We first ob- tain bipartite matching of objects /exxt labels between prediction and ground- truth via Hungarian matching algorithm, using BLEU-1 (Papineni et al., 2002) as the match-
Layout Recall
LLM
Object
Text
Overall
LLaMA3 8B
24.9
32.2
29.2
VPGen (Vicuna 13B)
30.6
12.9
GPT-3,55 Turbo
82.5
54.6
74.7
GPT-4 (default)
84.1
60.1
78.4
Table 4: Ablation of different LLMs for the planner and auditor.
ing score. Then we calculate layout recall metric: ΣN match(obju) where N is the total number of ground-truth object/ text labels, objn is nth the ground-truth object/ text label, and match outputs 1 if the object teet label is matched in the predicted diagram plan and 0 if not.
N
Table shows a comparison of four LLMs: LLaMA3 8B, GPT-3.5 Turbo, GPT-4, and Vicuna 13B (VPGen checkpoint). As indicated by GPT-3.5 Turbo's and GPT-4's high 'overall' performance (74.7 and 78.4 respectively), both models are capable of generating accurate diagrams; however, GPT-4 is slightly better in both the 'object' and 'text' metrics, we use GPT-4 as our main LLM. LLaMA3 and fine-tuned Vicuna are not able to perform well for the task with both getting overall score below 30.
6 Conclusion
In this work, we propose DiagrammerGPT, a novel two-stage text-to-diagram generation framework that leverages the knowledge of LLMs for planning and refining the overall diagram plans. We demonstrate that our DiagrammerGPT achieves more semantically accurate layouts in diagram generation than baseline models in both quantitative and qualitative analysis. In addition, we provide comprehensive human error analysis, ablation studies, and analysis about open-domain diagram generation, vector graphic diagram generation, human-in-the-loog diagram plan editing, and multimodal planner/ auditor L.L.s. We hope work inspire future research diagram generation.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank the reviewers for the thoughtful discussion and feedback. This work was sup- ported by DARPA ECOLE Program No. HR00112390060, NSF-AI Engage Institute DRL- 2112635, DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031, ARO Award W911NF2110220, ONR Grant N00014-23-1-2356, Accelerate Foundation Models Research program, and a Bloomberg Data Science Ph.D. Fellowship. The views contained in this article are those of the authors and not of the funding agency.
References
Jaided AL Easyocr: Ready-to-use ocr with 80 supported languages and all popular writing scripts including latin, chinese, arabic, devanagari, cyrillic and etc., 2023. URL
com/
Al@Meta. Llama 3 model card, 2024. URL |tt.t//ee////tttttooo///aaaaaaaa MODEL_ md.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. ArXiu, htt//2307.0000 2023. Uttpeuttappppppppaaaaaaaaaaaaae wet.eeeeaaaaannaaaaaeeeeeeeeeeee org/
CorpuaID: : 263334353.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS, 2020. URL http: //arxiv. v. org/ abs/ abs/a/a5..00... 14165.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text- Texmaeo Generation via Masked Generative Transformers. In ICML, 2023. URL
http://arxiv. org/ abs /2//2.201.00000
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters. ArXio, 2bb5/230.111 10855, 2023a. URL https:://app www.ppppaaaaaaam......... .org/
CorpuaID: 258762187.
Jingye Chen, Yupan Huang, Tengehao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power of language models for text rendering. arxiv preprint ar arXie:2311.1666, 2023b.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. ArXiv, abs/ 2502.200,,,,88 2023c. URL https //apt/ www.sseeaaatciiieeeeeeeeeee 258059885.
Corpus
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: htpp:://mmsy/ org/blog /log/ 2023-03-30 vicuna/.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV, 2023a.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation. In NewrIPS, 2023b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon
11

---

Published as a conference paper at COLM 2024
Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meeir-HHelsseen Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, abs 0200,, 2022. URL http://aap....mtaaeeee
org/ Perg/rrpuslai ID: 247951931.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Ja- son Wei. Scaling instruction-finetune language models. ArXit, abs abs/2210.11416, 2022. URL httpa://api.
semanticscholaa wttpiiooorrrrrrrrrroooeeeeee org/CorpusID 253018554.
Alex Clark. Pillow (pil fork) documentation, 2015. URL ptthaa/pd/ppllls////o
org/
pdf.
Xiaoliang Dai, Ji Hou, Ya Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Enhancing image generation models using photogenic needles in haystack, 2023.
Parikh. Emu:
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. NewrlPS, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic Object Accuracy for Generative Text-to- Image Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020. ISSN 0162-8828, doi: 10.1109 /T/////A222020002222000000 URL https: //teex//o//.. 9184960/.
leee. org/document
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring Semantic Layout for Hierarchical Text-to-lmage Synthesis. In CVPR, 2018
Amy Johnson, Kirsten Butcher, Gamze Ozogul, and Martin Reisslein. Introductory circuit analy- sis learning from abstract and contextualized circuit representations: Effects diagram labels. Education, IEEE Transactions on, 57:160-168, 08 2014. doi: 10.1109/TE. TET.2T1..2.222222222
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV, pp. 235-255-5555 251, 2016. ISBN 978-3-319-46493-0.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume abs/2301.12597. 2023a. URL https: //ttpps:///pp/ /ppi semanticscholaat
htg//oorpusID:253390555
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, pp. 22511-22521, 2023b.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint ar wwXie:2305.16653 2023.
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via Ilm-guided planning. Xiv preprint XXix:2309.1509/ 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16270-16297, Toronto, Canada, July 2023b. Association for Computational I Linguistics. doi: 10.18653 v1 /22.6663/11122.32//22/ a202..ccllooo...900 URL. https: anchooyyy org/2023. scl-long.900 900.
12

---

Published as a conference paper at COLM 2024
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In International Conference on Learning Representations, 2019.
Aman Madaan, Niket landon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. ArXis, 23b./b303337111 2023. htRp:ttps://app. httpppppe/appp..mmaaiccttooooooe semanticscholar............ org/ org/CorpusID: 257900871.
Ning Miao, Yee Whye Ieh, and Tom Rainforth. Selfcheck: Using Ilms to zero-shot check their own step-by-step reasoning. ArXie, abs/2308.00456, 2023. URL https //api.ppppppp semanticscholatt org/ :220p50900.0
OpenAI. Dall-e system card, 2023a. URL https:////ccc http://ppppp.coppappp/////LLLLL. pard.pdf.
OpenAL. Gpt-4 technical report. ArXio, 257532816.
2023b. URL
OpenAL. Gpt-4v(ision) system card, 2023c. URL Card.pdf.
OpenAl. Chatgpt. attps::/opennaee hwt.:ppann..cccccaaaaaaeeeee www.chtttttcccoooeeeaa 2023d.
http:://
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-ling Zhu, Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002. URL atttps://ap.. http:///ppp.ssmntiiicaiiiie semanticscholar org/ /tgp/orruusDDD ID: 11080756.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents, 2022. URL http: attt://aaxivv wwwpa/awwwwwrr.aaaaraaraaeaaeeee wwwp/b//g/////////o////// 06122.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL https://api www.nticscool......omom..mo..oo. org/ CorpuaID: 245335280.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, lim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic lext-to-lmage Diffusion Models with Deep Language Understanding. In NeurlPS, 2022. URL http://///// www.a/www.....aaooaaaaa hbtp:/ab/222000
11487.
Till Tantau. pgf a portable graphic format for tex, 2007. URL httLLtppee//gthhbb.../////ggt-tp
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lan- guage models. ArXio, abs/2302. 1397, 2023a. 257219404.
Drg/CorrusII:
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshomn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Laviil, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh lang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXie, abs/ 2307.0988, 2023b. URL httph:ttt///tt :/apli api. semanticscholar wttpr/accccooooooorororraaaaa 259950998.
13

---

Published as a conference paper at COLM 2024
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pP. 4566-4575, 2014. URL https: /ttpp:://a..... www.a//scaaattnccossooooee.....e httpe/cchhaaaorrroo.oooooo orgg/orpustii 9026666.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-lmage Generation. Transactions on Machine Learning Research, 2022. URL http: //axiv.. org/an abaaaaaa.6 10789.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi BERIScore: Evaluating text generation with BERT. Proceedings of the International Conference on Learning Representations, 2020.
Appendix
In this appendix, we provide L.LM prompt templates and diagram plans used in the diagram planning stage (Appendix A), A12D-Caption collection details,(Appentt DiagramGLIGEN details (Appendix C), experimental setup details (Appendix D), human evaluation setup details (Appendix E), additional results/ /aalls//sslls////// analysis/visualizatiii visualizaaiinss caaatttiinaaalatiioon ablations details (Appendix F), and limitations (Appendix G).
B), additional
A
LLM Prompt Templates and Diagram Plans
Prompt templates. In Fig. 9 and Fig. 10, we show the prompt templates for the planner LLM and auditor LLM, respectively. For both LLMs we provide 10 in-context examples followed by the inputs. For planner LLM, we give it the caption and the topic of the caption (e.g,, astronomy, biology, etc.). For auditor LLM, we give the diagram plan generated by the planner and ask if there are any issues.
Diagram plans. A diagram plan consists of three components: (1) entities a dense list of
objects (egg,, larva in Fig. 2) and text labels (eg.,, "egg" in Fig. 2); (2) relationships complex relationships between entities (eg,, object-object relationship "[obj_0] has an arrow to [obj_1]" or object-text label relationship "[textlabel.00) label.0] labels [obj_01" (3) layouts 2D bounding
boxes of the entities (e.g,, "[obj-0]: 220, 14, 14 in Fig. 2). For object-object relationships, we utilize two types: line and arrow (a line with explicit start and end entities), which are useful when specifying object relationships in diagrams such as flow charts or life cycles. For object-text label relationships, we specify which object each label refers to. For layouts, use the [x, y, w. h] format for 2D bounding boxes, whose coordinates are normalized and integer-quantized within {0, I,.. 100}, in accordance with VPGen (Cho et al., 2023b).
Full diagram plan example. Fig. 11 shows an example of fully generated diagram plan by our LLM planner for the prompt "A diagram showing the life cycle of a butterfly, going from an egg to larva to pupa an adult butterfly and repeating." This diagram plan corresponds to the second example in Fig. 13.
LLM API Costs The average input token length for the planner stands at 4.6K, while the average output token length is 0.5K. Generating diagram plan using GPT-4 costs $0.17 USD.
B A12D-Caption Collection Details
To create the A12D-Caption dataset described in main paper Sec. 4.1, employ LLaVA 1.5 (Liu et al., 2023a), state-of-the-art multimodal language model, to generate captions and bounding box region descriptions in AI2D diagrams. The original Al2D dataset provides region annotations for diagrams, including titles, bounding boxes for object/ text labels, and object linkages (e.g. arrows/line between objects). However, since the dataset is designed for
14

---

Published as a conference paper at COLM 2024
Instructions:
Given caption diagram and topic, generote the diagram layout then list required entiles that would needed to create the described diagram. Then generate list of the relationships between the entilis (iee. which ones are connected labeling each otherl. Finally, generate location of each entit.. An entity can be an image or tnxt Entity locations should be generated [s, width, height) format, where 0,0 is the top image size.
left comer and 100,100
Think step-by-step. break each caption into parts and generate the required entites, lationships, and locations for each port.
Here are some rules to follow:
All numbers should be positive, de not generate negative numbers. always generate list entties, even the list empty.
Entities should not be outside the bounds [0, 100. 100] coordinate the width of entity should not exceed 100. The coordinate the height of an entiy should not exceed 100. Entiies of the type should not overiap.
In-context examples;
Capton:
diagram showing the pas itions of the Earth, the Moon, and the Sun during lunar eclipse. The Earth is directly between the sun and moon. The moon the earth's shadow.
Topic:
Diagram Layout:
Required Enttlie: Image
"Sun" text label (TO)
Entity Relationships: between 10 and
Entiy Locations:
10 is located at 56, 13, 28]
Input
Caption:
|insert caction)
Topic:
[insert topic of caption)
Figure 9: Input prompt given to the planner LLM during the initial diagram plan generation step.
Instructions:
Given a caption and layout of a diagram. you should determine something is wrong the diagram based on the caption. you should explain your
Think step-by-step The diagram will be described in terms of entilles in the diagram, the relationships between the entites, and the location of each entity.
why the diagram correct
An entity can be an image or text. Entity locations wil be in [x. y, width, height] format, where 0,0 the top left carner and 100,100 image size.
Here some rules the diagrams should follow:
numbers should be positive, no negative numbers.
Entilie should not be outside the bounds [0. 0, 100, 100).
The x coordinate
entity should exceed 100. The coordinate + the height of an entity should not exceed 100. Entilie same type should overlap.
width
examples;
ddaggam m plans]
What is wrong with this diagram?
"The moon phases are out of order,
Input text:
[insert diagram plan]
What wrong with this diagram?
Figure 10: Input prompt given to the auditor LLM during the diagram plan refinement step.
15

---

Published as a conference paper at COLM 2024
Required Entities:
egg image (10)
larva image (11)
pupa image (12)
adult butterfly image (I3)
"egg" text label (TO)
"larva" text label (T1)
"pupa" text label (12)
"adult butterfly" text label (13)
Entity Relationships:
I3 has an arrow to I0
T3 labels
11 has an arrow to 12
TO labels I0
I0 has arrow to I1
T2 labels
has an arrow to 13
T1 labels 11
Enfity Locations:
I0 located at [24, 50, 14, 14]
located at [50, 74, 14, 14]
is located at [74, 50, 14, 14]
13 is located at [50, 24, 14, 14]
TO is located at [20, 44, 10, 4]
T1 is located
[44, 80,
T2 is located at [80, 44, 10,
T3 is located at [44, 20, 10,
Figure 11: Example diagram plan from our LLM planner for the prompt "A diagram showing the life cycle of a butterfly, going from an egg to larva to pupa to an adult butterfly and repeating."
task rather than diagram generation, the diagram titles often too short and don't provide enough information produce meaningful diagram plans. Additionally, the dataset doesn't 't include descriptions for each object (i.e., each object bounding box is labeled simply as 'blob'). To generate captions for each AI2D diagram, we present the diagram to LLaVA 1.5 and prompt it with the question "What is this diagram showing?". To collect region descriptions of the bounding boxes in AI2D, we first overlay the bounding box annotations of each object on the diagram, by assigning each box a label (i.e., box 1 would get label "B1", , box 2 would get label "B2", etc.... Then we provide this annotated image to LLaVA 1.5 and ask the model to describe each box's s content (e.g., "what the object labeled by 'B1'"?,. An example of this annotation is shown Fig. 4.
the diagram
To ensure the quality of A12D-Caption annotations, we perform a human evaluation of 50 LLaVA 1.5 annotations. For captioning, LLaVA generates very good captions 80% of the time, and for labeling, LLaVA generates very good bounding box region descriptions 68% of the time. We find that when LLaVA makes a captioning error, usually minor points, and for bounding box region errors, sometimes it may give nearby boxes the same description. While automatic annotations have some errors, we find that they are good enough for the domain adaption of DiagramGLIGEN; However, for the test set of AI2D-Caption and in-context learning annotations for the planner LLM, we manually annotate them to ensure correctness (Sec. 4.1).
C Additional DiagramGLIGEN Details
Training.
The parameters of DiagramGLIGEN are initialized from the GLIGEN (Box+Text checkpoint) and trained the AI2D-Caption dataset (see main paper Sec. 4.1 and Ap-
16

---

Published as a conference paper at COLM 2024
pendix B for details) for 15k steps (batch size of 5 per GPU), which takes 12 hours with 8 A6000 GPUs (each 48GB memory). Other hyperparameters include:
Optimizer: A (Loshchilov & Hutter, 2019)
Learning Rate: 5e-5
Warmup Steps: 2500
Image Size: 512x512
GLIGEN gated self-attention layers. The high-level intuition of the gated self-attention layer is similar to tross-attention layers, which are used to incorporate extra information (egg, text, layouts) into the model. The cross-attention layer in the image generation backbone is used to incorporate text tokens for text-guided image generation, while the gated self- attention layer takes the grounding tokens for layout-guided image generation. Additional details are clarified in the GLIGEN paper (Li et al., 2023b).
D Experimental Setups
D.1 Metrics
VPEval (Objects, Counts, Relationships, Texts). We evaluate the diagrams in terms of the presence of objects (object evaluation), the number of objects (count evaluation), the correctness of spatial and connection relationships (relationship evaluation), and the presence of correct text labels (text evaluation) using the VPEval metric (Cho et al., 2023b). VPEval works by first generating evaluation programs that call specific evaluation modules (eg,, Object, OCR, VQA) and then running the modules to evaluate the image. Each module evaluates different parts of the image (e.g,, Object checks object presence). For object, count, and text rendering evaluation, we use the ground-truth diagram plan to determine which evaluation programs to use. For relationship evaluation, following VPEval, we use an LLM (GPT-4) to generate VQA questions that evaluate the spatial connne/ioononoonnn relationships of objects in the diagrams (e.g., if the moon is in between the sun and earth or if a light bulb is connected to a battery). While the original VPEval uses BLIP-2 (Li et al., 2023a) as its visual reasoning model, we employ the recently released LLaVA 1.5 (Liu et al., 2023a) model as our visual reasoning model, as we find it is more faithful in diagram question answering in our initial experiments and an overall stronger model than BLIP-2 (Liu et al., 2023a). In our initial experimentation, we found that LLaVA 1.5 sometimes struggles with relationship evaluation, so we fine-tune the model on relationships to ensure accurate evaluation.
For object evaluation, LLaVA determines if the object is present. For count evaluation (i.e., a diagram requires multiple instances an object), we ask LLaVA if there are exactly N instances of the object in the diagram, where N represents the count of that object in the ground-trum -oound-truth diagram. For relationship evaluation, we ask LLaVA the relation (spatial or connection) is true. In experiments, we find that LLaVA often can generate false positives during relation evaluation, as such we also show human evaluation for object relationships (main paper Sec. 4.4). For text evaluation, in alignment with VPEval, we utilize EasyOCR (AI, 2023) as the OCR model and check if the target text is detected.
E Human Evaluation Setup Details
We employ crowd-workers from Amazon Mechanical Turk (AMT) for our human preference study. To ensure high-quality annotations, we set the following requirements for the workers: they must possess an AMT Masters qualification, have completed more than 1000 HITs, maintain an approval rating above 95%, and come from the United States, Great Britain, Australia, or Canada, given that our task is in English. We pay workers $0.06 to compare two diagrams (roughly $14-15/hr). For each prompt, we show diagrams generated by both our DiagrammerGPTT and the fine-tuned Stable Diffusion v1.4 (the order of diagrams are randomly shuffled every time to prevent selection biases) and ask five annotators to indicate their preference based on (1) the accuracy of the generated relationships between objects
17

---

Published as a conference paper at COLM 2024
Instructions
diagrams describes diagram different diagrams represent
you should
1. The first criterin, Object Relationships, is measure of which diagram better captures the proper relationships (Le. spacinglposiion of them, arrows lines between them, etc.) of objects.
2. The second criterion, Alignment, messure of which diagram better representatin the sentence.
If you cannot tell which is better for criterion, then you can choose "Equal". But please use sparingly.
Your Turn
S(caption)
Diagram
Diagram B
Which dagramm has better object relationships?
Diagram
Diagram B
Equal
Which digram better aligned with the sentence?
Diagram
O Diagram B
Equal
Submit
Figure 12: Interface provided to annotators for human evaluation.
(egg,, spatial relationships and arrows/ arrows/lines) and (2) alignment to the input prompt (egg,, how well does the generated diagram reflect the input prompt). Then, we take the agreement of the annotators. The task is described to the annotators as such:
1. Object Relationships is measure of which diagram better captures the proper relationships of the objects (i.e,, them, etc.).
of them, arrows between
2. Alignment is measure of which diagram is better representation of the input sentence.
Fig. 12 shows the interface provided to the annotators during the human evaluation process.
F Additional Experiment Results
F1 Qualitative Results and Analysis
Qualitative comparison to baselines. Fig. 13 shows example diagrams generated by the baselines (Stable Diffusion v1.4 and AutomaTikZ) and our DiagrammerGPT (both diagram plan and final generation diagram) on the A12D-Caption test split. Our diagram plans strongly reflect the prompts and the final diagrams are more aligned the input prompts, In Fig. 13 top example, our diagram correctly shows the earth in four phases revolving around the sun and in the second example, our diagram plan correctly represents the life cycle of butterfly and the generated diagram captures the circular flow of the diagram plan as well most aspects of the life cycle. Stable Diffusion vl.4 either over- or under-generates objects in the image (e.g,, too many earths in the first example and missing egg/larva, larva/pupa pupa stages in the second example), and Automa fails to generate proper layouts and objects. Although our generated diagram plans generally correct, however, sometimes DiagramGLIGEN can fail to properly follow all aspects (e.g., the egg is misdrawn and the larva pupa are swapped in Fig. 13 bottom example). As noted in main paper
18

---

Published as a conference paper at COLM 2024
Prompt
Fine-tuned SD v1.4
AutomaTikZ
Diagram Plan (Ours)
DiagrammerGP PT (Ours)
showing the phases around
Earth's
tm
A diagram showing the ll cycle of butterlly, going from an egg larva to edult butterly and repeeling.
sptyllatryy
Figure 13: Example diagram generation results from baselines (fine-tuned Stable Diffusion v1.4 and AutomaTikZZ) and our DiagrammerGIT on the A12D-Caption test split. In the first example, our DiagrammerGPPt correctly gets the object count right and has clear text, whereas Stable Diffusion v1.4 overpopulates the entities orbiting around the sun. In the second example, our DiagrammerGPT generates an accurate diagram plan and diagram that mostly reflects the plan, whereas Stable Diffusion v1.4 fails show life cycle (i.e., missing the egg, pupa, and larva). As noted in main paper Sec. 5.2, once better backbone becomes available, our DiagrammerGPT can produce better diagrams based on the diagram plans. AutomalTikZ struggles to generate the proper layouts and objects for both examples.
Input Prompt
Initial Diagram Plan
After Refinement
wwich
mobrr
A diagram showing a simple circuit with switch and a motor connected to a battery.
wWhh
19/27
F22
baltrr
tukars
diagram showing the eight phases of the with labels as it revalves around Earth. also Indicates direction of the sunlight.
Figure 14: Examples from our diagram refinement step. Our auditor LLM can help reorganize the connections between the components to be more clear in the first example and prevent overlaps of objects in the second example.
19

---

Published as a conference paper at COLM 2024
Input Prompt
DALL-E 3
Diagram Plan (Ours)
DiagrammerGPT (Ours)
diegian showing tiree mas of recks. Each ros has mois The tist moe including open dfigeous besalt,
and otsidann The secord ros stowrs
DOWert PELIY
tleeent typs sestrecttry matks,
show, limestiite, dobertiee The thie ma shows di fboenn types metermrpht rocks, iecluking slow.
mertse tnckade lated ics the of rock exchise shows and eech rock.
diagram showing the layers of the earth. it includes outer cores, the mantle, the
A diagram shoving three rows of plants. The first row has cactus, sunflower, dandelion. The second row has rose, tulip, and daisy, The third row has pine tree, palm tree, and a fern. Include text labe's for each.
Examples of open-domain generation demonstrate that our DiagrammerGPT can create diagrams that adhere to the input text prompts. Although DALL-E 3 yields images with superior visual quality, it tends to generate diagrams with redundant and crowded objects and also struggles to follow the prompt accurately (egg,, in the second example, it is not clear where the locations of layers such as the 'inner core', 'outer core', and 'mantle' are. In the third example, it generates too many objects that are not in rows).
Figure 15:
Sec. 5.2, once a better backbone becomes available, our DiagramGLIGEN can produce better diagrams following the diagram plans.
Diagram plan refinement. In Fig. 14, show how our diagram refinement step (see main paper Sec. 3.1) improves the diagram plans. In the top example, the switch is not connected to the battery, thus does not affect the circuit. After refinement, the connections are corrected so the switch is now also connected to the circuit and the layouts are adjusted to have a more straightforward flow. In the bottom example, the moon phase of 'New Moon' is too low and overlaps with the 'Earth' object. After refinement, there is no more overlap.
F2 Additional Analysis
Open-domain diagram generation. Our main diagram generation experiments are con- ducted on diverse domains such as astronomy, biology, and engineering which are included in the LLM planner's 's in-context examples. However, given that the in-context examples do not encompass all diagram domains, we experiment with generating diagrams in areas not covered by our LLM in-context examples, such as geology and botany, to assess whether our DiagrammerGPT maintains ability to produce more accurate diagrams in previously unseen domains.
In Fig. 15, we show examples of comparing our open-domain diagram generation to DALL- E3 While our DiagramGLIGEN struggles in some cases (like the third example), it is able to strongly adhere to the diagram plan. Fig. 16 (bottom) also shows our LLM planner is easily able to generalize to completely new domains (e.g., neural networks and vacation planning). As mentioned in main paper Sec. 5.2, once a stronger layout-guided image generation model than GLIGEN with Stable Diffusion v1.4 backbone is available, our DiagrammerGPT
20

---

Published as a conference paper at COLM 2024
Many Enttilies
Diagram Plan
Rendered via PowerPoint
mode
Relationship Example
diegrem
showing
tiree Food
chains. The
frot one is
com,
mouse,
qwl The
second one
is comotto
rhott
tox, to lion. The third one
is grass, a grasshopper, to frog. snake, to hawk.
Fight
Inser
diagrem network with nodes, hidden nodes, and node
A diegram infographic showing take vecation, include visuelize the
Open- Domain Diagram
kg
Figure 16: Additional examples. Our planner LLM is effective at generating dense (top) and open-domain (bottom) diagram plans.
can produce higher quality results. We find that DALL-E 3 generally produces images with good aesthetic style but tends to generate diagrams with redundant and crowded objects (egg,, excessive unnecessary text descriptions in the rock and Earth examples, and overabundance of plants in the third example). It also continues to struggle with creating accurate diagrams that adhere example and generating three columns of plants instead of three rows in the plant example). The DALL-E system card (OpenAT, 2023a) also notes that DALL-E tends generate scientifically inaccurate information in diagrams.
prompt (e.g,, generating incorrect layers in the earth
Vector graphic diagram generation in different platforms. We render our diagram plans in Microsoft PowerPoint via VBA language, Inkscape via Python scripting extension and Adobe Illustrator' via JavaScript. We represent objects using icons, which are retrieved via the Noun Project Icons API based on corresponding text descriptions." Fig. 16 (top) and Fig. 18 show additional examples of diagram plans rendered in the other platforms.
GPT-4 vs. GPT-4Vision for diagram plan creation and refinement. As described in main paper, our DiagrammerGPT employs a text-only GPT-4 model for diagram planning and refinement. To explore whether multimodal language model can offer improvements over text-only GPT-4, we experiment with using the recently introduced GPT--Vissio (GPT-4V) model (OpenAI, 2023c) the planner and auditor LLM during the diagram generation and refinements steps. As the GPT-4V model does not provide API access yet, we conduct small-scale qualitative study via the ChatGPT web UI. In our experiments, for the diagram plan creation stage (see main paper Sec. 3.1), GPT-4V does not provide improvements over
"hccps://eearn.niccrot www.roosttoooooooeooooeoeeoeeeoe http://tw...ccaooeeeeeee/eee///e
hhttps:/iinkscape.
'nttps://github. fattppgs//gthan....oooaaaaaaappp
7 /https://www adobe. aabbb...a/prrdicct//iuuseeteeeee html
hactpa//ttenonngrojett.coo
21

---

Published as a conference paper at COLM 2024
Input Prompt: a diagram showing the first four planets of the solar system.
Farth
OF
Venu
Exit
Rin
Generated Diagram Plan 1
Generated Diagram Plan 2
Figure 17: Example of two GPT-4 generated diagram plans. Given the same prompt, GPT-4 can generate diverse plans between runs.
Rendered Microsoft PowerPoint
Rendered with Inkscape
Rendered Adobe
daprom
leod The lell lood chan, slariang forn the goes fur
too. righttedd chain, startang fur the
botom, ases form algas, to
craylih, allgator
degram showing the eight presess of moon wound Barth It also insicates the dirzetion of the sunight
Figure 18: Examples of vector graphic diagrams generated with our diagram plans and exported into Microsoft PowerPoint, Inkscape, and Adobe Illustrator.
text-only GPT-4. In Fig. 19, we present a comparison between the diagram plans generated by GPT-4 and GPT-4V. GPT-4V does not produce diagram plans that are better than text- only GPT-4, suggesting that our text-only representation robust enough until better or fine-tuned versions of GPT-4V become available for diagrams. Similarly, during the diagram refinement step (see main paper Sec. 3.1), we observed that GPT-4V tends to overestimate correctness when compared to text-only GPT-4, further indicating the strength of our text representation. Fig. 20 shows two examples comparing the models. While text-only GPT-4 is not perfect, it can identify some errors, whereas GPT-4V says the diagram does not need improvement.
Using model based text rendering instead of Pillow. We also experiment with using a model-based text renderer, TextDiffuser-2 (Chen et al., 2023b) instead Pillow. Fig. shows that while TextDiffuser-2 is capable of producing good text labels, however, it can sometimes merge letters (e.g., the "mm" in summer). Pillow guarantees there no rendering error (and can easily allow font color/size adjustments). Due to the modular nature of
22

---

Published as a conference paper at COLM 2024
Input
GPT-4 (Default)
GPT-4Vision
A diagram snowing an electrical circutt with light bulb, bettery, and a switch.
carrt
sertery
Dateery
diagram showing the life cycle of grasshapper and mealworm oeetle,
rymat
gruat uppir
Illsstating transfrmation they undergo during development. The grashhapee goes from an egg, to nymphh to an adultan and then the meal worm beetle egg, pupa.
26:0
6647
Figure 19: Comparison examples of diagram plans generated by GPT-4 and GPT-4Vision (GPT-4V). GPT-4 creates diagram plans that are sufficiently accurate capturing the pres- ence of objects and their relationships and GPT-4V does not provide plans that are better.
Input Prompt
Initial Diagram Plan
Generated Feedback
arrows between each phase to indicate the progression of the life cycle.
diagram showing the life cycle ladybug, specifically focusing on the different stages from eggs adult.
luna
GPT-4Vision:
The diagram correctly aligned with caption, and all entitiss are in circular layout.
+30
GPT-4: The entities in the diagram connectdd in lnop, which is for circuit.
A diagram showing simple circuit with switch and a mator connected battery.
GPT-4Vislon: From the given information, nothing seems to be wrong with diagram. adheres all the rules and carrectly represents the caption.
Figure 20: Comparison examples of GPT-4 vs. GPT-4Vision for diagram plan refinement. While text-only GPT-4 not perfect, it can identify the errors, whereas GPT-4Vision says the diagram does not need improvement.
DiagrammerGPT, using text rendering model can easily be incorporated if the end user wants.
Is the LLM capable of generating diverse diagram plans? We find that having an LLM generate the diagram plans results in a wide diversity of diagrams. As we show in Fig. 13, 14, 15, and 16, the LLM can generate plans for many different prompts and domains. In Fig. 17 we show an example of the LLM generating variations the same prompt, further indicating the LLM (GPT-4 in case) is capable of producing diverse diagram plans.
23

---

Published as a conference paper at COLM 2024
Our Explicit Text Rendering
Text Rending via TextDifluser-2
Sumeee
STPR
Au
Winter
Figure 21: Explicit text rendering via Pillow compared to model-based text rendering via TextDiffuser-2 (Chen et al., 2023b).
F3
Ablation Studies
We show ablation studies design choices with DiagramGLIGEN: the number of denoising steps with layout guidance and whether to only update the gated self-attention layers parameters.
VPEval
# Layout guidance steps
Object
Count
Text
Relationships
a 5 steps
88.0
50.6
48.0
84.9
i 10 steps
86.4
49.4
47.0
86.6
a 15 steps (default)
86.4
57.0
47.5
87.9
Table 5: Ablation of denoising steps with layout guidance. DiagramGLIGEN uses 50 denoising steps in total. We use a 15 steps as our default setting.
Number of denoising steps with layout guidance. The number of denoising steps with layout guidance (i.e., with the gated self-attention layer activated in each transformer block of the diffusion UNet), denoted as is a crucial hyper-parameter in DiagramGLIGEN, A larger value indicates stronger layout control. Table 5 presents ablation study using varying & values. A smaller & value enhances object generation, while a larger value improves count performance. This observation aligns with intuition: rigorous layout control more effectively prevents the generation of extraneous objects in the background but may detract from the visual realism of the generated objects, which is also observed in (Li al., 2023b; Lin et al., 2023). We set the default value for & as 10 steps, as it ensures good balance of accuracy for objects and counts while achieving optimal performance in depicting object relationships.
VPEval
Captioning BERTScore
CLIPScore
Updated parameters
Overall
Img-Txt
None
68.5
88.9
29.3
GatedSA Layers only
67.4
88.9
30.0
All Layers (default)
71.2
89.4
32.1
Table 6: Ablation of fine-tuning different layers of DiagramGLIGEN. We use the fully fine-tuned model our default setting. GatedSA: Gated Self-Attention.
24

---

Published as a conference paper at COLM 2024
VPEval
Captioning
CLIPScore
Diagram Plan Source
Overall
BERTScore
Img- Txt
Ground-truth + DiagramGLIGEN (oracle)
81.9
89.8
32.3
GPT-4 DiagramGL.GHN
71.2
89.4
32.1
Table 7: Ablation of using ground-truth plans from AI2D-Caption (egg, oracle performance) instead of GPT-4.
Fine-tuning: all layers vs. layout layers. In Table 6, we present an ablation study compar- ing the fine-tuning of only the layout control layers in DiagramGLIIGN with fine-tuning of the entire model, including the Stable Diffusion backbone. Full fine-tuning enhances performance and improves the visual quality of the diagrams. Therefore, we employ the fully fine-tuned version as our default model for all subsequent experiments.
Using ground-truth diagram plans. We experiment with generating diagrams using ground-truth diagram plans from AIDD-Caption. Doing this allows to measure the upper bound of DiagramGLiGEN and see how much room our stage 1 has for improvement. Table 7 shows that using ground-truth plans does indeed do better than GPT-4 and that DiagramGLIGEN is able to perform better when using ground-truti plans. However, it is interesting to know that using GPT-4 performs very closely to the oracle score.
G Limitations
Our framework can benefit many educational applications, such as presentation/pa /paper creation, and human-in-the-loop diagram generation/ modiificaiio... However, akin to other text-to-diagram/ teaeddiirrat/eet-t--iiiaaa text-to-image generation frameworks, our framework can also make some errors and be utilized for potentially harmful purposes (e.g., creating false information or misleading diagrams), and thus should be used with caution in real-world applications (with human supervision, e.g., described in Sec. 5.4 human-in-the-low editing). Also, generating diagram plan using the strongest LLM APIs can be costly, similar to other recent L.M--based frameworks. We hope that advances in quantization / /ditiillation and open-source models will continue to lower the inference cost of LLMs. Lastly, Dia- gramGLIGEN is based on the pretrained weights of GLIGEN and Stable Diffusion v1.4. Therefore, we face similar limitations to these models, including deviations related to the distribution of training datasets, imperfect generation quality, and only understanding the English corpus.
diagram plan
25


Reviews:
[{"review_id": "Vf4lgwKDxW", "reviewer": "Reviewer_Lutp", "summary": "This paper presents DiagrammerGPT, which leverages large language models (LLMs) to produce text-visual diagrams. With the powerful capability of layout planning, they first derive and refine a diagram plan via LLMs. Then, they can follow the plan to generate the resulting diagram via the layout-grounded diffusion model.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Below we address your concerns with further clarifications.\n\n**W1. Paper contributions:**\n\nWe would like to point out that the previous works mentioned do not tackle diagram generation, which requires fine-grained/dense layout and relationship control. They also do not use iterative refinement. Below, we summarize our strong contributions:\n\n1. A new text-to-diagram generation framework that leverages the knowledge of LLMs for planning & refining the open-domain diagrams (Sec. 3), which is more effective than existing T2I methods (Table 1 & 3; Fig. 4 & 5).\n2. Iterative self-refinement can help create better diagram plans (Table 2; Fig. 13).\n3. Our diagram plans can be rendered via any vector graphic software (Fig. 6, 15, & 16) and T2I models, allowing for easier uses and human-in-the-loop editing (Fig. 7).\n4. New dataset (AI2D-Caption; Sec. 4.1) with evaluation suites (VPEval + LLaVA 1.5 finetuned on diagrams; Sec. 4.3) that can help the community benchmark and improve the text-to-diagram methods.\n5. Improved control of text labels/entity relations (Sec. 3.1).\n\n**W2. Iterative refinement effectiveness:**\nIn Table 2, we show that refinement can help improve object relationships (4.56 $\\rightarrow$ 4.72). We find the initial diagram plans are mostly correct, and the refinement cleans minor details. Fig. 13 also shows examples of how refinement helps.\n\n**W3. Can LLMs generate diverse diagram plans?:**\nYes, LLM can generate meaningful variations from the same prompt, as in this **[example link](https://shorturl.at/UpKig)**. We also show that the LLM can generate plans in diverse domains (Fig. 4-7, 10, 12-16). The backbone T2I model is still free to generate the diagrams in any art style, allowing further diversity.\n\n**W4. Visual quality:**\nWe agree that the visual appeal of the diagrams can be improved. However, please note that we also proposed diagram rendering via vector graphic software, such as PowerPoint (Sec. 5.4; Fig. 6, 15, & 16). This can create high-quality diagrams that are more visually appealing and easy to adjust (Sec. 5.4; Fig. 7). \n\nIn addition, the current backbone is SD v1.4, and we expect that a bigger backbone (e.g., SDXL, SD3) would yield more visually appealing results. However, such scaling is expensive for our academic lab, and we focus on introducing the first LLM-based diagram generation framework. Our methods' flexibility allows adapting to bigger models with sufficient GPU resources.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and easy to follow.\n- The targeted text-to-diagram is a novel task, which is a crucial application of the general text-to-image synthesis (T2I).\n- The proposed iterative refinement is well-motivated, and the grounded diffusion can also facilitate the output quality.\n- They provide comprehensive human evaluations as well as qualitative examples.", "reasons_to_reject": "- Since there are already previous works [1,2,3] adopting LLMs to improve layout planning, the novelty of this paper can be an issue. In addition, the proposed DiagramGLIGEN is an extension of the existing GLIGEN [4].\n- Although the iterative feedback seems to be helpful, there should be an empirical study to show the effectiveness of this refinement.\n- One powerful feature of current T2I is the diversity generation. Can LLMs also produce diverse yet meaningful diagram plans? And can those layouts further generate different visual results? Will this text-to-layout technique, which helps better controllability, hurt the diversity of T2I?\n- From those presented visualized cases, they look not that visually appealing for actual human usage (correct me if this is not true). I am worried about the real-world practicality with the current pipeline. \n\n**Reference**\n- [1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models\n- [2] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models\n- [3] VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning\n- [4] GLIGEN: Open-Set Grounded Text-to-Image Generation", "questions_to_authors": "Please see Reasons To Reject", "extra_scores": {}}, {"review_id": "mUSql6reEd", "reviewer": "Reviewer_GeX9", "summary": "The paper proposes DiagrammerGPT, a method to generate diagram images using a combination of LLMs and diffusion models. The method first generates layouts of the objects, including bounding boxes, paired text annotations, and arrows or lines between objects. This stage involves in-context learning and self-correction loops to refine the layouts. Then a GLIGEN-based layout-to-image model is fine-tuned and used for generating the diagram image based on the layouts. A new dataset named AI2D-Caption is generated based on an existing diagram dataset for the task and experiment. Experimental results show a clear improvement of the method over existing baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Below we address your concerns with further clarifications and experiments.\n\n**W1. Paper contributions:**\nWe would like to remind you of the strong contributions of our project:\n\n1. We present a new text-to-diagram generation framework that leverages the knowledge of LLMs for planning & refining the open-domain diagrams (Sec. 3), which is more effective than existing T2I methods (Table 1 & 3; Fig. 4 & 5).\n2. We show that iterative self-refinement can help create better diagram plans (Table 2; Fig. 13).\n3. We show that our diagram plans can be rendered via any vector graphic software (Fig. 6, 15, & 16) as well as T2I models, allowing for more flexible real-world use (e.g., for teachers) and human-in-the-loop editing (Fig. 7).\n4. We produce a new dataset (AI2D-Caption; Sec. 4.1) with evaluation suites (VPEval + LLaVA 1.5 finetuned on diagrams; Sec. 4.3) that can help the community benchmark and improve the text-to-diagram methods.\n5. We train DiagramGLIGEN to have improved control of text labels/entity relations  (Sec. 3.1).\n\n**W2. VPGen additional analysis:**\nOur improvement over VPGen primarily comes from stage 1. Following your suggestion, we compare fine-tuned VPGen to our DiagrammerGPT in the first stage.\n\n|  |  | Layout Recall |  |\n|-|:-:|:-:|:-:|\n|  | Object | Text | Overall |\n| Fine-tuned VPGen | 30.6 | 0 | 12.9 |\n| DiagrammerGPT (ours) | **84.1** | **60.1** | **78.4** |\n\nLayout recall compares GT diagram plans to the generated diagram plan.\n\nAs shown in the table, the fine-tuned VPGen struggles to produce the required entities for a diagram, while DiagrammerGPT does better.\n\nThe reason the fine-tuned VPGen does worse sometimes compared zero-shot VPGen is likely due to the diagram planning stage. After fine-tuning, VPGen generates objects in the right domain (as reflected by the increased object score 55.8 $\\rightarrow$ 62.8 in the main paper Table 1). However, potentially due to limited training data, it often generates too many objects, overlaps objects, etc. which leads to lower scores in certain metrics.\n\n**Q1. Arrow/line annotations:**\nThe original AI2D dataset labels which objects are connected via a line/arrow.\n\n**Q2. Fine-tuned GLIGEN in VPGen vs. DiagramGLIGEN:**\nWe fine-tune the GLIGEN model in VPGen like we do for DiagrammerGPT to give a fair comparison. DiagrammerGPT still has a different diagram planning stage than VPGen and also explicitly renders text labels, whereas VPGen does not.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem of diagram creation as image generation is interesting and has practical values. Treating it as an image generation has some advantages over program-based rendering like image diversity, flexible control, and having open-domain objects. \n- The proposed method, while similar to existing ones, involves arrows or lines for object relation in the first stage, which introduces more fine-grained elements to the layouts. The method also largely outperforms existing baselines, showing its effectiveness. The proposed dataset is also useful for future work.", "reasons_to_reject": "- Technical novelty is limited. There are some new elements like the use of arrows/lines or the iterative refinement process in using LLMs to generate layouts. But most of the techniques are not new like the in-context learning design or DiagramGLIGEN (basically a simple variant of GLIGEN). \n- Lack of in-depth experiment/analysis. While I appreciate the significant improvement shown in Table 1&2, it is not quite obvious where the improvement comes from. Why would the fine-tuned VPGen get decreased performance in count, CIDEr, and BERTScore compared to its zero-shot version? Besides, does the improvement of DiagrammerGPT over fine-tuned VPGen come from the first or second stage? I think it would be better to compare the first stage performance of fine-tuned VPGen and the proposed method.", "questions_to_authors": "- Does the AI2D dataset originally include arrows/lines between objects? If not, how did you get the annotation for the dataset?\n- Could you clarify the difference between fine-tuned GLIGEN in VPGen and the DiagrammarGLIGEN? For input, GLIGEN is like box+text while DiagrmmarGLIGEN is box+text+relation? Is GLIGEN also fine-tuned on all layers like in Table 5?", "extra_scores": {}}, {"review_id": "aXwB5En6LE", "reviewer": "Reviewer_SF1t", "summary": "This paper presents a text-to-diagram approach named DiagrammerGPT. It uses a two-step process to create open-domain diagrams from textual prompts, where GPT4 is prompted to generate a diagram plan followed by a visual generator creating the actual diagram from the generated diagram plan. \n\nInitially, the authors use GPT-4 to create diagram plans from text prompts during the diagram planning stage, which includes listing entities, their relationships, and layouts. Afterward, leveraging recent advances, another Language Model acts as an auditor to identify errors such as misplaced objects. The plan is then refined based on this feedback to correct inaccuracies like incorrect sizes, resulting in notably accurate LLM-generated diagram plans with minimal errors. \n\nDuring stage two, a new model called DiagramGLIGEN is designed for creating diagrams from plans and clearly rendering text labels for legibility. It's built upon the GLIGEN framework and trained using the AI2D-Caption dataset, which includes gated self-attention layers in the Stable Diffusion v1.4 model. To overcome the difficulty of text rendering in existing T2I models, the authors directly overlay text on diagrams to ensure clarity. \n\nBoth human evaluation and LLaVA/ClIP based evaluation show that the proposed approach can generate diagrams with better quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Below we address your concerns with further clarifications.\n\n**W1. In-context details:**\nFor open-domain diagram generation (Sec. 5.4 / Fig. 5 &14), we use all 30 in-context examples mentioned in Sec. 4.1. These examples do not change (i.e. static). We find that the LLM (GPT-4) can generalize these examples into other domains well (see Fig. 5, 14, & 15). We will clarify this in the paper. \n\n**Q1. GLIGEN details:**\nThe high-level intuition of the gated self-attention layer is similar to cross-attention layers, which are used to incorporate extra information (e.g., text, layouts) into the model. The cross-attention layer in the image generation backbone is used to incorporate text tokens for text-guided image generation, while the gated self-attention layer takes the grounding tokens for layout-guided image generation. \n\nAdditional details are clarified in the GLIGEN paper (Li et al., 2023b). However, following your suggestion, we can clarify the important architectural details in our paper.\n\n**Q2. AI2D-Caption details:**\nFor the test set of AI2D-Caption, we randomly choose 25 diagrams from astronomy (e.g., solar system, etc.), biology (e.g., life cycles, etc.), and engineering (e.g., circuits, etc.) for a total of 75 diagrams. We will add the selection criteria to the paper.\n\n**Q3. Human evaluation instructions:**\nHere is how we describe the image-text alignment and object relationships to annotators:\n\n- Object Relationships is a measure of which diagram better captures the proper relationships of the objects (i.e. spacing/positioning of them, arrows/lines between them, etc.).\n- Alignment is a measure of which diagram is a better representation of the input sentence.\n\nThe annotators are then shown a diagram from DiagrammerGPT and SD v1.4 and provided 3 options: Diagram A, Diagram B, and Equal. The diagrams are _randomly shuffled_ so there is no pattern between which model produces diagram A or B.\n\nWe will add a screenshot of the setup to our paper to make sure these details are clarified.\n\n**Q4. Vector graphic generation implementation:**\nWe take the diagram plan generated by the LLM and convert it into a format that is accepted by the related vector graphic platform (Appendix F.2; e.g., VBA for PowerPoint) via a script. Objects are represented using icons automatically pulled from the public icon library “The Noun Project” (Appendix F.2). We will release all related code for render vector graphic rendering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Text-to-diagram is a relatively new topic, and it is very useful in practice. \n2. The proposed two-stage approach is technically reasonable.", "reasons_to_reject": "Some important details are missing. For example, the quality of generated plan largely depends on the in-context examples. However, the authors do not introduce how to pick up a set of in-context examples in an open-domain scenario. Are the set of in-context examples selected on-the-fly or it is a static set?", "questions_to_authors": "As I mentioned above, a lot of important details are missing. Besides the forementioned in-context examples, here are a list of points need to be further clarified:\n- The description of the model in Section 3.2 is somehow vague. I suggest the authors can provide a detailed model architecture diagram about GLIGEN and how the new gated self-attention layer is attached. The authors need to explain more about the intuition behind adding the gated self-attention layer, as this is an important model contribution. \n- AI2D-Caption is an important part of the paper. The authors state that they select 75 diagrams. What are the selection criteria?\n- Human evaluation for pairwise preference is mentioned at Section 4.4. Please describe how the labelling task is described to annotators to avoid potential bias, as this is important for the faithfulness of the result. \n- For the vector graphic diagram generation, it is not clear how this is implemented. The current version is not replicable.", "extra_scores": {}}, {"review_id": "ajwnQJVGYq", "reviewer": "Reviewer_Bhqq", "summary": "This paper introduces DiagrammerGPT, a novel framework for the challenging task of diagram generation. The authors developed AI2D-Caption, a new dataset to benchmark text-to-diagram generation. DiagrammerGPT employs a two-stage process: the first stage uses LLMs to create and refine 'diagram plans' through a planner-auditor feedback loop, while the second stage utilizes DiagramGLIGEN and a text label rendering module to generate diagrams based on these plans. The framework also supports features such as open-domain generation, vector graphic creation, human-in-the-loop editing, and multimodal planning.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Below we address your concerns with further clarifications & experiments.\n\n**W1. Ablation studies:**\nWe would like to bring your attention to Appendix F.3, where we present ablation studies for different number of denoising steps, choosing which layers to fine-tuning for DiagramGLIGEN, and different LLMs for planning.\n\n**W2. Comparison to diagram generation from GT layouts:**\nFollowing your suggestion, we generate diagrams from GT diagram plans and present the results here:\n\n|  |  |  |  |\n|-|:-:|:-:|:-:|\n|  | VPEval Overall | BERTScore | CLIP Img-Txt |\n| GPT-4 + DiagramGLIGEN | 71.2 | 89.4 | 32.1 |\n| GT Plans + DiagramGLIGEN (Oracle) | 81.9 | 89.8 | 32.3 |\n\n\nFrom this, we see that using GT diagram plans leads to higher overall performance, however, using diagram plans from our first stage is not too far behind. This is also reflected in Table 2 where we show that humans label GPT-4 diagram plans very highly (4.96 / 5 object presence and 4.72 / 5 object relations).\nThanks for the suggestion, and we will add the results to the paper.\n\n**W3. Different LLMs:**\nIn Appendix F.3, we present layout generation experiment results from GPT-3.5 as well. As mentioned in the section, we also tried LLaMA2-Chat (13B) via both zero-shot and fine-tuning, but we found it does not perform well, often missing objects or failing to generate meaningful plans.\n\nWe also just tried the recently released LLaMA3 (8B) model but even with fine-tuning, it fails to produce good diagram plans, often just repeating the same object, missing objects, or sometimes failing to produce meaningful plans. We summarize the result below:\n\n\n|  |  | Layout Recall |  |\n|-|:-:|:-:|:-:|\n|  | Object | Text | Overall |\n| LLaMA3 8B | 24.9 | 32.2 | 29.2 |\n| GPT-3.5 Turbo | 82.5 | 54.6 | 74.7 |\n| GPT-4 (default) | **84.1** | **60.1** | **78.4** |\n\n\nLayout recall checks whether certain objects/texts present in GT diagram plans are present in the generated diagram plan.\n\nThe LLaMA models may potentially be suffering from the limited training data.\n\n**Q1. In-context details:**\nFor each of the three domains (astronomy, biology, engineering), we use 10 in-context examples for each prompt. For open-domain prompts, we use all 30 in-context examples. The examples chosen for in-context use were randomly selected from the AI2D dataset.\n\n**Q2. Figure enlargement:**\nWe will try to increase the text label/figure size where needed to improve readability. Thanks for the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) **Novelty and Relevance**: \nThis paper addresses a unique and challenging problem in diagram generation, presenting a fresh perspective on a multimodal task that encompasses diverse challenges such as commonsense reasoning, layout planning, and image generation.\n\n(2) **Initial Contributions**: \nAs a pioneering work in a new research area, this paper conducts foundational experiments demonstrating the effectiveness of the proposed method, including LLM planning with iterative feedback and the DiagramGLIGEN module.\n\n(3) **Clarity and Structure**: \nThe manuscript is well-structured and clear, with effective use of illustrations that enhance the understanding of core concepts and methodologies, making the proposed method more accessible to readers.", "reasons_to_reject": "(1) **Lack of Ablation Studies:** The paper does not include sufficient ablation studies to analyze the various factors influencing the performance of this complex multimodal task, making it difficult to fully understand the strengths and limitations of the proposed pipeline (further details are provided in subsequent points).\n\n(2) **Lack of Baseline Comparison:** Table 1 presents results based on constraints generated by the LLM in stage 1, but lacks a baseline comparison using ground truth diagram planning annotations. Including this baseline would clarify the T2I performance with accurate diagram plans.\n\n(3) **Limited Discussion on Diagram Planning:** The paper discusses the performance of DiagramGLIGEN in stage 2 but lacks a thorough exploration of the diagram planning in stage 1. Specifically, there is no discussion of related work in the planning stage, and only GPT-4 results are presented. It would be beneficial to evaluate other LLMs for this task and consider if state-of-the-art models in procedure planning could be applicable to diagram planning.", "questions_to_authors": "Many questions are raised in the weakness section, please refer above.\n\n(1) In-Context Learning Details: The authors mentioned annotating 30 diagrams for in-context learning. How many diagrams were used to prompt each example? How were the supportive examples for ICL selected—randomly or using similarity functions?\n\n(2) Figure Clarity (Comment): This won't affect my rating, but some figures have text that is difficult to read due to its small size. It would be beneficial to enlarge the figures or at least the text labels for better readability.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes DiagrammerGPT, a two-stage framework to generate diagrams from text using LLMs.  The framework first generates a plan (roughly layout of where diagram elements  go) and then generate the diagram from the plan, and adding text labels as a last step.  For plan generation, an iterative refinement strategy using the LLM is proposed.  For diagram generation from the plan, the work uses DiagramGLIGEN - a version of the GLIGEN diffusion model trained on diagrams with captions.  To train DiagramGLGEN, the paper introduces AI2D-Caption, which add captions to an existing dataset of diagrams from AI2 (AI2D) using LLaVA 1.5.  For evaluation and in-context learning, a small subset is further manually annotated.   Experiments against baselines show the proposed framework can generate diagrams that score higher on automated metrics and are also preferred more by humans.

Reviewers are favorable toward the work as the problem it addresses (text-to-diagram) is novel and under-studied.  They also found the paper to be well-written, easy-to-follow, with a well-motivated approach, and convincing experiments and results.  All reviewers are positive, and the AC also agrees the work presents an novel task that would be of interest to the community.  Thus the AC recommends acceptance.

The AC encourages the authors to incorporate feedback from the reviewers and clarifications from the rebuttal for the camera ready, including:
- VPGen analysis (GeX9) and comparison to using GT layout and different LLMs (Bhqq)
- Details for in-context examples, GLIGEN, AI2D-Caption selection, and human evaluation instructions (SF1t)
- Other clarifications and suggestions by reviewers
Decision: Accept