AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ
Authors: Jonas Belouadi
Anne Lauscher
Steffen Eger
Venue: ICLR 2024 poster (2024)
Keywords: Vector Graphics Generation
Code Generation
Science Generation
TikZ
Text-to-Image
Primary area: generative models
TL;DR: We train large language models on TikZ code, conditioned on captions, to automatically generate scientific vector graphics.

Abstract: Generating bitmap graphics from text has gained considerable attention, yet for scientific figures, vector graphics are often preferred. Given that vector graphics are typically encoded using low-level graphics primitives, generating them directly is difficult. To address this, we propose the use of TikZ, a well-known abstract graphics language that can be compiled to vector graphics, as an intermediate representation of scientific figures. TikZ offers human-oriented, high-level commands, thereby facilitating conditional language modeling with any large language model. To this end, we introduce DaTikZ the first large-scale TikZ dataset, consisting of 120k TikZ drawings aligned with captions. We fine-tune LLaMA on DaTikZ, as well as our new model CLiMA, which augments LLaMA with multimodal CLIP embeddings. In both human and automatic evaluation, CLiMA and LLaMA outperform commercial GPT-4 and Claude 2 in terms of similarity to human-created figures, with CLiMA additionally improving text-image alignment. Our detailed analysis shows that all models generalize well and are not susceptible to memorization. GPT-4 and Claude 2, however, tend to generate more simplistic figures compared to both humans and our models. We make our framework, AutomaTikZ, along with model weights and datasets, publicly available.

Published as a conference paper at ICLR 2024
AUTOMATIKZ: TEXT-GUIDED SYNTHESIS OF SCIENTIFIC
VECTOR GRAPHICS WITH TIKZ
Jonas Belouadi
Anne Lauscher
Natural Language Learning Group Bielefeld University, Germany jonas.
Data Science Group
University of Hamburg, Germany
anne. anne. www.cherucuuuniiiee Shatuuurrihamuurggggg
Steffen Eger
Natural Language Learning Group University of Mannheim, Germany steffen. egerêuni mmannheimmmmmmmmmm de
ABSTRACT
Generating bitmap graphics from text has gained considerable attention, yet for scientific figures, vector graphies are often preferred. Given that vector graphics are typically encoded using low-level graphics primitives, generating them directly is difficult. To address this, we propose the use of TikZ, a well-known abstract graphics language that can be compiled to vector graphics, an intermediate representation scientific figures. TikZ offers human-oriented, high-level commands, thereby facilitating conditional language modeling with any large language model. To this end, we introduce DATuKZ, the first large-scale TikZ dataset consisting of 120k TikZ drawings aligned with captions. We fine-tune LLAMA on DATixZ, as well as our new model CLIMA, which augments LLAMA with multimodal CLIP embeddings. In both human and automatic evaluation, CLiMA and LLAMA outperform commercial GPT-4 and CLAUDE 2 in terms of similarity to human- created figures, with CLIMA additionally improving text-image alignment. Our detailed analysis shows that all models generalize well and are not susceptible to memorization. GPT-4 and CLAUDE 2, however, tend generate more simplistic figures compared to both humans and our models. We make our framework, AUTOMA TiKZ, along with model weights and datasets, publicly available."
INTRODUCTION
Recent advancements in text-to-image generation have facilitated the generation of detailed images from simple natural language descriptions (Esser et al., 2021; Ramesh et al., 2021; 2022; Saharia al., 2022; Rombach et al., 2022; Zhang et al., 2023a). Models like Stable Diffusion (Rombach et al., 2022) and DALL-E (Ramesh et al., 2021; 2022) often yield results comparable to real photographs human-created artworks. However, these models primarily generate low resolutions, which are not ideal for scientific figures. Researchers use scientific figures to convey complex ideas or present critical findings, making them central to scientific research (Tufte, 1992; Hsu et al., 2021). Consequently, they demand a high degree of geometric precision and legible text, even at small font sizes, areas where raster graphics fall short. As result, many research conferences advocate the use of vector graphics, 2 which decompose information into geometric shapes, allow searchable text, and usually have smaller file sizes.
graphics, typically
Automated vector graphics generation is growing research area well (Lopes et al., 2019; Carlier al., 2020; Aoki Aizawa, 2022; Ma et al., 2022: Frans al., 2022: Jain et al., 2023; Wu et al., 2023), but current methods have their own share of limitations, Specifically, they mainly generate low-level path elements of the Scalable Vector Graphics (SVG) format, either failing to maintain
/https://github. http:///ppoaaaiieestttttaaaaaaaa >https://acl-org. git.gggttttb...... httpaa/./..c...////aaaaatittiia.

---

Published as a conference paper at ICLR 2024
3D contour plot of a loss function, show- casing global and local minima. The color gradient indicates function depth, providing insight challenges in machine learning.
optanization
I CLIMA
Visual representation multi-layer pereeptron: an intercomeetedd network of nodes, showcasing structure input, hidden, and ougpu layers that facilitate complex pattern recognition.
LLAMA
T CLIP
Bur chart comparing BLEU scores of ALPRA, BETA, and GAMMA models across ORION, NERULA, and PULSAR datusets, with ALPHA consistenlly leading.
Exemplary scientific figures generated with CLIMA. CLIMA takes the captions as input, processes them with CLIP and LLAMA, and generates TikZ drawings that compile to vector graphies.
Figure I:
accurate geometric relations (Ma al., 2022; Frans al., 2022: Jain al., 2023) only generating outputs of limited complexity such as single icons or font characters (Lopes et al., 2019; Carlier et al., 2020; Aoki & Aizawa, 2022; Wu et al., 2023).
To address these limitations, we explore the use of graphic languages, which abstract from lower-level vector graphics formats by providing high-level constructs that can be compiled to such formats (Van Zandt, 2007; Hobby, 2014; Tantau, 2023). Language models show potential in learning these languages to solve simple tasks (Bubeck al., 2023; Zhang et al., 2023b), but the depth of this capability, i.e., whether it can produce scientific figures, remains unexplored. Due to its expressiveness and emphasis on science, which enables the creation of complex figures with only a few commands, we focus on the graphics language 7ikZ in this work (Tantau, 2023). We aim to understand whether language models can capture the nuances of TikZ and automatically generate scientific figures based on image captions, analogous to text-to-image generation. This could not only enhance productivity and foster inclusiveness (aiding researchers less versed in programming-like languages, such as social scientists), but also aid education by creating tailored TiKZ examples. The use case for this is demonstrated by the TąX Stack Exchange3, where nearly 10% of the asked questions pertain to TikZ, making it the most frequently discussed topic on the platform. Our key contributions are follows:
(i) As part of our AUTOMATIKEE project, we create DATIIZ, the first large-scale TikZ dataset to our knowledge, featuring approximately 120k paired TikZ drawings and captions.
(ii) We fine-tune the large language model (LLM) LLAMA (Touvron et al., 2023a) DATIKZ and compare its performance to general-purpont LLMs, specifically GPT-4 (OpenAI, 2023) and CLAUDE 2 (Anthropic, 2023). Both automatic and human evaluation agree that scientific figures generated by fine-tuned LLAMA resemble human-created figures more closely.
(iii) We further develop CLiMA, variant of LLAMA augmented with multimodal CLIP embeddings (cf. Figure 1; Radford et al., 2021). This enhancement allows CLiMA to visually interpret input captions, thereby improving text-image alignment. It also enables the use of images as supplementary inputs, leading to a further boost in performance.
(iv) In addition, we demonstrate that all models exhibit few memorization problems and generate novel outputs. However, GPT-4 and CLAUDE tend to generate simpler outputs than LLAMA and CLIMA, sometimes producing degenerate solutions that maximize text-image similarity by visibly copying the input caption into the output image.
2 RELATED WORK
Our work connects to several distinct but interrelated fields, namely text-to-image generation, vector graphics generation, scientific figure understanding, and code generation. For each field, we provide comprehensive review of the most relevant prior work.
hnttps//tee.staccecchngg.e stackexchaage..com
2

---

Published as a conference paper at ICLR 2024
Text-to-Image & Vector Graphies Generation The evolution of text-to-image generation can be characterized by three development stages: Generative Adversarial Networks (Reed et al., 2016; Zhang et al., 2017; Brock et al., 2019; Kang et al., 2023), auto-regressivve models (Ramesh et al., 2021; Ding et al., 2021; 2022; Chang et al., 2023), and diffusion models (Rombach et al., 2022; Ramesh al., 2022; Saharia al., 2022; Zhang al., 2023a). Although Rodriguez al. (2023a;b) explore their use for scientific figures, these approaches are inherently limited to generating raster graphics.
Vector graphics generation has evolved as a parallel field. Building upon innovative work in sketch generation (Ha & Eck, 2018), Lopes et al. (2019) generate single SVG font characters made up of straight lines and Bézier curves. Carlier et al. (2020) extend this approach to include SVG icons. However, none of these models support text conditioning. Another branch of research focuses on vectorizing text-to- rext-to-image models (Ma et al., 2022; Frans et al., 2022; Jain et al., 2023). Although these methods enable text conditioning, text-to-image models typically have difficulties producing flat-colored SVG-style images, and the vectorized results tend to have imprecise geometric relations and jagged paths (Wu et al.. 2023). Addressing these challenges, Cai et al. (2023) and Wu et al. (2023) investigate auto-regressive language modeling directly on SVG representations. Even though these approaches better capture the aesthetics of vector graphics, they are limited to editing existing graphics or generating monochrome icons of limited complexity.
Scientific Figure Understanding Despite the limited number of approaches dedicated to generating scientific figures, scientific figure understanding is a subject of extensive research. Arguably, the task that is inverse to ours is the captioning of scientific figures. Expanding on prior work in scientific visual qnestion-answerin Chen al. (2019a;b; 2020) train a captioning model using a corpus of synthesized cqure-caption pairs. Hsu et al. (2021) extend this approach to real scientific figures, noting substantial challenges. To improve performance, Yang et al. (2023) reformulate the task, augmenting captions with OCR tokens and paragraphs referencing the figures. Singh et al. (2023) take a different approach and utilize reinforcement learning to consider the quality of the captions during training. In addition to such task-oriented models, recent advancements in multimodal large language modeling (MLLM; Liu et al., 2023; Dai et al., 2023; Yin et al., 2023) allow for generalized visual reasoning about scientific figures (Ye et al., 2023; Zhang al., 2023c; Horawalavithana al., 2023).
(Siegel et al., 2016; Kahou et al., 2018; Kafle et al., 2018),
Code Generation As graphics languages are a subset of programming languages, our work is closely related to code generation (Xu et al., 2022). At its core is the ongoing research on pre-training or fine-tuning LLMs on code (Rozière et al., 2023; Li et al., 2023; Fried et al., 2023; Li et al., 2022b; Chen al., 2021), commonly with multitask objective (Fried al., 2023) consisting of causal language modeling and infilling prediction (Bavarian et al., 2022). Despite the significant amount of recent progress, the primary focus of code generation remains on high-resource programming languages such as Python, Java, or JavaScript (Zan et al., 2023). TikZ commands, in comparison, are invoked as TEX macros, and the TEX programming language is considered low-resource and typically overlooked in model evaluations. Yet, TEX may still exist in training corpora, as evidenced by GPT--''s ability to comprehend TEX and TikZ (Bubeck et al., 2023; Zhang et al., 2023b). As far as we know, there has been no comprehensive evaluation of this capability, which also address in this work.
3 THE DATIKZ DATASET
DATIKZ is, to our best knowledge, the first large-scale dataset of TikZ drawings with corresponding captions. TikZ is well-known within the TEX community, but its resources are scattered across the internet, making the creation of a large dataset a fundamental challenge of our work. Consequently. we gather TikZ drawings and their captions from a variety of online resources, as detailed below.
3.1 DATA ACQUISITION
We collect the data from dedicated websites, online repositories, the TEX Stack Exchange, arXiv papers, and also artificial examples. A comprehensive overview of our data collection is provided in Table 1. We gather TikZ drawings created between November 2006 and June 2023 that successfully compile with T&X Live 2023.4 Ablation studies and examples can be found in Appendices C and E.
"https://tug.org/teellii
3

---

Published as a conference paper at ICLR 2024
Curated Examples Several websites and line repositories' focus on collecting and sharing TikZ drawings for educational purposes and to showcase high-quality examples. Through web seraping, we retrieve any TikZ. drawings from these sites that have associated captions.
Source
Size
Augmented
Curated Examples
981
63.2%
TEX Stack Exchange
29 238
51.31%
ArXiv Papers
85 656
67.75%
Artificial Examples
3914
50%
All
119789
62.71%
TEX Stack Exchange We also source TikZ drawings from TEX Stack Exchange (cf. $1). We examine the quarterly data dump and extract questions tagged with TikZ and relevant answers with a minimum score of To convert textual questions into image captions, we utilize WIZARDLM (Xu et al., 2023), LLM trained to follow arbitrary instructions. Using the title and body of a question as context, we task WIZARDLM with creating descriptive caption for the figure provided in the answer. More details on the caption generation procedure can be found in Appendix B.
Table 1:
Detailed breakdown of DATIKZ showing size and percentage of augmented data for the whole dataset and each source individually.
ArXiv Papers ArXivó is widely-used open-access archive for scientific articles. As arXiv encourages researchers to upload their papers alongside their source files, it serves as a valuable resource for obtaining TikZ drawings. Initially, we retrieve all papers with TEX source files and retain those that use the TikZ package. Subsequently, we expand any include directives and extract all TikZ environments using regular expressions. To ensure compilability, we additionally preserve all required preamble elements. For that, we first establish set of rules by analyzing documents obtained from other sources that determine which package imports and configuration options should be retained. We then parse all macro definitions and keep for each TikZ drawing the macros it uses. Finally, we exclude any TikZ. drawings that fail to compile after this extraction process (around 120k).
Artificial Examples GPT-4 has demonstrated the emergent ability to generate simple, tangible objects (e.g., unicorns) in TikZ (Bubeck et al., 2023). While not the primary focus of this work, we seek to transfer this ability to our models through knowledge distillation (Bucila et al., 2006). To this end, we compile a diverse set of object descriptions derived from the object categories in the MS COCO, LVIS, and VISOR datasets (Lin al., 2014; Gupta al., 2019; Gokhale et al., 2023). Moreover, we sample emoji descriptions from the OpenMoji database. 7 Following this, we instruct GPT-4 to generate TikZ drawing for each description, using a thain-of-thoughh prompt (Wei et al., 2022) we adopt from Zhang et al. (2023b), as detailed in Appendix B.
3.2 DATA AUGMENTATION
Prior research indicates correlation between caption length and caption quality (Gelman et al., 2002; Hartley, 2003; Huang et al., 2023). Notably, Huang et al. (2023) propose a heuristic to judge scientific captions with less than 30 tokens as being of poor quality. Given the recent advancements in MILM, and notably in MLLM with a focus on science (cf. $2), we propose the automatic augmentation of such captions (Belouadi & Eger, 2023a;b). For an in-depth analysis of implications, refer to Appendix C.
Specifically, we leverage LLA VAR (Zhang et al., 2023c), instructing it to generate short descriptions for TikZ drawings with captions containing fewer than 30 tokens (cf. Appendix B).* Inspired by the CAPFILT method (Li et al., 2022a), we generate five candidate descriptions and rank them based on their text-image similarity using CLIPSCORE (Hessel et al., 2021). The final augmented caption is then formed by concatenating the original caption with the top-ranked description. For GPT-4, cannot rely on the heuristic, as the captions used are not scientific. Instead, we augment all captions to increase diversity in our dataset while retaining the original captions well. Table displays the percentage of augmented captions in our dataset. On average, this method increases the CLIPSCORE of captions with originally fewer than 30 tokens from 24.76 to 29.12, substantial improvement in text- text-image similarity, especially considering that CLIPSCORE typically ranges between zero and 40 (Hessel et al., 2021). The CLIPSCORE for original captions exceeding 30 tokens is 27.06.
shttps: https/////eaaapee.eeeeee net, https::/ttkk. http:/eete.tttsss//pllleeeee
& ht //gi thub. com projects
org
7https://openmoj] org iI this work, we use the MOSES tokenizer (Koehn al., 2007) to count tokens.
4

---

Published as a conference paper at ICLR 2024
4 METHODS
We leverage LLAMA (Touvron et al., 2023a) as the base model in most experiments, using captions from DATINZ as model imput and TikZ code as ground truths. Since TEX source files from arXiv were included in LLAMA's pre-training data, it may have prior knowledge beneficial for this task. We choose the original LLAMA release over its updated revisions, LLAMA (Touvron et al., 2023b) and CODELLAMA (Rozière et al., 2023), as their training data is not as clearly specified. This uncertainty and their more recent release would make it difficult to create a test set without training-to- -tottssttttttttttt data leakage. We also experiment with GPT-4 and CLAUDE 2, as earlier research hints at inherent potential for our task (cf. $3.1 and $2), and employ the same chain-of- -ftho-tt approach outlined in $3.1. However, they proprietary, can only address data leakage for LLAMA (Aiyappa al., 2023).
4.1 CLIMA
A potential drawback of vanilla LLAMA is that it may not understand visual concepts, given that it was not designed to process image data. However, this ability could significantly enhance the creation of scientific figures. Therefore, we modify LLAMA by combining it with CLIP ViT-H/14 model (Cherti et al., 2023). CLIP is frequently employed to establish a bridge between vision and natural language, thereby facilitating the creation of MLLMs (Yin al., 2023).
However, unlike most MLLM methods, to our knowledge we are the first to use CLIP's s multimodal projection layer, which allows us extract visual information from both text and images in a common embedding space (cf. Figure 1). This approach is akin to text-to-imags models like DALL-E and CLIP-GEN (Wang et al., 2022b), that make use of this duality to generate raster graphics. In our case, our primary objective is to provide LLAMA with visual interpretation of the input caption, anticipating that this adjustment will boost the alignment with generated TikZ drawings. addition, additional input (cf.
it also enables to experiment with supplying rasterized scientific figures $5). As this new model can be described as using CLIP inside LLAMA, we refer to it as CLIMA.
We accomplish this integration by connecting CLIP's output with LLAMA's input via soft prompt- ing (Lester et al., 2021); i.e., we prepend CLIP's embedding vector to LLAMA's input embeddings of the caption. This requires adding ferd-forwrrd layer with dimensions ōv.T-H/14 X ÒLLAMA connect image features of dimension δVYT:H/14 with LLAMA's word embedding space of dimension SLLAMA- Following insights from Liu al. (2023), we pre-train this adaption layer on dataset of 595K generic text-image pairs for one epoch while keeping both CLIP and LLAMA frozen during the process.
4.2 ERROR HANDLING & CORRECTION
A general issue with our language modeling approach to TikZ generation is that outputs may violate the syntactic and semantic rules of TEX, potentially leading errors and uncompilable documents. While there are constrained decoding algorithms that can force models to form valid programs (Poesia et al., 2022; Scholak et al., 2021), they depend on parse trees and are only useful for languages with a context-free grammar. TEX, however, has flat, unstructured syntax that is generally impossible to parse (Erdweg & Ostermann, 2010), rendering these methods unsuitable for our approach.
As an alternative, we propose iterative resampling method, leveraging the diagnostic data produced during compilation. If an error arises during compilation, we analyze the logfile to identify its source. Rather than resampling from the start, we then reverse the generation process to just before the error line and continue sampling from there. the error persists, we infer that the origin of the problem lies earlier in the code and reverse further back, specifically 4(1-1) lines above the error, with i denoting the current iteration. While this method does not guarantee error-free results, it provides a more efficient and targeted strategy than simply reinitiating sampling from the beginning.
5 EXPERIMENTS
Before tine-tuning our models on DATIKZ, we extract a sample of 1k human- created items to serve as test set. As LLAMA's 's training started in December 2022, we only sample items introduced after this date to avoid data leakage. We conduct both automatic ($5.1) and human evaluations ($5.2). Additional results and instances of generated TikZ. drawings are available in Appendices C. and E.
5

---

Published as a conference paper at ICLR 2024
Model Sizes In terms of model size, fine-tune LLAMA7, and CLiMA1g, each with 7 billion parameters (7b), as well as 13 and CLiMA13s with 13 billion parameters (13b), respectively. During inference, additionally evaluate CLiMA with CLIP receiving compiled human-create TiKZ. drawings as input instead of captions, which we refer to as CLiMApM: for elarity (cf. $4.1).
Training Given the size of these models, we introduce trainable low-rank adaption weights (LoRA; Hu et al., 2022) while keeping the base model weights frozen and in half precision (Micikevicius et al., 2018). Following Dettmers al. (2023), we apply LoRA to all linear layers. In addition, we find that training the embedding layer and language modeling heads is crucial for successful fine-tuning. Since we are not aware of any studies applying LoRA to these layers, we make them fully trainable and leave this investigation to future work. In line with Liu et al. (2023), ADAMW (Loshchilov Hutter, 2019) and a batch size of 128, but increase the learning rate to 5e-4 as this leads faster convergence. As form of data augmentation only possible for CLiMA, randomly replace the captions forwarded to CLIP with the reference image in 50% of the cases.
train for 12 epochs with
5.1 AUTOMATIC EVALUATION
We use a variety of automatic evaluation metrics to evaluate the performance of our models on our test set in terms of code, image, and caption-image similarity. In particular, we use the following metrics:
CLIPSCORE calculates the similarity between image and text, as outlined in $3.2. We utilize it to evaluate the correlation between a rasterized TikZ drawing and its corresponding caption.
CLIPScOREMe:
technically the same metric as CLIPScORE, but with human-made TiAZ drawings as a reference input. Therefore, it assesses the similarity of two images rather than an image and a caption. To our best knowledge, we are the first use CLIPScoRE in this configuration.
Kernel Inception Distance (KID) assesses the quality of generated TikZ drawings by comparing their distribution with the distribution of real images in the test set (Binkowski et al., 201 8). This comparison helps determine how realistic the generated images appear in general. We extract image features using the same CLIP model utilized in CLIPSCORE.
CRYSTALBLEU is an n- g--rrmm- baandd metric designed to measure textual similarity (Eghbali & Pradel, 2023). As variant of BLEU (Papineni et al., 2002), optimized for evaluating code, employ it to assess the similarity between human-create and machine-produced TiKZ. code.
Extended Edit Distance (EED) is a metric dedicated assessing string similarity (Stanchev et al., 2019), much like CRYSTALBLEU. We utilize it to determine the minimum number of operations needed to convert the machine-generated code into the reference code.
Compilation Sampling Rate (CSR) measures how frequently we need sample from model to yield compilable TikZ code that outputs an image. This is crucial as some metrics depend on images. With LLAMA and CLIMA, we use iterative resampling (cf. $4.2) and account for partial samples. This is not feasible with GPT-4 and CLAUDE 2 due to their chain-of-thought prompt, which generates code across multiple steps. We take a relaxed stance, counting a sample successful if it results in an image, even if there errors.
Results We compute the above metrics for each model and present the system-level scores in Figure 2. The radar chart on the left illustrates that there are small but noticeable score differences between the LLAMA and CLIMA models, revealing some intriguing trends. Aligning with the intuitive expectation that larger models yield better performance (Kaplan et al., 2020), the 13b models clearly outperform the 7b models on all points). A notable exception is CSR, where all models perform comparably. This shows that all models require approximately 1.2 samples per caption to generate compilable TiAZ. drawing.
and image- metrics by 0.2-0.5pp (percentage
Within model sizes, CLIMA7, outperforms LLAMA7, on CRYSTALBLEU, EED, CLIPSCORE, and CLIPSCORE by up to 0.4pp, suggesting that, even when only text inputs are involved, integrating CLIP into the model has a predominantly positive effect. CLiMAIsa continues this trend, showing a O. 0.p higher CLIPSCORE than LLAMAABs- However, we also see that this does not necessarily have to increase the similarity with a reference image well, LLAMA13e has a 0.1pp higher CLIPSCORE On CRYSTALBLEU and EED, CLIMA13, again fares better, although, with 0.1p the gap is not as pronounced as for the 7b models, possibly due to diminishing returns (Hong al., 2023).

---

Published as a conference paper at ICLR 2024
LL & CLIMA7w/30
CLiMA 33/MM GPT-4 & CLAUDE 2
CSR
CSR
1,15
23
CRYSTALBLED
CRYSTALBLED
FED
EED
1:45
55/82
2.87
56.0
3.05
1.25
56.05
1.85
1.67
57.2
2.82
56.28
0.82
58.4
26.78
0.08
27.38
0.3
26.85
80.4
0.05
28.05
76.93
0.2
26.92
0.02
28.72
CLIPS
80.7
CLIPSCORE
78.95
KD
KID
80.97
CL DCCOR
CI CILORCOR
CLiMA7a
CLIMA 13B
CLIMAMGG
CLAUDE
LLAMA
LLAMA
GPT-4
Figure 2: Automatic evaluation results for LLAMA7w13a. CLIMAn GPT-4, and CLAUDE 2. Axes representing metrics where lower values are better (CSR, EED, and KID) have been inverted. Detailed scores are provided in Appendix C for further reference.
The right radar chart compares our best text-only model, CLiMA with CLiMA GPT-4, and CLAUDE 2. As before, all models perform roughly the same on CSR, except for CLAUDE 2, which needs noticeably more samples. expected, CLiMA having access to reference images, improves upon CLIMA in CLIPScORF by 1.2pp. However, this does not lead to an improvement in CLIPSCORE, echoing our earlier observation that image and caption-image image similarity do always correlate. also does not improve KID, demonstrating that the overall quality of the images remains constant. Nevertheless, the string-based metrics are 0. 1-0.ppp higher, indicating that conditioning on reference images positively impacts code similarity.
We also observe that CLAUDE performs much worse than GPT-4, and both perform noticeably worse than both CLIMA13e and CLIMAsGG most metries. The drastically lower CRystALBLED and EED (up to 3.9pp) suggest that GPT-4 and CLAUDE 2 generate fundamentally different code (in Appendix show that it exhibits lower level of complexity). The up to 6.6pp lower CLIPSCORE, and over six times as large KID indicate that not only do the generated images look different from human ones, but also that the general quality of images much different from the human distribution. However, most strikingly, both models achieve an up to 1. higher CLIPScoRE. Upon investigation, we find that both models tend to produce degenerate images, which visibly copy the input caption into the output image. Since the outputs of CLIP (and by extension CLIPSCORE) can be controlled with images of text (Ramesh et al., 2022), CLAUDE 2, and in particular GPT-4, essentially employ such typographic attacks to achieve exceptional caption-image similarities. We further explore this phenomenon in $6.
Overall, CLIMA7s and CLiMA13, outperform their respective LLAMA models in five out of seven metrics each, with CLAUDE 2 and GPT-4 substantially underperforming all of them. While CLIMA unsurprisingly improves upon CLiMA13n CLiMA is the best model with only textual inputs.
5.2 HUMAN EVALUATION
To further evaluate the effectiveness of our models, conduct human annotation campaign using best-worst scaling (BWS; Louviere et al., 2015). As a form of comparative annotation, BWS yields high-quality results even when the number of annotations is low (Kiritchenko & Mohammad, 2016;
7

---

Published as a conference paper at ICLR 2024
Caption Similarity
Reference Similarity
0.2
Density
0.1
0.0
-1.0
-0.5
0.0
0.5
1.0
-10
-0.5
0.0
0.5
1.0
CLiMA
LLAMA13a 13B
GPT-4
Human
CLIMA
Figure 3: Distributions of BWS scores per model for caption and reference similarity. Scores span from 11 (poor) to (excellent). The "*" markers denote expected values, and "x" signifies the mode.
2017). Within this method, annotators are tasked to compare tuples of n = 4 items, identifying the best and the worst item based on given property. Real-valued scores, ranging from -1 (poor) to 1 (excellent), are then computed by subtracting the fraction times an item is chosen as the worst from the fraction of times is chosen as the best (Orme, 2009).
In this work, we focus on caption similarity (CS) and reference similarity (RS), In CS, annotators evaluate image tuples based on text-image similarity with captions (similar CLIPSCORE), We construct 4-tulles consisting of our two leading text-only models from automatic evaluation (CLiMA13 and LLAMA13s). GPT-4, and human reference images. In RS, the human reference images are used as the standard of comparison instead (similar to CLIPSCORFMac), so we replace them in the tuples with CLiMA while leaving the other models unchanged. Each property is then annotated by four unique expert annotators with relevant research experience (cf. Appendix D). 9 To ensure a manageable workload for the annotators, we create our tuples from a subset of 100 items sampled from our test set. We assess the consistency of the annotators using split-half reliability (SHR; Kiritchenko & Mohammad, 2017). This method involves randomly splitting all annotations into two sets, calculating scores for each set, and then determining the correlation between them using Spearman's p.
Results For CS, the SHR is = 0.6, indicating moderate but adequate consensus among annotators. Figure (left) exhibits kernel density estimates for the computed scores, with marked modes and expected values. Unsurprisingly, humans perform best with a mode near 1. CLiMAJaa is the only other model with mode above O, followed by LLAMA136- while GPT-4 lags behind. This indicates that when sampling once with given caption, CLIMA is most likely to generate the best image. Since CLIMA13R and LLAMA retain their earlier CLIPSCoRE ranking, but GPT-4 drops substantially, we hypothesize that human annotators are not as prone to typographic attacks as metrics. However, we still observe a slight bias towards images text. In 75% of cases where GPT-4 selected as the best model, it copies more grgmms from the caption into the image than the worst-ranked image, potentially leading to outliers and thus a slightly higher expected value than CLiMA or LLAMA J38-
Regarding RS, record similar SHR, with p 0.58. For LLAMAJ3.. CLIMA13s and CLIMA the distributions in Figure (right) are almost normally distributed, with the mode and expected value being nearly identical. As with CLIPSCORE,we. LLAMA13e is ranked marginally higher than CLIMA indicating CLIPSCORE correlates well with human rankings (Appendix C details exact correlations). On similar scale, CLiMA outperforms LLAMA 13n- In contrast, GPT-4 follows nearly uniform distribution, with a slight downward trend for better scores. Therefore, its mode is noticeably lower than for the other models. The expected value, albeit only slightly, is also the lowest.
In summary, our human evaluation aligns well with our automatic metrics, with the added benefit of lower susceptibility typographic attacks. CLiMA13a outperforms LLAMA on CS, while CLIMAnMa surpasses LLAMA13a on RS. GPT-4 shows peculiar distributions, with the mode (and also the expected value for RS) lagging behind, highlighting the effectiveness of our models.
oWe tried crowdsourcing well, but due low agreement with experts, concluded that crowdworkers lack the necessary expertise for tasks (cf. Appendix D).
8

---

Published as a conference paper at ICLR 2024
Code Novelty
Caption Copying
80%
9%
Human
CLiMA
6%
60%
LLAMA
Human
GPT-4
3%
40%
CL
CLAUDE 2
1L
20%
GPT-4
CLAUDE 2
0%
0%
1-gram
3-gram
5-gram
7-gram
9-gram
1-gram
3-gram
5-gram
5
7-gram
9-gram
Figure 4: Proportion of unique code n-gramss (n E [1. 10)) that do not appear in the training data (left), and proportion of caption n-grams that were copied into the output image (right).
ANALYSIS
The issue of language models memorizing and copying training data is a prevalent concern (McCoy et al., 2023; Carlini et al., 2023; Raunak & Menezes, 2022; Meehan al., 2020). Similarly, we discovered in $5.1 that GPT-4 and CLAUDE tend to perform typographic attacks by memorizing and copying input captions. In this section, analyze the extent of these issues concept n-gram novelty (McCoy et al., 2023). Specifically, to measure code novelty, we determine the proportion of n-grams in the model-generated TikZ. code that not found in the training data. To measure caption copying, we calculate the proportion of n-grams from the caption that were copied verbatim into the output code. For comparison, we also calculate both metrics for human references.
test set using the
Figure 4 displays the results for both metrics (n E [1, 10]) after filtering code comments. In terms of code novelty, models tend to generate less novel code than humans for smaller n-grams (n 4). However. for π > 6, models become more novel, with more than 80% of all model n-grams being McCoy et al. (2023) observe the same phenomenon in all datasets investigated and
novel for conclude that this ratio is the normal case when model is not affected by memorization of the training data. Regarding caption copying, GPT-4 and CLAUDE 2 copy considerably more n-grams from the caption than our models. For 1-grams (i.e., 1), CLIMA13a and LLAMA13s copy around 6.5% of n-grams, while GPT-4 and CLAUDE copy more than 10%. For n > 5, CLiMA13s. LLAMA 133. and humans practically stop copying, but CLAUDE 2 and especially GPT-4 continue with almost linear trend, hinting at instances where they might copy the entire caption (cf. Appendix for examples). This reinforces hypothesis from $5.1 and points towards CLIPSCORE more robust metric for assessing the visuals of text-rich images since seems less susceptible to typographic attacks.
7 CONCLUSION & FUTURE WORK
In this work, we present AUTOMATIıZ, a project for automatically generating TiAZ drawings based on natural language descriptions. As part of AUTOMATIıZ, we release TATıZZ, a pioneering dataset of aligned TikZ drawings and captions, and CLIMA, novel model architecture which integrates multimodal CLIP embeddings into LLAMA. By fine-tuning CLiMA on DATIKZ, we demonstrate that outperforms LLAMA on several metries and also surpasses proprietary GPT-4 and CLAUDE In addition, CLiMA can also process images, potentially extending its application to vectorization and conditioning on sketches. Important finds are that (i) integrating CLIP can lead to improvements, when only text inputs involved, provided the task relates to visual concepts, and that (ii) attention should be paid to typographic attacks when evaluating models that generate text-rich images.
In future research, we aim to incorporate insights from the caption generation community and enrich our input texts with other
sections of the source documents (cf. $2). We also plan to enhance extraction pipeline, especially since we had to exclude over 120k TikZ images from arXiy that failed to compile. We hope that these modifications will bring us step closer bridging the gap human performance.

---

Published as a conference paper at ICLR 2024
8 ETHICS STATEMENT
We ensure that the TikZ drawings we gather from online sources are licensed a manner that permits us to copy and redistribute them. Most sources license their content under Creative Commons attribution license, the GNU Free Documentation License, or the MIT license." 12 ArXiv an exception in that, even though it allows licensing under Creative Commons license, the majority of papers are published under a non- non-exclusive license, which does not grant us permission to redistribute." 13 As result, we exclude any TikZ drawings from arXiv that use this license in the public release of DATIxZ. Nevertheless, we do release AuTOMA TikZ in conjunction with the dataset generation code, enabling anyone to recreate the full version of DATIKZ. themselves. As for auto-generated samples, OpenAl prohibits the use of GPT-4 for creating competing services, restricting this part of our dataset to non-commercial applications." 14
Apart from our dataset, in this work compare openly developed models with the proprietary GPT-4 and CLAUDE 2, whose full training details and hyperparameters have not been published. While we strive for fair evaluation of all models, the lack of transparency in proprietary systems inevitably hinders comparative evaluations and reprodueibility.
Within the scope of these limiting aspects, however, models perform best in generating TikZ across the tested conditions. Yet, our models should not be used as a substitute for human judgment and critical thinking. They may carry any biases, flaws, gaps that exist in the base models and training data and could potentially misinterpret the input, fabricate non-existent details, or overlook crucial information. Users should be aware of potential differences between the results they expect and the output the models generate.
Furthermore, while our models are designed to aid in the production of legitimate scientific figures. they could potentially be used to generate disinformation and fake science in the hands of malicious actors.
ACKNOWLEDGMENTS
We gratefully thank, in no particular order, Timm Dill, Yanran Chen, Daniil Larionov, JiWoo Kim, Vivian Fresen, Martin Kerscher, Christoph Leiter, and Ran Zhang for their help with our human evaluation campaign, proofreading, discussions, and comments on our work. We further thank the BMBF for its support via the grant METRICS4NLG and the Ministry of Culture and Science of the State of North Westphalll for the grant no. NW21-0594 (SAIL). The second author is funded under the Excellence Strategy of the German Federal Government and the Länder. The last author supported by DFG grant EG 375/5-1. We also thank Hugging Face for providing a community GPU grant. Any icons used in this work were designed by OpenMoji, the open-source emoji and icon project.
REFERENCES
Rachith Aiyappa, Jisun An, Hacwoon Kwak, and Yong-yeol Ahn. Can we trust the evaluation on ChatGPT? In Proceedings of the 3rd Workshop Trustworthy Natural Language Processing (TrustNLP 2023), pp. 47-54, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/112223.rrsmll
URL https: hattps://aclanthologg org/2023. trustnlp-
Anthropic. Model card and evaluations for Claude models, 2023. URL https://wwwffile anthropic. com/ coom/prouuction images/Model-Carde -awaard----aae2222 (etaaudee22ppp.. 2.pdf.
Haruka Aoki and Kiyoharu Aizawa. SVG vector font generation for chinese characters with transformer. In 2022 IEEE International Conference on Image Processing, ICIP 2022, Bordeaux, France, 16-19 October 2022, pp. 646-650. IEFE, 2022, doi: 10.1009/C11455572222222099 9897633.
URL
wottpss//creativecooonssuu http://mmoonooggg///eneeese
html
10

---

Published as a conference paper at ICLR 2024
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022.
Jonas Belouadi and Steffen Eger. ByGPTS: End-to-end style-conditioned poetry generation with token-free language models. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7364-7381. Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 118653/1/2223..aar https://aalaanthologg wwg.o/g........oo acl 1on.llog.4400
406. URL
Jonas Belouadi and Steffen Eger. UScore: An effective approach fully unsupervised evaluation metrics for machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 358-374, Dubrovnik, Croatia, May 2023b. URL https:
Association for Computational Linguistics. doi: 10.
//aclanthology. . org/2023 .al.0...aclmmain..... 27.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL / net
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/opeerreieew..ttt net/ forum?id-B1xsqj090
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395. doi: 10.114551150000. URL / 11451111111111111144111 1584222 1150464.
Mu Cai, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Lec, Leveraging large language models for scalable vector graphics-driven image understanding, 2023.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. DeepSVG: A hierarchical generative network for vector graphics animation. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Flort Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: //tt::::://roooe wwwp:ceeedinnn.nniiiii..... paper/2020/ 001222//aah//e httph/hhhhhhhhhhhehheehh frash//bff9dbbd 0aw.eeedd42228866eee000 4a285866ce3:500002244 Abstrratt............. html.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. net wwtp://tdpppp//dff/.///// wwtp:fiddaattTTttt
OpenReview.net, 2023. URL
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming- Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 4055 4055-4075. PMLR, 2023. html.
URL
mlr, ctress///2//ccre
Charles Chen, Ruiyi Zhang, Sungchul Kim. Scott Cohen, Tong Yu, Ryan A. Rossi, and Razvan C. Bunescu. Neural caption generation over figures. Robert Harle, Katayoun Farrahi, and Nicholas D. Lane (eds.), Proceedings the 2019 ACM International Joint Conference on Pervasive
11

---

Published as a conference paper at ICLR 2024
and Ubiquitous Computing and Proceedings of 2019ACM, International Symposium Wearable Computers, UbiComp/ISWC 2019 Adjunct, London, UK, September 9-13, 2019, pp. 482 092-485.22 ACM, 2019a. doi: 10.1145/3341112..3355555
URL https: /ttttp:://oi.orrrr 18. 11/43/3/11111 16244166..31 162. 3345601.
Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Tong Yu, Ryan Rossi, and Razvan Bunescu. Figure captioning with reasoning and sequence-level training, 2019b.
Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, and Ryan A. Rossi. Figure captioning with relation maps for reasoning. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pp. 1526-1534. IEEE, 2020, doi: 10. /1111///55555222200000000 909359.... URL https://doi.org// 1109/ WACV45572. 2020. 9093592.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilbarco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2829, June 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIPP Towards general-pune purpose vision- vision-language models with instruction tuning, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized Ilms, 2023.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. VigView: Mastering text-to-image generation via transformers. In Marc'Aure Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 199822-19833 2021. URL /ttp:://///////eeeeeeee neurriinnnnneuniinnee ips. wttpccccppppce////////// haah/hahh/ 6d Abstract. html.
Ming Ding, Wendi Zheng, Wenyi Hong. and Jie Tang. Cog View2: Faster and better text-to-image gen- eration via hierarchical transformers. In NewrIPS, 2022. URL http: http:://apprrs.nnnnn nips cc/paper_
html.
Aryaz Eghbali and Michael Pradel. CrystalBLEUU Precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASB '22, New York, NY, USA, 2023, Association for Computing Machinery. ISBN 9781450394758, doi: 10. 011444/55114493555555 3556903.
URL httpp:://don org/ /0/ 1141/35553399
Sebastian Thore Erdweg and Klaus Ostermann. Featherweight TeX and parser correctness. In Brian A. Malloy, Steffen Staab, and Mark van den Brand (eds.), Software Language Engineering Third International Conference, SLE 2010, Eindhoven, The Netherlands, October 12-13, 2010, Revised Selected Papers, volume 6563 of Lecture Notes in Computer Science, pp. 397-416. Springer, 2010. doi: 10.1007/978- 3-77778----4--8- 19440-51, _6. URL http:: http:///oooooog///// 1007/978- 00----2111990 26.
12

---

Published as a conference paper at ICLR 2024
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-
resolution image synthesis. In IEEE Conference on Computer Vision and Pat-
tern Recognition, CVPR 2021. virtual, June 19-25, 2021, pp. 12873-12883.
Computer Vision Foundation / IEEE, 2021. doi:
00.10990CVPR46437.220
01268. URL. https://openaaceess. thecvf. www.nteen///raa22//ttm///////// _aaming Transformers forrrrs__oo_rrroe
Kevin Frans, Lisa B. Soros, and Olaf Witkowski. CLIPDraw: Exploring texptorrnntetttttooddaaiinn synthesis through language-image encoders. In NeurlPS, 2022. URL http:://apprrrs...t.. nips. cc.ppprr. html.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A sarge-scale study of human evaluation for machine translation.
Tronsactions of the Association for Computational Linguistics, 9:1460-1474, 2021. doi: 10.1162 tacl _00_a_00437. URL ttttps://aclantthll.g... org/ 2/2g/ tacl-1.87
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: https::/openreview.nettt id=hQwb- 11MMEL.
Andrew Gelman, Cristian Pasarica, and Rahul Dodhia. Let's practice what we preach: Turning tables into graphs. The American Statistician, 56(2):121-130, 2002. doi: 10.119880000311302317777777 URL httph:t/t////tt org/10 10. 11 0099//000310023777777777
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation, 2023.
Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 5356-5364. Computer Vision Foundation / IEEE, 2019. doi: 10. 109/CVPR. 2019.00550. URL http: /wpp//ccssccccsccceeeeepc thecvf. cww//ooneett_CVPP 201 2019/2t0111/mmlmmm 9t/hmlltmm/mmmm /atppappppaaa_aa htt.:p/e.wr....alluaaa.a...aaann
wapeeeeppppppppeeeee html.
David Ha and Douglas Eck. A neural representation of sketch drawings. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: https:://peereeiew.nrr net/ Horum?id-Hy6GHppCHH
James Hartley. Single authors are not alone: Colleagues often help. Journal of Scholarly Publishing, (2):108 110008-11,,, 113, 2003. doi: 10.3 0..1138/jp.344.2.008 URL https: wrg.o..g///ooo..oo 3138/jsp 2. 108.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: URL. htsp:ttts:://allltttoloyyy 2trg//0211 emnlp-main..5. 595.
10. 10018665//112221.emmmmmmaa
John D. Hobby. MetaPost, 2014. URL /tttps://tupt org/metapost..
Zhi Hong, Aswathy Ajith, James Pauloski, Eamon Duede, Kyle Chard, and lan Foster. The diminishing returns of masked language models science. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1270-1283, Toronto, Canada, July 2023. Association for Computational URL /tttps://aclantti
Linguistics. doi: 10. 118653/11/2223 1t..16551//222..mddnnn......... findings-acl.Cr 82.
org org/2033.
Sameera Horawalavithana, Sai Munikoti, Ian Stewart, and Henry Kvinge. SCITUNE: Aligning large language models with scientific multimodal instructions, 2023.
13

---

Published as a conference paper at ICLR 2024
Ting-Yar Yao Hsu, C Lee Giles, and Ting-Hao Huang. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 3258-3264, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (0t.6///w/2222..nnnnnn----ee 2021. medinngs-mmlp.... 277.
URL https::/aclanthholgg)) org/
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. net/ OpenReview.net, 2022. URL /tt.:tttp///ppmeeeee..o////orro wttp://eeee..oom.odo////eeee///e
Chich Yang Huang, Ting- Yao Hsu, Ryan A. Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk- Yin Chan, Eunyee Koh, C. Lee Giles, and Ting-Hao (Kenneth) Huang. Summaries as captions: Generating figure captions for scientific documents with automated text summarization. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.). Proceedings of the 16th International Natural Language Generation Conference, INLG 2023, Prague, Czechia, September 11 15, 2023, pp. 80-92. Association for Computational Linguistics, 2023. URL https: httsss///aaatthlogggo org/ 2023. innlnmaiinn.. 6.
Ajay Jain, Amber Xie, and Pieter Abbeel. VectorFusion: Text-to-SVG by abstracting pixel-based diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 1911-1920, IEEE, 2023. doi: 10.1 10.1109/CVPR5279.2223.22
URL https: //t:::////ii/// org/10. 109/ 2023. .01990.
Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT; USA, June 18-22, 2018, pp. 5648- 5656. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CYPR. 2018.00592. URL /tLhtttp:/openaccesss thecvf. www.//ooeentccprr.ree... 201 /tttt///taaaaeeeeee/eeeee httprssttniiin_Ddta____P 2atpppp__221__ppprpr__ppp html.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https: /ppnr//veeeeeeveeeeeee net/ mz00yDz..
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for text-to-image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 10124-10134 IEEE, 2023. doi: 10.1 D0110999CCPPS572222200000000 URL https: //doi.org/// / /0//CCPR222299 00976.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Svetlana Kiritchenko and Saif Mohammad. Best- worst scaling more reliable than rating scales: A case study sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 465-470, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 18653/v URL https://aclanttoll.e org/P17 PeggPP7-2007 7-744.
Svetlana Kiritchenko and Saif M. Mohammad. Capturing reliable fine- ine-grained sentiment associations by crowdsoureing and bost-worst scaling. In Proceedings of the 2016 Conference of the North American Chupter the Association for Computational Lingusticc:: Human Language Technologies, pp. 81 811-817, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v 1/t///////11--/ 1095. URL https: hatps//////aaaaoooooooooo org/N16-1095.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burn Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine
14

---

Published as a conference paper at ICLR 2024
translation. In Proceedings of the 4Sth Annual Meeting of the Association for Computational Linguisties Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177-180, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: l/aclanthology.. org wwwggpp7-2200 2045.
Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punla Cana, Dominican Republic, 7-11 November, 2021, pp. 3045-3059. Association for Computational Linguistics, 2021. doi: 10. URL https: /tctps://do.... org/ 10. 18653/v1 /2011 emnlp- main. 243.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-VIL VL: A universal encoder for vision and language by cross- cross- modal pre- pre-training. In The Thirty-Fourde AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 11336-1133 1833-11334, 11344. AAAI Press, 2020. doi: 10.1609/AAALV331107
URL https: /ttp:://oooooooo org 0
v34107. 6795.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 12888-12900 PMLR, 2022a. URL https: / ptt.//inseeeeden.cooooo/ooo..... wwlr.press//ee 01/1122222222 htm........m..............mm....
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas. Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: may the source be with you!, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Ré mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. Science, (0926666)::0221000) dec 2022b. doi: 10.1 1..11/cceeccce.aag 158. URL https:///ol. or0/10. 11 112%%Fsccencce abq1158.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision FECV 2014 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V. volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014. doi: 10. 01177797883319--005 L844. URL https://6oi.org//e.
org/10. 0001/97877773800 3-319-10602-1-00
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty- seventh Conference Neural Information Processing Systems, 2023. URL htt.ttts://opeereviee.
net//formmidd=WHHXxGHH)
Raphael Gontijo Lopes, David Ha. Douglas Eck, and Jonathon Shlens. learned representation for scalable vector graphics In 2019 JEEE/CVF International Conference on Computer Vision. ICCV 2019, Seoul, Korea (South). October 27 November 2, 2019, pp. 7929-7938, IEEE, 2019. doi: 10.1 109//CCC.0019.0000000 URL https: /ettps://doi... http:/ooioooot////// 1110//ICCV 2019 00802.
15

---

Published as a conference paper at ICLR 2024
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019. URL https:/ http://wwweeeeoorrtrrt//r///eEC.
Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press, 2015. doi: 10.1017/CB097811000 107337855.
Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. Towards layer-wise image vectorization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. (6293-16302. IEEE, 2022. doi: 10. R...10999VP228888...2 20222.00882222155666 URL https: /tt.//oii.ooo http::::/doioor//1.... 1199/CVPR52288. 0153..
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN. Transactions of the Association for Computational Linguistics, 11:652-670, 2023.
doi: 10.1166222 162/tacccccccc a_00567. URL https: /tttts://aclanee http:///aaaaatlloaaaa/e/ org 2023.023. tacl-1.38.
Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A three sample hypothesis test for evaluating generative models. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 3546-3556. PMLR, 26-28 Aug 2020. URL https:/ http:://proceedinggs. mlr www.rrrressvvvvv 0
http:///v0mmmmeeemeeeme meehan20a html.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
net//orum?id=r11g550
OpenAL. GPT-4 technical report, 2023.
Bryan K. Orme. MaxDiff analysis: Simple counting, individual-level logit, and HB. Sawtooth Soft- ware Research Paper Series, 2009. URL https://conteni sawtoothsof tware. com/ con/assets/ 8e69929d- 1119929d-a0894400 -088-4493--80 a9f3 044644115664..
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002. wwwphhhooogooorg/PP2220 wtwp://-200//02100000/000 1040/.
doi: 10.3115/1073033.1000 10731333 URL
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In The Tenth International Conference on Learning Representattinns ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
htwpreenrreieeeeeeeeeeooeoe net. www..n/eoormiiiinnnnt/9//11/11
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021. URL http:/tttt///t http::::/poceeeigigggg/
ww.. www.pppre///...e 39/radford211 a. html.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831 PMLR, 2021. URL httphhtt://peeee http://proceeddingss mlr. press/v1333
/ https//s/v/3/ammm/12a1a rat.s/mmeesaaaaa html.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents, 2022.
16

---

Published as a conference paper at ICLR 2024
Vikas Raunak and Arul Menezes. Finding memo: Extractive memorization in constrained sequence generation tasks. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5153-5162, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https:/t//aaaae wwthhoooooooooooooooooooooooooo. org/ org/2022.0. sindings-emnllp 378. 378.
Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, html.
pp. 1060-1069. JMLR. org, 2016. URL
wwogg..ll...... www.mlr.pp. www.mmmwr.......mmmmmmmmmmmmm pprsss/44//reet
Ricardo Rei, Ana C Farinha, José G.C. de Souza, Pedro G. Ramos, André F.T. Martins, Luisa Coheur, and Alon Lavie. Searching for COMETINHO: The little metric that could. In Helena Moniz, Lieve Macken, Andrew Rufener, Loie Barrault, Marta R. Costa-jusså, Christophe Declercq, Maarit Koponen, Ellie Kemp, Spyridon Pilos, Mikel L. Forcada, Carolina Scarton, Joachim Van den Bogaert, Joke Daems, Arda Tezcan, Bram Vanroy, and Margot Fonteyne (eds.), Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pp. 61- 70, Ghent, Belgium. June 2022. European Association for Machine Translation. URL https: //acl facllanholoogy.orge wwwp:ey..rrgg/2222///// e..
Juan A. Rodriguez, David Vázquez, Issam H. Laradji, Marco Pedersoli, and Pau Rodríguez. FigGen: Text to scientific figure generation. In Krystal Maughan, Rosanne Liu, and Thomas Burns (eds.), The First Tiny Papers Track at ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023. OpenReview.net, 2023a. URL httpp::///ppere/v///
Juan A. Rodriguez. David Vázquez. Issam H. Laradji. Marco Pedersoli, and Pau Rodríguez. OCR- VQGAN: Taming text-within-ing -imgee generation. In JEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, Jamuary 2-7, 2023, pp. 3678-3687, IEEE, 202.5666688..80000 2033.0036.
2023b. doi:
URL https://ddd org/10. wtt.:///....1///5555...... 1199/WACV56668..
2023. 00368.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10688.. IEEE, 2022. doi: 10. BLL119/CVVPR22888222222.. 1189/CVPR52688. 2022. 01042.
URL htttps://dos or./10.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code LLaMA: Open foundation models for code, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NewrIPS, 2022. URL http: nipperrs...ii.....mm.po.... cc/paper_files// files/ paper htt022//aash/ec75aaaaeeeeee oatpaa/eeeeeeaccaooooooeoaaaaaae wwweeea--tttaaa-ooeeeeeeeeee
html.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 9895-9901, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v 1tt.33/111222.mmmlgp watmmnlpmmiin.7.1. 779. URL https: /awaaaaoaaaalaayyayaaooooooo org/ 2021 emnlp-maing 779.
Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Kumar Divvala, and Ali Farhadi. FigureSeer: Parsing result-figures in research papers. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII, volume 9911 of Lecture Notes in Computer Science, pp. 664 680. Springer, 2016. doi: 10.1007/978-3 3-1-7788--331--6668- 41. URL https://doi. org/ 1--11//9788--119--47--------1111
17

---

Published as a conference paper at ICLR 2024
Ashish Singh, Prateek Agarwal, Zixuan Huang, Arpita Singh, Tong Yu. Sungchul Kim, Victor Bursztyn. Nikos Vlassis, and Ryan A. Rossi. FigCaps-Hi A egure-to-caption generative framework and benchmark with human feedback, 2023.
Peter Stanchev, Weiyue Wang, and Hermann Ney. EED: Extended edit distance measure for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shured Task Papers, Day 1). pp. 514-520, Florence, Italy, August 2019. Association for Computational URL
Linguisties. doi: 10. 11111//5/11111155501
Till Tantau. The TikZ and PGF Packages, 2023. URL htttp:/://gittuu..om////gg--eeee
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. LLaMA 2: Open foundation and fine-tuned chat models, 2023b.
Edward Rolf Tufte. The visual display of quantitative information. Graphics Press, 1992. ISBN 978-0-9613921-0--9
Timothy van Zandt. PSTricks: PostScript macros for Generic TeX, 2007. URL https://ttu..rr/ PSTricks.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through simple sequence-to-sequene to-sequence learning framework. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári. Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 23318-23340 PMLR, 2022a. URI. https: //proceedings. wwt.mmmr.......... wtt.ppres//v622wwngg2222 html.
Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. CLIP-GEN: Language-free training of a text-to-image generator with CLIP, 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 wwwppnppp...p.eeeeeeeeeepppppppe cctpaper_pppppeeeeeeeeeee cccc/pppr_fille/peppeeeeepe /11ls//paprr/222//hahhh
December 9, 2022, 2022. URL http:://pappeee 9d5669613524ecF444 Fawoff4411ffffff3f 7b31aafffebbaaa AbCe444TTTTTTTTTTaaaaaaeetertttr
html.
Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. IconShop: Text-guided vector icon synthesis with autoregressive transformers, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardI.M: Empowering large language models follow complex instructions, 2023.
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Swarat Chaudhuri and Charles Sutton (eds.), MAPS@ PL.DI
18

---

Published as a conference paper at ICLR 2024
2022: 6th ACM SIGPLAN International Symposium Machine Programming. San Diego, CA. USA. 13 June 2022, pp. 1-10. ACM, 2022. doi: 10.1144/3520012235548822 URL https:
//doi. /145///28112355344222 3534862.
Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. SciCapt: Knowledge augmented dataset to study the challenges of scientific figure captioning, 2023.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUG-DocOwl: Modularized multimodal large language model for document understanding, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, A survey multimodal large language models, 2023.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large language models meet NL2Code: survey. Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7443-7464, Toronto, Canada, July 2023, Association for Computational Linguisties. doi: 10.18653/v1/2033.1 aallo/000....llooon11111111 URL http::::/aaaaaatllllaa// wrg.hloooy.og///// www./og.....2... acl-long 411.
Han Zhang, Tao Xu, and Hongsheng Li. StackGAN: Text to photo-rcalistic image synthesis with stacked generative adversarial networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp- 5908-5916 IEEE Computer Society, 2017.
doi: 10. 10.1109//CCV.2017..62
URL https://ddol /ttts:://doi.or//10000 118./ICCV. 2017.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023a.
Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. Controllable text-to-image generation with GPT-4, 2023b.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LLa LAR: Enhanced visual instruction tuming for text-rich image understanding, 2023c.
19

---

Published as a conference paper at ICLR 2024
A CODE & IMAGE COMPLEXITY
An interesting phenomenon we observe in our experiments in $5 is noticeable differences in TiKZ code size. As shown in Table 2, human-created TikZ drawings, after filtering comments, contain twice the number of tokens C and 1L which in turn have twice as many tokens as GPT-4 and CLAUDE 2. We investigate how this discrepancy is reflected in the compiled images, in particular, whether images created from code with fewer tokens are less complex and whether this also affects image quality. To this matter, gather another round of BWS scores. Analogous to $5.2, we form tuples with CLiMA13s LLAMA13e- GPT-4, and humans. We assign single annotator to evaluate image tuples in terms of complexity and, due to the potentially subjective nature of the task, four annotators evaluate image quality. Since we obtain an SHR of 0.68 for image quality, we conclude that all annotators have similar idea of what constitutes
Source
Tokens
Cplx
Qity
Human
916.56
0.76
0.83
CLiMA13m
422.56
0.46
0,4
LLAMA13a
420.28
0.46
0.39
GPT-4
186.73
0.31
0.38
CLAUDE 2
179.42
Table 2: Average number of tokens in TikZ documents, along with averaged min- max normalized BWS scores for image complexity (Cmplx) and quality (Qlty).
We display averaged and min-max normalized BWS scores in Table 2. Unsurprisingly, there is clear correlation between code size and image complexity. Humans score over twice as high as GPT-4 with CLiMA13s and LLAMA13e falling between. A link to quality is also visible, although less pronounced. Humans excel distinctly, but CLIMA LL and GPT-4 are not far apart. Nonetheless, in absolute terms, CLIMA13a still ranks higher than LLAMA and GPT-4 comes in last. Overall, this confirms our observations in $5.1 1 that GPT-4 and CLAUDE 2 produce code most different from humans, yielding figures that simpler than other models. Consequently, believe that our tool can assist humans better in creating more detailed TikZ drawings.
B PROMPT ENGINEERING
Our work involves developing a series of prompts guide general-purpose language models in accomplishing specific tasks. In this section, we discuss each model and the corresponding prompts we designed. Within a prompt, terms enclosed in curly braces symbolize placeholder variables that substituted during inference.
WIZARDLM We employ WIZARDLM to create captions for TEX Stack Exchange content based on question and its title (cf. $3.1). We force the generated tokens start with "Desired outcome:" which we subsequently strip from the output, along with any text generated after the first newline:
Create clear and specific caption for image that depicts the desired outcome of the following question. Utilize all relevant details provided in the question, particularly focusing on the visual aspects. Avoid referencing any example images or code snippets. Ensure that your caption is comprehensive and accurate:
(TITLE)
(QUESTION)
We use LAR generate short descriptions for scientific figures (cf. $3.2). We initially tried to provide LLA VAR with the original caption as context to improve it directly, but in many cases this seemed to confuse the model. Therefore, we decided to create short descriptions based solely on the figure and later append them to the original caption. We found that short, concise prompts work best for generating these descriptions:
LLAVAR
[IMAGE}
Write short description for the image.
20

---

Published as a conference paper at ICLR 2024
Model
CER₁
C
EED,
KID,
CLIP IMtt
CLIP
BLEU,
LL
0.644
1.183
56.393
0.059
80.237
26.733
2.732
LLAMA
0.552
1.178
55.762
0.053
81.12
26.898
3.258
CLiMA7a
0.545
1.202
56.045
0.068
80.671
26.776
2.82
CLIMA1SA 13s
0.656
1.184
55.708
0.05
81,017
26.966
3.369
CLIMA 0.658
1.175
55.264
0.057
82.252
26.994
3.469
CLAUDE 1.115
1.758
59.135
0.333
75.589
27.753
0.137
GPT-4
0.384
1.147
58.667
0.222
78.63
29.115
0.181
Table 3: Precise system-level scores for automatic evaluation metrics. CLIPSCORE abbreviated with CLIP and CRYSTALBLEU with BLEU. CER shows the average number of compile time errors. Arrows indicate whether larger or smaller scores are better, and the best scores are visually highlighted.
To generate TikZ drawings (cf. $3.1, $4, and $5). we use the same prompt for GPT-4 and CLAUDE 2, which was originally designed by Zhang et al. (2023b) and slightly modified for our task. After a comparison with our own prompts and prompts from other works (Bubeck et al., work best our initial experiments. As chain-of-thoughh prompt it breaks
GPT-4 & CLAUDE 2
2023), we found down the task of code generation into a series of logical steps. By default, the models tend to copy the input caption into the output image. To counter this, we additionally instruct the models not to include the caption in generated images. However, instances of caption copying still occur, discussed in $6:
Draw a TikZ picture for the following caption: (CAPTION). First, you need to provide an abstract step-by-step drawing guide. Then, you need to generate the full code (beginning with "Adocumentclass" and ending with "Nend(document)") following the guide. Avoid any direct inclusion of the caption any lengthy text sequences wi thin the code. Finally, summarize the drawing.
C ADDITIONAL DETAILS ON EXPERIMENTS
In this section, conduct additional ablation studies and provide more specific aspects of experiments in $5. This will provide clearer understanding of our methodologies and their respective considerations and limitations.
C.1 TRAINING CHALLENGES
During the training phase, we limit the context size of our models to 1200 tokens due constraints of our existing GPU resources, and filter out all examples that exceed this limit. Although most examples were within these limits, they were generally very close to the maximum allowable length. This prevented us from training our models to perform functions such as multi-turn conversations like chatbots, which could have allowed compiler logs to be fed back into the model to obtain improved outputs. Although did consider this approach in preliminary experiments with vanilla LLAMA and its instruction-tund derivatives in a zero-shot setting, the baseline performance of these models was poor produce usable TikZ. output. Instead, by drawing parallels to text-to-image generation models that condition the output solely on the input caption and using our iterative resampling technique improve the outputs, we provide more resource-cfficient alternatives with good performance.
C.2 IN-DEPTH EVALUATION METRIC SCORES
Table 3 presents the precise system-level scores from our automatic evaluation in $5.1. As observed in that section, CLIMA and CLiMAjaa generally outperform the other models, with CLIPScoRE being notable exception due GPT-4 and CLAUDE being prone to caption copying. The table also contains the Code Error Rate (CER), which indicates the average number compile time errors of our models. As evidenced, it largely reflects the trends of CSR. Most models generate less than
21

---

Published as a conference paper at ICLR 2024
Training Configuration
CER₁
CSR
EED=
KID,
CLIP nMat
CLIP
BLEU,
Full Training
0.545
1.202
56.045
0.068
80.671
26.776
2.82
Caption Augmentation
0.603
1.169
56.384
0.056
78.312
25.703
2.714
Image Training
0.681
1.178
56.336
0.08
80.129
26.604
2.736
-ArXiv Papers (86k)
2.145
1.546
60.552
0.229
75.15
25.328
0.886
-TEX Stack Exchange (29k)
0.624
1.207
56.274
0.071
80,461
26.958
2.936
AAriiiciil Examples (4k)
0.503
1.194
56.51
0.079
79.912
26.68
2.537
-Curated Examples (Ik)
0.656
1.22
56.246
0.072
80.346
26.815
2.711
Table 4: Ablation study results for CLiMA7s demonstrating how omitting either caption augmentation or randomly forwarding images to CLIP instead of captions during training impacts performance on the test set. The study also evaluates the impact of each data source by training CLiMA7, on subsets of DATINZ, omitting training (taken from Table 3) are shaded in red and green, respectively. Color intensity mirrors each score's proportion to the minimum and maximum values of its metric.
data source each time. Scores lower higher than those from full
one error on average, with CLAUDE being the only exception. Overall, GPT-4 achieves the best score, possibly because it has fewer opportunities make mistakes by generating shorter and less complex code than other models (cf. SA). Since CLIPSCORE, at fundamental level, evaluates similar aspects to CS from our human evaluation in $5.2 (caption-image similarity), and CLIPSCOREAN is the intuitive counterpart to RS (image similarity), we also calculate their respective Spearman correlations. At the segment-level, the correlation coefficients are 0.23 for CLIPSCORE and CS, and 0.25 for CLIPSCORE and RS (the degrees of correlation aligning with values commonly found for machine translation metries; Freitag et al., 2021; Rei et al., 2022). The system-level correlations are 0.5 and 0.8, respectively. The higher correlations witnessed for image caption- -imdio similarity lends credibility to our hypothesis regarding CLIPSCORE's biases favoring images of text.
C.3 ABLATION STUDIES
In this subsection we delve into ablation studies performed understand the degree to which data augmentation and the different data sources of DATIKZ contribute to the test set performance of our models.
Impact of Data Augmentation In our work, we incorporate two key data augmentation techniques: we (i) automatically augment image captions in DATIKZ (cf. $3.2), and we (ii) randomly replace image captions forwarded to CLIP with reference images during the training of CLIMA (cf. $5). Even though we show that caption augmentation improves caption-image alignment, we have yet to conclusively quantify its effect model performance. Similarly, although mixing text-only and text-image modalities is common when training models accepting both unimodal and multimodal input (Li et al., 2020; Wang et al., 2022a), the implications of our particular training strategy remain to be explicitly measured. To investigate these matters, we conduct an ablation study using CL.MA as a representative model for efficiency reasons. particular, we construct a version of CLiMA7, that uses DATIXZ without caption augmentation and another version where we train without sampling images. The performance on our evaluation metrics compared to full training results of CLIMA can be seen in Table 4.
Although eliminating caption augmentation reveals subtle improvements for CSR and KID, the model noticeably underperforms for all other metrics, particularly CLIPSCORE (1.1pp lower) and CLIPSCORFIee (2.4pp lower). Interestingly, the metrics that show an improvement do not compare model outputs with individual references but either do not use references at all (CSR) or use them only to evaluate general characteristics (KID). On the other hand, metrics with worse results are mostly reference-based. We attribute this observation the reduced amount of information carried by nug-angeetted captions, most likely leading to a decline in CLIMA's 's ability to capture the connection between captions and images and, therefore, delivering lower performance in reference-based metrics. Conversely, with less information that might otherwise constrain the output space, the model likely gain more flexibility in producing figures. We conjecture that this allows the model to concentrate
22

---

Published as a conference paper at ICLR 2024
more aspects that contribute positively to metries without direct references, such CSR and KID. This hypothesis provides a plausible explanation for our observed results.
Upon examining the impacts of not sampling images during the training process, we find that most metrics exhibit worse performance, with only CSR yielding comparable results. These findings support choice of training strategy and show that integrating images throughout the training process can bolster performance even when only textual inputs are considered at the evaluation stage.
In summary, we observe that both data augmentation techniques contribute positively to model performance. Caption augmentation enhances the caption-image alignment, benefiting CLiMA while leveraging images during training results in improvements well.
Importance of each Data Source Given the diverse data sources making up DATIKZ (arXiv, TpX Stack Exchange, curated examples, and artificial examples; cf. $3.1), we aim to assess the contribution of each source to the test set performance. This exploration serves multiple purposes: firstly, it enables us to justify the integration of each source; secondly, it can inform future data collection initiatives. To realize this objective, we develop four additional variants of CLiMA7,, where each variant is trained by omitting one data source and utilizing only the remaining three. As before, we plot the difference in performance compared to the full training results in Table 4, which should expose the importance of each data source.
The greatest performance decline occurs with the removal of arXiv, consistent outcome across all metrics. As the primary contributor of scientific figures totaling 86k, its absence instigates a decline of 4.5pp for EED, 5.5pp for CLIPSCOREAME 1.4pp for CLIPSCORE, and 1.9pp for CRYSTAI.BLEU Similarly, CER inflates nearly fourfold, and CSR experiences a noticeable increase, as well, Strikingly, however, results diverge for DATIZZ's s second largest source, the TEX Stack Exchange, which provides 29k examples. Contrary to expectations, the general downward trend is comparatively small, with EED and CLIPScORE only suffering a 0.2pp decrease and CLIPSCORE and CRYSTALBLEE even improving by 0.2pp and 0.1p respectively. We attribute this outcome to TEX Stack Exchange's unique focus on providing problem-specific minimum working examples rather than scientific figures like the rest of DATiKZ. Although this could still provide valuable training signals, we did not design our test set (only ~100 of its Ik random samples originate from this source) to evaluate the ability of models to follow technical advice given in the figure captions. Consequently, we plan to improve our testing framework in future work. Next, although we exclude artificial examples from our test set due to their non-human origin and their removal from the training data improves CER and CSR, their absence still has a negative impact on all other metrics. Without artificial examples, EED drops by 0.5pp. CLIPSCORFMaC by 0.8pp. CLIPSCORE by 0.1pp and CRYSTALBLEU by 0.3pp. showing that distillation of GPT-4 is a useful component in the training process. Lastly, even though curated examples constitute the smallest data source (less than 1k examples), their omission from the training data induces greater negative performance impacts than the removal of TEX Stack Exchange (0.2pp for EED, 0.3pp for CLIPSCORF and pp for CRYsTAIBBEEU We attribute this disproportionate value per size to the likely higher quality of code and captions inherent to curated examples.
Overall, the results are examined independently, we observe that each source has primarily positive effects. However, when evaluated in the context of the source sizes, we find that curated examples create particularly noticeably positive impact, whereas TEX Stack Exchange exhibits smaller effects. These insights will guide us to focus more on collecting such curated examples in future work.
D ANNOTATOR DEMOGRAPHICS
Our annotators are proficient in English, possessing a C1 level or above per the Common European Framework of Reference for Languages. is They all come from a science and technology background with research experience. They specifically consist of one female and one male faculty member, one female PhD student, three male PhD) students, and two female and two male assistants from other institutions.
In addition to these expert annotators, we also conducted preliminary tests with crowdworkers via Amazon Mechanical Turk, 16 gathering ten crowd-annotations per tuple for each task. However, the
>hhttps:/www.co coe. wtt.:/.o....nnnnnneeeeeeee int/lang-cefr whhttps://www.ttue www.:k.wmuukccccoo
23

---

Published as a conference paper at ICLR 2024
Belonane
LSTML
ESTM
LSTM.
LSTML
Bi p)
2
(a) diagram representing recurrent neural network consisting of several LSTM blocks, processing the input sequence simultancously forwards and backwards (to exploit both directions of temporal dependence). Con- tains rather tight manoeuvering.
(b) plot comparing the distribution functions Einstein, Boltzmann, and Fermi-Dirac statistics as a function of the reduced chemical potential B(c μ). This visualiation highlights the differences between the three types of distribution functions, which used descrihe the behavior of particles different statistical systems.
K
(6721
TD
4
1
(672)
SOS, of
Grea
(c) Tree with aligned matrix. A probability tree with an aligned matrix listing the possible outcomes, their probabilities and three columns for events described in later tasks. uses the grahdrawing library and requires LuaLaTeX.
(d) Our approach is modified version of meta-seq transformer decoder (TD) is trained to produce a se- quence actions o ..... m given a query instruction 12, The context are demonstrations (4 A&) produced by our generative model. We transformer encoder- decoder (T) encode instructions and state and transformer encoder (TE) to encode actions. The trans- formers that process instructions (pink blocks) receive state S as the input of the encoder.
Figure Illustrative examples from the DATiKZ, licensed for free distribution. The examples are from https: //gi://ittt http//ggttttttt..e com/ janos///s///tkz https://tikk..n
https://githul and httpp:://axiis org, respectively.
correlation between the results of the crowdworkers and the experts was strikingly low (p < 0.1). Since the crowdworkers also showed low agreement among themselves, we decided to discontinue further experiments with crowdsourcing.
E EXAMPLES
In Figure 5, we provide human-created examples from DATIKZ. Further, Figure contains examples of scientific figures generated by CLiMA LLAMA and GPT-4 for our human evaluation (cf. $5.2). Each model represented by one high-scoring and one low-scoring instance, as assessed by expert annotators in terms of caption similarity. is worth noting that the low-scoring example of GPT-4 suffers from caption copying. provide additional examples of caption copying Figure Instances of generated code can be found in Figure 8.
24

---

Published as a conference paper at ICLR 2024
2n
LL (goody; ansatz (5.17) (5.18) points are evenly spaced, and the blac points scale quadratically with dots scattered
ClaMApo (bad): the scheme, regular footprint
standard five-point Laplacian, and if any point in the foorprint is
n. The image is a white background with blue it The dots placed various urrungements, creating visually interesting pattern. Some dots are clustered tggtherr while others are spaced further uport, covering
cut cell, then "irregular" Cut colls are shown with dark shading.
irregular cells with light shuding, and the remaining while cells are "regular."
backgrond.
L (z)
K
Ri
R3
T&P
= (z)
R5
h
R₂
12
LLAMA13a (goody: Local configuration of poth foliation (in red) around the point (2)... The image displays complex mathematical formuta with various symbols and notation, including black dot, and physics or engineering, and it is wristen on a white background for easy
LLAMA13u (bady: Illustration the proof Theorem 2.7. The image displays tree with blue and orange labels white background. The tree hus unique structure, with branches that don't follow a typical layout. The labels instructions, and the tree is accompanied by several equations in the surrounding
The formula
rolated
he representing formula
readobility.
Averuee Reiative loss encoder recall accuracy on NQ by recall set size depth
0.8
2xy
0.4
Recall set size depti
GPT-4 (goody; Expression graph for algebraic expression x2 2 GPT-4 (bad): Average Relative loss in bi-cucoder recall accuracy on with symbol
The image shows mathematical concopt. The tree has veries of logical connections, and there are voriables mathematical symbols throughout structure. The image conveys sense of order and organization in the presentation of the mathematical concapt.
representing
recall set size depth the baseline, Pretrained Alignment (PT), Data Augmentation (DA), and Contrastive Alignment Post Training (CAPOT) noisy queries.
Figure 6: Examples of model-generatee generated scientific figures that received high ratings (good) and low ratings (bad) according to the perception of our expert annotators for caption similarity. The sections of the captions that have been augmented (cf. $3.2) are emphasized.
25

---

Published as a conference paper at ICLR 2024
GPT-4
optimal tomeports are My Wamarttein space. T; adied multiphied
Vi
T + T2
T2
V
(Ti
T2)/2
V2
CI
2
2
V1
Ti
CLiMA
(a) Motivating the definition of the addition @ and scalar multiplication @ operations for < a < 1 in the Wasserstein optimal transport space for transports Ti, (right), while in R² optimal transports defined by vectors Vj, V/2 (left).
Infleence uaining Scale Dowastcam Perfemanne
55
INFoGPT-ViTT'
Method 2
Method
downstream AP
6
54
.3
52
10
100
1000
2000
GPT-4
training Scale
CLiMA13a
pre-training scale
(b) The influence of pre-training scale on downstream performance. The value of each method is the average metric values across all datasets the privacy protection setting with LIGHTGBM.
An overlay TikZ picture below the colored layer appearing below the text rather than above it. Achieving this requires adjusting the z-order of the TikZ pieeue and
ordinary text,
GPT-4
CLiMA138
(c) An overlay TikZ picture below the ordinary text, colored layer appearing below text rather than above it. Achieving this requires adjusting the z-order of the TikZ picture and the text.
Figure 7: Additional examples of GPT-4 that suffer from caption copying. For reference, also provide the output of CLIMA13a for the same captions. The underlying TikZ code of Figure 7c can be found in Figure 8.
26

---

Published as a conference paper at ICLR 2024
idccumentclass((atticle))
tusepackage(tikz)
ppagestyle(eppty)
\begin(document)
\sffamily\Huge
Abegin( tikapicture)/oovrllay remember picture]
Anode at (current page.center)
Abegin(tikzpicture,
11
\fill[red] (0,0) (5,7) (10,0) cycle;
12
lend(tikzpicturg)
};
tend(tikzpicture)
15
16 \begin(center)
17 An overlay Tikz picture below the ordinary text, with the colored layer appearing below the text rather than above it. Achieving this requires adjusting the z-order of the TikZ picture and the text. Nend(center)
18
19
(eccmmncm)ttt)
GPT-4: An overlay TikZ picture below the ordinary text, with the colored layer appearing below the text rather than above it. Achieving this requires adjusting the z-order of the TikZ picture and the text.
Adocumentclass<(attiilee tusepackage(tikz)
lusepackage(mmsmatth)
(usetikelibrary((:kkkkr)))
\begin(document))
Abegin(alignx)
\sep(2+2=4)
Nend(align+)
11
kend(minipage))%
12
13
15
16
17 Valkmmmrroooode3333///uee22223an
18
Vend(minipaage)
picture]
20 \draw[red] (-1,-3) (3, (-1,-2) (5,-2) (-1,-1) (5,-1)
21
(-1,0) (0,0) (1,0) (2,0) (3,0) (5,0)
22
(-1,1) (0,1) (1,1) -(2,1) (3,1) - (4,1) (5,1) (5.5,1)
(-1,2) (0,2) (1,2) -(2,2) (3,2) - (3.5,2) (4,2) - (5,2)
(-1,3) (0,3) (1,3) -(1.533) (2,3) - (2.5,3) (3,3) (4,3)
25
(5,3) (5.5,3);
26
tend(tikzpicture)
27
\end(document)
CLIMA133 An overlay TikZ picture below the ordinary text, with the colored layer appearing below the text rather than above Achieving this requires adjusting the z-order of the TikZ picture and the text.
Figure 8: The generated code of GPT-4 (top) and CLiMA13a (bottom) from Figure 7c. Both models correctly use the overlay option, but they are not devoid issues. particular, GPT-4 utilizes nested tikzpicture environments which is generally discouraged and the output of CLiMA seems particularly verbose.
27


Reviews:
[{"review_id": "QCvba4mv11", "reviewer": "Reviewer_1jqj", "summary": "In this paper, authors collect 119k TiKz datat to fine-tune LLaMa, and achieve better performance compared to GPT4 and Claude2 in TiKz generation. \nAlso, authors propose a method to prevent the TiKz code from not compiling.", "questions": "1. Any code error rate comparsion? (compiling success rate)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good approach error handling method. Quite interesting study: on a specific domain, with relatively small data scale (119k) and LoRA, the author's approach can actually perform pretty well.\n2. Evaluation metrics are extensive and conving.\n3. Presentation is clear.", "weaknesses": "1. Seems like author didn't show any conversation example where users can edit or optimize the code while in the chat. \n2. Recently, [1] propose to leverage SVG, a similar format to TiKz, to conduct image understanding, generation and editing. How do authors judge on this?\n3. The error handling strategy seems quite standard and wildly used in programming languages. \n\n[1] Cai, Mu, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Lee. \"Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding.\" arXiv preprint arXiv:2306.06094 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wXDaEm95WG", "reviewer": "Reviewer_L9z3", "summary": "The paper proposes an approach to generating scientific images, by generating their TiKZ code, and open-sources the training dataset for this task. The paper investigates two solutions, namely LLaMa and LLaMa combined with CLIP-based image adapter. Both models are LoRA-tuned. The comparison with promping GPT-4 and Claude-2 is performed.", "questions": "How did the authors arrive at the specific prompt that was used for GPT-4 / Claude-2 and how do they know if that is the one that indeed yields a reasonably high performance from the models?\nGiven the iterative re-generation, the models could have also made use of the TikZ compiler error / stack trace, what was the rationale for not using this information?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: While this is not the first paper to do vector graphics generation with language models, the specific focus on TikZ is novel, and the published dataset is first of its' kind.\n\nQuality & Clarity: The paper is of high quality, with good related work section, clear metrics,and detailed analysis of the performance, including a human study. It provides the code accompanying the submission, and suggests the availability of the models (link removed for anonymity). The writing is mostly clear, doesn't have typos, the dataset composition is clearly outlined, as are many of the choices made by the authors (such as the choice of LLaMa vs LLaMa-2, the specific prompts used for the models, etc.).\n\nSignficance: The main point of signficance is the release of the dataset, fostering future work in this direction, and I believe it will be useful for the community.", "weaknesses": "The main weakness of the paper is the limited ablation study: The authors compare and contrast the two versions of the model, LLaMa and CLiMA (LLaMa + CLIP adapter) but don't highlight other important choices made, ex. \n* The effect of the aritifical samples. 50% of the proposed dataset is the artificial samples, but the evaluation is done on real samples. The effect of augmentation with artificial samples is not measured.\n* The effect of re-sampling the generation. The authors re-generate the output multiple times until the result is compilable, and the average number of re-generations is > 1.5, but the performance without regeneration is not reported.\n* The effect of data augmentation for CLiMa. As specified in section 5, during training CLIP is given either the input caption or the reference image (50-50%) and during inference it is given the input caption. The effect of this data augmentation is not reported.\n* The performance of vanilla LLaMa or prompt-tuned LLaMa is not reported, making the comparison between fine-tuned versions and prompted GPT-4 / Claude-2 not exactly fair.\n\nThe second issue is the limited model output results reported in the paper - the authors only show 6 examples in the appendix, and out of 3 labeled \"good\", the first one shows the image that, in my perception, doesn't really match the label (\"some dots are clustered together\" in the prompt, but shows just rows of points in the image)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XoQzeskdRu", "reviewer": "Reviewer_stBp", "summary": "The paper presents AutomaTikZ, a project aimed at automatically generating TikZ drawings from natural language descriptions. The authors introduce a novel dataset, DaTikZ, which consists of aligned TikZ drawings and captions, and a new model architecture named CLiMA that integrates multimodal CLIP embeddings into LLaMA. The results show that CLiMA outperforms both LLaMA and other proprietary models like GPT-4 and Claude 2 in various metrics. CLiMA's capability to process images opens potential applications in vectorization and sketch conditioning.", "questions": "- Can you please provide a couple of examples of different models' behavior to typographic attacks?\n- Have others considered comparing with ChatGPT? Maybe fine-tuned on a very small subset of the dataset.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The paper introduces a pioneering dataset and a novel model architecture, which is a significant contribution to the field of automatic drawing generation from textual descriptions.\n- The proposed model, CLiMA, demonstrates superior performance over existing models, including well-known ones like GPT-4 and Claude 2, across several evaluation metrics.", "weaknesses": "- Since the main contribution of the study is introducing DatikZ, it would be better to add a couple of simple captions to the code/image examples in the main draft", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tI2VD32CIO", "reviewer": "Reviewer_hZYD", "summary": "This paper focuses on TikZ, a language for vector graphics, and provides a dataset, DaTikZ, consisting of 120,000 pairs of TikZ and their captions. The authors are also evaluating the application of LLaMA and CLIP on the dataset. Experimental results show that models trained on DaTikZ can generate more complex vector graphics compared to those synthesized by closed-source LLMs such as GPT-4 and Claude 2.", "questions": "- The reviewer expects the authors to respond to the points listed in Weaknesses.\n- Figure 2 shows two radar charts. Although the main purpose of the radar charts is to make relative comparisons within the same chart, it would be possible to make some comparisons among the method in the left and right charts if the ranges of each score were adjusted.\n- When the authors conduct a human annotation campaign, they could also evaluate how well the automatic evaluation metrics used in section 5.1 correlate with subjective evaluation; are there any plans to do so?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- It is the largest scientific vector image and caption dataset to the best of the reviewer's knowledge.\n- Benchmarking with multiple baseline models is reported.\n- While automatic evaluation of generative models often encounters difficulties regarding their quality, this paper uses multiple scores for automatic evaluation to make the comparison meaningful. Multiple human subjective evaluations have also been considered and performed, as described in Sections 5.2 and D.", "weaknesses": "- As shown in Table 1, DaTikZ consists of multiple data sources. If DaTikZ consists of multiple data sources, the authors should evaluate which data sources contribute to the accuracy of the generation on the test data and to what extent. This would not only justify the use of each data source, but may also suggest what further data should be collected in the future. The contribution of data augmentation should also be evaluated.\n- Since this paper is also about a novel dataset for scientific vector graphics and their benchmarking, the technical contribution of the baseline method is modest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)