DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ
Authors: Jonas Belouadi
Simone Paolo Ponzetto
Steffen Eger
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Vision Language Models
Code Generation
Image Understanding
Vector Graphics Generation
Primary area: natural_language_processing
TL;DR: We train vision language models on TikZ code to automatically convert sketches and existing scientific figures into editable, semantics-preserving graphics programs.

Abstract: Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy. Furthermore, recreating existing figures that are not stored in formats preserving semantic information is equally complex. To tackle this problem, we introduce DeTikZify, a novel multimodal language model that automatically synthesizes scientific figures as semantics-preserving TikZ graphics programs based on sketches and existing figures. To achieve this, we create three new datasets: DaTikZv2, the largest TikZ dataset to date, containing over 360k human-created TikZ graphics; SketchFig, a dataset that pairs hand-drawn sketches with their corresponding scientific figures; and MetaFig, a collection of diverse scientific figures and associated metadata. We train DeTikZify on MetaFig and DaTikZv2, along with synthetically generated sketches learned from SketchFig. We also introduce an MCTS-based inference algorithm that enables DeTikZify to iteratively refine its outputs without the need for additional training. Through both automatic and human evaluation, we demonstrate that DeTikZify outperforms commercial Claude 3 and GPT-4V in synthesizing TikZ programs, with the MCTS algorithm effectively boosting its performance. We make our code, models, and datasets publicly available.

DETIKZIFY: Synthesizing Graphies Programs for Scientific Figures and Sketches with TikZ
Jonas Belouadi'
Simone Paolo Ponzetto'
Steffen Eger
Natural Language Learning Group, Data and Web Science Group" University of Mannheim," University of Technology Nuremberg www.mmiinnnn.mmmmmmmm de, steffen. eger@utn.
{jonas. belouadi, fonzetto]@uni y
Abstract
Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy. Furthermore, recreating stored in formats preserving semantic information is equally complex. To tackle this problem, we introduce DETIKZIFY, a novel multimodal language model that automatically synthesizes scientific figures semantics-presst preserving TikZ graphics programs based on sketches and existing figures. To achieve this, we create three new datasets: DATuxZv2. the largest TikZ dataset to date, containing over 360k human-createe TikZ graphics; SKETCHFIG, a dataset that pairs hand-drawn sketches with their corresponding scientific figures; and METAFIG, collection of diverse scientific figures and associated metadata. We train DETIKZIrY on METAFIG and DATIKZ,2, along with synthetically generated sketches learned from SKETCHFIG. We also introduce an MCTS-bar based inference algorithm that enables DETiKZIFY to iteratively refine its outputs without the need for additional training. Through both automatic and human evaluation, we demonstrate that DETIKZIPY outperforms commercial ClaUde 3 and GPT-4V synthesizing TikZ programs, with the MCTS algorithm effectively boosting its performance. make our code, models, and datasets publicly available.'
existing figures that
Introduction
Creating high-quality scientific figures similar typesetting scientific documents in many ways. When it comes to typesetting, markup languages like LSTEX enjoy widespread popularity, as exemplified by major machine learning conferences that either mandate strongly encourage SifX-ffrmmated submissions. 2 The advantages of using such languages go beyond producing high-quality outputs; documents expressed as high-level, semantics- preeiieiipeeeeiiggggg programs enhance accessibility, serve archival purposes, and remain easily editable and human-readabl (facilitating language modeling applications: Moosavi al., 2021; Lu et al., 2023). Consequently, efforts have been made to recover this type of information from outputs stored in lower-level vector graphics formats like PDF or SVG, or raster graphics formats (Desai et al., 2021: Blecher et al., 2024). At the other end the spectrum, the versatility of ыTEX comes with a steep learning curve, and typesetting can often be challenging for end users. In response, researchers have been working on assisting authors with certain aspects of the problem, such typesetting math based on hand-draw sketches (Kirsch, 2010; Wu et al., 2020).
Just like documents, scientific figures can also be created using markup languages. popular example is the TikZ graphics language (Tantau, 2023), which can be integrated into IeEX documents, providing comparable benefits and encountering similar challenges. However, unlike 18TEX, the prospects of TiKZ in research contexts remain largely unexplored. Although the promise of simplifying editing and
icml, http:///..eeeeeecccccccccccccccs
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

DETIKZIFY
Generate TikZ
LLAMA
BAEE Engine
AC
Linear 0
=
SIGLIP
Harness Reward
Figure I: Overview of the DETIKZIFY architecture: A multimodal language model converts sketches or figures into TikZ programs, which are compiled by a L8TX engine. This provides a reward signal to the model via MCTS, allowing it to iteratively refine the output until satisfactory results are achieved.
enabling applications in visual understanding (Masry et al., 2022; Huang et al., 2023) is evident, there are currently no viable solutions for recovering graphics programs from compiled figures. Moreover, there is lack of tools that assist in creating graphics programs, e.g., based on hand-drawn sketches, despite the clear demand for such approaches on the TІX Stack Exchange (TE:X.SE), where nearly 10% of all questions revolve around TikZ. making it the most frequeetly discussed topic on the site. Addressing this gap could greatly improve the accessibility of existing figures and support researchers all levels of programming proficiency when creating new ones, fostering diversity and inclusion. In response, we introduce DETIKZIFY, a multimodal language model that automatically synthesizes Tikz programs for scientific figures and sketches (cf. Figure 1). Our key contributions are as follows:
(i) As part of DETUxZIFY, we introduce (a) DATIKZA2, a large TikZ dataset with over 360k human-created TikZ graphics; (b) SKETCHFIG, a dataset of human-created sketches with paired scientific figures; and (c) METAFIG, large meta-dataset of scientific figures and associated texts.
(ii) We train DETIKZIFY on METAFIG and DATIKZ,2. augmented with synthetic sketches that mimic SKETCHFIG. We demonstrate that DETIKZIYY can effectively synthesize TikZ programs for both existing scientific figures and sketches, outperforming the commercial large language models (LLMs) GPT-4V and CLAUDE 3 (OpenAI, 2023b; Anthropic, 2024).
(iii) We also present an inference algorithm based on Monte Carlo Tree Search (MCTS) that is tailored to graphics programs and allows DETIKZIFY to iteratively refine its own oulpuly for a given computational budget, further improving performance without additional training.
2 Related Work
Image-to-ENT EXTxXXX- Conversion A closely related task is the translation of mathematical illustrations into I8TEX markup. In inspirational work, Kirsch (2010) tackle the recognition of single hand-drawn symbols to find corresponding ŁTTX commands. Subsequent works by Deng ct al. (2017); Zhang et al. (2017, 2019); Wu et al. (2020); Wang and Liu (2021) expand this concept to handle and scanned math formulas. Suzuki et al. (2003); Wang and Liu (2020); Blecher et al. (2024); Lv et al. (2023) further extend the scope by extracting LTEX formulas alongside text from entire documents.
Image Vectorization Similarly, converting (rasterized) figures into TikZ programs can be char- acterized form of image vectorization (Sun al., 2007: Diebel, 2008; Ganin et al., 2018; Li et al., 2020; Ma et al., 2022: Zhu et al., 2024). Most existing methods vectorize images into low-level graphics primitives in the SVG format (Tian and Günther, 2024). Although this works well for specific domains like fonts, icons, and emoji (Lopes et al., 2019; Carlier et al., 2020; Reddy, 2021; Rodriguez al., 2023b), it does not capture higher-level semantics and does not generalize well to our scientific context (cf. Appendix B). Closer to our work, Ellis et al. (2018) generate vector representations as graphics programs based on limited subset of ЕTTX commands. Their approach even handles

---

sketches, but their experiments are restricted to synthetic dataset with only basic shapes of limited complexity. Belouadi et al. (2024) also generate TikZ programs, but their primary emphasis is on conditioning the generation on textual descriptions, with images serving only as secondary input.
Code Generation As TikZ is implemented the Turing- complete TEX macro system (Erdweg and Ostermann, 2011), 1), our work is also closely tied to code generation (Xu et al., 2022). Despite continuing progress in this field (Chen et al., 2021; Li et al., 2022, 2023; Guo al., 2024; Lozhkov et al.. 2024), most research concentrates on high-resource languages like Python, Java, and JavaScript (Zan et al., 2023), typically overlooking TEX in evaluations. However, TEX and TikZ may still find their way into the training data, as demonstrated by the zhro ability of some models to understand and generate code in these languages (Bubeck et al., 2023; Belouadi et al., 2024; Sharma et al., 2024).
3 Datasets
We introduce DATIRZv2, to our knowledge, the most com- prehensive dataset of TiKZ graphics to date; SKErcHFIG, the first dataset comprising human-created sketches of scientific figures; and METAFIG, a large-scale scientific figure dataset with rich metadata. See Appendix E for examples.
Source
DATIKZA1
DATIKZN2
curated
981
1 1666
TEX.SE
29 238
30
arXiv
85 656
326450
artificial
1 1957
1958
serves as the primary source of TikZ. graphics for training DETIxZIPY. It is an expanded version of DA 1 (Belouadi et al., 2024), incorporating graphics from the same sources, namely curated repositories, Ti:X.SE, arXiv papers, and artificial examples, The key difference that DATIKZ,, includes all TikZ programs that compile with TEX Live 2023,4 regardless of whether they have associated captions, which was requirement for inclusion in DATIxZ1 but is not needed for DETIKZIFY. This approach allows us to create a dataset that is more than three times as large as its predecessor (cf. Table 1).
DATIxZ DA
all
117 832
360 583
Table 1: Breakdown of the number of unique TikZ graphics in DATIKZv2 com- pared to its predecessor DATiKZw1-
SKETCHFIG To create realistic synthetic sketches of scientific figures in DATIRZv2, we rely on examples of real human-created sketches, TEX.SE is a suitable source for collecting these, as users often illustrate their questions with sketches, and the answers provide the desired figure. We semi- automatically extract these figure-sketch pairs by first ranking all questions on the site that contain images based on their similarity to the string "a sketch of scientific figure" using a multimodal vision encoder (Zhai et al.. 2023). We retain the ones with high similarity scores, manually filter for true positives, and align them with the best matching figure provided in the answers. In total, we collect 549 figure-sketch pairs this way. As we also want to use this dataset for evaluation (cf. $6), we ensure that for subset of these sketches, no code provided in the answers included in DA DATIKZ
Beyond Tikz graphics, there is much larger pool of figures where the underlying source not available. Existing datasets that collect such figures frequently come with rich metadata, such as captions, OCR tokens, and paragraphs that mention the figures (Hsu et al., 2021; Karishma et al., 2023; Rodriguez et al., 2023a). Since such high-level descriptions are useful for pretraining (ef. $4; Liu et al., 2023b), we collect these datasets and merge them with the subset of figures in D DATINZa2 that have captions. This results in over 734k figure-text pairs, more than twice the size of DATIKZ.2.
METAFIG
The DETIKZIPY Model
Building previous work (Liu al., 2023b,a; Dai al., 2023; McKinzie et al., 2024), we build DETIKZIPY by combining a pretrained vision encoder with a pretrained language model (cf. Figure 1), where the vision encoder receives figures sketches generates corresponding TikZ programs as output. We focus on code language models that have been pretrained on TF:X, as this prior knowledge may be helpful for our task. All the models we end up using follow the LLAMA architecture (Touvron et al., 2023): CODELLAMA (Rozière et al., 2023) has likely been trained TEX code from arXiv (Touvron al., 2023), has been TINYLLAMA (Zhang et al.,
input images, and the language model
"https://uu..or//een
3

---

2024), while DEEPSEEK (code variant; Guo et al., 2024) was trained on TEX code from GitHub. For the vision encoder, we use SIGLIP (Zhai al.. 2023), which has been trained on OCR annotations (Chen et al., 2023c) and demonstrates state-of-the-ar understanding of text-rich images (Tong et al., 2024; Chen et al., 2023b), a crucial skill for our task. We then condition the LLMs on StGLIP's patch embedding vectors. To reduce the prompt length, we concatenate adjacent patch embeddings (Chen et al., 2023a). A feed-forward layer with dimensions 20 X LLLM serves as a connector, mapping image features of dimension SSILII to the LLM word embedding space of dimension ÔLLM.
Model Training We experiment with TIN and DEEFSEEK 1.30 (approximately billion parameters each) and CODEL and DEEPSERKT, (7 billion parameters each). When referring to specific variants of DETIRZIFY, we use the names DETiKZIFY-TLI. CLaa, and DETIsZuYy-SS; respectively. For all models, we use the SoViT 400m variant of SIGLIP as the vision encoder. Following Liu et al. (2023b,a), we first pretrain the connector with other model parameters frozen. We pretrain for one epoch on METAFIG with ADAMW (Loshchilov and Hutter, 2019), a batch size of 256, a learning rate of 1e-3, and a cosine learning rate decay with a 3% warmup ratio. Next, we unfreeze the language model (keeping the vision encoder frozen) and fine-tune on examples from DaTIKZ22 that fit within a 2048 token context window. We use a batch size of 128, a learning rate of 5,-5, and train for three epochs. Training data ablations can be found in Appendix B.
DETixZIFY-DS DETIKZIFY-
Synthetic Sketches When training DETIKZIFY on DATINZ,2, we randomly replace figures with synthetic sketches 50% of the time. Sketches are generated on the fly, meaning that each time figure is sampled as a sketch, a different synthetic sketch will be generated. Creating realistic sketches requires high-level image manipulation methods that go beyond traditional transformations like zooming or cropping. We, therefore, adopt INSTRUCT-PIx2P1 (Brooks et al., 2023), a model capable of diversely editing images based human instructions. We chose this model due to its remarkable zero-shot performance in generating synthetic sketches during our initial experiments. By then fine-tuning the model on SKETCHFIG, we further improve its performance (cf. $7 and Appendix C).
5 Iterative Refinement with Monte Carlo Tree Search
Due to the inherent probabilistic nature of language models, generating valid TikZ programs during inference can be a challenging task. The generated code may not always comply with the syntactic and semantic rules of TEX and TikZ, potentially leading to compilation errors. While constrained decoding algorithms can assist in guiding models towards generating valid programs (Ugare et al., 2024; Poesia et al., 2022; Scholak et al., 10, these approaches are limited to programming languages defined by context-free grammars (CFGs). However, TEX and TikZ are not defined by CFGs (Erdweg and Ostermann, 2011), rendering these methods ineffective for our purpose. Moreover, even if the generated code compiles successfully, fidelity errors such as misaligned elements, inconsistent scaling, repetitions, or mislabeling may only become apparent in the rendered output.
Despite these challenges, which make difficult guide DETIxZIFY based on intermediate states, we can still analyze completed outputs in a straightfor manner (c.g., by examining compiler diagnostics or comparing rendered outputs to the input image), allowing us to make informed decisions during subsequent sampling iterations. This concept of making decisions based on random sampling of the search space forms the core of Monte Carlo Tree Search (MCTS; Coulom, 2007). By integrating DETL with MCTS and adapting the standard MCTS algorithm to our problem domain, we can iteratively steer DIFY towards more promising regions of the output space (cf. Figure 1). In the following, we outline our fundamental approach, with further extensions discussed in Appendix A.
5.1 Integrating MCTS into DETIKZIFY
MCTS is versatile search algorithm that has been successfully applied to various domains, including board games (Silver et al., 2016, 2017), procedural content generation (Kartal et al., 20,6a,b; Summerville et al., 2015), and more recently, guiding language models to achieve long-term goals (Brandfonbrener et al., 2024; Zhang et al., 2023b; Chaffin et al., 2022). The algorithm incrementally builds a search tree and repeatedly runs simulations until an exit condition is met or a computational budget is exhausted. In our context, at depth n, each node's 's state consists of n lines of TiKZ. code, and edges represent continuations for generating the next line. Initially, MCTS starts with only empty root node and then iteratively performs the following four steps (cf. Figure 2):

---

Selection
(ii) Rollout
(iii) Expansion
(iv) Backpropagation
Figure 2: An example of the four steps of an MCTS simulation: The selection policy (i) reaches green backtracking node (normal nodes are blue), causing new nodes from the rollout (ii) to be added to the parent node during expansion (iii). The reward is backpropagated (iv) accordingly.
Selection Each simulation starts at the root node and successively selects child nodes based on a selection policy until leaf node is reached. The policy determines which parts of the tree should be explored further, balancing the exploitation of high-value regions and exploration of less-visited areas. Following previous work, as our selection policy, iteratively selecting the successor node i that maximizes the formula
Upper Confidence Trees (UCT; Kocsis and Szepesvári, 2006)
- ViJ + +c
In(Rptn)
UCT(1)
(1)
ni
where Ve, E [-1,
1J is the estimated value of í the jth visit, n, and Rp(r) are the visit counts i and its parent p(i), respectively, and is coefficient that controls the degree of exploration.
Rollout Once leaf node is selected, we utilize DETIKZIFY rollout policy. By conditioning it on the node's state, we continue to sample TikZ code until the -nd-of-seuuence token is encountered. This so-called rollout is then stored for reuse in the subsequent steps.
Expansion Next, the tree expanded by adding nodes from the rollout as new leaf nodes. While most implementations add only one node (i.e., one line of TikZ code) per simulation, computing rollouts with LLMs is computationally expensive. Therefore, inspired by MCTS for real-time settings (Soemers et al., 2016), we instead add multiple nodes. Specifically, we add v/rl - d, new nodes, where |r| is the number of lines in rollout r and di is the depth of the old leaf node l. This approach allows our tree to grow quickly in early simulations while converging to the standard case in the long run. To enable the tree to grow in multiple directions, nodes (Brandfonbrener et al., 2024: Chaslot et al., 2008). For each added node i, we add a backtracking node as sibling that mirrors the parent node p()). When backtracking node is expanded, its descendants are added to p(i) so that the backtracking node remains a leaf. This enables practically infinite search space anywhere in the tree while still maintaining a bounded branching factor.
also introduce backtracking
Backpropagation Finally, we calculate the value for rollout r using a predefined reward function (cf. $5.2) and backpropagate it to every node the path from the root node to the newly added nodes by appending it to Vi,- We also increment the visit counts n; for the same nodes. For backtracking nodes, only the visit counts are updated. Finally, we check any exit conditions. If MCTS terminates, we return the TikZ, program of the rollout that achieved the highest value.
5.2 Reward Functions
We explore two distinct reward functions to guide the search process. The first reward function utilizes compiler diagnostics identify documents that compile successfully. The second reward function provides visual signal based perceptual image similarity, which, in addition, helps find TiAZ programs that better match the input image. We explore further reward functions in Appendix A.
Compiler Diagnostics The biagnostics-based reward function based on analyzing the log file from compiling the generated TikZ program. We assign rewards according to the error state and whether an output file was produced. The reward function is defined as follows:
if the code compiles without issues, 0 if the code compiles with recoverable errors,
Vi.-
(2)
if compilation fails due a fatal error.
5

---

Reference Vigures
Synthelic Sketches
Models
MTT,
cBLEUU
TED,
KTD,
AVG₁
MTE₁
cBLFU
TED
DS04
SSIMey
KID,
AVG
CLAuDe:
51.812
0.111
57.389
64.896
83.372
17.822
0.148
50.156
0,024
59.731
59.102
73,954
29.541
0.189
GPT-4V
61,975
0.286
57.178
69.741
86,215
6.714
0.612
54.126
0.024
60.298
61.98
75.687
3332203
0.15
DT-ILI.M Iw
88.03
1.168
58.815
65.538
84.161
15.747
0.207
90.597
0.502
60.202
60.585
77.947
21.851
0.454
D Sx
83,771
1.336
57.661
68.659
86.079
11,536
0.572
87,446
0,541
60.112
62.756
79.097
17.334
0.642
DT-CC.
88.593
1.477
56.893
72.315
87.466
8,301
0.869
91.221
0.555
59.563
65.
79.717
12.207
0.941
DT-
82.366
1.815
57.227
73.01
88.323
5,951
0.965
89.299
0.69
59.693
65,198
80.207
12.207
0.965
Table 2: System-level scores for output-driven inference (DETIıZIFY abbreviated as DT). Bold and underlined values indicate the best and second-bos best scores for each metric column, respectively. Cell shading reflects the relative score magnitudes across input types. Arrows indicate metric directionality.
Self-Assessed Perceptual Similarity (SELESIM) SELFSIM computes the reward as the perceptual similarity (Zhang et al., 2018) between the input image and the compiled output figure. We hypothesize that DETIkZIFY itself can assess this similarity, enabling the model to guide its own search process. To achieve this, we encode both images into embedding vectors using DrTIKZIFY's s vision encoder and calculate SELISIM as their cosine similarity (Fu et al., 2023; Hessel et al., 2021). In cases where compilation fails, we assign reward of -1, In $7, we demonstrate that SELISIM correlates well with human judgments and outperforms other baseline methods.
6 Experiments
Before training on DATIXZy2, we extract Ik samples to serve as our test set for an automatic evaluation and generate corresponding synthetic sketches. To mitigate data leakage from pretraining to testing, we only include items created after the cut-off date of CODELLAMA and exclude repositories that may have been used in training DREPSEEK. We also use an n-gram matching algorithm to prevent cross-contamination with our train split (OpenAI, 2023a). For human evaluation involving human- created sketches, we also select 100 items from SKETCHFiG that do not overlap with DA D (ef. $3). Across all models, we set the temperature to 0.8 and the exploration coefficient c to 0.6. We provide examples of real and synthetic sketches as well as generated outputs in Appendix E and Table 4.
Baselines Given CLAUDE 3 and GPT-4V's potential for our task (cf. $2), we use them as baselines. Similar to DETIxZIPY, we instruct these models to generate TikZ programs for given images. However, as proprietary chatbots, they often mix code and natural language (Zhang et al., 2023e; Belouadi et al., 2024) and do not expose the internals needed to compute SELFSIM. This makes it impractical to apply our MCTS-based refinement algorithm, which is designed for code-only outputs and open models. Instead, compare approach to equivalent chat-oriented refinement methods, i.e., we use Self-Refine as an alternative to diagnostics-based MCTS and Visual Self-Refine as an alternative to SELrSIM-based MCTS (Madaan et al., 2023; ef. Appendix C for additional inference details). In Appendix B, we also explore SVG as an alternative to TikZ but find it less effective for our domain.
6.1 Automatic Evaluation
We introduce two inference tasks to automatically evaluate our models on the test split of DATIKZ,2. During output-driven inference (OI), we employ the diagnostics- based reward and use successful compilation as an early exit condition (we consider compilation successful if an output artifact is produced). For time-budgeted inference (TI), we use the more fine-grained SuLrSIM-basee reward and continue from OI until a computational budget of 10 minutes is exhausted (cf. Brandfonbrener et al., 2024), investigating the extent of achievable improvement. We report results for the two use cases where either (rasterized) reference figures or (synthetic) sketches serve as model inputs (cf. $1). Due to high inference costs, we only evaluate commercial CLAUDE 3 and GPT-4V in OI using Self-Refine. leaving TI with Visual
for human evaluation. We evaluate the following properties
Code Similarity To measure the similarity between generated and reference TiKZ programs, we use CRYSTAI.LLEU (cBLEU), variant of BLEU optimized for evaluating code (Eghbali and Pradel, 2023; Papinnni et al., 2002), and the TEX Edit Distance (TED), our adapted version of the Extended Edit Distance (Stanchev et al., 2019) combined with a TEX tokenizer.

---

Reference Figures
Synthetic Sketches
Models
MST+
cBLEUU,
TED
DSIM
S801,
KID,
AVG
NST+
cBLEU,
TED,
DS
SSiMG
KID,
AYG
DT
33.775
-0.011
-2.001
+8.704
+5.561
-12.146
0.128
35,975
+0.094
-0.628
+5.82 +3.026
+0.854
0.014
DT-DS
29.975
-0.028
-1.303
+8.464 +5.108
-8.728
0.531
32.429
+0.061
-0.504
45.573
+2.685
+5.493
0.22
DT-CLa
25.124
+0.07
-1.351
+7.797
+4.93
-4,868
0.876
26.219
+0.073
-0.468
+5.079
+2.455
+5,493 0.681
DT-DSt
24.145
-0.073
-1.542
+6.974
+3.893
-0.946
0.76
26.
+0.054
-0.696
+4.887
+2,241 +1,099 0.994
Table 3: System-level scores for time-budgeted inference, displaying relative changes for metrics shared with output-driven inference (Table 2: colored green for improvements and red for declines) and absolute scores for independent metrics. Bold and underlined values indicate the best and second-best absolute scores for each metric column, respectively. Arrows indicate metric directionality.
Image Similarity In addition to SELFSIM (SSIM), which can also be used a metric, we report DREAMSIM (DSIM; Fu et al., 2023), a fine-tuned metric for perceptual similarity. We also compute the Kernel Inception Distance (KID x 10²; Birikowski et al., 2018), which assesses the overall quality of generated figures by comparing their distribution with the distribution of reference figures. These metrics are always computed by comparing the generated figures to the reference figures, regardless of what the model receives as input.
Average Similarity To offer holistic view of each model's performance, we also compute the arithmetic mean (AVG) of all code and image similarity metrics. Given that these metrics operate on different scales, we min-max normalize their scores before calculating the average.
Efficiency For OI, we compute the Mean Token Efficiency (MTE) as the 10% winsorized mean of the ratio of the number of tokens in the final TikZ program to the total number of tokens generated arrive at that program. For TI, we instead compute the Mean Sampling Throughput (MST), measuring the throughput of unique TikZ graphics for the given budget.
Results Table 2 presents the system-level metric scores for OI As expected, the scores for reference figures are, on average, 38% higher than those for synthetic sketches, but similar patterns emerge across both input types. DETIxZIFY- -CL and DETIKZIFY-DD DD2% consistently outperform all other models, achieving AVG scores of 0.869 & 0.965 for figures and 0.941 & 0.965 for sketches, respectively. In contrast, GPT-4V reaches AVG scores of only 0.612 and 0.15, placing it in competition with the smaller 1b models: for figures. GPT-4V surpasses DETIxZIFY-TL, II and D 1.3 which achieve scores of 0.207 and 0.572, respectively. However, these smaller models outperform GPT-4V on sketches, where they achieve scores of 0.454 and 0.642. CLAUDE 3 trails behind all our models, with an AVG of only 0.148 and 0.189. When examining individual similarity metrics, DrTuZZry-DS,, the top-performing ZETLKZIFY model overall, surpasses GPT-4V, the best baseline, by more than 3pp (percentage points) on average for DREAMSIM and SELESIM, while maintaining noticeably lower KID. In terms of cBLEU, GPT-4V, and CLAUDE 3 only reach 6.5-18.5% of the performance achieved by the lowest-scoring DETIxZIFY model (DETinZury-TLLLs. less pronounced, possibly due to the influence of boilerplate code, which cBLEU inherently ignores.
The differences in TED are
For efficiency, all DETIKZIFY models demonstrate an MTE of 82-91%. indicating that only 1-2 out of 10 inference runs require a second simulation to generate compilable TikZ program. Interestingly. the model size does not seem to particularly influence this score, with the pretraining setup appearing to be the key factor instead. For instance, DETIRZIFY-TLL, and DETIXZIFY- -CZz share a similar pretraining setup and exhibit comparable MTE values, as do DITIRZIPY-DS, and DETIZZIPY-DSA We can further observe that (i) MTE generally higher for sketches compared to figures, and (ii) for figures, the MTE of similarly pretrained models is inversely correlated with their scores other metrics. These phenomena likely stem from models making fewer mistakes when the input is less detailed when their understanding of is limited finding that aligns well with other studies (Tong et al., 2024). Compared to DETIKZIFY, CLAUDE 3 and GPT-4V perform considerably worse, with an MTE of only 50-62%. Notably, for these models, 98.5% of the items already compile after the initial Self-Refine step, meaning that this inefficacy primarily originates from the natural language texts surrounding the code and that Self-Refine is nearly equivalent to regular sampling based inference.
The results for ZETY on TI are presented Table Remarkably, increasing the computational both reference figures and sketches as input
budget for MCTS improves nearly all metrics for without requiring access to any additional knowledge. The improvement with sketches is particularly noteworthy, as it demonstrates that the refinement process enhances the desired properties even when
7

---

Sampling
X
MCTS
Human Sketches
GPT-4V (Ol)
SELFSSM
44
GPT-4V (TT)
42
DE DrTwZnnv-Ss (TT)
40
200
400
600
Reference Figures
Time (seconds)
Figure 3: Bivariate distributions of BWS scores (higher is better) using kernel density estimation (left) and log-linear regression over Tl reward scores for different generation strategies over time (right).
the model input type differs from the one used for evaluation. The 2.2-5.6pp increase of SELrSIM for all models is not surprising since it serves the reward signal we optimize, but DREAMSIM and TED also increase by 4.9-8.71p 7p and 0.5-2pp, respectively, demonstrating the efficacy of our approach. While KID improves by 1-12.1 points with reference figures, it drops by 0.9-5.5 points with sketches. We believe this is because sketches often omit minor details, such as axis tick labels, which is reflected more in the output of the TI models. biasing their overall output distributions. Therefore, we consider the substantial improvement of metrics capturing instance-leved similarities to be more important. For cBLEU, we observe only minor changes (less than =0.Ipp). aligning with findings that BLEU-based metrics become less effective as performance increases (Ma et al., 2019). The MST and AVG reveal that, although 1b models produce more unique outputs within the time frame compared to their larger 7b counterparts (30-36 vs. 24.1 24.1226.2) they still fail to close the overall gap in performance, with AVG scores ranging between 0.014 0.531 compared to 0.681-01 189999994 for 7b models.
Overall, all DETIxZIFY models are capable of generating compilable outputs with reasonable efficiency, Upon examination of these outputs, it becomes evident that the 7b models, particularly DETIxZIPY- DS consistently outperform both CLAUDE 3 and GPT-4V, whose performance is more comparable the 1b range. Increasing the computational budget for DETIxZIFY further improves performance.
6.2 Human Evaluation
To further assess the quality of the generated figures, we perform a human evaluation on SKETCHFIG using Best-Worst Scaling (BWS; Louviere et al., 2015; Kiritchenko and Mohammad, 2016, 2017). In this process, for each reference figure, we present annotators with tuple of generated figures and ask them to identify the most and least perceptually similar figure. We then transform this data into scores ranging from -1 (poor) to 1 (excellent) by calculating the difference between the proportion of times a figure is selected the best and the proportion of times it chosen the worst (Orme, 2009). To keep the workload manageable, we focus on the most promising DETIxZIFY model (DETIxZIFY-DS,, and the strongest baseline (GPT-4V). Building upon the automatic evaluation, we assess these models in the OI and TI configurations, using either reference figures or human-created sketches as input. For each input type, we engage six unique expert annotators (cf. Appendix D for more details).
Results Figure 3 (left) shows kernel density estimates for the computed BWS scores, revealing intriguing findings that are consistent across input types. In contrast the automatic evaluation, DeTixZIFY-DSr performs worse (mean score f = -0.32) than GPT-4V (μ 0.09) in OI. This could be attributed the fact that TEX.SE, the sole source of SKETCHFIG, emphasizes minimum working examples, type which GPT-4V particularly excels (Belouadi et al., 2024). However, when we increase the computational budget, as DrTikZiry-DS, (TI), it not only improves over OI results (μ 0.39; in line with automatic evaluation) but also surpasses GPT-V in both configurations by a considerable margin. Interestingly, GPT-4V's performance TI (μ = -0. 16) is lower than its performance in OI, indicating that GPT-4V (TT) struggles to refine its own outputs effectively and quickly deteriorates. Overall, this shows how difficult is for models to refine their own outputs and highlights the effectiveness of our MCTS-based approach. Example outputs are provided in Table 4.

---

Input
GPT-4V (01)
GPT-4V (TI)
DeTuZwr--Sy, (01)
DrTwZuv-DSh, (TI)
Z/nZ
R/rd
2/42
7/1Z
eVR.
QNR
On.R. QuR
Q
Q.
Q
QNR.+1
Table 4: Examples of model inputs and generated outputs from our human evaluation, where annotators rated GPT-4V (OI) higher than DE (OI) but ranked DETUZZIFyDDS. (TI) as the overall best model, illustrating our findings $6.2. See Appendix E for more examples.
7 Analysis
In this section, we take a closer look at our methodologies and evaluation strategies, correlating evaluation metries with human judgments, quantifying the quality of synthetic sketches, and examining the rate of convergence our MCTS algorithm. We also demonstrate that our models are not affected by memorization of the training data, shown in Appendix B.
Correlating Humans and Metrics To assess the reliability of our human evaluation results, we investigate the agreement between annotators. To this end, we calculate the split-half reliability (SHR; Kiritchenko and Mohammad, 2017) by randomly splitting our annotations into two subsets, computing BWS scores for each subset, and measuring their correlation with Spearman's p. The SHR values of 0.69 for sketches and 0.75 for images indicate a moderate to strong correlation between annotators, supporting the validity of our human evaluation results. Motivated by these findings, we explore whether metrics that also assess perceptual similarity (i.e., SELFSIM and DREAMSIM) correlate with these human judgments. We again calculate Spearman's 's and show the average correlations (David M. Corey and Burke, 1998) at the segment and system level in Table 5. For comparison, et al., 2018; Ding et al., 2020). At the segment level, SELFSIM outperforms all other metrics, which is remarkable considering it is the only untrained metric. Segment-leve performance particularly important for fine- -naiinai reward functions, justifying our choice SELSSIM our MCTS algorithm. At the system level, DREAMSIM performs the best, showcasing its strength in evaluation settings.
Metric
Segment
System
LPIPS
0.224
0.642
DISTS
0.32
0.642
DSIM
0,424
0.954
SSIM
0.436
0.642
Table 5: Correlations of image similarity metrics with humans the segment and system level.
also include the popular LPIPS and DISTS metrics (Zhang
Synthetic Sketch Quality We also assess the quality of our synthetic sketches by measuring their congruence coefficient (Lorenzo-Seva and Berge, 2006) with real sketches. We embed human- created figure-sketch pairs from SKETCHFiG using SIGLIP, subtract each sketch embedding from the corresponding figure embedding to obtain local sketch vectors, and perform a component Principal Component Analysis to derive a global sketch vector (Zou et al., 2023). We repeat this process synthetic sketches generated for the test split of DATiKZaa and compare the global vectors using cosine similarity. Base PATRUCTT-Pxx2P. generates synthetic sketches with congruence coefficient of 0.66, which increases to after fine-tuning. These results demonstrate high correlation with human human-created sketches, suggesting that generated sketches are of good quality.

---

MCTS Convergence To gain insights into the long-term characteristics of our MCTS algorithm. we visualize the trends in achieved TI reward scores over time in Figure 3 (right) and compare them to conventional sampling-based inference. As expected, sampling does not lead to improvements over time due to the absence of feedback loop. In contrast, MCTS consistently improves throughout the entire time frame, and even at the end of our budget of 10 minutes, it does not appear to converge, suggesting potential additional gains for larger budgets. Apart from this, MCTS is not only more effective but also faster. With an average MST of 25.17, compared to 18.7 for sampling, our MCTS algorithm generates considerably more unique TikZ programs within the same amount of time.
8 Conclusion
In this work, we showcase the potential of DETIRZIFY in generating TikZ programs for two practical use cases. First, it can convert existing figures from lower-level formats into Tikz, paving the way for semantic image editing and downstream tasks (Zhang et al., 2023a). Second, it can develop hand-drawn sketches into TikZ graphics, which could aid researchers in creating illustrations. In both cases, DETIKZIFY substantially outperforms the commercial LLMs GPT-4V and CLAUDE 3 despite its presumably much smaller size. We hope that our datasets (DATiKZv2, SKETCHFIG, and METAFIG), our method for generating synthetic sketches, and our MCTS-based inference algorithm will pave the way towards future research on graphics program synthesis and bolster the cause of open science.
scientific
Looking ahead, we plan to extend our approach to other graphics languages, such as MetaPost, PSTricks or Asymptote (Hobby, 2014; Van Zandt, 2007; Hammerlindl et al., 2024). We also intend to explore alternatives to perceptual similarity as an MCTS reward signal, including per-pixel measures and point cloud metries (Wang and Bovik, 2009; Wu et al., 2021). In addition, we aim to investigate reinforcement learning from reward functions, for example, using Direct Preference Optimization (Rafailov al., 2023; Xu al., 2024). Finally, while this work focuses on visual inputs, we plan to explore additional modalities, such as text and mixed -mxxedyy inputs, in future work.
Limitations
In this work, we compare openly available models with proprictary systems that lack transparency in their training details and internal workings and whose performance is not stable over time. This inevitably complicates efforts to address concerns such as data leakage or cross-contamination and limits the fairness and reproducibility of experiments. Nevertheless, under these adverse conditions, our open models and methods demonstrate favorable performance. Users should be aware, however, that models might inherit biases, flaws, other limitations present in the training data, potentially leading to discrepancies between expected results and generated outputs. Furthermore, given the inouurce-intesive nature of LLMs, many of our training and inference were adopted from related work or chosen based on general intuition. Although LLMs are generally robust to
selection (Beyer al., 2024), conducting thorough byper-parameter search might enhance their performance further. Finally, it should be noted that our models could potentially be misused by malicious actors to produce misinformation and fake science.
Another important consideration is that the public release of DATIKZv2 does not include some TikZ programs from internal version due to licensing restrictions. These programs distributed under the arXiv perpetual, non-exclusivg license, which prohibits redistribution. Nonetheless, we provide our dataset creation scripts alongside usage instructions, enabling anyone reproduce the full version of DATIKZ.22 independently. The remaining TikZ programs in DATiXZu2 are licensed under Creative Commons attribution licenses, 5 the GNU Free Documentation License, 6 or the MIT license,? 7 and their respective terms and conditions apply. Regarding artificially created examples, OpenAl's terms of use restrict the use of their services for creating competing products, limiting this subset of DA TixZz2 to non-commercial applications.*
http://wwww....oog/iic////////// e...1n.......... html
10

---

Acknowledgments
We would like to express sincere gratitude to the following individuals for their contributions our work: JiWoo Kim, Tommaso Green, Christoph Leiter, Ines Reinig, Martin Kerscher, Margret Keuper, Christopher Klamm, Daniil Larionov, Yanran Chen, Tornike Tsereteli, and Daniel Ruffinelli. Their assistance with our human evaluation campaign, proofreading, insightful discussions, and constructive comments have been invaluable. The last author is supported by the Federal Ministry of Education and Research (BMBF) via the research grant METRICS4NLG and the German Research Foundation (DFG) via the Heisenberg Grant EG 375/5-1. We would also like acknowledge the OpenMoji project for providing the open-source icons used throughout this work and Hugging Face for their generous community GPU grant.
References
Anthropic. 2024. The Claude 3 model family: Opus, Sonnet, Haiku.
Jonas Belouadi and Steffen Eger. 2023. UScore: An effective approach to fully unsupervised evaluation metries for machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 358-374, Dubrovnik, Croatia. Association for Computational Linguistics.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. 2024. AutomaTikZ Text-guided synthesis of scientific vector graphics with TikZ. In The Twelfth International Conference on Learning
Representations.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey Gritsenko, Neil Houlsby. Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender. Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. 2024. PaliGemma: versatile 3b VLM for transfer. Preprint, arXiv:2407.07726.
Mikotaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton, 2018. Demystifying MMD GANs. In International Conference on Learning Representations.
Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2024. Nougat: Neural optical understanding for academic documents. In The Twelfth International Conference Learning Representations.
Ali Borji. 2023, Qualitative failures of image generation models and their application in detecting deepfakes. Image and Vision Computing, 137:104771.
David Brandfonbrener, Sibi Raja, Tarun Prasad, Chloe Loughridge, Jianang Yang, Simon Henniger, William E. Byrd, Robert Zinkov, and Nada Amin. 2024. Verified multi-step synthesis using large language models and Monte Carlo tree search. Preprint, arXiv:2402.08147.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023. InstructPix2Pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18392-18402.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4, Preprini, arXiv:2303.12712
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. 2020. DeepSVG: A hierarchical generative network for vector graphics animation. In Advances in Neural Information Processing Systems, volume 33, pages 16351-16361. Curran Associates, Inc.
Nicholas Carlini, Daphne Ippolito. Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference Learning Representations.

---

Antoine Chaffin, Vincent Claveau, and Ewa Kijak. 2022. PPL-MCTS: Constrained textual generation through discriminator -eriidd MCTS decoding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2953-2967, Seattle, United States. Association for Computational Linguistics.
Guillaume M. J-B. Chaslot, Mark H. M. Winands, H. Jaap Van Den Herik, Jos W. H. M. Uiterwijk, and Bruno Bouzy. 2008. Progressive strategies for Monte-Carke tree search. New Mathematics and Natural Computation (NMNC), 4(03))343-357.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023a. MiniGPT-v2: large language model unified interface for vision-languaa multi-task learning. Preprint, arXiv:2310.0947..
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. 2023b. PaLI-3 vision language models: Smaller, faster, stronger. Preprint, arXiv:2310.09199
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2023c. PaLI: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations.
Rémi Coulom. 2007. Efficient selectivity and backup operators in Monte-Carlo tree search. In Computers and Games, pages 72-83, Berlin, Heidelberg. Springer Berlin Heidelberg.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards general-purpot vision- languaee models with instruction tuning. In Seirth- Conference on Neural Information Processing Systems.
Giannis Daras and Alex Dimakis. 2022. Discovering the hidden vocabulary of DALLE-2. In NeurIPS 2022 Workshop on Score-Based Methods.
William P. Dunlap David M. Corey and Michael J. Burke. 1998. Averaging correlations: Expected values and bias in combined pearson rs and fisher's z transformations. The Journal of General Psychology, 125(3):245-211.
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. 2017. image-to-markun generation with coarse-to-fine attention. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 980-989.
PMLR.
Harsh Desai, Pratik Kayal, and Mayank Singh. 2021. TabLeX: A benchmark dataset for structure and content information extraction from scientific tables. In Document Analysis and Recognition ICDAR 2021, pages 554-569, Cham. Springer International Publishing.
12

---

James Richard Diebel. 2008. Bayesian image vectorization: The probabilistic inversion of vector image rasterization. Ph.D. thesis, Stanford University, Stanford, CA, USA. AA13332816.
Keyan Ding, Kede Ma, Shiqi Wang. and Eero Simoncelli. 2020. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2567-2581
Aryaz Eghbali and Michael Pradel. 2023. CrystalBLEU: Precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ASE '22, New York, NY, USA. Association for Computing Machinery.
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning to infer graphics programs from hand-drawn images. In Thirty-second Conference on Neural Information Processing Systems, pages 6062-6071.
Sebastian Thore Erdweg and Klaus Ostermann. 2011. Featherweight TeX and parser correctness, In Software Language Engineering, pages 397-416, Berlin, Heidelberg. Springer Berlin Heidelberg.
Stephanie Fu, Netanel Yakir Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. 2023. DreamSim: Learning new dimensions human visual similarity using synthetic data. In Thirty-seventh Conference on Neural Information Processing Systems.
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. 2018. Synthesizing programs for images using reinforced adversarial learning. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1666-1675. PMLR.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coderr When the large language model meets programming the rise of code intelligence. Preprint, 1411:2
Andy Hammerlindl, John Bowman, and Tom Prince. 2024. Asymptote: The Vector Graphics Language.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi, 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
John D. Hobby. 2014. MetaPost.
Ting-Yao Hsu, Lee Giles, and Ting-Hao Huang. 2021. SciCap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics; EMNLP 2021, pages 3258-3264, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Chich-Yang Huang. Ting-Yao Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin Chan, Eunyee Koh, Lee Giles, and Ting-Hao Huang. 2023. Summaries as captions: Generating figure captions for scientific documents with automated text summarization. In Proceedings of the 16th International Natural Language Generation Conference, pages 80 92, Prague, Czechia Association for Computational Linguistics.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024. Prismatic VLMs: Investigating the design space of models. In Forty-first International Conference on Machine Learning.
language
Zeba Karishma, Shaurya Rohatgi, Kavya Shrinivas Puranik, Jian Wu, and C. Lee Giles. 2023. ACL-Fig: A dataset for scientific figure classification. In Proceedings of the Workshop Scientific Document Understanding co-located with 37th AAAI Conference on Artificial Inteligence (AAAI 2023), Remote, February 14, 2023, volume 3656 of CEUR Workshop Proceedings. CEUR-WS.cr
Bilal Kartal, Nick Sohre, and Stephen Guy. 2016a. Data driven Sokoban puzzle generation with Monte Carlo tree search. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 12(1):58-64.
13

---

Bilal Kartal, Nick Sohre, and Stephen Guy. 2016b. Generating Sokoban puzzle game levels with Monte Carlo tree search. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, UCCAI-1, The IJCAI-16 Workshop on General Game Playing, pages 47-54. International Joint Conferences on Artificial Intelligence Organization.
Svetlana Kiritchenko and Saif Mohammad. 2017. Best worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1-178 San Diego, California. Association for Computational Linguistics.
Daniel Kirsch. 2010. Detexify: Recognition of hand-drawn LaTeX symbols. Diploma thesis,
University of Münster, Münster, Germany, October.
Levente Koesis and Csaba Szepesvári. 2006. Bandit based Monte-Carlo planning. Machine Learning: ECML 2006, pages 282-293, Berlin, Heidelberg. Springer Berlin Heidelberg.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 957-966, Lille, France. PMLR.
Mengtian Li, Zhe Lin. Radomir Mech, Ersin Yumer, and Deva Ramanan. 2019. Photo Photo-Sketching: Inferring contour drawings from images. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1403-1412.
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehacne, Joel Lamy-Poirier, Joao Monteiro. Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavine Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. 2023. StarCoder: may the source be with you! Transactions Machine Learning Research. Reproducibility Certification.
Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kellee 2020. Differentiable vector graphics rasterization for editing and learning. ACM Trans. Graph., 39(6).
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago. Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-Ievel code generation with AlphaCode. Science, 378 1092-
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. Preprint, arXiv:23 10.00344.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In seirth- Conference on Neural Information Processing Systems.
Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. 2019. A learned representation for scalable vector graphies. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV).
14

---

Urbano Lorenzo-Sewe and Jos M. F. ten Berge. 2006. Tucker's 's congruence coefficient as meaningful index of factor similarity, Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2(2):57-64.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015, and Applications. Cambridge University Press.
Scaling: Theory, Methods
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jía Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain. Yixuan Su, Xuanli Hc, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebare, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2024. StarCoder 2 and The Stack V2: The next generation. Preprint, arXiv:2402. 19173.
Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. SCITAB: A challenging benchmark for compositional reasoning and claim verification on scientific tables. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7787-7813, Singapore. Association for Computational Linguistics.
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, and Furu Wei. 2023. Kosmos-2.5: A multimodal literate model. Preprint, arXiv:2309. 1149..
Qingsong Ma, Johnny Wei, Ondiej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy. Association for Computational Linguistics.
Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. 2022. Towards layer-wise image vectorization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 163 1-32441
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Systems.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguisties: ACL 2022, pages 2263-2279, Dublin, Ireland. Association for Computational Linguistics.
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2023. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN. Transactions of the Association for Computational Linguistics, 11:652-677
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, Anton Belyi. Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang. Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. 2024. MM1: Methods, analysis & insights from multimodal LLM pre-training Preprint, 240.:2403309966
15

---

Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. 2020. A three sample hypothesis test for evaluating generative models. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3546-3556. PMLR.
Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021. SciGen: a dataset for reasoning-aware text generation from scientific tables. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
OpenAl. 2023a. GPT-4 technical report. Preprint, arXiv:2303.08774.
OpenAL. 2023b. GPT-4V(ssion) system card.
Bryan K. Orme. 2009, Max analysis: Simple counting, individual-level logit, and HB, Sawtooth Software Research Paper Series.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Lingusttics.
Sayak Paul. 2023. Instruction- tuning Stable Diffusion with InstructPix2Pix. Hugging Face Blog. -onnitiioo--tnnn-s.-.....
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE
Press.
Vikas Raunak and Arul Menezes. 2022. Finding memo: Extractive memorization in constrained sequence generation tasks. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5153-5162, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Pradyumna Reddy, 2021. Im2Vee: Synthesizing vector graphics without vector supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2124-2133.
J. A. Rodriguez, D. Vazquez, I. Laradji, M. Pedersoli, and P. Rodriguez. 2023a. OCR-VQGAN: Taming tixt-within-imee image generation. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3678-3687, Los Alamitos, CA, USA. IEEE Computer Society.
Juan A. Rodriguez, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, David Vazquez. Christopher Pal, and Marco Pedersoli. 2023b. Statorctor: Generating scalable vector graphics code from images. Preprint, arXiv:2312.1
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Seialom, and Gabriel Synnaeve. 2023. Code LLaMA: Open foundation models for code. Preprint, arXiv:2308. 12550.
Y. Rubner, C. Tomasi, and L.J. Guibas. 1998. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision (IEEE Cat. No. 9.CC8127271,, pages 59-66.
16

---

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph., 35(4).
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto- rego-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 0895-99001 Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Pratyusha Sharma, Tamar Rott Shaham, Manel Baradad, Stephanie Fu, Adrian Rodriguez-Munee Shivam Duggal, Phillip Isola, and Antonio Torralba. 2024. A vision check-up for language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14410-14411
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche. Julian Schrittwieser, Ioannis Antonogiou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominík Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587)::88) 489.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang. Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. 2017. Mastering the game of go without human knowledge. Nature, 550(7676):3544 359.
Dennis J. N. J. Soemers, Chiara F. Sironi, Torsten Schuster, and Mark H. M. Winands. 2016. Enhancements for real-time monte-carlo tree search in general video game playing. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8.
Yurun Song, Junchen Zhao, and Lucia Specia. 2021. SentSim: Crosslingual semantic evaluation of machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3143-3156, Online. Association for Computational Linguistics.
Peter Stanchev, Weiyue Wang, and Hermann Ney. 2019, EED: Extended edit distance measure for machine translation. In Proceedings of the Fourth Conference on Machine Translation ( (Volume 2: Shared Task Papers, Day 1), pages 51 514-520, Florence, Italy. Association for Computational Linguisties.
Adam Summerville, Shweta Philip, and Michael Mateas. 2015. MCMCTS PCG 4 SMB: Monte Carlo tree search guide platformer level generation. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 11(3):68-74.
Jian Sun, Lin Liang, Fang Wen, and Heung-Yeung Shum. 2007. Image vectorization using optimized gradient meshes. ACM Trans. Graph., 26(3):11-e) l-es.
Masakazu Suzuki, Fumikazu Tamari, Ryoji Fukuda, Seiichi Uchida, and Toshihiro Kanahori. 2003. INFTY: an integrated OCR system for mathematical documents. In Proceedings of the 2003 ACM Symposium on Document Engineering. DocEng '03, page 95-104, New York, NY, USA. Association for Computing Machinery.
Till Tantau. 2023. The TikZ and PGF Packages.
Xingze Tian and Tobias Günther. 2024. A survey of smooth vector graphics: Recent advances in representation, creation, rasterization, and image vectorization. IEEE Transactions Visualization and Computer Graphics, 3063):155--1671.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9578.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. ILaMA: Open and efficient foundation language models. Preprint, 13777.
17

---

Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2024. Improving LLM code generation with grammar augmentation. Preprint, arXiv:2403.01632.
Timothy van Zandt. 2007. PSTricks: PostScript macros for Generic TeX.
Zelun Wang and Jyh- Liu. 2020. PDF2LaTTeX: A deep learning system to convert mathematical documents from PDF to LaTeX. Proceedings of the ACM Symposium on Document Engineering 2020, DocEng '20, New York, NY, USA. Association for Computing Machinery.
Zelun Wang and Jyh-Charn Liu, 2021. Translating math formula images to LaTeX sequences using deep neural networks with sequence-level training. International Journal on Document Analysis and Recognition (IJDAR), 24(1):63-75.
Zhou Wang and Alan C. Bovik. 2009. Mean squared error: Love it or leave it? new look at signal fidelity measures. IEEE Signal Processing Magazine, 26(1):98-111
Jin-Wen Wu, Fei Yin, Yan-Ming Zhang, Xu-Yao Zhang, and Cheng-Lin Liu. 2020. Handwritten mathematical expression recognition via paired adversarial learning. International Journal of Computer Vision, 128 (28(10):2386244) 2401.
Tong Wu, Liang Pan, Junzhe Zhang, Tai WANG, Ziwei Liu, and Dahua Lin. 2021. Balanced Chamfer distance as a comprchensive metrie for point cloud completion. In Advances in Neural Information Processing Systems.
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In MAPS@ PLDI 2022: 6th ACM SIGPLAN International Symposium Machine Programming, San Diego, CA, USA, 13 June 2022, pages 1-10. ACM.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023. Large language models meet NL2Code: A survey. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7443-7464, Toronto, Canada. Association for Computational Linguistics.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975-11986.
Jianshu Zhang, Jun Du, and Lirong Dai. 2017. A GRU-based encoder-decoded approach with attention for online handwritten mathematical expression recognition. In 20/7 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 902-907.
Peiying Zhang, Nanxuan Zhao, and Jing Liao. 2023a. Text-guided vector graphics customization. In SIGGRAPH Asia 2023 Conference Papers, SA '23, New York, NY, USA. Association for Computing Machinery.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. TinyLlama: An open-source small language model. Preprint, arXiv:2401.00 003355..
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023b. Planning with large language models for code generation. In The Eleventh International Conference Learning Representations.
Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. 2023e. Controllable text-to-image generation with GPT-4. Preprint, arXiv:2305.1855 18583.
18

---

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations.
Wei Zhang, Zhiqiang Bai, and Yuesheng Zhu. 2019. An improved approach based on CNN-RNNs for mathematical expression recognition. In Proceedings of the 2019 4th International Conference on Multimedia Systems and Signal Processing, ICMSSP '19, page 57-61, New York, NY, USA. Association for Computing Machinery.
Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the limitations of pross-lingual encoders as exposed by reference-free machine translation evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1656-1671, Online. Association for Computational Linguistics.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 563-578, Hong Kong, China. Association for Computational Linguistics.
Haokun Zhu, Juang Ian Chong, Teng Hu, Ran Yi, Yu-Kun Lai, and Paul L. Rosin. 2024. SAMVG: A multi-stage image vectorization model with the aggnent-anything model. ICASSP 2024 2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 4350 4354.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. 2023, Representation engineering: A top-down approach to Al transparency. Preprint, 10.04455
19

---

A Further Details on MCTS
In this section, discuss several extensions MCTS algorithm that aim to improve its performance and efficiency. We also explain alternative reward functions that we experimented with but ultimately found less effective than chosen approaches.
A.1 MCTS Enhancements
Building on our base MCTS implementation, we introduce several enhancements, namely dynamic rescaling of visual rewards, node deduplication, and preemptive stopping of faulty rollouts.
Dynamic Rescaling One challenge when using SELFSIM is that MCTS expects values to be in the range of [-1, while deep encoders often work with much narrower range in practice (Hessel al., 2021; Zhang et al., 2020). Furthermore, this range may vary depending whether the input image is a real figure or sketch. To address this discrepancy, we propose dynamically min-max normalizing the visual reward scores whenever they (re)computed, ensuring that MCTS always operates on the full range. The modified reward formula is as follows:
V.y-min(V, (-1)) max V.) min IIM
if Vi. # 1 and max max(Vt.) min VxA((-))),
Vi
if V # - and max max (Vi.) min VrM--1)
(3)
otherwise.
Node Deduplication During a rollout for a backtracking node, is possible to generate code that already exists elsewhere in the tree (i.e., in siblings and their descendants). To prevent the duplication of nodes, we always merge identical node states before adding any nodes to the tree.
Preemptive Stopping If the code generated in rollout cannot be compiled due fatal error, we record the rollout, including the state which the faulty line of code was first introduced. the same (intermediate) state is sampled again during subsequent rollouts, we know that the completed output will fail to compile. In such cases, we preemptively abort the rollout and reuse the previously recorded rollout for the remainder of the simulation. To further prevent continuations from faulty code, during the expansion phase, we only add nodes to our tree whose node states do not contain any lines of code with fatal errors.
A.2 Additional Reward Functions
Taking inspiration from popular machine translation metrics (Belouadi and Eger, 2023; Zhao al., 2019, 2020; Song et al., 2021), which compute the Earth Mover's Distance (EMD; Rubner et al., 1998; Kusner et al., 2015) between word embeddings, we also explore with measuring perceptual image similarity as the EMD between SIGLIP's image patch embeddings. Given the distance matrix D, where Di,j cos(x, y)) and x, are the patch embedding vectors of the input and output images of simulation j with lengths |xr and [y], respectively, EMD is defined as follows:
Σ Σ 3
5 Fi.
I Di,
Σ
Fi.
EMD(x, y)
with
min
Vi.j
1 2 j=1
xxl
Fi.j
Fi,
We define Vr, 2 tanh( - EMD(x, y)) [-1, if compilation produces any output. If compilation fails, set the reward to -1. We empirically tune the hyperparameter which layer to extract the patch embeddings using the perceptual similarity dataset of scientific figures from Belouadi et al. (2024). We find that extracting embeddings after the 24th layer yields the best results. However, when evaluated on our data, this reward function achieves a segment-level correlation of only 0.425 (cf. $7), which is lower than for SELFSIM while being computationally more expensive. Consequently, do not employ this reward function in further experiments.
B Additional Experimental Results & Analyses
In Table 6, we compare LIVE (Ma et al., 2022), state-of-the-art method for generating SVG, with our TikZ-based approach. In Figure 4, we additionally investigate the extent to which our models memorize the training data. We also perform training data ablation studies, presented in Table 7.
20

---

Reference Figures
Synthetic Sketches
Models
DSIM,
S
KID
DSIM,
SSIM,
KID]
LIVE
57.078
69.253
324.219
49.455
64.998
416.016
CLAUDE 3
64.896
83.372
17.822
59.102
73.954
29.541
GPT-4V
69.741
86.215
6.714
61.98
75.687
33.203
DETisZIFY-LLI.I
65.538
84.161
15.747
60.585
77.947
21.851
DETINZIEY-DSI.5 68.659
86.079
11,536
62.756
79.097
17.334
DETIKZIFY-CL7 72.315
87.466
8.301
65.118
79.717
12.207
DETIwZEYY-S77 73.01
88.323
5.951
65.198
80.207
12.207
Table 6: System-level scores for LIVE, an SVG-generating model, compared with TikZ-based models from output-driven inference. Scores for TikZ-based models are copied from Table 2 for easy reference. Bold and underlined values indicate the best and second-best scores for each metric column, respectively. Cell shading reflects the relative score magnitudes across input types. Arrows indicate metric directionality.
B.1 Comparing TiAZ and SVG
Since LIVE generates SVG code instead of TikZ, do not report cBLEU and TED scores. Additionally, because it optimizes Bézier curves rather than gen- erating tokens, we exclude MTE, leaving only the image similarity metrics DREAMSIM, SELESIM, and KID. Table 2 shows that LIVE underperforms all other models in our evaluation. On reference figures, it scores over 7.8p 7.8pp and 1p lower than the worst baseline model on DREAMSIM and SELFSIM, respec- tively, and its KID is more than 18 times higher. This subpar performance can be attributed the complex- ity of scientific figures saved as SVGs. While we use LIVE in its default configuration, generating eight paths with four segments each, our scientific figures consist of over 110 paths on average with an arbi- trary number of segments, not counting deduplicated paths, which LIVE cannot detect. Although we could theoretically configure LIVE to generate more paths, this would linearly increase inference time, quickly becoming intractable. LIVE already requires over 18 hours to complete the test set for one input type, whereas DETiKZIFY-CS: (OI), for example, takes less than 5 hours. Furthermore, since LIVE attempts to vectorize the input directly without semantic interpretation, it performs even worse on synthetic sketches. We conclude that SVG, and, by extension, models that generate SVG, not well-suited for our problem domain and objectives.
80%
60%
40%
Human
DETuZZIFY-DSI.a DeTuxuyy DS
20%
CLAUDE: 3
GPT-4V
0%
1 1-gram 3-gram 5-gram 7-gram 9-gram
Figure Proportion of generated code n- grams with E [0] that are novel (i.e., not present in the training data). Results for human-created code are included as reference point for comparison.
B.2 Memorization
Memorization of training data a common concern in language models (MeCoy et al., 2023; Carlini al., 2023; Raunak and Menezes. 2022: Meehan et al., 2020). To assess the extent of this issue in our models, we calculate the n-gram novelty (McCoy et al., 2023). Specifically, we determine the proportion of n-grams, with n E [1, 10], in the model-generated TikZ programs that are not present in the training data. We perform this analysis on the test split of DATUKZ,2 for our baselines and DEEPSEEK-baRd DrTIxZIrY models conditioned on reference figures, as well human generaede code, as shown in Figure 4. All models initially exhibit similar novelty and are slightly less novel than humans for n < 7. However, starting from n 7, all models except DETIxZIRY-DD -D surpass human novelty, with more than 80% of all model-generated n-grams being novel for n >= 8. This phenomenon of models becoming more novel than humans is commonly observed and is considered an indicator that language models are not significantly affected by memorization (McCoy al.,
21

---

Reference Figures
Syuthetic Sketches
Models
cBLEU
TED
DS
S851,
KID
MTF;
cBLEU
TED
DS11
S8m4,
KTD
Full Training
83.771
1.336
57.661
68.659
86.079
11.536
87.446
0.541
60.112
62.756 79.097
17.334
-Synthetic Sketches
-1.957
-0.822
+1.318
+3.296
-133388
+0.171
-3,991
-3,332
+34.18
-MeTAFiG
-1.846
-0.096
-0.356
40.398
-0.046
40.115
-0.132
40.053
-0,378
40.084
-0.181
42.773
Table 7: Ablation study results for DT-TLL.1s (OD), showing the relative impact test set performance when either sketch-based training or connector pretraining omitted, compared to full training. Improvements are highlighted in green, and declines in red, with reference scores taken from Table 2.
2023; Belouadi et al., 2024). Interestingly, for larger n-grams, DEThxZIeY-DST demonstrates higher novelty than its smaller counterpart, suggesting that despite its larger capacity, it does not overfit and generalizes well. The most novel models are GPT-4V and CLAUDE 3, possibly because they were not trained on DaTiKZa2 and might have been trained on data that has been prepared differently.
B.3 Training Data Ablation Studies
To better understand the impact of training with synthetic sketches and pretraining using METAFIG on test set performance, we conducted ablation studies with DETiRZiry-SS,a in the OI configuration representative model, following the experimental setup detailed in 1..1. In partieular, Table 7 compares full training with variations where synthetic sketches are excluded and the step of pretraining the connector omitted. The results from excluding synthetic sketches align with expectations: although this approach slightly improves performance on reference figures on average, it substantially reduces performance sketches. Therefore, for models expected to perform well both figures and sketches, we recommend our original training methodology. Conversely, for models focused solely on figures, training exclusively on figures may be advantageous. The findings related to skipping connector pretraining are less definitive as the score differences are minimal, reflecting the lack of consensus in related literature about the benefits of connector pretraining for downstream performance (Liu et al., a, Karamcheti et al., 2024). However, average, we observe positive impact, especially on MTE and KID, where consistent improvements are noted for both reference figures and synthetic sketches input. Thus, we advocate incorporating a dedicated pretraining step in the training protocol. future work, we also plan to investigate the impact pretraining dataset size and quality.
C Additional Training & Inference Details
In this section, provide supplementary information on the training and inference procedures for all our models. For training and inference of our local DETIKZIPY models, we utilize a compute node equipped with four Nvidia A40 GPUs and 448 gigabytes of RAM. We access CLAUDE 3 and GPT--V through their respective official API endpoints.
C.1 DETIKZIFY
Complementing the information provided in $4, our lb models require approximately two days of fine-tuning on our hardware. For the 7b models, we employ optimizer state and gradient partitioning (Rajbhandari et al., 2020) to accommodate them within the available resources, resulting in an extended training time of 21 days. Generating sketches for the training runs takes additional 1.5 days, but since we cache our sketches, these costs are incurred only once. Output-driven inference takes 4 8 hours, depending the model and input type, and time- budgeed inference extends the runtime by a further 1.5 days.
C.2 INSTRUCT-PIX2PIX
As SKETCHFiG with only 549 examples may be considered too small for fine-tuning INstRUCT-Pix2Pix we augment our training data with 4000 additional sketches of natural images (Sangkloy et al., 2016; Li et al., 2019) and 2000 synthetic sketches of scientifie figures generated with base INsTRUCT-Pixx2ix We then oversample SKETCHFIG at 5:1 ratio and, following Paul (2023), train for 15k steps with
22

---

This the TikZ/LaTeX code for the scientific figure shown in the picture labeled "Input". Can you improve it better resemble the provided reference [ sketch picture ]? First, analyze the "Input" picture to understand its components and layout. Then, consider how the scientific figure can be enhanced to more closely match the reference sketch picture ]. Finally, rewrite the Tikz code to impl emete these improvements, making the image more similar to the reference. Ensure that the LaTeX code is self- 12 contained and does not require any packages except TikZ-related batch size of 8 and learning rate of 5e-5. We select "turn it into doodle" as our initial prompt. which also appears in INSTRuCT-PIx2PIx's pretraining dataset and demonstrates the most promising zero-shot performance.
C.3
CLAUDE 3 & GPT-4V
Building upon the experiments described $6, we derive all our Self-Refine prompts from the official examples provided by Madaan et al. (2023) for generating TikZ programs, with only minor modifications. In particular, we employ the following prompt template in the initial step of both Self-Refine and Visual Self-Refine, substituting "sketch" "picture" appropriate:
This is a [ sketch picture ] of a scientific figure. Generate LaTeX code that draws this scientific figure using TikZ. Ensure that the LaTeX code is self-contained and does not require any packages except TikZ-related imports. Don't forget include \usepackage
I understand that this is challenging task, so do your best. Return your result in "latex code block.
We then extract the first LTEX code block from the generated text. In the rare cases where GPT-4V incorrectly classifies input images as unsafe, we add a small amount of Gaussian noise to the image pixels to bypass the issue. If compilation fails due a fatal error (which occurs in only 1.5% of all cases) without producing an output artifact, we repeatedly use the following prompt template until all issues resolved, replacing <code> with the generated code and <error> with the corresponding error message:
Given the error message:
<error>
And the problematic code:
"latex
<code>
7 First, identify the issue based on the error message. Then, determine the cause of the error in the code. Finally, propose and implement a solution. Return the fixed code in a 'latex code block.
For Visual Self-Refine, we additionally use the following prompt template to visually refine the output. Since we provide two input images (the initial figure sketch and the current output), we label one "Input" and the other as "Reference". CLAUDE 3's API has built-in mechanism for labeling images, while for GPT-4V, we embed the labels directly into the images:
lateex
<code>
10
13 imports. Don't forget include (usepackagettik2)! Return your " result in a lltex code block.
Following the findings of Madaan et (2023), we visually refine for a maximum of four iterations, as they observe diminishing returns beyond that point, and it helps reduce inference costs. Although this means that in most cases, we terminate before the 10-minute timeout is reached (cf. $6. 1), we believe this is sensible decision, observe that GPT-4V is unable to visually refine its
23

---

outputs successfully in any case. We hypothesize that this limitation is due to general purpose chat models requiring too much explicit context for this task. These models receive the entire previously generated code as input, along with two input images and complex textual prompt, which may be too challenging for them to process effectively. Preliminary experiments with more elaborate prompts did not seem to mitigate the subpar performance, likely due to this reason.
general-purpose
D Annotator Demographics
Our annotator team consists of eleven experts with extensive research experience in science and technology. The team comprises one male faculty member, two female PhD students, seven male PhD students, and one male research assistant from another institution. We chose to work exclusively with expert annotators based on the findings of Belouadi et al. (2024), which demonstrated that crowd annotators often lack the necessary research background to produce reliable annotations.
E Examples
To provide better understanding of our work, we present a variety of examples in this section. Table 8 displays exemplary figures and real sketches from SKETCHFIG, while Table 9 shows figures and synthetic sketches from DATIKZv2. Additionally, Tables 10 & 11 present sample outputs generated by our systems during our human and automatic evaluations. Figure 5 provides a closer look at generated code.
When comparing the real sketches in Table 8 to their corresponding reference figures, it becomes evident that the sketches often contain less detail. For instance, sketches may lack colors or grids and feature less precise lines. Moreover, the handwritten nature of the sketches can sometimes make the text within them harder to read. These characteristies are also present in the synthetic sketches shown in Table 9. However, the problem of illegible text is more pronounced in these sketches, as generating readable text remains a common challenge for image generation models (Borji. 2023). While the text may still retain its meaning in hidden way (Daras and Dimakis, 2022), this could lead to hallucinated text in the generated TikZ programs. Nonetheless, we believe that this aspect can still be advantageous for end users, as it enables them to quickly add scribbles to indicate the desired text placement. By doing so, DETIKZIPY can generate code for the overall structure and layout, allowing users to easily modify and replace the text afterward.
The randomly selected generated figures from our human and automatic evaluations (cf. $6.2 & 16.1) shown in Tables 10 & 11 corroborate our quantitative findings. DETIxZIFY-SS; (TT) demonstrates the best overall performance and shows the least amount of fidelity errors, confirming the effectiveness of our SELFSIM- MCTS refinement algorithm. However, we still observe some inconsistencies, such as in layout and axes labeling, although to a lesser extent compared and GPT-4V. We attribute the prevalence of this problem partly to our focus on perceptual similarity rather than, e.g., pixel-level similarity, which allows the models greater flexibility in interpreting the general semantics of the input figures and sketches. While optimizing pixel-level similarity could be an alternative approach, we argue that perceptual similarity can serve as a more meaningful measure, especially when considering sketches. We believe that real users who provide rough sketches of unfinished ideas will find the generated outputs that interpret and refine their concepts to be inspirational. However, we acknowledge the potential benefits of exploring more rigorous similarity measures and plan to investigate this in future research. Interestingly, GPT-4V occasionally generates outputs that may not be appropriate in scientific context, such as mistakenly embedding smiley face in the fourth example in Table 10. Instead of resolving such issues, GPT-4V (TI) further emphasizes these details, distancing the output from the actual reference.
DETIzZIFyDSS,, (OI)
Figure 5 provides a byde-b comparison of the generated TikZ. programs corresponding to the first row in Table 10. DrTieZiry-DS; demonstrates its ability to utilize advanced abstractions and control flow statements, generating code that free of compile-time errors in both OI and TI configurations. On the other hand, GPT-4V (OI) incorrectly uses an undefined arrow tip kind stealth in lines 9 and 10, resulting in recoverable compile-time errors. GPT-4V (TI) contains the same error in line and introduces additional errors in lines 16 and 26, where the * symbol would have to be removed from the loop lists for successful expression evaluation.
24

---

Reference Figures
Real Sketches
Arll
SUN
One Dimegnsion
One Dimomsion
A
Second Dimersion
A
A
Secoad Mimenetoo
AA
AB
A
2%
3%
47
AA
B 0
BA
BB
2%
13%
dt
e
K
(()
0
extinction
extiactinn
d
SYSTEM
System
HEASOREMENT
Measuzenenl
M
E
MODEL
Model
Table 8: Representative examples of reference figures paired with real sketches from the SKETCHFIG dataset.
25

---

Reference Figures
Synthetic Sketches
www. wWW.MMAAAAAA WWW.IIAAAAUUURAAA i
4p
0
7 /
-1
Theory/ model model
Inference
06s
Obsrvations
Predictions
Daffiins
hypothesik testing
Experiments
loae cnnd
Obelanom
(oy
1971
Table 9; Illustrative examples of reference figures and corresponding synthetic sketches from the subset the DATIKZ,, dataset that is licensed for redistribution.
26

---

Input
GPT-4V (O1)
GPT-4V DeTuZZwy-SS; DeTuZuuy-Dy% (TI)
Te ire
Garenc bipott
CONTRIC
13
32
Table 10: Alternating rows of randomly selected reference figures and real sketches (first column) alongside corresponding scientific figures generated by GPT-4V and DiTiiZIrY-DS, in output- driven (OI) and bungeeed (TI) configurations (columns 2-4), taken from human evaluation campaign (cf. $6.2).
27

---

Inpet
CLAUDE (OI)
GPT-4V (O))
DeTlsZur-DSS, DrTuZZey-DS% (T)
Dor
chat
dha
Table 11: Alternating rows of randomly selected reference figures and synthetic sketches (first column) alongside corresponding scientific figures generated by CLAUDE (OI), GPT-4V (OI), and DETIKZIPY-DS; (OI & TI) in columns 2-4, taken from our automatic evaluation (cf. $6.1).
28

---

ddocumentclass
(horree Senorddd--prrtieeee [standalone]
Vdocumentel ass iizz,, border-3. 14enn] (standalone) \usepackage (tikz- 3dplot)
Nusepackageftike waw.ccagggeikikklleee
(document))
Lkset brary (arrows, bng
ttéplorssttaiincoords///7
(10)
Vbeginí (document)
rotated coards
tedplatsetssinnconndd (116)
[trppott_mein cot.o_iiinnoooreeeeee line join:round]
(beginttkkrpicturer ttikppicture) wttp:oootrinccoodssscooooo thick]
1pg f//m/sher format/.cd.gren fixed)
Vdraw
Deraw[-[Laexxiiiee [Latexx [bend]] (8,6,0) (3,0,0) node [poe-1.1111111 (5x55);
Adraw[ (Latex(bend))
(8,3,8) [pos-1.11[55]]]
\deFun(12)
(8.0.0) (0.0.3) 3) node [pos:1-1]]33:3
12
(daf(b[2)
(foreach VX [renenber-V (X as Mastt tinitially 831 in (1....,
12.
13
(Nbegin (scope) [canvas plane =>LastX/22]
13
canvas plane
circle(radius=21 M XLfnun\X-1
y-81
15
[1ltte] (hh (bb
Adraw [- (bend)) oorange, shorten se-1pt]
16
16
Vdraw[-laters [-latex] (0.00 -1 (1.0):
sirclefrddius-1.c
17
Moreach (8.....88)
\else
17
18
Adraw (orange, (orange,sborten aborten >e-1pt] circle[radius=1.8
18
19
[draw[thick] (VEMA/Vn,8) circlefradiss:1);
VEI
19
20
/end( scope))
20
Send(scope)
tend[tikapicture
22
Vend (dncumentt
23
(foreach V in Aa)
24
/drawT-Iated -Ttee, thick, brown] (MM///A,N
(h'M//nn Ab, Ab/mm//n);
26
2?
\beginfscope [toppl wtw.llot.tttttd.cco coords, canvas plane at y=\b]
29
\draw [ateex] (0,9) -- (1,0):
30
(foreach (B....
32
[draw(thick] (W/M//n,9) circle(radius=1);
33
34
(scop)
36
[1atex] (0,0,0) (1,0.0) (so==ss11.1]3333
37
\drael [-latex] (0,0,3) (8.11.) [se[pos-1.1][yy33
38
Vdraw [-latex] (8.9,3) (8.9.1) ppoe(pos:1.1]]
41
(ennddeeeeent))
TuzZar DS,, (OI)
DrTixZer DS
dlocumentclass (tike border-16pt: (standalone)
(documentclase, [tikz, baaderborrr-11pt bocder=11pt3[sssmalloor
\usepackage( (tikk)
Jusepackage(tiin
lusepackage amsmath)
\usetikelibrarf 3d, calc, neta))
sieetikalle (brrry dec.otttooossiinns markings)
Vbegin (document)
document)
\beging ctuuei [scale=1.5]
Abeginit
Styles for and
Styles http:/styllleeeeeeeeee (thick, "5.sestellth")
(axis/ aty.eethhicc....ssstee.tt
/sikzset(axis// wwtp///ttlle/etteeeeeeoe
teikzseetveecterr serettr/.sstylle sttyles poyslates orange))
10
Neikzset silksse(vvector// sttyle:tthiiceeeee 22, pestealth' .orange))
\tikzset (helix/ sttyle: width- black,
10
"[Trinngle (vidth- Jan, smnghhanmm)33
12
Braw
13
Mfreacch \t in (8.19.....60;
$ Define the number of turs for the helix (ded (denns))s((3)
12
14
/draw (daratnsktt//ee11 smooth, variableswy, black)
13
15
plot (()z/368"0) (( (sic))))))
14
belix Xt......turnss55e (turns"36e
17
Vforeach Ve in (8,38
16
18
s Braw azis
[draw[belix] (sl/366)),(cin(tt)),......
Vdraw [belix] (ent-))))) (ennttt))),
17
19
(draw[axis] (0,0,0) (5,9,6) modeaanchoranurth
(5x5]
20
[draw[axis] 0,0) -- (0,2,6) modessnchoc-north
19
west] (521];
x Draa axes Adraw [anchor-weest
20
\draw[axis, [axis]
(0)0,-2) node [yd
(47)turns11.5,e,)
25
X Braw vactors
Adraw [axis] (9,8,8) - (9.1.5,8) 8) node (anchorrsoottaa(Sy11
24
Mfrracch \t in (8, 326)
326)
\draw (9,9,0) -- (9,8,1.5) node [anccoooeeesss]](55533
23
25
ydraw [vectrr) (1tt:366")).sinnnt
24
(E(ccos())) (s)
Dray vectors
26
Woresch VE in. (9,45....> Mturns"66e
26
27
Calculate the noma) vector components
27
vend ttizzpettur)
Apgfnattseetacrro (-sin()))
28
Lend document))
\podfathhectaccro((y// (cos(
Npoffaathsetmacrooeee (9)
30
31
the normal
32
[draa[vector] C(Nt/36844), (sta()))
(xdr, Vay, Adz):
(dy,)dz))
34
Nend [1ikapicture) (docueent)
GPT-4V (OI)
GPT-4V (TT)
Figure 5: TikZ programs generated by ZeFTRZIFY-DSS -D (top) and GPT-4V (bottom) corresponding to the figures in the first row of Table 10. Lines with compile- time errors are highlighted in yellow.
29

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate " "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should diseuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

reproducible, they Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https://nips. cccccc//puu ctc//ublic/gguue
for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Reporting егror bars for rung-runnig experiments involving proprictary language models is challenging, primarily due the high costs and time required for repeated model runs. Instead, we quantify the reliability of human judgments and calculate their correlation with evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
32

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www/catm/aa//e dataatsss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/moodl as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No].
Justification: We describe our human evaluation in detail, and the human subjects are exclusively colleagues who participate voluntary basis.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
The participants in our human evaluation exposed any risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "1C2vSBEGUq", "reviewer": "Reviewer_bXux", "summary": "To create high-quality scientific figures the work trains a multimodal language model, DETixZirv, a new multimodal language model from sketches and existing figures. Trained on ATIXZ.2 (over 360k human-created TikZ graphics), SKETCHFIG (hand-drawn sketches paired with scientific figures), and SciCAr++ (diverse scientific figures with metadata) datasets, MCTS-based inference algorithm to refine outputs it outperforms CLAUDE 3 and GPT-4V in creating TikZ programs.", "questions": "Please see the weakness section.", "limitations": "Please see the weakness section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The work addresses a novel and innovative challenge the approach seems to be easily adapted to downstream tasks.\n\n- The work introduces several new datasets to further scientific figures generation The work efficiently combines vision and language models for sketch to TiKz output, along with Instruct-Pix2Pix to convert instructions to draw sketch to sketch.\n\n- Identifying the challenges pertaining to the problem statement, the work effectively dismisses certain solutions and opts for the Monte Carlo Tree Search.\n\n- For efficient training, the work uses 2 losses one based on the success of compilation and another to measure the similarity between generated images and input images.\n\n- To validate the performance of the model, the work proposes several metrics which though seeming trivial, fit the task well.", "weaknesses": "- Evaluation relies on compiler diagnostics and perceptual similarity, which may need to fully capture the quality and usability of generated TikZ code. Thus, visual outputs are necessary to evaluate the performance efficiently.\n\n- Generated code might still contain subtle errors or inconsistencies that are not easily detectable through automated metrics. \n\n- Though the proposed model works better than GPT-4V when run for 10 minutes, it might limit the use cases of the work especially considering the computational requirements necessary. \n\n- From the provided metrics it could be inferred that the proposed model performs better than other models, however, the improvement provided considering the time and computational requirements makes the model less effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and their feedback on our work. We noted a few misunderstandings in the reviewer's summary, as well as the listed strengths and weaknesses, which we would like to clarify.\n\nFirst, while we generally agree with the summary, for accuracy and to prevent any confusion we want to point out that instead of DETixZirv, ATIXZ.2, and SciCAr++, our artifacts are called DeTikZify, DaTikZv2, and SciCap++.\n\nSecond, although we are grateful for the recognition of our work's strengths, we must clarify that Instruct-Pix2Pix does not \"convert instructions to draw sketch to sketch,\" but rather converts figures into sketches (according to edit instructions). Moreover, we do not use two losses for training; the described functions serve as reward functions for MCTS, which does not involve additional training. We will now address the perceived weaknesses.\n\n> Evaluation relies on compiler diagnostics and perceptual similarity [...]. Thus, visual outputs are necessary to evaluate the performance efficiently.\n\nOnly the image similarity metrics depend on visual outputs (and not all of them are based solely on perceptual similarity). Code similarity and efficiency metrics do not rely on visual outputs. Thus, this statement is not fully accurate and we do not consider this a weakness. We will clarify this point in the revised version of the paper.\n\n> Generated code might still contain subtle errors or inconsistencies that are not easily detectable through automated metrics.\n\nWhile we acknowledge that automatic metrics are not flawless, this is a challenge common across all fields, which is why human evaluations are typically included-and we have followed this approach as well. Further, while we agree that such fidelity errors are difficult to detect in code space, they become apparent in the compiled outputs, which our image similarity metrics are designed to account for. Thus, we do not view this issue as having significant implications, and we will make this clearer in the revised version of our paper.\n\n> Though the proposed model works better than GPT-4V when run for 10 minutes, it might limit the use cases of the work especially considering the computational requirements necessary.\n\nWe agree that there is a quality versus run-time trade-off in time-budgeted versus output-oriented inference. However, this applies not only to our models but also to the baselines, including GPT-4V; and our results indicate that our models are competitive or outperform the baselines given equivalent inference methods (i.e., comparable run-time). Also, even when comparing output-oriented GPT-4V with time-budgeted DeTikZify across inference modes, as the reviewer suggests, given GPT-4V's presumably much larger size, DeTikZify still likely requires much fewer computational resources.\n\n> From the provided metrics it could be inferred that the proposed model performs better than other models, however, the improvement provided considering the time and computational requirements makes the model less effective.\n\nAs mentioned in the previous point, this trade-off is not unique to our models but applies to all evaluated models. Given this context, we kindly request the reviewer to reconsider their evaluation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PW9COGAUEo", "reviewer": "Reviewer_gKyN", "summary": "This paper proposes DeTikZify, a multimodal language model that generates scientific graphics in TikZ from hand-drawn illustrations. The authors provide three datasets: DaTikZ v2 (TikZ source code and figures), SketchFig (TikZ and hand-drawn sketch pairs), and SciCap++ (figures and captions). They also propose a method to iteratively improve generated TikZ using Monte Carlo Tree Search (MCTS) based on two reward functions. Experimental results show that combinations of multiple VLMs with the proposed method outperform closed-source LVMs like GPT and Claude. Additionally, iterative exploration using MCTS is shown to further improve figures efficiently.", "questions": "A response to the point raised in the Weaknesses section is expected.", "limitations": "As stated in the main text, the authors provide a sufficient discussion of limitations in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Tackles the challenging task of generating TikZ from hand-drawn illustrations.\n- Provides three new datasets for this task.\n- Experimental results sufficiently demonstrate the contribution of the proposed method. The authors conducted preliminary studies on using SVG in addition to TikZ for figure generation, and whether the evaluation metrics correlate with human subjective evaluations, before quantitatively demonstrating the effects of iterative improvement and MCTS.", "weaknesses": "A dataset named SciCap++ already exists and is not cited in this paper. The authors need to appropriately acknowledge this, explain the differences, and if necessary, rename their dataset.\n[a] Yang et al., SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning. AAAI workshop SDU, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and are pleased to know they generally liked our work.\n\n> A dataset named SciCap++ already exists and is not cited in this paper.\n\nWe acknowledge the potential for confusion regarding similar dataset names, despite a slight variation in the suffix (our dataset is SciCap++, while Yang et al.'s is SciCap+). However, it is important to note that these datasets serve distinct purposes: SciCap+ is derived from the original SciCap dataset and exclusively contains scientific figures of a specific type, namely graph plots. On the other hand, our meta-dataset, SciCap++, aggregates various types of scientific figures from multiple sources, including an updated version of the original SciCap dataset (which we do cite) but excluding SciCap+ due to its narrow scope. The broader range of figures in SciCap++ makes it especially well-suited for our pretraining phase aimed at generating diverse scientific figures. Additionally, our dataset is considerably larger, containing 734k figures compared to SciCap+'s 414k. In light of the reviewer's advice, we will rename our dataset in the revised version of our paper to avoid any future confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlwg1ju8JX", "reviewer": "Reviewer_uo3w", "summary": "This paper introduces DeTikZify, a multimodal model that automatically generates TikZ code for scientific figures and sketches. The authors create and leverage three novel datasets: DaTikZv2 (a large dataset of TikZ graphics), SketchFig (paired human-drawn sketches and scientific figures), and SciCap++ (a meta-dataset of scientific figures and associated text). They demonstrate that DeTikZify outperforms GPT-4V and Claude 3. They also show that a novel MCTS-based iterative refinement algorithm brings gains.", "questions": "1. The paper focuses on scientific figures, which often exhibit structured layouts and simple shapes. However, TikZ's capabilities extend to much more complex diagrams and illustrations. How well would DeTikZify generalize to more diverse TikZ use cases beyond scientific figures? Are there inherent limitations in the approach that hinder its application to more free-form diagrams?\n2. While the paper argues for the importance of segment-level performance, it's unclear how this translates to actual user experience. Can the authors elaborate on how segment-level improvements manifest in visually perceptible differences in the generated figures?\n3. The paper mentions that SelfSim outperforms other metrics in segment-level correlation. However, the difference in Spearman's ρ appears relatively small (Table 4). Could the authors comment on the practical significance of this difference?\n4. How sensitive is MCTS to the exploration coefficient c? Could the authors discuss the process of selecting this parameter and its effect?\n5. Providing visual examples of successful and unsuccessful cases for DeTikZify, Claude 3, and GPT-4V would offer valuable insights into their strengths and weaknesses.\n6. What types of figures or TikZ constructs does DeTikZify struggle with? What are the common failure modes of the system?", "limitations": "The limitations and societal impacts have been addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: This paper introduces DeTikZify, a novel approach to automatically generate TikZ code from both existing figures and hand-drawn sketches. Prior work mostly explored image vectorization and image-to-LaTeX conversion, but this paper tackles the underexplored problem of synthesizing semantically TikZ programs from visual input. This is a valuable contribution to the field. Although the individual components are not novel, their combination is (SigLIP, CodeLLaMA/TinyLLaMA/DeepSeek and MCTS-based refinement algorithm).\nThe authors introduce three new datasets which are increasingly valuable contributions in the field. I also find several ideas in the paper, e.g. to randomly replace figures with 50% synthetic sketches very interesting, original and practical.\nQuality: I find the scientific standards of this paper to be very high. Firstly, it provides a thorough evaluation of DeTikZify on various metrics, including code similarity, image similarity, and efficiency. The human evaluation further strengthens the findings. The ablations are insightful and further support the design choices.\nClarity: The paper is very well-structured, clearly written and easy to follow. The figures aid understanding of the method and the results.\nSignificance: This paper tackles a very practical problem that can have a large impact on the workflows of figure creation, editing, etc. DeTikZify is a valuable step towards improving this process. Making the code, models and datasets publicly available enables further research and development and aids everyone working in this area.", "weaknesses": "1. The main issue that I have with this work is the potential data leakage. It is not clear how much of the training / evaluation data is in the training set of models. How can we be sure about this?\nFor the public models, it is said “we only include items created after the cut-off date of CodeLLaMA and exclude repositories that may have been used in training DeepSeek.” but how to ensure this for the private models? I feel that using the “𝑛-gram matching algorithm” does not really tell us a lot about this. It would be great if the authors can provide some further thoughts / explanations on this.\n2. From the text I find it hard to understand the intuition behind and the exact computation of “Average Similarity” (L249). Would it be possible to improve clarify this in the main text?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough review and are pleased that our work is generally well received. We now address the remaining questions and perceived weaknesses.\n\n> The main issue that I have with this work is the potential data leakage. [...] For the public models, it is said “we only include items created after the cut-off date of CodeLLaMA and exclude repositories that may have been used in training DeepSeek.” but how to ensure this for the private models?\n\nThe proprietary nature of the private models indeed makes it difficult to address data leakage and cross-contamination as we cannot make a lot of assumptions about their training data. On the upside, however, this means the performance of these private models should be interpreted as an upper bound, and since our public models manage to outperform them this strengthens our findings. We will add this to our limitations section and move it to the main text in our revised version.\n\n> From the text I find it hard to understand the intuition behind and the exact computation of “Average Similarity” (L249). Would it be possible to improve clarify this in the main text?\n\nAverage scores are commonly employed to provide a holistic view of a system's performance across various metrics \\[1,2]. In our case, since the metrics are on different scales, 0-1 normalization is necessary before computing their average (cf. Section 6.1). This gives a clear picture that our models dominate overall. We will elaborate on this more in the revised version.\n\n> How sensitive is MCTS to the exploration coefficient c? Could the authors discuss the process of selecting this parameter and its effect?\n\nThe performance of MCTS is indeed sensitive to the exploration coefficient. A high value for c favors exploration of less-visited regions, whereas a low value emphasizes exploitation of high-value areas. Although theory provides some guidance, in practice, this parameter is empirically determined. For our work, we opted for a lower value to prioritize exploitation over exploration, due to the high computational costs of rollouts with LLMs. This consideration will be included in the revised version of our paper.\n\n> The paper mentions that SelfSim outperforms other metrics in segment-level correlation. However, the difference in Spearman's ρ appears relatively small (Table 4). Could the authors comment on the practical significance of this difference? [And] while the paper argues for the importance of segment-level performance, it's unclear how this translates to actual user experience. Can the authors elaborate on how segment-level improvements manifest in visually perceptible differences in the generated figures?\n\nStatistically, the differences in segment-level Spearman's rho in Table 4 are significant (based on a Fisher's z-Tests with a significance level of 5%). Practically, this means that when using SelfSim in MCTS, the computed rewards for rollouts better align with human perception, resulting in higher-quality outputs when exploiting high-value regions and, therefore, improved user experience (this is in contrast to system-level performance, which assesses the overall quality of whole systems and not individual outputs).\n\n> Providing visual examples of successful and unsuccessful cases for DeTikZify, Claude 3, and GPT-4V would offer valuable insights into their strengths and weaknesses.\n\nAlthough we already provide examples in Table 8, we agree that additional examples would offer more valuable insights, a point also raised by reviewer tgj9. Please refer to the additional examples in the supplementary pdf provided in the general response. These will be included in the revised version of our paper.\n\n> What types of figures or TikZ constructs does DeTikZify struggle with? What are the common failure modes of the system?\n\nThe failure modes of DeTikZify result primarily from the composition of our dataset. For instance, humans sometimes compose independent TikZ pictures to create more complex figures. Since most examples in DaTikZ treat each TikZ picture separately, in practice it might be difficult to generate such composed figures in one go.\n\n> The paper focuses on scientific figures, which often exhibit structured layouts and simple shapes. However, TikZ's capabilities extend to much more complex diagrams and illustrations. How well would DeTikZify generalize to more diverse TikZ use cases beyond scientific figures? Are there inherent limitations in the approach that hinder its application to more free-form diagrams?\n\nAs mentioned previously, the limits of DeTikZify are largely defined by our datasets, and DaTikZ contains a large portion of intricate TikZ pictures, including free-form diagrams and state machines. Some examples on how DeTikZify handles this are provided in Figure 1 and Table 8. However, even outside these bounds, we observe rudimentary emergent capabilities, for example generating TikZ code for photorealistic images. Understanding the boundaries of DeTikZify, e.g., through detailed dataset analysis and exploration of its emergent capabilities is a key aspect for our future research.\n\n\\[1]: Yuan, Weizhe, Graham Neubig, and Pengfei Liu. \"Bartscore: Evaluating generated text as text generation.\" Advances in Neural Information Processing Systems 34 (2021): 27263-27277.\n\n\\[2]: Liu, Haotian, et al. \"Visual instruction tuning.\" Advances in neural information processing systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2xhKxkZ6Zb", "reviewer": "Reviewer_tgj9", "summary": "The paper proposes a technique for generating TikZ graphics based on handwritten sketches. It open-sources the datasets used for this task, including a large TikZ dataset, a small sketch-to-TikZ dataset, and a large dataset of scientific figures with accompanying texts.\n\nThe model uses a vision-language model to take as input an image of the sketch, and potentially the output of the LaTeX compiler from the previous step, to generate the code that is attempted for compilation in the successive step. To further improve the model, Monte Carlo Tree Search is employed for more efficient iteration in finding a compilable and high-quality output.\n\nThe evaluation is done via several automated metrics such as code and image similarity, as well as wall time for generation, accompanied by the human evaluation, which positively correlated with the used automated metrics.", "questions": "No particular questions, see suggestions in \"weaknesses\".", "limitations": "The authors have adequately addressed the limitations in the checklist.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is clearly written, the experimentation setup, ablation and analysis is of the high quality, the task of sketch-to-tikz is somewhat novel, and it has high significance, thanks to the datasets being shared, and the bits about the generation of synthetic dataset using Instruct-Pix2Pix and using MCTS to improve the quality are valuable for many other applications, and having the openly shared code for this is also a great contribution.", "weaknesses": "I think it would strengthen the paper to provide more examples of the model inference (similar to page 29 of the Appendix) to better gauge the performance of the models and to see examples without MCTS to understand how much does it contribute to the downstream quality (Fig. 3, right, paints some picture, but the SelfSim metric is hard to interpret). More generally, ablation study likely can also be strengthened by highlighting the amount of contribution brought in my MCTS, synthetic sketches, SciCap++, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and are delighted to see that they generally like our work.\n\n> I think it would strengthen the paper to provide more examples of the model inference [...]. More generally, ablation study likely can also be strengthened by highlighting the amount of contribution brought in my MCTS, synthetic sketches, SciCap++, etc.\n\nWe agree with the reviewer that providing more examples would be helpful, an opinion also shared by reviewer uo3w. We therefore provide more randomly selected examples which also exemplify the contributions brought by MCTS in the pdf of the general response (we will also add this to the revised version of our paper). However, we want to remark that since MCTS only has an effect on image similarity when the SelfSim reward is used, comparing examples with and without MCTS is equivalent to comparing examples from output-driven inference with time-budgeted inference. This also means that we can refer the reviewer to Sections 6.1 (image similarity metrics) and 6.2 for a quantitative comparison. We will clear this up in our revised version.\n\nTo assess the impact of synthetic sketches on training, we conducted an additional ablation study using DeTikZify-DS-1.3b, comparing fine-tuning with a 50% random replacement of figures with sketches (w/ sketches) against no sketch replacement (w/o sketches):\n\n|              | Figures  |          |          |          |          |          | Sketches |          |          |          |          |          |\n| ------------ | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n|**Training**  |**MTE**   |**cBLEU** |**TED**   |**DSim**  |**SSim**  |**KID**   |**MTE**   |**cBLEU** |**TED**   |**DSim**  |**SSim**  |**KID**   |\n| w/ sketches  |**83.771**|  1.336   |  57.661  |  68.659  |  86.079  |**11.536**|**87.446**|  0.541   |**60.112**|**62.756**|**79.097**|**17.334**|\n| w/o sketches |  81.814  |**1.663** |**56.839**|**71.092**|**87.397**|  14.832  |  74.088  |**0.712** |  61.481  |  58.763  |  75.765  |  51.514  |\n\nThe results align with expectations: omitting sketches slightly enhances performance on reference figures but significantly diminishes performance on sketches. For a model that should perform adequately on both tasks, our original training recipe is recommended, whereas for figure-only input, exclusive training on figures may be preferable. These findings will be\nintegrated into our revised paper.\n\nRegarding the impact of pre-training on SciCap++, we direct the reviewer to existing literature on MLLM development, which shows that pre-training vision-language adapters generally improves downstream performance \\[1,2,3]. Nonetheless, we agree on the importance of independent verification and will train a model without this pre-training step and include the results in the revised version of our paper.\n\n\\[1]: Tong, Shengbang, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" arXiv preprint arXiv:2406.16860 (2024).\n\n\\[2]: Liu, Haotian, et al. \"Visual instruction tuning.\" Advances in neural information processing systems 36 (2024).\n\n\\[3]: Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to extend our gratitude to all the reviewers for their feedback and we are delighted that our work was generally well-received. In response to the comments from reviewers uo3w and tgj9, we have uploaded a PDF containing additional examples along with this general response. For more details, see our individual responses below.

Meta-review: Three of the four reviewers gave either a strong accept or accept for this paper.  Those reviewers appreciated the novelty of the work to automatically generate TikZ code from both existing figures and hand-drawn sketches, and they felt the results and evaluations were conceiving.

There was one dissenting reviewer.  This review had some inconsistencies in their comments that don't quite line up with the paper or are unclear and the reviewer didn't engage in the discussion with the authors or other reviewers.  They noted there were no visual results, however those appear in the appendix. 

Given the discussion with the other reviewers and the consideration of the rebuttal which other reviewers did respond to and felt satisfied, the AC agrees that this paper is a strong contribution and is accepted.  

Congratulations!

In revising your paper for the camera-ready, please include and comments and clarifications that were made or proposed during the rebuttal and discussion phase.  Also, the AC recommends including some visual results that are in the appendix into the main body of the paper.
Decision: Accept (spotlight)