DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ

Authors: jonas  belouadi
simone paolo ponzetto
steffen  eger
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 13:19:21

Abstract: Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy. Furthermore, recreating existing figures that are not stored in formats preserving semantic information is equally complex. To tackle this problem, we introduce DeTikZify, a novel multimodal language model that automatically synthesizes scientific figures as semantics-preserving TikZ graphics programs based on sketches and existing figures. To achieve this, we create three new datasets: DaTikZv2, the largest TikZ dataset to date, containing over 360k human-created TikZ graphics; SketchFig, a dataset that pairs hand-drawn sketches with their corresponding scientific figures; and MetaFig, a collection of diverse scientific figures and associated metadata. We train DeTikZify on MetaFig and DaTikZv2, along with synthetically generated sketches learned from SketchFig. We also introduce an MCTS-based inference algorithm that enables DeTikZify to iteratively refine its outputs without the need for additional training. Through both automatic and human evaluation, we demonstrate that DeTikZify outperforms commercial Claude 3 and GPT-4V in synthesizing TikZ programs, with the MCTS algorithm effectively boosting its performance. We make our code, models, and datasets publicly available.

Main paper: 
 introduction: 
 Creating high-quality scientific figures is similar to typesetting scientific documents in many ways. When it comes to typesetting, markup languages like L A T E X enjoy widespread popularity, as exemplified by major machine learning conferences that either mandate or strongly encourage L A T E X-formatted submissions.2 The advantages of using such languages go beyond producing high-quality outputs; documents expressed as high-level, semantics-preserving programs enhance accessibility, serve archival purposes, and remain easily editable and human-readable (facilitating language modeling applications; Moosavi et al., 2021; Lu et al., 2023) . Consequently, efforts have been made to recover this type of information from outputs stored in lower-level vector graphics formats like PDF or SVG, or raster graphics formats (Desai et al., 2021; Blecher et al., 2024) . At the other end of the spectrum, the versatility of L A T E X comes with a steep learning curve, and typesetting can often be challenging for end users. In response, researchers have been working on assisting authors with certain aspects of the problem, such as typesetting math based on hand-drawn sketches (Kirsch, 2010; Wu et al., 2020) .Just like documents, scientific figures can also be created using markup languages. A popular example is the TikZ graphics language (Tantau, 2023) , which can be integrated into L A T E X documents, providing comparable benefits and encountering similar challenges. However, unlike L A T E X, the prospects of TikZ in research contexts remain largely unexplored. Although the promise of simplifying editing and enabling applications in visual understanding (Masry et al., 2022; Huang et al., 2023) is evident, there are currently no viable solutions for recovering graphics programs from compiled figures. Moreover, there is a lack of tools that assist in creating graphics programs, e.g., based on hand-drawn sketches, despite the clear demand for such approaches on the T E X Stack Exchange (T E X.SE),3 where nearly 10% of all questions revolve around TikZ, making it the most frequently discussed topic on the site. Addressing this gap could greatly improve the accessibility of existing figures and support researchers at all levels of programming proficiency when creating new ones, fostering diversity and inclusion. In response, we introduce DeTikZify, a multimodal language model that automatically synthesizes TikZ programs for scientific figures and sketches (cf. Figure 1 ). Our key contributions are as follows:(i) As part of DeTikZify, we introduce (a) DaTikZ v2 , a large TikZ dataset with over 360k human-created TikZ graphics; (b) SketchFig, a dataset of human-created sketches with paired scientific figures; and (c) MetaFig, a large meta-dataset of scientific figures and associated texts. (ii) We train DeTikZify on MetaFig and DaTikZ v2 , augmented with synthetic sketches that mimic SketchFig. We demonstrate that DeTikZify can effectively synthesize TikZ programs for both existing scientific figures and sketches, outperforming the commercial large language models (LLMs) GPT-4V and Claude 3 (OpenAI, 2023b; Anthropic, 2024) . (iii) We also present an inference algorithm based on Monte Carlo Tree Search (MCTS) that is tailored to graphics programs and allows DeTikZify to iteratively refine its own outputs for a given computational budget, further improving performance without additional training.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Image-to-L A T E X Conversion A closely related task is the translation of mathematical illustrations into L A T E X markup. In inspirational work, Kirsch (2010) tackle the recognition of single hand-drawn symbols to find corresponding L A T E X commands. Subsequent works by Deng et al. (2017) ; Zhang et al. (2017 Zhang et al. ( , 2019)) ; Wu et al. (2020) ; Wang and Liu (2021) expand on this concept to handle hand-drawn and scanned math formulas. Suzuki et al. (2003) ; Wang and Liu (2020) ; Blecher et al. (2024) ; Lv et al. (2023) further extend the scope by extracting L A T E X formulas alongside text from entire documents.Image Vectorization Similarly, converting (rasterized) figures into TikZ programs can be characterized as a form of image vectorization (Sun et al., 2007; Diebel, 2008; Ganin et al., 2018; Li et al., 2020; Ma et al., 2022; Zhu et al., 2024) . Most existing methods vectorize images into low-level graphics primitives in the SVG format (Tian and Günther, 2024) . Although this works well for specific domains like fonts, icons, and emoji (Lopes et al., 2019; Carlier et al., 2020; Reddy, 2021; Rodriguez et al., 2023b) , it does not capture higher-level semantics and does not generalize well to our scientific context (cf. Appendix B). Closer to our work, Ellis et al. (2018) generate vector representations as graphics programs based on a limited subset of L A T E X commands. Their approach even handles sketches, but their experiments are restricted to a synthetic dataset with only basic shapes of limited complexity. Belouadi et al. (2024) also generate TikZ programs, but their primary emphasis is on conditioning the generation on textual descriptions, with images serving only as a secondary input.As TikZ is implemented in the Turing-complete T E X macro system (Erdweg and Ostermann, 2011), our work is also closely tied to code generation (Xu et al., 2022) . Despite continuing progress in this field (Chen et al., 2021; Li et al., 2022 Li et al., , 2023;; Guo et al., 2024; Lozhkov et al., 2024) , most research concentrates on high-resource languages like Python, Java, and JavaScript (Zan et al., 2023) , typically overlooking T E X in evaluations. However, T E X and TikZ may still find their way into the training data, as demonstrated by the zero-shot ability of some models to understand and generate code in these languages (Bubeck et al., 2023; Belouadi et al., 2024; Sharma et al., 2024) . (Belouadi et al., 2024) , incorporating graphics from the same sources, namely curated repositories, T E X.SE, arXiv papers, and artificial examples. The key difference is that DaTikZ v2 includes all TikZ programs that compile with T E X Live 2023,4 regardless of whether they have associated captions, which was a requirement for inclusion in DaTikZ v1 but is not needed for DeTikZify. This approach allows us to create a dataset that is more than three times as large as its predecessor (cf. Table 1 ).To create realistic synthetic sketches of scientific figures in DaTikZ v2 , we rely on examples of real human-created sketches. T E X.SE is a suitable source for collecting these, as users often illustrate their questions with sketches, and the answers provide the desired figure. We semiautomatically extract these figure-sketch pairs by first ranking all questions on the site that contain images based on their similarity to the string "a sketch of a scientific figure" using a multimodal vision encoder (Zhai et al., 2023) . We retain the ones with high similarity scores, manually filter for true positives, and align them with the best matching figure provided in the answers. In total, we collect 549 figure-sketch pairs this way. As we also want to use this dataset for evaluation (cf. §6), we ensure that for a subset of these sketches, no code provided in the answers is included in DaTikZ v2 .MetaFig Beyond TikZ graphics, there is a much larger pool of figures where the underlying source is not available. Existing datasets that collect such figures frequently come with rich metadata, such as captions, OCR tokens, and paragraphs that mention the figures (Hsu et al., 2021; Karishma et al., 2023; Rodriguez et al., 2023a) . Since such high-level descriptions are useful for pretraining (cf. §4; Liu et al., 2023b) , we collect these datasets and merge them with the subset of figures in DaTikZ v2 that have captions. This results in over 734k figure-text pairs, more than twice the size of DaTikZ v2 .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
the detikzify model: 
 Building on previous work (Liu et al., 2023b,a; Dai et al., 2023; McKinzie et al., 2024) , we build DeTikZify by combining a pretrained vision encoder with a pretrained language model (cf. Figure 1 ), where the vision encoder receives figures or sketches as input images, and the language model generates corresponding TikZ programs as output. We focus on code language models that have been pretrained on T E X, as this prior knowledge may be helpful for our task. All the models we end up using follow the LLaMA architecture (Touvron et al., 2023 ): CodeLLaMA (Rozière et al., 2023) has likely been trained on T E X code from arXiv (Touvron et al., 2023) , as has been TinyLLaMA (Zhang et al., 4https://tug.org/texlive 2024), while DeepSeek (code variant; Guo et al., 2024) was trained on T E X code from GitHub. For the vision encoder, we use SigLIP (Zhai et al., 2023) , which has been trained on OCR annotations (Chen et al., 2023c) and demonstrates state-of-the-art understanding of text-rich images (Tong et al., 2024; Chen et al., 2023b) , a crucial skill for our task. We then condition the LLMs on SigLIP's patch embedding vectors. To reduce the prompt length, we concatenate adjacent patch embeddings (Chen et al., 2023a)  
iterative refinement with monte carlo tree search: 
 Due to the inherent probabilistic nature of language models, generating valid TikZ programs during inference can be a challenging task. The generated code may not always comply with the syntactic and semantic rules of T E X and TikZ, potentially leading to compilation errors. While constrained decoding algorithms can assist in guiding models towards generating valid programs (Ugare et al., 2024; Poesia et al., 2022; Scholak et al., 2021) , these approaches are limited to programming languages defined by context-free grammars (CFGs). However, T E X and TikZ are not defined by CFGs (Erdweg and Ostermann, 2011), rendering these methods ineffective for our purpose. Moreover, even if the generated code compiles successfully, fidelity errors such as misaligned elements, inconsistent scaling, repetitions, or mislabeling may only become apparent in the rendered output.Despite these challenges, which make it difficult to guide DeTikZify based on intermediate states, we can still analyze completed outputs in a straightforward manner (e.g., by examining compiler diagnostics or comparing rendered outputs to the input image), allowing us to make informed decisions during subsequent sampling iterations. This concept of making decisions based on random sampling of the search space forms the core of Monte Carlo Tree Search (MCTS; Coulom, 2007) . By integrating DeTikZify with MCTS and adapting the standard MCTS algorithm to our problem domain, we can iteratively steer DeTikZify towards more promising regions of the output space (cf. Figure 1 ). In the following, we outline our fundamental approach, with further extensions discussed in Appendix A.MCTS is a versatile search algorithm that has been successfully applied to various domains, including board games (Silver et al., 2016 (Silver et al., , 2017)) , procedural content generation (Kartal et al., 2016a,b; Summerville et al., 2015) , and more recently, guiding language models to achieve long-term goals (Brandfonbrener et al., 2024; Zhang et al., 2023b; Chaffin et al., 2022) . The algorithm incrementally builds a search tree and repeatedly runs simulations until an exit condition is met or a computational budget is exhausted. In our context, at depth 𝑛, each node's state consists of 𝑛 lines of TikZ code, and edges represent continuations for generating the next line. Initially, MCTS starts with only an empty root node and then iteratively performs the following four steps (cf. Figure 2 ): The selection policy (i) reaches a green backtracking node (normal nodes are blue), causing new nodes from the rollout (ii) to be added to the parent node during expansion (iii). The reward is backpropagated (iv) accordingly.+ + + + (i) Selection (ii) Rollout (iii) Expansion (iv) BackpropagationSelection Each simulation starts at the root node and successively selects child nodes based on a selection policy until a leaf node is reached. The policy determines which parts of the tree should be explored further, balancing the exploitation of high-value regions and exploration of less-visited areas. Following previous work, we use Upper Confidence Trees (UCT; Kocsis and Szepesvári, 2006) as our selection policy, iteratively selecting the successor node 𝑖 that maximizes the formulaEQUATIONwhere 𝑉 𝑖, 𝑗 ∈ [-1, 1] is the estimated value of 𝑖 at the 𝑗th visit, 𝑛 𝑖 and 𝑛 p(𝑖) are the visit counts at 𝑖 and its parent p(𝑖), respectively, and 𝑐 is a coefficient that controls the degree of exploration.Rollout Once a leaf node is selected, we utilize DeTikZify as a rollout policy. By conditioning it on the node's state, we continue to sample TikZ code until the end-of-sequence token is encountered. This so-called rollout is then stored for reuse in the subsequent steps.Expansion Next, the tree is expanded by adding nodes from the rollout as new leaf nodes. While most implementations add only one node (i.e., one line of TikZ code) per simulation, computing rollouts with LLMs is computationally expensive. Therefore, inspired by MCTS for real-time settings (Soemers et al., 2016) , we instead add multiple nodes. Specifically, we add √︁ |𝑟 | -𝑑 𝑙 new nodes, where |𝑟 | is the number of lines in rollout 𝑟 and 𝑑 𝑙 is the depth of the old leaf node 𝑙. This approach allows our tree to grow quickly in early simulations while converging to the standard case in the long run. To enable the tree to grow in multiple directions, we also introduce backtracking nodes (Brandfonbrener et al., 2024; Chaslot et al., 2008) . For each added node 𝑖, we add a backtracking node as a sibling that mirrors the parent node p(𝑖). When a backtracking node is expanded, its descendants are added to p(𝑖) so that the backtracking node remains a leaf. This enables a practically infinite search space anywhere in the tree while still maintaining a bounded branching factor.Backpropagation Finally, we calculate the value for rollout 𝑟 using a predefined reward function (cf. §5.2) and backpropagate it to every node 𝑖 on the path from the root node to the newly added nodes by appending it to 𝑽 𝑖,: . We also increment the visit counts 𝑛 𝑖 for the same nodes. For backtracking nodes, only the visit counts are updated. Finally, we check any exit conditions. If MCTS terminates, we return the TikZ program of the rollout that achieved the highest value.We explore two distinct reward functions to guide the search process. The first reward function utilizes compiler diagnostics to identify documents that compile successfully. The second reward function provides a visual signal based on perceptual image similarity, which, in addition, helps find TikZ programs that better match the input image. We explore further reward functions in Appendix A.The diagnostics-based reward function is based on analyzing the log file from compiling the generated TikZ program. We assign rewards according to the error state and whether an output file was produced. The reward function is defined as follows:𝑉 𝑖, 𝑗 =       1 if the code compiles without issues, 0 if the code compiles with recoverable errors, -1 if compilation fails due to a fatal error.(2) Self-Assessed Perceptual Similarity (SelfSim) SelfSim computes the reward as the perceptual similarity (Zhang et al., 2018) between the input image and the compiled output figure. We hypothesize that DeTikZify itself can assess this similarity, enabling the model to guide its own search process.MTE ↑ cBLEU ↑ TED ↓ DSim ↑ SSim ↑ KID ↓ AVG ↑ MTE ↑ cBLEU ↑ TED ↓ DSim ↑ SSim ↑ KID ↓ AVG ↑ ClaudeTo achieve this, we encode both images into embedding vectors using DeTikZify's vision encoder and calculate SelfSim as their cosine similarity (Fu et al., 2023; Hessel et al., 2021) . In cases where compilation fails, we assign a reward of -1. In §7, we demonstrate that SelfSim correlates well with human judgments and outperforms other baseline methods. 
experiments: 
 Before training on DaTikZ v2 , we extract 1k samples to serve as our test set for an automatic evaluation and generate corresponding synthetic sketches. To mitigate data leakage from pretraining to testing, we only include items created after the cut-off date of CodeLLaMA and exclude repositories that may have been used in training DeepSeek. We also use an 𝑛-gram matching algorithm to prevent cross-contamination with our train split (OpenAI, 2023a). For a human evaluation involving humancreated sketches, we also select 100 items from SketchFig that do not overlap with DaTikZ v2 (cf. §3). Across all models, we set the temperature to 0.8 and the exploration coefficient 𝑐 to 0.6. We provide examples of real and synthetic sketches as well as generated outputs in Appendix E and Table 4 .Baselines Given Claude 3 and GPT-4V's potential for our task (cf. §2), we use them as baselines.Similar to DeTikZify, we instruct these models to generate TikZ programs for given images. However, as proprietary chatbots, they often mix code and natural language (Zhang et al., 2023c; Belouadi et al., 2024) and do not expose the internals needed to compute SelfSim. This makes it impractical to apply our MCTS-based refinement algorithm, which is designed for code-only outputs and open models. Instead, we compare our approach to equivalent chat-oriented refinement methods, i.e., we use Self-Refine as an alternative to diagnostics-based MCTS and Visual Self-Refine as an alternative to SelfSim-based MCTS (Madaan et al., 2023; cf. Appendix C for additional inference details). In Appendix B, we also explore SVG as an alternative to TikZ but find it less effective for our domain.We introduce two inference tasks to automatically evaluate our models on the test split of DaTikZ v2 .During output-driven inference (OI), we employ the diagnostics-based reward and use successful compilation as an early exit condition (we consider compilation successful if an output artifact is produced). For time-budgeted inference (TI), we use the more fine-grained SelfSim-based reward and continue from OI until a computational budget of 10 minutes is exhausted (cf. Brandfonbrener et al., 2024) , investigating the extent of achievable improvement. We report results for the two use cases where either (rasterized) reference figures or (synthetic) sketches serve as model inputs (cf. §1). Due to high inference costs, we only evaluate commercial Claude 3 and GPT-4V in OI using Self-Refine, leaving TI with Visual Self-Refine for human evaluation. We evaluate the following properties:Code Similarity To measure the similarity between generated and reference TikZ programs, we use CrystalBLEU (cBLEU), a variant of BLEU optimized for evaluating code (Eghbali and Pradel, 2023; Papineni et al., 2002) , and the T E X Edit Distance (TED), our adapted version of the Extended Edit Distance (Stanchev et al., 2019) combined with a T E X tokenizer. Table 3 : System-level scores for time-budgeted inference, displaying relative changes for metrics shared with output-driven inference (Table 2 ; colored green for improvements and red for declines) and absolute scores for independent metrics. Bold and underlined values indicate the best and second-best absolute scores for each metric column, respectively. Arrows indicate metric directionality.MST ↑ cBLEU ↑ TED ↓ DSim ↑ SSim ↑ KID ↓ AVG ↑ MST ↑ cBLEU ↑ TED ↓ DSim ↑ SSim ↑ KID ↓ AVG ↑ DT-TL 1.1bIn addition to SelfSim (SSim), which can also be used as a metric, we report DreamSim (DSim; Fu et al., 2023) , a fine-tuned metric for perceptual similarity. We also compute the Kernel Inception Distance (KID × 10 3 ; Bińkowski et al., 2018) , which assesses the overall quality of generated figures by comparing their distribution with the distribution of reference figures. These metrics are always computed by comparing the generated figures to the reference figures, regardless of what the model receives as input.Average Similarity To offer a holistic view of each model's performance, we also compute the arithmetic mean (AVG) of all code and image similarity metrics. Given that these metrics operate on different scales, we min-max normalize their scores before calculating the average.For OI, we compute the Mean Token Efficiency (MTE) as the 10% winsorized mean of the ratio of the number of tokens in the final TikZ program to the total number of tokens generated to arrive at that program. For TI, we instead compute the Mean Sampling Throughput (MST), measuring the throughput of unique TikZ graphics for the given budget.Table 2 presents the system-level metric scores for OI. As expected, the scores for reference figures are, on average, 38% higher than those for synthetic sketches, but similar patterns emerge across both input types. DeTikZify-CL 7b and DeTikZify-DS 7b consistently outperform all other models, achieving AVG scores of 0.869 & 0.965 for figures and 0.941 & 0.965 for sketches, respectively. In contrast, GPT-4V reaches AVG scores of only 0.612 and 0.15, placing it in competition with the smaller 1b models: for figures, GPT-4V surpasses DeTikZify-TL 1.1b and DeTikZify-DS 1.3b , which achieve scores of 0.207 and 0.572, respectively. However, these smaller models outperform GPT-4V on sketches, where they achieve scores of 0.454 and 0.642. Claude 3 trails behind all our models, with an AVG of only 0.148 and 0.189. When examining individual similarity metrics, DeTikZify-DS 7b , the top-performing DeTikZify model overall, surpasses GPT-4V, the best baseline, by more than 3pp (percentage points) on average for DreamSim and SelfSim, while maintaining a noticeably lower KID. In terms of cBLEU, GPT-4V, and Claude 3 only reach 6.5-18.5% of the performance achieved by the lowest-scoring DeTikZify model (DeTikZify-TL 1.1.b ). The differences in TED are less pronounced, possibly due to the influence of boilerplate code, which cBLEU inherently ignores.For efficiency, all DeTikZify models demonstrate an MTE of 82-91%, indicating that only 1-2 out of 10 inference runs require a second simulation to generate a compilable TikZ program. Interestingly, the model size does not seem to particularly influence this score, with the pretraining setup appearing to be the key factor instead. For instance, DeTikZify-TL 1.1b and DeTikZify-CL 7b share a similar pretraining setup and exhibit comparable MTE values, as do DeTikZify-DS 1.3b and DeTikZify-DS 7b .We can further observe that (i) MTE is generally higher for sketches compared to figures, and (ii) for figures, the MTE of similarly pretrained models is inversely correlated with their scores on other metrics. These phenomena likely stem from models making fewer mistakes when the input is less detailed or when their understanding of it is limited-a finding that aligns well with other studies (Tong et al., 2024) . Compared to DeTikZify, Claude 3 and GPT-4V perform considerably worse, with an MTE of only 50-62%. Notably, for these models, 98.5% of the items already compile after the initial Self-Refine step, meaning that this inefficacy primarily originates from the natural language texts surrounding the code and that Self-Refine is nearly equivalent to regular sampling-based inference.The results for DeTikZify on TI are presented in Table 3 . Remarkably, increasing the computational budget for MCTS improves nearly all metrics for both reference figures and sketches as input without requiring access to any additional knowledge. The improvement with sketches is particularly noteworthy, as it demonstrates that the refinement process enhances the desired properties even when the model input type differs from the one used for evaluation. The 2.2-5.6pp increase of SelfSim for all models is not surprising since it serves as the reward signal we optimize, but DreamSim and TED also increase by 4.9-8.7pp and 0.5-2pp, respectively, demonstrating the efficacy of our approach.While KID improves by 1-12.1 points with reference figures, it drops by 0.9-5.5 points with sketches. We believe this is because sketches often omit minor details, such as axis tick labels, which is reflected more in the output of the TI models, biasing their overall output distributions. Therefore, we consider the substantial improvement of metrics capturing instance-level similarities to be more important. For cBLEU, we observe only minor changes (less than ±0.1pp), aligning with findings that BLEU-based metrics become less effective as performance increases (Ma et al., 2019) . The MST and AVG reveal that, although 1b models produce more unique outputs within the time frame compared to their larger 7b counterparts (30-36 vs. 24.1-26.2), they still fail to close the overall gap in performance, with AVG scores ranging between 0.014-0.531 compared to 0.681-0.994 for 7b models.Overall, all DeTikZify models are capable of generating compilable outputs with reasonable efficiency. Upon examination of these outputs, it becomes evident that the 7b models, particularly DeTikZify-DS 7b , consistently outperform both Claude 3 and GPT-4V, whose performance is more comparable to the 1b range. Increasing the computational budget for DeTikZify further improves performance.To further assess the quality of the generated figures, we perform a human evaluation on SketchFig using Best-Worst Scaling (BWS; Louviere et al., 2015; Kiritchenko and Mohammad, 2016, 2017) . In this process, for each reference figure, we present annotators with a tuple of generated figures and ask them to identify the most and least perceptually similar figure. We then transform this data into scores ranging from -1 (poor) to 1 (excellent) by calculating the difference between the proportion of times a figure is selected as the best and the proportion of times it is chosen as the worst (Orme, 2009). To keep the workload manageable, we focus on the most promising DeTikZify model (DeTikZify-DS 7b ) and the strongest baseline (GPT-4V). Building upon the automatic evaluation, we assess these models in the OI and TI configurations, using either reference figures or human-created sketches as input.For each input type, we engage six unique expert annotators (cf. Appendix D for more details).Figure 3 (left) shows kernel density estimates for the computed BWS scores, revealing intriguing findings that are consistent across input types. In contrast to the automatic evaluation, DeTikZify-DS 7b performs worse (mean score 𝜇 = -0.32) than GPT-4V (𝜇 = 0.09) in OI. This could be attributed to the fact that T E X.SE, the sole source of SketchFig, emphasizes minimum working examples, a type on which GPT-4V particularly excels (Belouadi et al., 2024) . However, when we increase the computational budget, as in DeTikZify-DS 7b (TI), it not only improves over OI results (𝜇 = 0.39; in line with automatic evaluation) but also surpasses GPT-4V in both configurations by a considerable margin. Interestingly, GPT-4V's performance in TI (𝜇 = -0.16) is lower than its performance in OI, indicating that GPT-4V (TI) struggles to refine its own outputs effectively and quickly deteriorates. Overall, this shows how difficult it is for models to refine their own outputs and highlights the effectiveness of our MCTS-based approach. Example outputs are provided in Table 4 . QN Rn QN R +1 n -1 QN R n QN R n+1 QN R n QN R n+1 +1 -1 Z/nZ Q ∩ N Rn Q ∩ N Rn+1 Q ∩ N Rn Q ∩ N Rn+1 Z/nZ -1 +1 -1 +1 Z/nZ QN n QN n+1 +1 -1 Z/nZ QN Rn+1QN Rn Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
analysis: 
 In this section, we take a closer look at our methodologies and evaluation strategies, correlating evaluation metrics with human judgments, quantifying the quality of synthetic sketches, and examining the rate of convergence of our MCTS algorithm. We also demonstrate that our models are not affected by memorization of the training data, as shown in Appendix B. To assess the reliability of our human evaluation results, we investigate the agreement between annotators. To this end, we calculate the split-half reliability (SHR; Kiritchenko and Mohammad, 2017 ) by randomly splitting our annotations into two subsets, computing BWS scores for each subset, and measuring their correlation with Spearman's 𝜌.The SHR values of 0.69 for sketches and 0.75 for images indicate a moderate to strong correlation between annotators, supporting the validity of our human evaluation results. Motivated by these findings, we explore whether metrics that also assess perceptual similarity (i.e., SelfSim and DreamSim) correlate with these human judgments. We again calculate Spearman's 𝜌 and show the average correlations (David M. Corey and Burke, 1998) at the segment and system level in Table 5 . For comparison, we also include the popular LPIPS and DISTS metrics (Zhang et al., 2018; Ding et al., 2020) . At the segment level, SelfSim outperforms all other metrics, which is remarkable considering it is the only untrained metric. Segment-level performance is particularly important for fine-grained reward functions, justifying our choice of SelfSim in our MCTS algorithm.At the system level, DreamSim performs the best, showcasing its strength in evaluation settings.We also assess the quality of our synthetic sketches by measuring their congruence coefficient (Lorenzo-Seva and ten Berge, 2006) with real sketches. We embed humancreated figure-sketch pairs from SketchFig using SigLIP, subtract each sketch embedding from the corresponding figure embedding to obtain local sketch vectors, and perform a single-component Principal Component Analysis to derive a global sketch vector (Zou et al., 2023) . We repeat this process for synthetic sketches generated for the test split of DaTikZ v2 and compare the global vectors using cosine similarity. Base Instruct-Pix2Pix generates synthetic sketches with a congruence coefficient of 0.66, which increases to 0.7 after fine-tuning. These results demonstrate a high correlation with human-created sketches, suggesting that our generated sketches are of good quality.To gain insights into the long-term characteristics of our MCTS algorithm, we visualize the trends in achieved TI reward scores over time in Figure 3 (right) and compare them to conventional sampling-based inference. As expected, sampling does not lead to improvements over time due to the absence of a feedback loop. In contrast, MCTS consistently improves throughout the entire time frame, and even at the end of our budget of 10 minutes, it does not appear to converge, suggesting potential additional gains for larger budgets. Apart from this, MCTS is not only more effective but also faster. With an average MST of 25.17, compared to 18.7 for sampling, our MCTS algorithm generates considerably more unique TikZ programs within the same amount of time.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Reporting error bars for long-running experiments involving proprietary language models is challenging, primarily due to the high costs and time required for repeated model runs. Instead, we quantify the reliability of human judgments and calculate their correlation with evaluation metrics. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 In this work, we showcase the potential of DeTikZify in generating TikZ programs for two practical use cases. First, it can convert existing figures from lower-level formats into TikZ, paving the way for semantic image editing and downstream tasks (Zhang et al., 2023a) . Second, it can develop hand-drawn sketches into TikZ graphics, which could aid researchers in creating high-quality scientific illustrations. In both cases, DeTikZify substantially outperforms the commercial LLMs GPT-4V and Claude 3 despite its presumably much smaller size. We hope that our datasets (DaTikZ v2 , SketchFig, and MetaFig), our method for generating synthetic sketches, and our MCTS-based inference algorithm will pave the way towards future research on graphics program synthesis and bolster the cause of open science.Looking ahead, we plan to extend our approach to other graphics languages, such as MetaPost, PSTricks or Asymptote (Hobby, 2014; Van Zandt, 2007; Hammerlindl et al., 2024) . We also intend to explore alternatives to perceptual similarity as an MCTS reward signal, including per-pixel measures and point cloud metrics (Wang and Bovik, 2009; Wu et al., 2021) . In addition, we aim to investigate reinforcement learning from reward functions, for example, using Direct Preference Optimization (Rafailov et al., 2023; Xu et al., 2024) . Finally, while this work focuses on visual inputs, we plan to explore additional modalities, such as text and mixed-modality inputs, in future work.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this work, we compare openly available models with proprietary systems that lack transparency in their training details and internal workings and whose performance is not stable over time. This inevitably complicates efforts to address concerns such as data leakage or cross-contamination and limits the fairness and reproducibility of our experiments. Nevertheless, under these adverse conditions, our open models and methods demonstrate favorable performance. Users should be aware, however, that our models might inherit biases, flaws, or other limitations present in the training data, potentially leading to discrepancies between expected results and generated outputs. Furthermore, given the resource-intensive nature of LLMs, many of our training and inference hyper-parameters were adopted from related work or chosen based on general intuition. Although LLMs are generally robust to hyper-parameter selection (Beyer et al., 2024) , conducting a thorough hyper-parameter search might enhance their performance further. Finally, it should be noted that our models could potentially be misused by malicious actors to produce misinformation and fake science.Another important consideration is that the public release of DaTikZ v2 does not include some TikZ programs from our internal version due to licensing restrictions. These programs are distributed under the arXiv.org perpetual, non-exclusive license, which prohibits redistribution. Nonetheless, we provide our dataset creation scripts alongside usage instructions, enabling anyone to reproduce the full version of DaTikZ v2 independently. The remaining TikZ programs in DaTikZ v2 are licensed under Creative Commons attribution licenses,5 the GNU Free Documentation License,6 or the MIT license,7 and their respective terms and conditions apply. Regarding artificially created examples, OpenAI's terms of use restrict the use of their services for creating competing products, limiting this subset of DaTikZ v2 to non-commercial applications.8In this section, we discuss several extensions to our MCTS algorithm that aim to improve its performance and efficiency. We also explain alternative reward functions that we experimented with but ultimately found less effective than our chosen approaches.Building on our base MCTS implementation, we introduce several enhancements, namely dynamic rescaling of visual rewards, node deduplication, and preemptive stopping of faulty rollouts.Dynamic Rescaling One challenge when using SelfSim is that MCTS expects values to be in the range of [-1, 1], while deep encoders often work with a much narrower range in practice (Hessel et al., 2021; Zhang et al., 2020) . Furthermore, this range may vary depending on whether the input image is a real figure or a sketch. To address this discrepancy, we propose dynamically min-max normalizing the visual reward scores whenever they are (re)computed, ensuring that MCTS always operates on the full range. The modified reward formula is as follows:𝑉 ′ 𝑖, 𝑗 =         𝑉 𝑖, 𝑗 -min(𝑽 𝑖,: \{ -1} ) max(𝑽 𝑖,: ) -min(𝑽 𝑖,: \{ -1} ) if 𝑉 𝑖, 𝑗 ≠ -1 and max(𝑽 𝑖,: ) ≠ min(𝑽 𝑖,: \{-1}), 0 if 𝑉 𝑖, 𝑗 ≠ -1 and max(𝑽 𝑖,: ) = min(𝑽 𝑖,: \{-1}), -1 otherwise.(3)Node Deduplication During a rollout for a backtracking node, it is possible to generate code that already exists elsewhere in the tree (i.e., in siblings and their descendants). To prevent the duplication of nodes, we always merge identical node states before adding any nodes to the tree.Preemptive Stopping If the code generated in a rollout cannot be compiled due to a fatal error, we record the rollout, including the state in which the faulty line of code was first introduced. If the same (intermediate) state is sampled again during subsequent rollouts, we know that the completed output will fail to compile. In such cases, we preemptively abort the rollout and reuse the previously recorded rollout for the remainder of the simulation. To further prevent continuations from faulty code, during the expansion phase, we only add nodes to our tree whose node states do not contain any lines of code with fatal errors.Taking inspiration from popular machine translation metrics (Belouadi and Eger, 2023; Zhao et al., 2019 Zhao et al., , 2020;; Song et al., 2021) , which compute the Earth Mover's Distance (EMD; Rubner et al., 1998; Kusner et al., 2015) between word embeddings, we also explore with measuring perceptual image similarity as the EMD between SigLIP's image patch embeddings. Given the distance matrix 𝑫, where 𝐷 𝑖, 𝑗 = cos(𝑥 𝑖 , 𝑦 𝑗 ) and 𝒙, 𝒚 are the patch embedding vectors of the input and output images of simulation 𝑗 with lengths |𝒙| and | 𝒚|, respectively, EMD is defined as follows:EMD(𝑥, 𝑦) = | 𝒙| 𝑖=1 |𝒚 | 𝑗=1 𝐹 𝑖, 𝑗 𝐷 𝑖, 𝑗 | 𝒙| 𝑖=1 |𝒚 | 𝑗=1 𝐹 𝑖, 𝑗, with minEQUATIONWe define 𝑉 𝑖, 𝑗 = 2 tanh(-EMD(x, y))+1 ∈ [-1, 1] if compilation produces any output. If compilation fails, we set the reward to -1. We empirically tune the hyperparameter on which layer to extract the patch embeddings using the perceptual similarity dataset of scientific figures from Belouadi et al. (2024) . We find that extracting embeddings after the 24th layer yields the best results. However, when evaluated on our data, this reward function achieves a segment-level correlation of only 0.425 (cf. §7), which is lower than for SelfSim while being computationally more expensive. Consequently, we do not employ this reward function in further experiments.In Table 6 , we compare LIVE (Ma et al., 2022) , a state-of-the-art method for generating SVG, with our TikZ-based approach. In Figure 4 , we additionally investigate the extent to which our models memorize the training data. We also perform training data ablation studies, as presented in Table 7 . Since LIVE generates SVG code instead of TikZ, we do not report cBLEU and TED scores. Additionally, because it optimizes Bézier curves rather than generating tokens, we exclude MTE, leaving only the image similarity metrics DreamSim, SelfSim, and KID. Table 2 shows that LIVE underperforms all other models in our evaluation. On reference figures, it scores over 7.8pp and 14.1pp lower than the worst baseline model on DreamSim and SelfSim, respectively, and its KID is more than 18 times higher. This subpar performance can be attributed to the complexity of scientific figures saved as SVGs. While we use LIVE in its default configuration, generating eight paths with four segments each, our scientific figures consist of over 110 paths on average with an arbitrary number of segments, not counting deduplicated paths, which LIVE cannot detect. Although we could theoretically configure LIVE to generate more paths, this would linearly increase inference time, quickly becoming intractable. LIVE already requires over 18 hours to complete the test set for one input type, whereas DeTikZify-CS 7b (OI), for example, takes less than 5 hours. Furthermore, since LIVE attempts to vectorize the input directly without semantic interpretation, it performs even worse on synthetic sketches. We conclude that SVG, and, by extension, models that generate SVG, are not well-suited for our problem domain and objectives.Memorization of training data is a common concern in language models (McCoy et al., 2023; Carlini et al., 2023; Raunak and Menezes, 2022; Meehan et al., 2020) . To assess the extent of this issue in our models, we calculate the 𝑛-gram novelty (McCoy et al., 2023) . Specifically, we determine the proportion of 𝑛-grams, with 𝑛 ∈ [1, 10], in the model-generated TikZ programs that are not present in the training data. We perform this analysis on the test split of DaTikZ v2 for our baselines and DeepSeek-based DeTikZify models conditioned on reference figures, as well as human-generated code, as shown in Figure 4 . All models initially exhibit similar novelty and are slightly less novel than humans for 𝑛 < 7. However, starting from 𝑛 = 7, all models except DeTikZify-DS 1.3b surpass human novelty, with more than 80% of all model-generated 𝑛-grams being novel for 𝑛 >= 8. This phenomenon of models becoming more novel than humans is commonly observed and is considered an indicator that language models are not significantly affected by memorization (McCoy et al., batch size of 8 and a learning rate of 5e-5. We select "turn it into a doodle" as our initial prompt, which also appears in Instruct-Pix2Pix's pretraining dataset and demonstrates the most promising zero-shot performance.Building upon the experiments described in §6, we derive all our Self-Refine prompts from the official examples provided by Madaan et al. (2023) for generating TikZ programs, with only minor modifications. In particular, we employ the following prompt template in the initial step of both Self-Refine and Visual Self-Refine, substituting "sketch" or "picture" as appropriate: 6We then extract the first L A T E X code block from the generated text. In the rare cases where GPT-4V incorrectly classifies input images as unsafe, we add a small amount of Gaussian noise to the image pixels to bypass the issue. If compilation fails due to a fatal error (which occurs in only 1.5% of all cases) without producing an output artifact, we repeatedly use the following prompt template until all issues are resolved, replacing <code> with the generated code and <error> with the corresponding error message:Given the error message:1 <error> 2And the problematic code: 10 For Visual Self-Refine, we additionally use the following prompt template to visually refine the output.Since we provide two input images (the initial figure or sketch and the current output), we label one as "Input" and the other as "Reference". Claude 3's API has a built-in mechanism for labeling images, while for GPT-4V, we embed the labels directly into the images: 14 Following the findings of Madaan et al. (2023) , we visually refine for a maximum of four iterations, as they observe diminishing returns beyond that point, and it helps reduce inference costs. Although this means that in most cases, we terminate before the 10-minute timeout is reached (cf. §6.1), we believe this is a sensible decision, as we observe that GPT-4V is unable to visually refine its outputs successfully in any case. We hypothesize that this limitation is due to general-purpose chat models requiring too much explicit context for this task. These models receive the entire previously generated code as input, along with two input images and a complex textual prompt, which may be too challenging for them to process effectively. Preliminary experiments with more elaborate prompts did not seem to mitigate the subpar performance, likely due to this reason.Our annotator team consists of eleven experts with extensive research experience in science and technology. The team comprises one male faculty member, two female PhD students, seven male PhD students, and one male research assistant from another institution. We chose to work exclusively with expert annotators based on the findings of Belouadi et al. (2024) , which demonstrated that crowd annotators often lack the necessary research background to produce reliable annotations.To provide a better understanding of our work, we present a variety of examples in this section. Table 8 displays exemplary figures and real sketches from SketchFig, while Table 9 shows figures and synthetic sketches from DaTikZ v2 . Additionally, Tables 10 & 11 present sample outputs generated by our systems during our human and automatic evaluations. Figure 5 provides a closer look at generated code.When comparing the real sketches in Table 8 to their corresponding reference figures, it becomes evident that the sketches often contain less detail. For instance, sketches may lack colors or grids and feature less precise lines. Moreover, the handwritten nature of the sketches can sometimes make the text within them harder to read. These characteristics are also present in the synthetic sketches shown in Table 9 . However, the problem of illegible text is more pronounced in these sketches, as generating readable text remains a common challenge for image generation models (Borji, 2023) .While the text may still retain its meaning in a hidden way (Daras and Dimakis, 2022), this could lead to hallucinated text in the generated TikZ programs. Nonetheless, we believe that this aspect can still be advantageous for end users, as it enables them to quickly add scribbles to indicate the desired text placement. By doing so, DeTikZify can generate code for the overall structure and layout, allowing users to easily modify and replace the text afterward.The randomly selected generated figures from our human and automatic evaluations (cf. §6.2 & §6.1) shown in Tables 10 & 11 corroborate our quantitative findings. DeTikZify-DS 7b (TI) demonstrates the best overall performance and shows the least amount of fidelity errors, confirming the effectiveness of our SelfSim-based MCTS refinement algorithm. However, we still observe some inconsistencies, such as in layout and axes labeling, although to a lesser extent compared to DeTikZify-DS 7b (OI) and GPT-4V. We attribute the prevalence of this problem partly to our focus on perceptual similarity rather than, e.g., pixel-level similarity, which allows the models greater flexibility in interpreting the general semantics of the input figures and sketches. While optimizing pixel-level similarity could be an alternative approach, we argue that perceptual similarity can serve as a more meaningful measure, especially when considering sketches. We believe that real users who provide rough sketches of unfinished ideas will find the generated outputs that interpret and refine their concepts to be inspirational. However, we acknowledge the potential benefits of exploring more rigorous similarity measures and plan to investigate this in future research. Interestingly, GPT-4V occasionally generates outputs that may not be appropriate in a scientific context, such as mistakenly embedding a smiley face in the fourth example in Table 10 . Instead of resolving such issues, GPT-4V (TI) further emphasizes these details, distancing the output from the actual reference.Figure 5 provides a side-by-side comparison of the generated TikZ programs corresponding to the first row in Table 10 . DeTikZify-DS 7b demonstrates its ability to utilize advanced abstractions and control flow statements, generating code that is free of compile-time errors in both OI and TI configurations.On the other hand, GPT-4V (OI) incorrectly uses an undefined arrow tip kind stealth' in lines 9 and 10, resulting in recoverable compile-time errors. GPT-4V (TI) contains the same error in line 8 and introduces additional errors in lines 16 and 26, where the * symbol would have to be removed from the loop lists for successful expression evaluation.Real Sketches EQUATION′ 2 ′ 3 ′ 4 ′ 5 ′ 6 ′ 7 ′ 8 ′ 9 ′ 10 ′ 11 ′ 12 ′ 13 ′ 14 ′ 15 ′ α β α + β -α -α -β -β Experiments Theory/model Predictions Observations inference hypothesis testing × × decay r = 0 r → +∞ r → -∞Table 9 : Illustrative examples of reference figures and corresponding synthetic sketches from the subset of the DaTikZ v2 dataset that is licensed for redistribution. * 2 q * Ll tl ⃗ xl c c ′ t0 ⃗ x0 ⃗ xh -→ τ ql -→ qh ⃗ xh ⃗ x ′ l ⃗ x ′′ l th τ -→ τ ′ 0 0 1 1 (1+ ) 1 (1+ )2Table 10 : Alternating rows of randomly selected reference figures and real sketches (first column) alongside corresponding scientific figures generated by GPT-4V and DeTikZify-DS 7b in outputdriven (OI) and time-budgeted (TI) configurations (columns 2-4), taken from our human evaluation campaign (cf. §6.2). ′ q q q ′ q ′′ η ′′ η ′′′ E A ⊗A⊗A E A ⊗A⊗A E C ⊗A⊗A E B ⊗A⊗A E A ⊗A⊗A E 1 1 ≃ 1 1 ≃ ≃ 1 1 ≃ ≃ ≃ ≃3https://tex.stackexchange.com5https://creativecommons.org/licenses 6https://www.gnu.org/licenses/fdl-1.3.en.html 7https://opensource.org/license/mit 8https://openai.com/policies/terms-of-use 


Decision: Three of the four reviewers gave either a strong accept or accept for this paper.  Those reviewers appreciated the novelty of the work to automatically generate TikZ code from both existing figures and hand-drawn sketches, and they felt the results and evaluations were conceiving.

There was one dissenting reviewer.  This review had some inconsistencies in their comments that don't quite line up with the paper or are unclear and the reviewer didn't engage in the discussion with the authors or other reviewers.  They noted there were no visual results, however those appear in the appendix. 

Given the discussion with the other reviewers and the consideration of the rebuttal which other reviewers did respond to and felt satisfied, the AC agrees that this paper is a strong contribution and is accepted.  

Congratulations!

In revising your paper for the camera-ready, please include and comments and clarifications that were made or proposed during the rebuttal and discussion phase.  Also, the AC recommends including some visual results that are in the appendix into the main body of the paper.

Reviews: review: paper_summary: paper_summary: To create high-quality scientific figures the work trains a multimodal language model, DETixZirv, a new multimodal language model from sketches and existing figures. Trained on ATIXZ.2 (over 360k human-created TikZ graphics), SKETCHFIG (hand-drawn sketches paired with scientific figures), and SciCAr++ (diverse scientific figures with metadata) datasets, MCTS-based inference algorithm to refine outputs it outperforms CLAUDE 3 and GPT-4V in creating TikZ programs.
strength_weakness: strength_weakness: - Evaluation relies on compiler diagnostics and perceptual similarity, which may need to fully capture the quality and usability of generated TikZ code. Thus, visual outputs are necessary to evaluate the performance efficiently.

- Generated code might still contain subtle errors or inconsistencies that are not easily detectable through automated metrics. 

- Though the proposed model works better than GPT-4V when run for 10 minutes, it might limit the use cases of the work especially considering the computational requirements necessary. 

- From the provided metrics it could be inferred that the proposed model performs better than other models, however, the improvement provided considering the time and computational requirements makes the model less effective.
questions: questions: Please see the weakness section.
limitations: limitations: Please see the weakness section.
paper_summary: paper_summary: This paper proposes DeTikZify, a multimodal language model that generates scientific graphics in TikZ from hand-drawn illustrations. The authors provide three datasets: DaTikZ v2 (TikZ source code and figures), SketchFig (TikZ and hand-drawn sketch pairs), and SciCap++ (figures and captions). They also propose a method to iteratively improve generated TikZ using Monte Carlo Tree Search (MCTS) based on two reward functions. Experimental results show that combinations of multiple VLMs with the proposed method outperform closed-source LVMs like GPT and Claude. Additionally, iterative exploration using MCTS is shown to further improve figures efficiently.
strength_weakness: strength_weakness: A dataset named SciCap++ already exists and is not cited in this paper. The authors need to appropriately acknowledge this, explain the differences, and if necessary, rename their dataset.
[a] Yang et al., SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning. AAAI workshop SDU, 2023.
questions: questions: A response to the point raised in the Weaknesses section is expected.
limitations: limitations: As stated in the main text, the authors provide a sufficient discussion of limitations in the appendix.
paper_summary: paper_summary: This paper introduces DeTikZify, a multimodal model that automatically generates TikZ code for scientific figures and sketches. The authors create and leverage three novel datasets: DaTikZv2 (a large dataset of TikZ graphics), SketchFig (paired human-drawn sketches and scientific figures), and SciCap++ (a meta-dataset of scientific figures and associated text). They demonstrate that DeTikZify outperforms GPT-4V and Claude 3. They also show that a novel MCTS-based iterative refinement algorithm brings gains.
strength_weakness: strength_weakness: 1. The main issue that I have with this work is the potential data leakage. It is not clear how much of the training / evaluation data is in the training set of models. How can we be sure about this?
For the public models, it is said “we only include items created after the cut-off date of CodeLLaMA and exclude repositories that may have been used in training DeepSeek.” but how to ensure this for the private models? I feel that using the “𝑛-gram matching algorithm” does not really tell us a lot about this. It would be great if the authors can provide some further thoughts / explanations on this.
2. From the text I find it hard to understand the intuition behind and the exact computation of “Average Similarity” (L249). Would it be possible to improve clarify this in the main text?
questions: questions: 1. The paper focuses on scientific figures, which often exhibit structured layouts and simple shapes. However, TikZ's capabilities extend to much more complex diagrams and illustrations. How well would DeTikZify generalize to more diverse TikZ use cases beyond scientific figures? Are there inherent limitations in the approach that hinder its application to more free-form diagrams?
2. While the paper argues for the importance of segment-level performance, it's unclear how this translates to actual user experience. Can the authors elaborate on how segment-level improvements manifest in visually perceptible differences in the generated figures?
3. The paper mentions that SelfSim outperforms other metrics in segment-level correlation. However, the difference in Spearman's ρ appears relatively small (Table 4). Could the authors comment on the practical significance of this difference?
4. How sensitive is MCTS to the exploration coefficient c? Could the authors discuss the process of selecting this parameter and its effect?
5. Providing visual examples of successful and unsuccessful cases for DeTikZify, Claude 3, and GPT-4V would offer valuable insights into their strengths and weaknesses.
6. What types of figures or TikZ constructs does DeTikZify struggle with? What are the common failure modes of the system?
limitations: limitations: The limitations and societal impacts have been addressed.
paper_summary: paper_summary: The paper proposes a technique for generating TikZ graphics based on handwritten sketches. It open-sources the datasets used for this task, including a large TikZ dataset, a small sketch-to-TikZ dataset, and a large dataset of scientific figures with accompanying texts.

The model uses a vision-language model to take as input an image of the sketch, and potentially the output of the LaTeX compiler from the previous step, to generate the code that is attempted for compilation in the successive step. To further improve the model, Monte Carlo Tree Search is employed for more efficient iteration in finding a compilable and high-quality output.

The evaluation is done via several automated metrics such as code and image similarity, as well as wall time for generation, accompanied by the human evaluation, which positively correlated with the used automated metrics.
strength_weakness: strength_weakness: I think it would strengthen the paper to provide more examples of the model inference (similar to page 29 of the Appendix) to better gauge the performance of the models and to see examples without MCTS to understand how much does it contribute to the downstream quality (Fig. 3, right, paints some picture, but the SelfSim metric is hard to interpret). More generally, ablation study likely can also be strengthened by highlighting the amount of contribution brought in my MCTS, synthetic sketches, SciCap++, etc.
questions: questions: No particular questions, see suggestions in "weaknesses".
limitations: limitations: The authors have adequately addressed the limitations in the checklist.

Comments: comment: I thank the authors for further clarifications.

I still think this is a strong paper and recommend its acceptance.