Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches
Authors: Itsumi Saito
Haruto Yoshida
Keisuke Sakaguchi
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal
large language model
diagram
vector graphics
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We address the challenge of automatically converting hand-drawn sketches into vector diagrams. We present SkeTikZ, a dataset of 3,231 sketch-diagram-code triplets, and ImgTikZ, a vision-language model that performs the sketch-to-diagram conversion.

Abstract: We address the challenge of automatically generating high-quality vector diagrams from hand-drawn sketches. 
Vector diagrams are essential for communicating complex ideas across various fields, offering flexibility and scalability. 
While recent research has progressed in generating diagrams from text descriptions, converting hand-drawn sketches into vector diagrams remains largely unexplored due to the lack of suitable datasets. 
To address this gap, we introduce SketikZ, a dataset comprising 3,231 pairs of hand-drawn sketches and thier corresponding TikZ codes as well as reference diagrams.
Our evaluations reveal the limitations of state-of-the-art vision and language models (VLMs), positioning SketikZ as a key benchmark for future research in sketch-to-diagram conversion.
Along with SketikZ, we present ImgTikZ, an image-to-TikZ model that integrates a 6.7B parameter code-specialized open-source large language model (LLM) with a pre-trained vision encoder. 
Despite its relatively compact size, ImgTikZ performs comparably to GPT-4o.
This success is driven by using our two data augmentation techniques and a multi-candidate inference strategy.
Our findings open promising directions for future research in sketch-to-diagram conversion and broader image-to-code generation tasks. SketikZ is publicly available.

Published as a conference paper at ICLR 2025
SKETCH2DIAGRAM: GENERATING VECTOR DIA- GRAMS FROM HAND-DRAWN SKETCHES
Itsumi Saito",,, Haruto Yoshida', Keisuke Sakaguchi" "Tohoku University, 'RIKEN AIP itsum... saa saitotcchouu.ac.
ABSTRACT
We address the challenge of automatically generating high-quality vector dia- grams from hand-drawn sketches. Vector diagrams are essential for communicat- ing complex ideas across various fields, offering flexibility and scalability. While recent research has progressed in generating diagrams from text descriptions, con- verting hand-drawn sketches into vector diagrams remains largely unexplored due to the lack of suitable datasets. To address this gap, we introduce SKETIAZ, a dataset comprising 3,231 pairs of hand-drawn sketches and thier corresponding TikZ codes as well as reference diagrams. Our evaluations reveal the limitations of state-of-the-art vision and language models (VLMs), positioning SKETIKZ a key benchmark for future research in sketch cketch-to-ding -o-iggraam conversion. Along with SKETIKZ, we present IMGTIKZ. an image-to- -ikkz model that integrates a 6. parameter code-specialized open-source large language model (LLM) with a pre- trained vision encoder. Despite its relatively compact size, IMGTTKZ performs comparably to GPT-4o. This success is driven by using our two data augmentation techniques and a multi-candidate inference strategy. Our findings open promising directions for future research in sketch-to-diagran conversion and broader image- to-code generation tasks. SKETIKZ is publicly available
INTRODUCTION
Diagrams serve as powerful visual tools widely adopted across academic and professional domains to communicate complex ideas effectively. They play a crucial role in clear communication and knowledge transfer by distilling complex information into more accessible visual formats. Vector graphics have become the standard medium for creating high-quality diagrams, primarily due to their inherent scalability and precision. The ability to resize and modify vector diagrams without degrading quality makes them especially valuable in academic and professional settings. These characteristics enable researchers and professionals to adapt diagrams seamlessly across different presentation formats and requirements, enhancing both the clarity and versatility of scientific com- munication. While established tools and languages such as TikZ and Graphviz are popular for creating high-quality vector graphies, they often require significant manual effort and specialized expertise. Recent developments in large language models (L.LMs), such as GPT-4o, have triggered growing interest automating the generation of vector graphic diagrams from textual descrip- tions (Belouadi et al., 2023 Zala et al. 2023 Zou et al. 2024). This emerging research area holds significant potential to enhance the efficiency diagram creation and improve accessibility high- quality visualizations. Despite the significant advancements in text-to-code codeode generation, generating diagrams from sketches remains largely unexplored. Sketch-based input often provides more intu- itive and user-friendly way to express visual ideas (Figure I. This approach leverages the inherent human ability to quickly and effectively communicate complex visual information through simple drawings. A primary reason for the limited research in this area is the lack of publicly available datasets that pair hand-drawn sketches with their corresponding codes. Such datasets are essential for training and evaluating models that translate sketch-based based input into structured diagram code. To address this gap, we introduce SKETIKZ, a new dataset designed for benchmarking sketch-to- diagram generation. SKETIAZ comprises 3,231 pairs of hand-drawn sketches and their correspond- ing TikZ codes. The sketches were created using several tools commonly employed in real-world scenarios: paper, whiteboards. and tablets. This diverse collection provides valuable resource for
https://sketiiex.githhi

---

Published as a conference paper at ICLR 2025
sketch to diagram
Agureitee Y
@+
tbcaincoccuentt
texk
toxt
boot
bock
rest
teat
promptist
Hgueedtes
Concrate TIK coce for dreated grohh with the following speciications:: Five nodes
abeled 0, 1, 2, 3, and Mode has
outyoing red
Nodes 0, cum.
Inode
toxt-to loxt-to-diegram
"res"
LLM
Figure I: Overview of sketch-to-diagram. We consider scenarios where users hand-draw diagrams that they want create. Sketch-to-diagras to-diagram models (e.g., VLM) take these sketches I, and pre- defined instructions X and then generate code Y for producing vector graphics. Y is subsequently rendered into generated image The process -an-t--diagrmm also provided for comparison.
advancing research in automated diagram generation from sketches. SKETIKZ aims to facilitate the development of models capable of generating high-quality diagrams from hand-drawn inputs for real-world applications. We also developed IMGTTKZ, Vision-Language Model (VLM) specifically designed for this task. Our model combines three components: an open-source LLM specialized in code generation, vision encoder, and an adapter. This combination is intended to create model capable of efficiently converting sketches into TikZ code. We evaluated the effectiveness of two strategies: expanding our dataset through data augmentation and employing inference strategy that generates multiple candidates and selects the best one. From the results, IMGTIKZ performed comparably to GPT-4o in subjective evaluations despite having relatively small model size of 6.7B parameters. However, both IMGTIKZ and the latest state-of-the-art models still struggle to accu- rately generate code that captures all elements and layouts of sketches, indicating the potential for further advances. We aim for our dataset and findings to drive future research and development in this field. Our contributions are summarized follows:
We introduce SKETIAZ: new dataset containing 3,231 pairs of hand-drawn sketches and their corresponding TikZ codes, addressing the lack of real-world data for sketch-to-vector diagram conversion.
We develop IMGTIKZ: An image-to-TikZ model that combines 6.7B parameter code- specialized LLM with a pre-trained vision encoder, achieving performance comparable to larger models despite its modest size.
We empirically demonstrate the effectiveness of two types of data augmentation and a multi-candidate inference strategy.
2 RELATED WORK
Vision and language models With advancements in LLMs, significant progress has been made in constructing VLMs that interpret images and generate text. A promising approach integrates vision encoders like CLIP (Radford et al. 2021) with LLMs via adapter modules. This method has demonstrated promising results (Liu al. 20233] Dai al. 2023 Ye et al.) 2023 Zhu et al. 2023 Li et al. 2024 Wang et al.) 2024), efficiently creating VLMs that leverage the extensive know ledge base pre-trained models. In this study, along the same line as these approaches, we build VLM to generate TikZ code from images.
Image-to-oode generation While VLMs are primarily designed generate natural language out- puts, such as answering questions and describing images, research on generating code for image rendering-sun such as HTMM,, LaTeXX, or SVG-has emerged as a valuable application. For instance, recent studies have introduced models capable of generating LaTeeX code from screenshots of math- ematical formulas or handwritten images (Deng al. 2016
Gervais et al. 2024), HTML code from
web page screenshots (Soselia al. 2023 Si et al. and SVG code from icon images (Rodriguez et al.
2024 Laurençon et al. 2024 Gui et al. 2024).
2023). While LaTeX code generation and TikZ
code generation similar in terms of code output, problems than previous formula-to- -oaal---t-TTeee- conversion studies. involves much longer output se- quences (739 tokens on average compared to 65 tokens in prior work) and requires an understanding
research tackles significantly more complex
2

---

Published as a conference paper at ICLR 2025
Pairing TiZ codes, skeich inages, and rendered images.
Table 1: Sketch Tool Usage Statistics.
Step1. Rendering
Step2. Filterlin
Step3. Annotation
Tool
Number
Proportion
TIKZ
Rendered Image
Rendered Image
Sketch Image
Paper
2,545
78.8%
Whiteboard
346
10.7%
Tablet
340
10.5%
Figure 2: Dataset construction process.
All 3,231
100%
of two-dimensional layouts. We introduce three key advances handle this increased complexity: code-specialized VLM, two data augmentation strategies, and caulti-anndddte generation.
Diagram understanding Understanding diagrams has been an important and long- -ta-sitndinn re- search topic, including question answering (Kembhavi et al. [0166] Lu et al. 2023 Wang ct al.),, caption generation (Hsu et al.] 2021] Singh et al.) [023 H Huang et al. 2023). and generating de- scriptions (Hu et al. 10233aBuusha & Lee [022] Bhushan et al. 2024). Recent research proposed
benchmark datasets to assess not only the understanding of diagram images but also the direct com-
prehension of vector graphics code (Zou et al. 2024 Qiu al. 2024). This expanding research area reflects the growing interest in understanding vector graphics diagrams.
Diagram generation Ellis et al. (2017) proposed a model generating TikZ. code for primitive ge- ometric sketches, focusing on circles, rectangles, and lines without text. We extend the approach to handle real-world diagrams with unrestricted shapes and text. Furthermore, our dataset reflects realistic environments by including sketch images from various sources such as paper, whiteboards. and tablets. Recent work has explored real-world diagram generation from text (Belouadi et al.) 2023 Zala al. 2023). Belouadi et al. (2023) proposed method for generating TikZ code to ren- der diagram images from caption text. Generating diagrams through code synthesis provides better controllability and editability than pixel-based image generation methods, while enabling L.M inte- gration. [Belouadi et al. (2023) also highlights the challenge of image-to-diagram generation, which remains limited due the scarcity of paired image-code data. Concurrent work by Belouudi et al. (2024) addresses the task of generating diagrams from images, which is closely related to our task. However, their evaluation of iketch-based generation is limited to a small dataset, which lacks cor- responding TikZ code and thus cannot be used for image- image-to-code to-code training. Our dataset provides the largest and most diverse sketch-to-diagran to-diagram dataset with TikZ code, captured under conditions. We also contribute novel data augmentation methods and multi-candidate generation strategies, providing new insights for future research directions this field.
DATASET AND TASK
3.1 TASK DEFINITION
We introduce sketch-to-diagran task (Figure []. where the input consists of sketch image of a diagram I, and a language instruction X, and the output is sequence of TikZ code Y. Then generated TikZ code Y are compiled to render the diagram image I.
3.2 DATASET CONSTRUCTION
We constructed our dataset in three steps: rendering, filtering, and sketch annotation (Figure
We first rendered diagrams from TikZ code in the DaTikZ (Belouadi et al. 2023 by using pdflatex. We then paired the rendered reference diagrams I, with the corresponding TikZ code Y,. We refer to the rendered diagrams as the reference images.
Step 1: Rendering diagrams from TikZ code
Step 2: Diagram classification and filtering Diagrams can be classified into various categories, as demonstrated by ACL-Fig (Karishma et al. 2023) with its 19-category dataset. For our sketch-to- diagram task, we focused on diagrams composed of geometric shapes and arrows, excluding those primarily based numerical data. We specifically targeted diagrams categorized Tree, Graph, Architecture Diagram, Neural Networks, and Venn Diagram according to ACL-Fig labels. We chose these categories because sketch-to to-iaga-- generation is particularly effective for visually oriented diagrams. These diagrams often involve complex combinations of shapes and interconnections, making manual creation time-consuming and precise linguistic instructions challenging. Using
3

---

Published as a conference paper at ICLR 2025
To
(9)G
Up
(#)
GE)
Figure sigaree:EExappeeo comptible Preserenccdi
Eanoenal
Figure 3; Examples of sketch images. Left: paper, Center: whiteboard, Right: tablet.
Table 2: Datasets used for training IMGTIKZ.
No
Name
Input
Output
Size
Stage1
Stage2
arXiv figure
Figure table image
OCR text
1.2M
2
arXiv figure
Figure table image
Caption text
1.1M
LLaVA-Prettain
Multi-domain image
Caption text
558K
SKETIKZ
Diagram sketch image
TiAZ. source code
2.6K
RenderTikZ
Diagram image
TiAZ source code
155K
AugTikZ
Diagram image
T source code
556K
ImgAug TikZ
Noised Diagram image
TiAZ source code
714K
8
DaTikZ-v
Diagram image
TikZ source code
46K
image classification model trained on the ACI.fig dataset (details in Appendix Ej. we extracted and sampled 4,000 diagram images from our targeted categories for annotation. We present the detailed breakdown of categories in Table 10 and Figure 8 Appendix
Step 3: Sketch data collection Twenty-eight annotators created sketch images I, on the basis of filtered reference images I,. Annotators used black pens primarily, with red, blue, and green for colored elements, excluding complex diagrams and ignoring color filling. Regarding the sketching tools, annotators freely selected available option from paper, whiteboard, their respective environments. Table shows the distribution of sketches by tool, with the paper being the most common. Figure 3 illustrates examples of sketch images drawn using each tool. The dataset includes diverse sketches mimicking real-world scenarios, with paper and whiteboard sketches showing varied lighting and backgrounds. We aligned sketches Ig with corresponding Tikz codes Y, and reference images I,., creating a dataset of 2,585 training, 323 validation, and 323 test samples. More examples are shown in Appendix FF)
tablet the basis of
IMGTIKZ: VISION-LANGUAGE MODEL FOR IMAGE-TO-TICE TIKZ GENERATION
4.1 MODEL STRUCTURE
We developed IMGTIKZ, VLM specifically designed for this task using the model architecture of LLaVA 1.5 (Liu et al. 2023). The model architecture comprises three key components: code- specialized LLM, a vision encoder, and adapter, illustrated in Figure (a). The model inputs diagram image and generates corresponding TikZ code. We employed the same architecture as LLaVA 1.5 for the adapter module- a simple two-layer multi-layer perceptron (MLP). While the original LLaVA 1.5 uses language model for natural language generation, we replaced it with 6.7B instruction-tunet DeepSeek Coder (Guo al., (224) for code generation. For vision encoder, used SigLIP model Zhai et al. 2003).. We trained our model in two stages: first updating only the adapter parameters, then training both adapter and LoRA (Hu et al.) 2021) parameters added to the LLM. The LLM and vision encoder parameters remained frozen throughout training. For more detailed information about the model hyperparameters, refer to Appendix B and Table
4.2 TRAINING DATA
Datasets used in stage training In stage training, we incorporated arXiv figure data (No. I and 2 in Table 2 in addition to LLaVA-pre VA-eerranii data (No. 3). This arXiv figure dataset was created by extracting figures, tables, and captions from arXiv paper PDFs arXiv bulk dataset

---

Published as a conference paper at ICLR 2025
TikZ code
figure.tex
1. Generate
3. Select one candidate
Ibegin(document
K candidates
2. Rendering
CodeLLM
LORA
TikZ
Diagram
Adapter
Instruction
Img TikZ
TikZ
Diagram
Diagram
Vision Encoder
Please generate Tikz code to draw the diagram of the giver image.
TikZ
Diagram
Image
Sketch
TikZ
Diagram
Calculate similarity using selector
(a) mmg TikZ model structure
(b) Multi-candidate generation
Figure 4: IMGTIKZ model structure (a) and multi-candidate generation process for inference (b).
using PDFFigure2.0 We also used Google Cloud Vision AP to extract text from these images. The arXiv data served two purposes: (1) generating optical character recognition (OCR) text from images to improve text recognition and (2) generating captions from diagram images enhance diagram image understanding.
Datasets used in stage 2 training In the second stage of training, we focused on enhancing the model's ability to generate TiKZ code. Given the limited size of the SKETIKZ dataset alone, we supplemented our training data by creating pairs of rendered diagram images and TikZ code col- lected from the arXiv source file in bulk data, which is referred to as RenderTikZ (No. 5). We implemented two data augmentation techniques to increase diagram and image variations. First, generated TikZ code using GPT-3.5 to increase the variety of diagrams, referred to as AugTikZ (No. 6). Second, we applied image augmentation technique, referred to ImgAugTikZ (No. 7), to simulate common sketch image noise such as background interference, lighting variations, and ro- tation. In addition these augmentation techniques, we also used existing pairs of TikZ code and images (No. 8), excluding data with arXiv IDs that overlap with our collected dataset.
Data augmentation for increasing diagram variations While we collected approximately 916K original TikZ codes from arXiv sources, many failed to be compiled during RenderTiÄZ creation. We used GPT-3.5 to fix these compilation errors with prompt such "Please modify the code to make it compilable." To increase diagram variety, we instructed GPT-3.5 to modify the origi- nal diagram into a different diagram, producing altered versions of the original diagrams. These augmentation techniques resulted in 556K AugTikZ data samples. Previous data augmentation for VLMM used other VLMM generate instruction-respons pairs from images, which was costly due to image processing. Instead, we generate data efficiently by modifying only TikZ code using text- based LLMs. This approach could be applied various image-to-code tasks. More details are in Appendix G.I
Data augmentation for increasing image variations Hand-drawn sketch diagrams inherently contain more im- age noise than rendered images. This noise can appear background interference or lighting variations when captur- ing sketches from paper or whiteboards. Furthermore, hand- written text and lines often exhibit significant distortions, and diagrams frequently stored with angular rotations.
Ihe il
To address these issues, we applied multiple image aug- mentation techniques to RenderTikZ and AugTikZ datasets, such as synthesizing notebook backgrounds, adding Gaus- sian noise, varying brightness and contrast, and introducing distortion. Figure |illustrates example of the augmented image. This augmentation approach general-purpose and be applied to various sketch-to torcc--t--iagram tasks. More de- tails are in Appendix G.2
Figure 5: Example of ImgAugTikZ. Top: original image, bottom: aug- mented image.
https://qithin http:////itthhhb..ommmaeaaaaaa//
https://cloud.gonnt www.vosiiiiiicoooooo..c docee//ddocchhcccc een
5

---

Published as a conference paper at ICLR 2025
4.3 INFERENCE
We implemented two inference methods: iterative generation and canti--a generation. In the paper, we refer to them as IMGTIAZ-IG and IMGTIAZ-MCG, respectively.
Iterative generation Iterative generation produces one candidate per test sample, regenerating upon compilation failure until success. We set a maximum number of generation attempts M to limit this process. This method straightforward and can be considered a baseline approach.
Multí-candidate generation Multi-candidate generation creates candidates simultancously, selecting the best one (Figure 4 (b)) using selector model. In study, generate multiple TikZ codes and render them as images. The selector selects the best candidate by maximizing the similarity between the input sketch image I, and the generated diagram image I. As general vision encoders cannot accurately measure diagram similarity, we propose D D-SigLIP DDigram-SSpcializie SigLIP) as our selector. D-SigLIP adds trainable linear layer a pre-trained SigLIP model, and we fine-tune only this layer through contrastive learning (Chen et al.] (020) with diagram pairs from RenderTikZ and AugTikZ. More details are in Appendix [ To calculate the similarity score, we computed the cosine similarity between the embedding vectors obtained by inputting the sketch image J and the generated diagram image into D-SigLIP.
Our task requires generating lengthy code sequences (averaging 739 tokens), making producing error-free code in a single-generation attempt challenging. Furthermore, since the model training is based next-token prediction loss for code sequences, metrics related to image quality are not explicitly considered during code generation. The multi-candidate generation and selection strategy allows to evaluate these metrics after code generation, which could not be considered during the training phase. While similar approaches have been proposed for text inference and coding tasks (Brown et al., 2024), our work is the first to image similarity for candidate selection in image-to-diagran conversion.
EVALUATION METRICS
5.1 AUTOMATIC EVALUATION
We used four aspects of automatic evaluation: compilation success rate, image similarity, code similarity, and character similarity.
Compilation success rate The compilation success rate (CSR) represents the percentage of gener- ated TikZ codes that are successfully compiled into images. In this study, we employ two CSR met- rics. The first is the averaged CSR, which calculates the ratio of successful compilations Nuucesss to the total number of generation attempts Ngen. expressed as CSRagg Nessus This metric indicates how often model succeeds in compilation represents the number of test samples that are compiled successfully through multiple iterations of iterative generation. It is defined as the ratio of successfully compiled samples, Nuss to the total number of test samples, Niett and is expressed CSR Nus Nis This metric shows the proportion of test samples that are correctly compiled through multiple attempts during iterative generation. Detailed examples are provided in Appendix
Ngo
average. The second is the cumulative CSR, which
Image similarity We used cosine similarity between image embeddings to measure the similarity between the generated image I and the reference diagram image I,-, We used our D-SigLIP (see Sec. (43) for calculating image embeddings. We also calculated the image similarity score using the original CLIP model; however, the similarity score computed with CLIP correlated less with human evaluations than the similarity calculated using D-SigL.I, If the compilation failed, we set the similarity score to O.
We used cosine similarity in the embedding space between Y and Y,. We gener- ated the code embeddings using OpenAl's 's text embedding model
Code similarity
Character similarity The character similarity calculates the similarity between the text in the generated image I and the text in the reference image I, using Rouge-1 score (Lin) 2004). We used the OCR included in the Google Cloud Vision API to extract text. This metric indicates how well the model can read and generate text from the sketch.
We used text-enbeedi hg-3 3-mammml version.

---

Published as a conference paper at ICLR 2025
5.2 SUBJECTIVE EVALUATION
We conducted subjective evaluation focusing two key aspects: alignment and quality following established practices in previous studies (Otani et al. [023] Ku et al. 2023). In our study, alignment measures the similarity between the generated and reference images, while quality assesses the coherence and appropriate arrangement of elements within the generated diagram. We employed a five-point scale for both metries ensure nuanced evaluation.
Aligument Annotators assessed alignment by visually comparing the generated diagram image to the reference diagram image 1, The sketch diagram image L, was also provided for evaluation. Score of 1 and indicated that the diagram's 's elements were completely misaligned and almost perfectly aligned, respectively. To illustrate score of I, randomly selected rendered diagram image from the training dataset was displayed.
Quality Annotators assessed the quality of the generated diagram images independently of the reference images, focusing on the structural clarity and arrangement of elements within the layout. A score was assigned to diagrams with poorly arranged, overlapping elements that were nearly unreadable. Conversely, score of 5 was given well-structured diagrams with logically arranged shapes and text that closely resembled human-created diagrams. The scale reflects the overall layout quality, ranging from incomprehensible highly coherent visual representations.
Annotation We comprchensively evaluated each model's outputs across the entire test set using Amazon Mechanical Turk. A total of 40 annotators conducted the annotation. For each test sample generated by each model, five annotators performed the evaluation. Diagrams that failed to be compiled were automatically assigned the minimum score of 1 for alignment and quality metrics. We computed the final score for each system and instance by averaging the three median evaluation scores, excluding potential outliers. A detailed description provided Appendix
EXPERIMENTAL SETUP
Models for Comparison We evaluated several state-of-ther -frtte-art models in our study GPT-4o, OpenAI's most efficient multimodal model. We also included GPT-4o mini, their top small model. From Anthropic employed Claude 3.5 Sonnet, the latest in their multimodal LLM series. Lastly, we assessed LLaVA-Next, a popular open-source model.
Training parameters for IMGTIKZ We set the LoRA tuning parameters for training to r=128 and a=256. Stage training was conducted with a batch size of 256 for 6,000 steps. Stage training used a batch size of 128 for 1 epoch. We used 8 A100 GPUs for training IMGTIKZ, and H100 GPU for inference. More details are in Appendix
Inference We applied iterative generation the baseline for the four comparison models (see Sec(o), while for IMGTIKZ, we implemented both iterative and multi cunti-canddddte generation. The maximum number of attempts M for iterative sampling was set 5, and the number of candidates K for multi-candidate generation was set to 20. More details are in Appendix A
7 RESULTS
7.1
MAIN RESULTS
Can models generate compilable TikZ code for diagrams? Table 3 presents the averaged CSR results (CSR avg), with IMGTIAZ significantly outperforming the other models. The remaining models showed relatively low CSR avg values (approximately 0.35-0.54), indicating insufficient adaptation to TiAZ data. Since averaged CSR directly impacts user convenience, achieving higher scores is crucial. Figure couure/illuee iillustrates the progression cumulative CSR across iterative generation attempts. IMGTIKZ achieved nearly 100% success after five attempts for the test data, while other methods leveled off at 0.8-0.9. These results indicate that 10-20% of samples remain uncompilable after five attempts with these models.
ver- We used the gpt 2ot-44-2024405-13 version for GPT-4o, the version for Claude 3.5, and the sion for GPT-4o mini, the cl claude-3 11ama3-1 1aaa--ext--e version, which trained the 8B Ilama 3 model, for VA-NNett

---

Published as a conference paper at ICLR 2025
Table 3: The results of the automatic (0-1) and subjective (1-5) evaluations. highlighted in bold.
The best results are
Automatic
Subjective
Model
ImageSim
CodeSim
CharSim
CSR avg
Alignment
Quality
Closed models
GPT-4o
0.695
0.821
0.611
0.479
3.00
3.20
GPT-4o-mini
0.595
0.814
0.514
0.376
2.39
2.71
Claude 3.5 Sonnet
0.753
0.813
0.671
0.544
3.32
3.54
Open-source models
LLaVe-Next
0.315
0.727
0.206
0.350
1.43
1.93
IMGTIIZA TIKZ-IG (ours)
0.734
0.815
0,503
0.767
2.78
2.92
IMG MIGTIZ--MGG (ours)
0,821
0.822
0.594
0.799
3.13
3,30
1.0
Image Similarity (ImageSim)
0.85
Cumulative CSR
0.9
0.80
0.8
0.75
0.7-
Claude 3.5 Sonnet
0.70
0.65
Next
ImgTikZ with D-SigLIP selector
TIk-IIG
ImgTikZ with CLIP selector
0.60
10
20
Number of Attempts
Number Generated Candidates
Figure 6: Progression of cumulative compi- lation success with varying number of attempts in an iterative generation.
Progression of image similarity with varying number of candidates in multi- candidate generation
Figure 7:
Can models generate diagram images close to the reference images? ImageSim and Alignment 3 present the similarity between generated and reference images. Claude performed the best in Alignment score, followed by IMGT47-MCG, In contrast, for ImageSim, IMGTTAZ-MCG outperformed the other models, with Claude 3.5 performing the second best. LLaVA-Next, which has comparable model size to IMGTIKZ but lacks TikZ-specifie training, performed poorly and rarely generated correct output. IMGTIAZ-MCGG performed comparably to GPT-4o in Alignment despite being smaller, highlighting the effectiveness of adaptation and multi-candidate genera- tion strategy. However, even the best-performing model, Claude 3.5, achieved an average Alignment score of only 3.3, indicating that the generated diagrams match only 50-60% of the reference dia- grams based on the subjective assessment. These results suggest that the task remains challenging, even for state-of-the-ar models.
Can models generate TikZ code close to the reference code? Table indicates that IMGTIKZ- MCG achieved the highest similarity scores for code similarity. However, code similarity scores are generally high with minimal inter-model differences. This indicates that high code similarity does not necessarily guarantee quality image generation. This discrepancy highlights critical insight for model training: generating code that closely resembles the ground truth is insufficient. Similar conventional VLMs, IMGTIKZ training relies loss based the next-word prediction of code. However, our findings suggest image similarity metrics need to be incorporated in training or infer- ence phrases. This result aligns with the significant performance improvements of IMGTıkZ-MCG.
Can models accurately render text in sketch images? The CharSim in Table provides in- sight into each model's ability to recognize characters in sketch images and render them accurately TiAZ diagram. Claude 3.5 achieved the highest CharSim score, followed by GPT-4o. While IMGTIKZ performed comparably to GPT-4o in Alignment, it significantly underperforms in Char- Sim. This suggests that IMGTIKZ has enhanced diagram shape recognition but struggles with de- tailed character recognition. This limitation may reflect the resolution constraints of the SigLIP vision encoder. However, the substantial improvement in CharSim with multi-candidee candidate generation indicates character recognition needs to be strengthened during training.
8

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation of the effectiveness of SKETIKZ as training data.
Table 5: Effectiveness of two data augmen-
tation: (a) ImgAug and (b) AugTikZ.
Model
ImageSim CharSim CSR avg
Model
ImageSim CharSim CSR avg
IMGTIiZ-IG
0.734
0.502
0.767
IMGTIAZ-IG
0.734
0.502
0.767
w/ SKETiKZ only
0.513
0.358
0.533
w/o (a)
0.668
0.457
0.635
VA- 8B
0.315
0.205
0.350
w/o (a) and (b)
0.601
0.439
0.541
Can models generate high-quality diagrams? Table evaluations. Claude 3.5 achieved the highest average score of 3.54 out of 5, followed by IMGTTAZ- MCG. Even the persfoerforn Claude model produces approximately 38% of samples with quality scores below 3 (indicating significant overlap of shapes and text), demonstrating that current VLMs still struggle with correct diagram layout rendering. This limitation in spatial reasoning is a common challenge among current VLMs. Our task and dataset be considered of the benchmark datasets for evaluating VLMs' spatial reasoning capabilities.
3 presents quality scores from subjective
How does the number of candidates in multi-candidate generation affect performance? Fig- ure 7 illustrates the image similarity trends for ImgTikZ-MCG as the number of candidates K in multi-candidate generation varies. The oracle represents the highest achievable performance by se- lecting the best candidate on the basis of image similarity to the reference diagram 4, Results show performance significantly improved when candidates were increased from one to five. Both ora- cle and IMGTIKZ demonstrate enhanced image similarity with more candidates. However, when replacing the selection model from DI to CLIP, performance does not increase beyond five candidates. This indicates the importance of selection model quality in multi-candidate generation.
Do subjective evaluations correlate with automated evaluations? We analyzed correlations be- tween subjective alignment ratings and automatic evaluation metrics. Pearson's correlation coeffi- cients were calculated between human--atee alignment and image similarity (0.759), code similarity (0.365), and character similarity (0.592). Image similarity correlated strongly with the subjective evaluation. while code similarity correlated weakly with it. Character similarity correlated moder- ately, highlighting the importance of textual information in diagram evaluation. Image similarity metrics often fail capture this local textual similarity.
Are the subjective evaluations consistent? To assess inter-annotator agreement in subjective evaluations, we employed Krippendorff's a (Krippendorfi) 1980), a measure commonly used in related research (Otani al. [023] Ku et al.) (023). The analysis showed Krippendorff's of 0.761 for alignment and 0.662 for quality, indicating substantial to moderate agreement among annotators in their subjective assessments.
7.2 DETAILED ANALYSIS
How effective is SKETIKZ alone training data? We evaluated the effectiveness of SKETIKZ dataset, comprising only 2.6k hand-drawn sketch samples, as training data. We evalu- ated the performance of a model trained solely SKETIkZ in step 2. Results T While the SKETIkZ-onll model underperforms compared to the full-data model, it signifi- cantly outperforms LLaVA-Next, indicating meaningful adaptation even with this limited dataset.
presented in
Is data augmentation effective? To assess the impact of our two data augmentation methods, trained models excluding ImgAugTikZ and both ImgAugTikZ TikZ and AugTiAZ Results are presented in Table The observed significant decrease in image similarity, character similarity, and CSR_ave when excluding these datasets demonstrates the effectiveness both augmentation methods.
To what extent does image augmentation improve sketch recognition? While the ablation study in Table confirmed image augmentation improved performance, we further investigated its impact on sketch recognition. Specifically, we compared the performance gap between using ren- dered reference images 1, and sketch images I, as input. The closer the performance of sketch input approaches that of rendered image input, the more robust the model's understanding of sketch noise can be considered. Results by approximately 12.5% and character similarity by 22.7%. In contrast, TikZ limited these re- ductions 6.97% for image similarity and 17.0% for character similarity. However, Img still
shown in Table 6 Without ImgAugTikZ, image similarity decreased

---

Published as a conference paper at ICLR 2025
Table 6: Performance gap between ren- dered and sketch image inputs: comparison IMGTIKZ-IG and IMGTIKZ-IG without Im- gAugTikZ data.
Table 7: Performance gap between ren- dered and sketch image inputs across differ- ent sketching tools. Evaluation conducted using the IMGTIAZ-IG.
Model
Tool
Metric
IMGTIZZ-GG w/o ImgAugTikZ
Metric
Paper
Whiteboard
Tablet
ImageSim Rendered Image
ImageSim
0.789
0.763
Rendered Image
0.793
0.796 0.716
0.754
Sketch Image
0.734
0.668
Sketch Image
0.735
0.740
Performance Gap
-6.97%
-12.5%
Performance Gap
-7.31 %
-10.1 %
-1.90%
CharSim
CharSim
Rendered Image
0.605
0.591
Rendered Image
0.587
0.627
0.581
Sketch Image
0.503
0.457
Sketch Image
0.502
0.451
0.570
Performance Gap
-16.9%
-22.7%
Performance Gap -14.5%
-28.1 %
-1.89 %
does not match rendered image input performance, suggesting the potential for further improving performance by constructing a more noise-robure model construction.
Does image augmentation improve performance for non-sketch images? Comparing Ima- geSim and CharSim results for Rendered Images in Table reveals that ImgTikZ outperforms the model without image augmentation. Image augmentation enhanced both ImageSim for clean, computer-
and CharSim (0.591 -0.655) scores, showing improved recognition rendered images.
Does image recognition difficulty vary across sketch tools? Table 7 presents the performance gap in image and character similarity when using rendered images versus sketches inputs across different sketching tools. Results show that tablet sketches maintain image and character similarity close to rendered images. However, sketches from paper and whiteboard tools show significant per- formance degradation, declining by 7-10% in image similarity and 14-28% character similarity. This performance drop suggests that paper and whiteboard sketches are more challenging for the model to process, likely due to their greater noise variety than tablet sketches. Whiteboard sketches showed the most significantly in performance. While image augmentation techniques have rela- tively minimized the gap with rendered image input, further performance improvements will require developing methods more robust real-world noise.
CONCLUSION
We introduced SKETIKZ, benchmark dataset with 3,231 pairs of hand-drawn sketches and their corresponding TiAZ codes for generating diagrams. Our experiments demonstrate that current VLMs face considerable challenges in this task, highlighting the value of SKETIKZ as benchmark for future research. We also developed IMGTIÁZ, an image-to-TikZ model. Despite being smaller, this model performed as well as GPT-4o in subjective evaluations. This success came from using two data augmentation techniques and generating multiple candidates during inference. SKETIKZ is publicly available, and we expect these data resources and insights to drive the development of more advanced and efficient methods for automating vector graphics creation from hand-drawn sketches.
LIMITATION
Currently, SKETIkZ is restricted to generating diagrams using TikZ. However, the methodology could be extended to other formats such as SVG, HTMLL, Python, and JavaScript for diagram gen- eration from code. Exploring these additional formats could enhance the dataset's generality and applicability. Transforming sketches into well-formed diagrams involves information completion, which can potentially lead to hallucination. An important direction for future work is developing an interactive system that allows users to modify generated diagrams through instructions. Further- more, while our multi-candidate generation strategy considers code correctness and image quality metrics after code generation, incorporating these metrics directly into the training phase could po- tentially lead to better generation results, representing promising direction for future work.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Were annotators for sketch creation told what the dataset would be used for, and did they consent? Yes. BAOBAB Inc. was fully responsible for managing the annotators. BAOBAB Inc. provides task descriptions, training, and agreements for each project with the annotators https: 7/baobab trees. eersviiccseeeeeee//e
Data License SKFTiKZ derived from a publicly available subset of DaTikZ (Belouadi et al, 2023 which permits copying and redistributing content under Creative Commons Attribution License the GNU Free Documentation License or the MIT License 10
Potential ethical considerations We believe that there are minimal ethical considerations within the scope of this current research. However, as more accurate automatic diagram generation be- comes feasible in the future, several issues may arise. These potential problems include the misuse of highly accurate auto-generated diagrams to spread misinformation, the risk of Al models per- petuating or amplifying biases from their training data, and the possibility of advanced systems inadvertently reproducing copyrighted diagram designs, thereby raising intellectual property and copyright infringement issues; all of these challenges necessitate the establishment of appropriate guidelines to address them effectively.
REPRODUCIBILITY STATEMENT
Dataset Distribution
SKETIKZ is available at
/o/
Details of models, hyperparameters, and manual evaluation Appendices and provide detailed information about the models developed in this study. Appendix A describes the specifics of our inference process. Appendix [H presents details regarding the subjective evaluation. Addi- tionally, Appendices Dand and G Gresennts details of the data creation process.
ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Numbers, 21 K23343 and 24K20829, and JST Moonshot R&D Program Grant Number JPMJMS2236. We thank BAOBAB Inc. for creating hand- drawn diagram images with high quality and accuracy. In this research work, we used the "mdx: a platform for building data-empowert empowered society". This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.
REFERENCES
Jonas Belouadi, Anne Lauscher, and Steffen Eger. AutomaTikZ: Text-guided synthesis of scientific vector graphics with TikZ. arXiv [cs.CL], 2023. URL http:///axiv..rg// http://waxxi..ooo/////331..
00367
Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ. arXiv CLJ) 2024. URL http:
/aaxiv.org/abs/////// http:/aabb22005..555
Shreyanshu Bhushan and Minho Lee. Block diagram-to-text: Understanding block diagram im- ages by generating natural language descriptors. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Findings of the Association for Computational Linguistics: AACL- IJCNLP 2022, pp. 153-168, Association for Computational Linguistics, 2022. URL https: //aclantholoc www..oooo..oooooooooooooooo
hwwp://tthhhooooo..oooooo/e. org/2022. findings aacl. 15
Shreyanshu Bhushan, Eun-Soo Jung, and Minho Lee. Unveiling the power of integration: Block diagram summarization through local-global fusion. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 13837- 13856. Association for Computational Linguistics, 2024. URL https:://claatthlloog
org/2024. f www.:affiiiingggge m 822,
httraa/ww...mm.or//ccceeeeee/// h.t.://ennaeee.............e... h.m.

---

Published as a conference paper at ICLR 2025
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv [cs. 2024. URL
org/
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv [cs.LG], 2020. URL httt:://axxxx.
org/aba 05709
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. arXiv
2023. URL http::t//xx...orggg
0as/22305.0500
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. Image-to-markup generation with coarse-to-fin to-fiee attention. arXiv [cs. CV]. 2016. URL http://aaxxi..orr//bb////
04938
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B Tenenbaum. Leamning to infer graphics programs from hand-drawn images. arXiv [cs.Al], 2017. URL htt.:tttt///aii.oottt
abs abs/1707.0962
Philippe Gervais, Asya Fadeeva, and Andrii Maksai. MathWriting: A dataset for handwritten mathematical expression recognition. arXiv [cs.CV], 2024. URL httpp::///rxii..oo///aa...
Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Shaoling Dong, Xing Zhou, and Wenbin Jiang. VISION2UI: A real-world dataset with layout for code generation from UI designs. arXiv [Vs.CV], 2024.
org abs/ / 2004/404..000 06369
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, Y K Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek-coder: When the large language model meets programming the rise of code intelligence. arXiv [cs.SE], 2024.
URL
org abs///b///4/0 14196
Ting- Yao Hsu, Lee Giles, and Ting-Hao Huang- SciCap: Generating captions for scientific fig- ures. In Mrani-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-Tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 202/, pp. 3258 3264. Asso- ciation for Computational Linguisties, 2021. URL Findings emnlp.277
0r0/2211,
Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. mPLUG-PaperOWws Scientific diagram analysis with the multi- modal large language model. arXiv Mcs. 2023.
18248
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, arXiv [cs. CLJ, 2021.
URL htlthttpp/aaxii/ org/ abs//106............bb.mmbm 09685
Chieh-Yare Huang, Ting-Yat Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin Chan, Eunyee Koh, Clyde Lee Giles, and Ting-Hao 'kenneth Huang. Summaries as captions: Generating figure captions for scientific documents with automated text summarization. arXiv [cs.CL), 2023.
http::/arxiv.orr/aaa//
12324
Zeba Karishma, Shaurya Rohatgi, Kavya Shrinivas Puranik, Jian Wu, and C Lee Giles. ACL-fig: A dataset for scientific figure classification. arXiv [cs.AI], 2023. URL htth//ttp///ar/// org/
abs/ 2301 12293
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision ECCV 2016, pp. 235-251. Springer International Publishing, 2016. URL http:// /it./ springer. com 10. 1007/
178-3-10 978-3-319-46493-0
12

---

Published as a conference paper at ICLR 2025
Klaus Krippendorff. Content analysis: An introduction to its methodology. SAGE Publications, 1980. URL htt.t:ttt///oooooo www.goooogll.ooooo wttp:///e..//oooooooo.oo///////.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagen- Hub: Standardizing the evaluation of conditional image generation models. arXiv [cs. CVs, 2023.
URL http::tt::/aart
org/ abs /2100 02233111130000 01596
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into HTML code with the WebSight dataset. arXiv [cs.HC), 2024. URL
org org/abs/24C0 09029
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision Easy visual task transfer. arXiv [cs.CV), 2024.
URL http://arxiv. org/ abs / 2408. 03326
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81. Association for Computational Linguisties, 2004. URL https://
acl anthology. wwwp:o/-gww0000000
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv /cs.CV 2023. URL http: wtrpiv.vviv.///ee//eea org/ 0ttp///31.037444
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng. Kai- Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv [cs. (Vs, 2023. URL /tRL/tttpp///xii..rrr////////
1310.02251
Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima. Esa Rahtu, J Heikkila, and Shin ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattenn Recognit, pp. 14277- 14286, 2023. URI.tttp:/// http:://ppeeaccessssss seaeeassssteeceeeee com Ccontent/CyPR202.. Otani Toward Verifiable and Reproducible Human Evaluation for Text to- Iarioo-mma-ee.-.... Image Generation CVPR 2023 paper.html
Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M Collins, Joshua Tenenbaum, Adrian Weller, Michael Black, and Bernhard Schölkopf. Can large language models understand symbolic graphics programs? arXiv [cs.LG], 2024. URL
abs abs/2408 08313
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 8748-8763, PMLR, 2021. URL hhttps://pptt http:////pocc.....e///////o mlr. press/vi39/ radfordd html
Juan A Rodriguez, Shubham Agarwal, Issam Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, and Marco Pedersoli. StarVector: Generating scalable vector graphics code from images. arXiv /cs. CV), 2023. URL httt:::/aaxiv.orr/aatte
11556.
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2Code: How far are from automating front-end engineering? arXiv [cs.CL), 2024. URL URL http::/aaa/: org/ ht5//bs/24D0 2403. 03163
Ashish Singh, Prateek Agarwal, Zixuan Huang. Arpita Singh, Tong Yu, Sungchul Kim, Vie- tor Bursztyn, Nikos Vlassis, and Ryan A Rossi. FigCaps-HF: A figure-to-caption genera- tive framework and benchmark with human feedback. arXiv /cs. CL], 2023. URL http:
//arxiv.oro
Davit Soselia, Khalid Saifullah, and Tianyi Zhou. Learning Ul-to-code reverse generator using visual critic without rendering. arXiv [cs. CVJ, 2023. wRUhttt://arxxv.orr www.wiiwww.....ooooooooooooo.eoe wttp:/iwworr/bb///30000 ab33335..
1 14637
13

---

Published as a conference paper at ICLR 2025
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision- language model's perception of the world at any resolution. arXiv [cs.CV], 2024. URL
http::/arxiv.cr abss 2409. 12191
Shaowei Wang, Lingling Zhang, Longji Zhu, Tao Qin, Kim-Hui Yap, Xinyu Zhang, and Jun Liu. CoG-DQA: Chain-of-guiddng learning with large language models for diagram question answer-
wwnttottt/////cc ing. https:/ theovf. http:///ooteeet///////////aaaepe
CoG DQA wfw.-hhaa-f---iuiinn Learning with Large Language Models
For aalaaaraa_uuustsii
2024 2224__apeer___________________22 wwwppppppppp... pddf
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUU-owl: Modularization empowers large language models with multimodality. arXiv [cs.CL), 2023.
2034/ 1 14178
Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. DiagrammerGPT: Generating open-domain, open-platform diagrams via LLM planning, arXiv [cs, CVJ, 2023. URL http://arxiv..rgg
abs/2310. 12128
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv [cx.CV], 2023. URL
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv [cs. CVs, 2023. URL.
http://arxiv. http://arxis org.
10592
Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. VGBench: Evaluating large language models vector graphies understanding and generation. arXiv [cs.CV], 2024. URL http:
A DETAILS OF INFERENCE
Inference Procedure We used pdflatex from TeX Live 202: into a diagram image. We first cropped the rendered image using pdferop and then converted it to PNG file to calculate image similarity.
[1
compile generated TikZ code
Hyperparameters for closed models We used the API's default parameters for the closed models GPT-4o, GPT-4o mini, and Claude. The mar loken parameter was set 2,048 for all models.
Hyperparameters for LLaVA1.6 and IMGTIKZ We set the maximum number of newly gener- ated tokens to 2,048 and generated the code through sampling. The sampling temperature was set to 0.6, value determined through evaluation using the validation set.
B HYPERPARAMETERS FOR TRAINING IMGTIKZ
We conducted the training using the official code LLaVA Table details the hyperparameters used for stage 2 training of IMGTIKZ. For stage 2 training, we used total batch size of 128. The stage training employed similar hyperparameters,, with few exceptions: we set the batch size to 32 with gradient accumulation over 4 steps, resulting in total batch size of 128, and we increased the menghh to 2048. These parameters were derived from the original implementation of VAI.5. The training process consisted of 6000 steps for stage and full epoch for stage 2. We conducted the training using A100 GPUs. The total training time was approximately 24 hours for stage and 60 hours for stage 2.
https: http://////tooggg/ee////// hhtpg://ggthhh.cmmmaaaannnlllll
14

---

Published as a conference paper at ICLR 2025
Table 8:
Configuration for the IMGTIKZ model training.
Option
Value
name (LLM)
d 7b- 100-instruce
model_nmme (Vision encoder) google/siglip-ooee so00 1ap0000-ppppeeet 30400m-patchhl 4-38 4
lora_r
128
lora-alpha
256
mm projector ir
2e-5
mm projector type
mip2x gelu
sroup_by_mddalle _length bf16
True
True
com_traii.epocs.
batch_size
16
gradient
8
weight decay
0
warmup ratio
0.03
tr_scheduler_tppe
cosine
model_max__eentth
4096
gradicnt.checkkoinning
True
C D-SIGLIP: AN SIGLIP MODEL. ADAPTED FOR DIAGRAM
We trained D-SigLIP using contrastive learning framework based on Hugging Face's code We used the google/ / During training, we applied augmentation twice each image, aiming to maximize the similarity between augmented versions of the same image within the batch. Image augmentation was per- formed on- on-the-fly using imgaug The noise pipeline applied through imgaug is detailed below.
4-334 version of SigLIP as the vision encoder.
Listing 1: Image Augmentation Pipeline for D-SigLIP Training
pipeline = iaa. Seq.eniint
Sla.Afffiee (scale= (0.7, 1.0), "y": (0.7, 1.0),, cval=255), iaa. (rotate=(-5, 5), eval=255),
iaa Afa.Afiin afffinettraassltteepercttttttee_
(-0.1, 0.1), "y": (-0.1, 0.1)
cval=255),
iaa Saa.timmee (0. Sae.CCaannccolllrTmppeertttrrrre (11000, 3000)
iaa Ses.ttmmmmmmsss (0. faa.Additiiveaaussiiiin se (scale-(10, 20))
iaa Scmetimmssss... (0.3, ja..hillidpyyymmdddoooorrgghhee (mu1=(0.8, 1.2), add
--5, )
iaa. Sca.Stmmimmss 30.3, iaa. CataGaamaCCnnttastt 1.2))),
iaa. www.etmmmmmss.....
iaa. aaaaBeeeeddApphaaammmeeeliii....
Moa.Multiplyy(1155 2.5), per_chammel-True), upscale_method=/ cubic', iterations= 2)
))
iaa www.Seemeemmme..... (0.1, iaa LinearCentrastt 80.8, 1.2))), iaa. in (alpha-= (15.0, 40.0), sigma- (5.0, 10.0)),
1)
The training conducted using four H100 80G GPUs. We set the batch size to 1024, the learning rate to 5e-5, and the warmup steps to O, with training carried out for 200 steps.
D DATASET COLLECTION PROCESS
First, we compiled the TikZ code from DaTiAZ (Belouadi et al. 2023) to render the diagram images. Then, we developed diagram classification model (See Section using the ACL-fig (Karishmal
pytorch/cont i texe
"https://imgaug readthedocs.... en/latest/
15

---

Published as a conference paper at ICLR 2025
et al. 2023) data, which was subsequently employed to classify the rendered diagrams from the DalikZ dataset. We then extracted diagrams with the predicted labels Tree, Graph, Architecture diagram, Neural networks, and Venn diagram and sampled 4,000 instances from them.
BAOBAB Inc. coordinated multiple annotators to create the corresponding sketches for sampled instances. We excluded diagrams that were complex to be sketched, diagrams of bar charts and line graphs that require numerical data, overly simplistic diagrams comprising only straight lines or dots, diagrams with illegible text, diagrams containing non-English text, and incomplete diagrams that were unnaturally truncated from the tasks during this process. The annotators selected one of the following tools create the sketches: paper, whiteboard, or tablet. When using paper or whiteboard, they captured photos of the hand-drawn images with smartphone camera. They used the drawing tool' save function for tablets to save images. All images were then converted to PNG format. As result of these processes, we ultimately created 3,231 instances.
E DIAGRAM IMAGE CLASSIFICATION MODEL FOR DATA CONSTRUCTION
We developed a model to classify diagram images into categories by fine-tuning a pre-trained vision transformer on the ACL-fig dataset For the pre-trained VIT, we used Google's wawparreepacch--22220
The training was conducted using Hugging Face's tools The parameters used for the training are listed Table 9 We trained the model using NVIDIA A100 GPU. The model achieved a classification accuracy of 0.886 on the evaluation dataset.
Table 9: Configuration for the image classification model.
Option
Value
model name
learning rate
2e-5
gum_train_epocht
5
batch_size
8
warmup ratio
0
Beight-decay 0
presents the breakdown estimated image labels within the sampled data. Furthermore, Figure illustrates example diagrams for each estimated label category. While these are estimated labels and may potentially include diagrams that do not strictly conform to any specific category or contain estimation errors, we confirmed that there are diverse types of diagrams in our dataset.
Table 10
F SKETCH IMAGE EXAMPLES
Figure 9 shows subset of the collected sketch images.
G DETAILS OF THE DATA AUGMENTATION
G.1 AUGTTKZ: THE AUGMENTATION FOR INCREASING DIAGRAM VARIATION
From the arXiv source files 20 we initially obtained 123.123 TikZ code samples. However, only 155K of these were successfully compiled. We utilized these compilable codes as RenderTikZ. While the remaining codes failed to compile, we recognized their potential to significantly increase diagram variations effectively utilized. To achieve this, we employed two types of augmentation prompts. The first prompt focused on code revision and was applied to the initially failed compila- tions. The second prompt, aimed at code modification, was applied to the entire dataset. The specific instructions provided were as follows. We used the -pt-3. aatuo-uu112 version of GPT-3.5 to create the augmentation data.
16

---

Published as a conference paper at ICLR 2025
Table 10: Proportion of estimated image labels in the sampled data.
Category
Number Proportion
Tree
1,799
45.0 %
Graph
1,046
26.2 %
Architecture diagram
646
16.2 %
Neural networks
459
11.5 %
Venn diagram
50
1.1 %
All
4,000
100%
MEAN
Lincarizability
RECORD
Applonaion lehhe
RSC
OSC(U)
VV Regularity
Apprt de pess
Sequential Consistency
MWR-R
MWR-NI
MWR-WO
Inpral de raiid
MWR-Weak
-eme Causal
a) Architecture diagram
b) Tree
U1
U2
U3
U4
h2
h3
L4
c) Graph
d) Neural networks
Venn diagram
Figure 8: Examples of estimated image labels and their diagrams.
Prompts for data augmentation
Please modify the given LaTeX source code to make it compilable, including only the required preamble statements. If any external files are referenced, please modify the code to avoid referencing external files and include the content directly. The output should consist solely of the code itself, without any supplementary text.
Please generate TikZ source code that modifies parts of the following code to create different diagram. Ensure the code is compilable and includes only the required preamble statements. If any external files are referenced, please modify the code to avoid referencing external files and include the content directly. The output should consist solely of the code itself, without any supplementary text.
We included only the code that successfully compiled and rendered images correctly in our dataset AugTikZ. Furthermore, excluded images that were rendered at extreme scales (either too large or too small) from the training dataset.
G.2 IMGAUGTIAZ: THE AUGMENTATION FOR INCREASING IMAGE VARIATION
To simulate the noise typically present in sketches, we applied several augmentation techniques to both RenderTikZ and TugTikZ. These included compositing with notebook background images, augmentation using imgaug, and white balance augmentation For notebook backgrounds, we created eight unique images independently of the sketch annotation process. The imgaug library
17

---

Published as a conference paper at ICLR 2025
choovalie
Computer Road
Reauronn ond
TP
DP
le
Sperlesbl
John
Husr Disomporiy
Cafe Uster la
DP
(cobe)
VP
(so,Y)
DP
likes
<liees
Fus
May
P
(xX)) Prac
Applicat rom starts
ProcedureAt)
Ploceduma
Phf
[a]
haad
Phat
f1p)
Compute
P/33
PB
(1-)
Frishok?
Yes
10
add
mapping
valoe
8
Point
Fuke
1/14
Eva
Figure 9; Examples of collected sketch images.
18

---

Published as a conference paper at ICLR 2025
was used to generate variations in rotation, distortion, Gaussian noise, brightness, and contrast. The specific augmentation pipeline created with imgaug is detailed below.
Listing 2: Image Augmentation Pipeline for Image Augmentation
pipeline iaa. Sequentint (1
iaa Pad (percent=0.33 3, pad__cde= pod__mdd=="neiian")
iaa. Scmetimmms (0.3, iaa isa..AdiitiveeGaussiinnlee (scale= (10, 20))
iaa. Semasemmmmmmsss (0.3, saa.hhullbppyaccccdooooggggttsss (mul= 1.2), add
=(-5, )
iaa. Sla.Scmmmmmss (0.3, iaa. CanmaCont ( (0.8, (.2)).2))) 1.2))),
iaa wer.SSmmtiime 30
iaa pww.heeendmlp
Maa.Multiply (1.5, 2.5), per_channel-Trus upscale mww.od= cubic', iterations=11, 2)
)),
iaa. Sometimes Laa..nnearCnntrast ((0.8, 1.2))),
iaa Affinn (rotate- (-5, 5)),
(alpha (15ppha--15.00 30.0), sigma= 10.0)),
iaa.CropTod i ze (width-int (witth+0.8)) 8), height-int (height.0.8)
1)
H SUBJECTIVE EVALUATION
For each test sample, annotators evaluated the alignment and quality of the six systems' outputs, GPT-4o, GPT-4o mini, Claude 3.5 Sonnet, L.AVA-Next, IMGTIKZ-IG, IMGTIKZ-MGG. We com- pensated annotators at rate of $1.5 per test sample.
We provided annotators with the following instructions for conducting their evaluations:
Instructions
For each image A-F, please assign a score from 1 to 5 based on the following two aspects. You may also use 0.5 increments, such 1.5 or 3.5.
Alignment: The extent to which the generated diagram image matches the layout and content of the hand-drawn image.
Quality: The overall completeness of the generated diagram image, regardless of the presence absence of the hand-drawn and reference image.
The specific evaluation criteria for alignment that we instructed the annotators to follow are as fol- lows:
Evaluation Criteria for Alignment
1: The elements of the diagram in the generated image and the hand-drawn image do not match at all.
2: The elements of the diagram in the generated image and the hand-drawn image match approximately 25%.
3: The elements of the diagram in the generated image and the hand-drawn image match approximately 50%.
4: The elements of the diagram in the generated image and the hand-drawn image match approximately 75%.
5: The elements of the diagram in the generated image and the hand-drawn image match almost perfectly.
The specific evaluation criteria for quality that instructed the annotators follow follows:
19

---

Published as a conference paper at ICLR 2025
Evaluation Criteria for Quality
1: Almost complete overlap of text or shapes, making the diagram unreadable.
2: Significant overlap of text shapes, and the arrangement of elements unnatural.
3: Significant overlap of text shapes, making some elements unreadable, some elements are arranged unnaturally.
4: Some overlap of text or shapes, but the arrangement of elements neat.
5: No overlap of text or shapes, and the arrangement of elements is as neat human- created diagram.
Figure 10 presents partial screenshot of the annotation system interface. The complete template file for the annotation system, which includes all instructions, can be accessed this link https:
202409 example. 1hmmllelemmm
Question:
Hand- drawn Imag
Seftrance image
Baseline iage (Allgnment scoreel)
610 5j0
6jr/+ejo--4
2.0 F-5
Aligemant for
Alignment for
Allgmment for
Allgnment for
Allgement for
Allgnment for
Quality for
Quality for
Quality for
Quality for
Quality for Quality for
Screenshot of the annotation interface: In the HTML, each image can be clicked to enlarge, allowing annotators to view the details of the diagrams.
Figure 10:
I GENERATED DIAGRAM EXAMPLES WITH EVALUATION SCORES
Tables II and 2 show some examples of generated diagrams. IMGTIKZ-MCG generally selects
better candidates compared to IMGT1KZ-IG.
J DETAILED EXPLANATION OF COMPILATION SUCCESS RATE (CSR)
To better illustrate the difference between CSR and CSReum we provide examples below.
CSRing represents the success rate across all generation attempts. For example, if a model attempts N generations for each of the 100 test samples and succeeds in compilation K times, then
N Nenceess
K
CSRavy
(I)
N
(100 x N)
gen
20

---

Published as a conference paper at ICLR 2025
Table 11:
Examples generated diagrams and their metric scores & indicates a compile error and, therefore, has no score.
(a)
sketch diagram
reference diagram
left
up. Rg
right
left
MP. Ps 2
R
P9
right
left
right
IMGTIKZ- IG
IMGTIKZ- MCG
Models
GPT-4o
GPT-4o mini
Claude 3.5
LLaVA
Diagram
B1
Alignment
3.67
2.83
3.83
1.67
4.00
4.67
Quality
3.67
2.67
4.17
2.17
3.67
5.00
ImageSim
0.86
0.69
0.82
0.38
0.81
0.91
(b)
sketch diagram
reference diagram
#2
As
Am
IMGTIKZ- IG
IMGTIKZ- MCG
Models
GPT-4o
GPT-4o mini
Claude 3.5
LLaVA
Diagram
SO
Alignment 3.83
N/A
4.50
1.00
4.17
4.67
Quality
4.00
N/A
4.83
1.00
4.83
4.83
ImageSim 0.79
N/A
0.92
0.05
0.87
0.92
To illustrate, if we make 10 generation attempts for each of the 100 test samples (totaling 1,000
generations) and achieve successful compilation in 400 cases, then
400
CSRagg
0.4.
(2)
1000
CSRuum: which is exclusively used for iterative generation, measures the cumulative proportion of test samples achieving successful compilation across multiple attempts. Consider the following sequential process for 100 test samples:
First generation: 50 of the 100 samples compile successfully
Second generation: 20 of the remaining 50 (100 -5) samples compile successfully
Third generation: 10 of the remaining 30 (50 20) samples compile successfully
21

---

Published as a conference paper at ICLR 2025
Table 12:
Examples generated diagrams and their metric scores V indicates a compile error and, therefore, has no score.
(c)
sketch diagram
reference diagram
hal
IMGTIKZ- IG
IMGTIAZ-
Models
GPT-4o
GPT-4o mini
Claude 3.5
LLaVA
MCG
Hel
Diagram
Alignment
3,83
3.17
4.67
2.33
3.00
3.83
Quality
4.17
3.50
4.83
3.83
3.33
4.17
ImageSim
0.70
0.76
0.88
0.27
0.77
0.81
(d)
sketch diagram
reference diagram
IMGTIKZ- IG
IMGTIKZ-
Models
GPT-4o
GPT-4o mini
Claude 3.5
LLaVA
MCG
Diagram
X
X
X
Alignment
N/A
N/A
N/A
N/A
2.83
4.33
Quality
N/A
N/A
N/A
N/A
4.33
3.50
ImageSim
N/A
N/A
N/A
N/A
0.75
0.86
In this scenario,
N Nest
50
20 + 10
CSR
0.8.
(3)
Nist
100
This metric specifically quantifies the proportion of test samples that eventually achieve successful compilation, independent of the total generation attempts.
The motivation for utilizing these two distinct evaluation metrics arises from their complementary analytical perspectives: CSRwg represents the average compilation success rate, enabling fair model comparison. CSR measures the proportion of successfully compiled test samples across multiple attempts, analogous recall metric.
22

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of IG and Step-by-step + IG approaches.
Model
IG
Two-stage +
GPT-4o 0.695
0.730
Claude 3.5 Sonnet 0.753
0.733
K EVALUATION OF THE EFFECTIVENESS OF A STEP-BY-STEP APPROACH: TEXT GENERATION FOLLOWED BY CODE GENERATION
Although this paper does not focus on this approach, we also investigate a step-by-step method for solving this task, where a textual description of the sketch image first generated, followed by code generation. To assess the effectiveness of this approach, we conducted experiments using Claude 3.5 Sonnet and GPT-4o. The results are presented Table 13 The evaluation was performed using ImageSim for automated evaluation. The results indicate that while Claude 3.5 did not improve performance, GPT-4o slightly increased performance. This suggests that text-mediated generation beneficial for models with relatively lower code generation capabilities but has limited impact models with stronger code generation abilities. One possible reason is that errors may occur during text generation, meaning that textual descriptions do not always positively contribute to subsequent code generation. Investigating more effective step-by-step methods for models with sufficiently high code generation capabilities remains important direction for future research.
23


Reviews:
[{"review_id": "acCSam0iCa", "reviewer": "Reviewer_jbiu", "summary": "The paper presents a system for generating high-quality vector diagrams from hand-drawn sketches based on the new SKETIkZ dataset. SKETIkZ has pairs of hand-drawn sketches and TikZ codes. The system uses data augmentation and a multi-candidate inference strategy to significantly improve output quality.", "questions": "Would it be better to add text descriptions as input along with the hand-drawn diagram for image input?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a novel important dataset. SKETIkZ fills a critical gap in publicly available data for sketch-to-diagram conversion, supporting future research.\n- IMGTIkZ shows comparable performance to larger models like GPT-4o, highlighting efficient architecture and training strategies.\n- Data augmentation and multi-candidate inference enhances performance.\n- This paper contributes SKETIkZ for evaluating vision-language models' capabilities in diagram generation from sketches.\n- Human evaluation is adopted to evaluate the generation results.", "weaknesses": "- No attempt was made to train with other open-source multimodal large models.\n- No performance comparison was made with other open-source multimodal large models.\n- Other input should be considered, such as a hand-drawn diagram with corresponding descriptions, where descriptions can be generated using a captioning model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCt9BW00Vj", "reviewer": "Reviewer_CrUP", "summary": "This paper generates TikZ code and render them as images from input hand-drawn sketch and a textual prompt (predefined instruction). The proposed VLM, ImgTikZ, combines three components: an open-source LLM from DeepSeek coder, a vision encoder from SigLIP, a trainable adapter network (i.e., a linear layer) added to a pre-trained SigLIP vision encoder which is trained using contrastive learning, and a LoRA appended to the language model. The resulting method is trained in two-stages: in stage-1, the adapter network weights are updated, whereas, in stage-2, both the adapter and LoRA are updated. Along with the ImgTikZ method, the authors proposed SkeTikZ, a new dataset comprising of 3,231 pairs of hand-drawn sketches and corresponding TikZ codes. The authors further augment this dataset such as synthesising notebook backgrounds, adding Gaussian noise, varying brightness and contrast, and introducing distortion. The proposed method and the impact of the dataset is measured using four automatic evaluation: compilation success rate, image similarity, code similarity, and character similarity.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "[+] I really want to appreciate the authors for highlighting concurrent works (Belouadi et al, 2024). This is something that should be celebrated more broadly, as it really helps readers understand the overall literature.\n\n[+] The proposed method is simple and intuitive, without unnecessary forced contributions.\n\n[+] The SkeTikZ dataset will hugely help the community in <query>-to-technical diagram generation.", "weaknesses": "[-] While the proposed method is simple and clear, the paper, however, is difficult to follow. For example, when describing \"model structure\" in Sec.4.1 or in \"Datasets used in stage 1 training\" the authors could describe the simple adapter network. I could not find the architecture of this linear layer until I went to Page-15 in supplementary. The same is for LoRA, where the authors have to wait till Page-15 in Tab.7 to know lora_r and lora_alpha. If space is the limitation, the authors could add the following in Sec.4.1 -- \"for more details on <xxx> architecture/designs, please refer to Tab.7\".\n\n[-] The dataset creation process has a caveat: images are rendered using pdflatex, after which a human annotator draws a sketch based on the rendered image. However, human sketching is inherently a lossy process, meaning that certain details may be ommitted or subtly altered. Consequently, when a VLM uses this dataset to perform sketch-to-image generation, there is a high risk of hallucination where the model might introduce small details into the generated image that were not present in the original sketch. Hallucinations may not always be a bad thing, but for generating code or technical diagrams, this can be detrimental.\n\n[-] How is the proposed method different from Gervais et al., 2024 and others, which generate LaTeX code from screenshots of mathematical formulas or handwritten images? Can they be applied to this same problem with minimal modifications?\n\nMinor: \n\n[-] Typo I presume? re: CSR_ave in Tab.2 whereas CSR_avg in Tab.3\n\n[-] All the tables look too big and consumes a lot of unnecessary space. I would suggest the authors to make the table font smaller and use the space to add details on adapter network, LoRA, and some training details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7t4u5zSJfR", "reviewer": "Reviewer_eF9v", "summary": "This work investigates the problem of sketch-to-diagram generation, which converts hand-drawn sketches into diagrams formatted as TikZ codes. As drawing sketches is a visual tool that is more intuitive and user-friendly for ideation, the authors think tackling the problem of sketch-to-diagram generation is meaningful but underexplored so far, compared to its text-to-diagram generation counterpart. The authors proposed using a vision language model to handle the interesting problem. A new dataset of paired sketch images, TikZ codes and reference images is also proposed.", "questions": "- Could the authors provide a detailed analysis of why the proposed method requires generating multiple candidates for the TikZ code and selecting the best one? Does this stem from the limitation of the vision encoder or the LLM-based code generator?\n- Follow-up question: It seems the output of the CodeLLM is somehow random, i.e., sometimes it works and sometimes not, so it requires generating until a satisfactory result is given (both the iterative and the multi-candidate generation falls in this case). Is there any strategy to improve the consistency and accuracy of the code generation process?\n- It is confusing to know the difference between CSR_avg and CSR_cum. How could N_gen be different from N_test? The authors might want to provide a concrete example that illustrates how these two metrics are calculated and why they might differ. This could help readers better understand and interpret the results.\n- Regarding all the competitors, did all of them fine-tune using the SkeTikZ dataset as well?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed dataset contains 3,231 sketch images associated with the TikZ codes, and the rendered images are valuable for sketch-to-diagram generation.\n- It is interesting and reasonable that a vision language model is utilised to tackle the problem of sketch-to-diagram generation. This work expands the usage of VLM into a new domain.\n- Extensive experiments are conducted, which is helpful in understanding the effectiveness and the limitations of the proposed method and other VLM models in the context of image-to-TikZ generation.", "weaknesses": "- The authors claim that the task of sketch-to-diagram has not been explored before. However, there are some existing works [a][b][c][d]. It would be nice if the authors discuss how this work differs from or improves upon these existing works, particularly in the context of TikZ code generation from sketches. This would help clarify the novelty and contributions of this work.\n- The authors could further improve the dataset section. For example, the authors could demonstrate all the types of diagrams this work focuses on and provide statistics about the data (e.g., how many for each category, etc). \n- The contribution of using data augmentation and multi-candidate inference tricks is minor.\n\n[a] Vitruvion: A Generative Model of Parametric CAD Sketches, ICLR 2022    \n[b] Learning to infer graphics programs from hand-drawn images, NeurIPS 2018   \n[c] Neurosymbolic Models for Computer Graphics, Computer Graphics Forum 2023   \n[d] SketchGen: Generating Constrained CAD Sketches, NeurIPS 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYla9kT5Pq", "reviewer": "Reviewer_SFio", "summary": "This article provides a solution for the task of generating high-quality vector graphics from hand-drawn sketches, a convenient method for conveying complex concepts across various fields. However, the problem of converting hand-drawn sketches to vector graphics remains inadequately addressed due to the lack of datasets. To address this issue, the article constructs a dataset called SKETIkZ, which contains 3,231 pairs of hand-drawn sketches, reference images, and corresponding TikZ code. The authors also commit to making the SKETIkZ dataset fully publicly available. Leveraging the SKETIkZ dataset, the article proposes a modest-sized method called IMGTIkZ that can compete with the performance of GPT-4o.", "questions": "1. I find the SKETIkZ dataset contributed by this article to be quite beneficial, which is why I am more concerned about this dataset. Although I have noticed that the article discusses how the use of the SKETIkZ dataset alone can enhance model performance, I am somewhat concerned about whether the SKETIkZ dataset will universally bring performance improvements in future research work.\n\n2. Regarding the vector graphics generated from hand-drawn sketches, if there are minor errors in the output, I wonder if there are any simple methods available for correction.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This article is written in a clear and concise manner, introducing the proposed method very clearly and discussing in detail some issues of concern to readers in the results section.\n2. The article has open-sourced a dataset for converting hand-drawn sketches to vector graphics, contributing usable foundational data for subsequent research on the same task.", "weaknesses": "1. The method proposed in this article for converting hand-drawn sketches to vector graphics relies more on the combination of existing technical solutions and does not introduce particularly innovative technical approaches.\n2. This article carries out extensive data augmentation and supplementation when using the SKETIkZ dataset, which raises curiosity about whether the SKETIkZ dataset itself could play a significant role in future research endeavors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)