Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches

Authors: Itsumi Saito, Haruto Yoshida, Keisuke Sakaguchi
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: multimodal, large language model, diagram, vector graphics

Abstract:
We address the challenge of automatically generating high-quality vector diagrams from hand-drawn sketches. 
Vector diagrams are essential for communicating complex ideas across various fields, offering flexibility and scalability. 
While recent research has progressed in generating diagrams from text descriptions, converting hand-drawn sketches into vector diagrams remains largely unexplored due to the lack of suitable datasets. 
To address this gap, we introduce SketikZ, a dataset comprising 3,231 pairs of hand-drawn sketches and thier corresponding TikZ codes as well as reference diagrams.
Our evaluations reveal the limitations of state-of-the-art vision and language models (VLMs), positioning SketikZ as a key benchmark for future research in sketch-to-diagram conversion.
Along with SketikZ, we present ImgTikZ, an image-to-TikZ model that integrates a 6.7B parameter code-specialized open-source large language model (LLM) with a pre-trained vision encoder. 
Despite its relatively compact size, ImgTikZ performs comparably to GPT-4o.
This success is driven by using our two data augmentation techniques and a multi-candidate inference strategy.
Our findings open promising directions for future research in sketch-to-diagram conversion and broader image-to-code generation tasks. SketikZ is publicly available.

Decision: accept (poster)
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730380155264

thread_type:
official_review

<thread_object>
creator: reviewer_sfio

<content>
summary:
This article provides a solution for the task of generating high-quality vector graphics from hand-drawn sketches, a convenient method for conveying complex concepts across various fields. However, the problem of converting hand-drawn sketches to vector graphics remains inadequately addressed due to the lack of datasets. To address this issue, the article constructs a dataset called SKETIkZ, which contains 3,231 pairs of hand-drawn sketches, reference images, and corresponding TikZ code. The authors also commit to making the SKETIkZ dataset fully publicly available. Leveraging the SKETIkZ dataset, the article proposes a modest-sized method called IMGTIkZ that can compete with the performance of GPT-4o.

strengths:
1. This article is written in a clear and concise manner, introducing the proposed method very clearly and discussing in detail some issues of concern to readers in the results section.
2. The article has open-sourced a dataset for converting hand-drawn sketches to vector graphics, contributing usable foundational data for subsequent research on the same task.

weaknesses:
1. The method proposed in this article for converting hand-drawn sketches to vector graphics relies more on the combination of existing technical solutions and does not introduce particularly innovative technical approaches.
2. This article carries out extensive data augmentation and supplementation when using the SKETIkZ dataset, which raises curiosity about whether the SKETIkZ dataset itself could play a significant role in future research endeavors.

questions:
1. I find the SKETIkZ dataset contributed by this article to be quite beneficial, which is why I am more concerned about this dataset. Although I have noticed that the article discusses how the use of the SKETIkZ dataset alone can enhance model performance, I am somewhat concerned about whether the SKETIkZ dataset will universally bring performance improvements in future research work.

2. Regarding the vector graphics generated from hand-drawn sketches, if there are minor errors in the output, I wonder if there are any simple methods available for correction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614983449

thread_type:
official_review

<thread_object>
creator: reviewer_ef9v

<content>
summary:
This work investigates the problem of sketch-to-diagram generation, which converts hand-drawn sketches into diagrams formatted as TikZ codes. As drawing sketches is a visual tool that is more intuitive and user-friendly for ideation, the authors think tackling the problem of sketch-to-diagram generation is meaningful but underexplored so far, compared to its text-to-diagram generation counterpart. The authors proposed using a vision language model to handle the interesting problem. A new dataset of paired sketch images, TikZ codes and reference images is also proposed.

strengths:
- The proposed dataset contains 3,231 sketch images associated with the TikZ codes, and the rendered images are valuable for sketch-to-diagram generation.
- It is interesting and reasonable that a vision language model is utilised to tackle the problem of sketch-to-diagram generation. This work expands the usage of VLM into a new domain.
- Extensive experiments are conducted, which is helpful in understanding the effectiveness and the limitations of the proposed method and other VLM models in the context of image-to-TikZ generation.

weaknesses:
- The authors claim that the task of sketch-to-diagram has not been explored before. However, there are some existing works [a][b][c][d]. It would be nice if the authors discuss how this work differs from or improves upon these existing works, particularly in the context of TikZ code generation from sketches. This would help clarify the novelty and contributions of this work.
- The authors could further improve the dataset section. For example, the authors could demonstrate all the types of diagrams this work focuses on and provide statistics about the data (e.g., how many for each category, etc). 
- The contribution of using data augmentation and multi-candidate inference tricks is minor.

[a] Vitruvion: A Generative Model of Parametric CAD Sketches, ICLR 2022    
[b] Learning to infer graphics programs from hand-drawn images, NeurIPS 2018   
[c] Neurosymbolic Models for Computer Graphics, Computer Graphics Forum 2023   
[d] SketchGen: Generating Constrained CAD Sketches, NeurIPS 2021

questions:
- Could the authors provide a detailed analysis of why the proposed method requires generating multiple candidates for the TikZ code and selecting the best one? Does this stem from the limitation of the vision encoder or the LLM-based code generator?
- Follow-up question: It seems the output of the CodeLLM is somehow random, i.e., sometimes it works and sometimes not, so it requires generating until a satisfactory result is given (both the iterative and the multi-candidate generation falls in this case). Is there any strategy to improve the consistency and accuracy of the code generation process?
- It is confusing to know the difference between CSR_avg and CSR_cum. How could N_gen be different from N_test? The authors might want to provide a concrete example that illustrates how these two metrics are calculated and why they might differ. This could help readers better understand and interpret the results.
- Regarding all the competitors, did all of them fine-tune using the SkeTikZ dataset as well?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625690423

thread_type:
official_review

<thread_object>
creator: reviewer_crup

<content>
summary:
This paper generates TikZ code and render them as images from input hand-drawn sketch and a textual prompt (predefined instruction). The proposed VLM, ImgTikZ, combines three components: an open-source LLM from DeepSeek coder, a vision encoder from SigLIP, a trainable adapter network (i.e., a linear layer) added to a pre-trained SigLIP vision encoder which is trained using contrastive learning, and a LoRA appended to the language model. The resulting method is trained in two-stages: in stage-1, the adapter network weights are updated, whereas, in stage-2, both the adapter and LoRA are updated. Along with the ImgTikZ method, the authors proposed SkeTikZ, a new dataset comprising of 3,231 pairs of hand-drawn sketches and corresponding TikZ codes. The authors further augment this dataset such as synthesising notebook backgrounds, adding Gaussian noise, varying brightness and contrast, and introducing distortion. The proposed method and the impact of the dataset is measured using four automatic evaluation: compilation success rate, image similarity, code similarity, and character similarity.

strengths:
[+] I really want to appreciate the authors for highlighting concurrent works (Belouadi et al, 2024). This is something that should be celebrated more broadly, as it really helps readers understand the overall literature.

[+] The proposed method is simple and intuitive, without unnecessary forced contributions.

[+] The SkeTikZ dataset will hugely help the community in <query>-to-technical diagram generation.

weaknesses:
[-] While the proposed method is simple and clear, the paper, however, is difficult to follow. For example, when describing "model structure" in Sec.4.1 or in "Datasets used in stage 1 training" the authors could describe the simple adapter network. I could not find the architecture of this linear layer until I went to Page-15 in supplementary. The same is for LoRA, where the authors have to wait till Page-15 in Tab.7 to know lora_r and lora_alpha. If space is the limitation, the authors could add the following in Sec.4.1 -- "for more details on <xxx> architecture/designs, please refer to Tab.7".

[-] The dataset creation process has a caveat: images are rendered using pdflatex, after which a human annotator draws a sketch based on the rendered image. However, human sketching is inherently a lossy process, meaning that certain details may be ommitted or subtly altered. Consequently, when a VLM uses this dataset to perform sketch-to-image generation, there is a high risk of hallucination where the model might introduce small details into the generated image that were not present in the original sketch. Hallucinations may not always be a bad thing, but for generating code or technical diagrams, this can be detrimental.

[-] How is the proposed method different from Gervais et al., 2024 and others, which generate LaTeX code from screenshots of mathematical formulas or handwritten images? Can they be applied to this same problem with minimal modifications?

Minor: 

[-] Typo I presume? re: CSR_ave in Tab.2 whereas CSR_avg in Tab.3

[-] All the tables look too big and consumes a lot of unnecessary space. I would suggest the authors to make the table font smaller and use the space to add details on adapter network, LoRA, and some training details.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730867579332

thread_type:
official_review

<thread_object>
creator: reviewer_jbiu

<content>
summary:
The paper presents a system for generating high-quality vector diagrams from hand-drawn sketches based on the new SKETIkZ dataset. SKETIkZ has pairs of hand-drawn sketches and TikZ codes. The system uses data augmentation and a multi-candidate inference strategy to significantly improve output quality.

strengths:
- This paper proposes a novel important dataset. SKETIkZ fills a critical gap in publicly available data for sketch-to-diagram conversion, supporting future research.
- IMGTIkZ shows comparable performance to larger models like GPT-4o, highlighting efficient architecture and training strategies.
- Data augmentation and multi-candidate inference enhances performance.
- This paper contributes SKETIkZ for evaluating vision-language models' capabilities in diagram generation from sketches.
- Human evaluation is adopted to evaluate the generation results.

weaknesses:
- No attempt was made to train with other open-source multimodal large models.
- No performance comparison was made with other open-source multimodal large models.
- Other input should be considered, such as a hand-drawn diagram with corresponding descriptions, where descriptions can be generated using a captioning model.

questions:
Would it be better to add text descriptions as input along with the hand-drawn diagram for image input?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732625607245

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank the reviewers for taking the time to provide constructive and valuable feedback on our manuscript. We also appreciate their positive comments on our dataset, methodology, and experiments. We have uploaded the revised paper incorporating the discussion feedback, with new additions highlighted in blue. We believe these revisions have enhanced both the completeness and clarity of our manuscript. All specific changes made to the manuscript have been detailed in our responses to the reviewers. We would be grateful if we could hear your feedback regarding our responses to the reviews.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733114325142

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Dear Reviewers,

comment:
We would like to express our sincere gratitude to the reviewers for their thoughtful evaluation of our work. We have carefully addressed all review comments and updated our paper accordingly. We believe that the concerns raised have been satisfactorily resolved. As we approach the end of the discussion period, we welcome additional comments and questions if you have any remaining concerns. Thank you once again for your time, effort, and valuable insights.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper initially received 2 borderline reject and two positive reviews. Almost all reviewers felt the proposed SKETIkZ dataset will be useful for the research community.  Also the authors response help the reviewer CrUP, and the score was increased to borderline accept (6).  Reviewer jbiu also satisfied with the responses. Unfortunately other two reviewers did not participate in the discussion and left the final scores unaltered. This paper finally ended with three supporting reviews and a borderline reject. Considering the reviews and responses, the AC panel acknowledged the importance of the introduced dataset for research community and decided to accept the paper. Authors should update the manuscript considering all the reviewers suggestions while preparing the camera ready.