Unveiling the Power of Integration: Block Diagram Summarization through Local-Global Fusion (2024)

Abstract: Block Diagrams play an essential role in visualizing the relationships between components or systems. Generating summaries of block diagrams is important for document understanding or question answering (QA) tasks by providing concise overviews of complex systems. However, it’s a challenging task as it requires compressing complex relationships into informative descriptions. In this paper, we present “BlockNet”, a fusion framework that summarizes block diagrams by integrating local and global information, catering to both English and Korean languages. Additionally, we introduce a new multilingual method to produce block diagram data, resulting in a high-quality dataset called “BD-EnKo”. In BlockNet, we develop “BlockSplit”, an Optical Character Recognition (OCR) based algorithm employing the divide-and-conquer principle for local information extraction. We train an OCR-free transformer architecture for global information extraction using BD-EnKo and public data. To assess the effectiveness of our model, we conduct thorough experiments on different datasets. The assessment shows that BlockNet surpasses all previous methods and models, including GPT-4V, for block diagram summarization.1

heading: 1 Introduction
text: Document Artificial Intelligence (Document AI) is gaining momentum across industries for streamlining document processes, enhancing efficiency, and extracting insights from unstructured data. Downstream tasks span various domains and paradigms, such as document QA (Tanaka et al., 2021), layout detection (Zhong et al., 2019), classification (Harley et al., 2015), and more. Documents contain not only text, images, charts, or tables but also the most important block diagram, which is often
1The dataset, source code, and model checkpoints are publicly available at https://github.com/shreyanshu09/ BlockNet.
overlooked but plays a crucial role in illustrating systems, processes, or relationships.
Block diagrams are visual representations of systems, processes, or relationships using blocks or boxes to represent components or elements, and lines or arrows to show the relationships between them. They are particularly useful in technical documentation, engineering reports, business models, scientific papers, and more. Block diagrams can vary in complexity from simple representations of basic systems to more complex systems with numerous interconnected components. Summaries from block diagrams offer numerous advantages, including reducing manual effort, enhancing productivity, integrating with AI systems for analysis, Document AI, QA, and knowledge representation. They improve accessibility for visually impaired individuals, facilitate efficient information retrieval, support collaboration among team members, aid decision-making, and streamline workflows. These summaries also convert complex information into concise descriptions, aiding comprehension and saving time compared to manual analysis.
Despite its numerous advantages and potential applications, the block diagram summarization problem remains relatively overlooked within the natural language processing (NLP) community. We encountered a scarcity of literature on block diagram summarization. Previous approaches primarily focused on detecting different shapes within the diagrams (Julca-Aguilar and Hirata, 2018) or converting handwritten block diagrams into electronic formats (Schäfer and Stuckenschmidt, 2019; Schäfer et al., 2021; Schäfer and Stuckenschmidt, 2021). However, approaches failed to address the vital task of linking text phrases with shapes and arrows, crucial for effective summarization. To the best of our knowledge, only a few methods have effectively accomplished diagram summarization by combining object detection with pre-trained language models (Bhushan and Lee, 2022). Neverthe-
13837
less, their results are subject to several limitations such as hallucination problem, OCR errors, lack of large dataset, and support limited diagrams. Even the latest powerful vision language model, GPT-4V (Yang et al., 2023), struggles with efficiency, particularly in the task of converting block diagrams to text, often lacking detailed information. Addressing block diagram summarization encounters two main challenges: the absence of a large dataset and the lack of robust baselines.
In this paper, we propose a fusion-based framework “BlockNet”, comprising three main components: a local information extractor, a global information extractor, and an integrator. The local extractor, based on our algorithm “BlockSplit”, follows the divide-and-conquer principle to split block diagrams into subunits, each containing two connected nodes or symbols. These subunits are then processed using OCR to extract triplets (head, relation, tail). The global extractor utilizes an OCRfree visual document understanding (VDU) transformer model, trained on both English and Korean diagram datasets. Lastly, the integrator, a large language model (LLM), interprets visual information and combines outputs from both extractors. We also introduce a new multilingual data collection approach for block diagram understanding and generation, enabling flexibility and scalability. With this approach, we compile a large, high-quality dataset, “BD-EnKo”, containing 83,394 diagrams with their summaries in English and Korean, expandable to any language using our approach.
To summarize, our contributions are below:
• We propose BlockNet, a fusion framework that combines local and global information from block diagram to enhance summary generation.
• We introduce a novel multi-lingual data collection approach specifically designed for block diagram understanding and generation, offering flexibility and scalability.
• Using our innovative data collection methodology, we produce the diverse and highquality benchmark dataset BD-EnKo in English and Korean languages.
• We conduct several automatic and human evaluations to check the performance of the proposed framework.

heading: 2 Related Work
text: Visual Document Understanding Several VDU methods have been developed to interpret semistructured documents and extract vital information, such as receipts and tables (Davis et al., 2019; Huang et al., 2019; Hong et al., 2022). Early approaches aimed to utilize OCR-independent visual backbones (Kang et al., 2014; Harley et al., 2015; Afzal et al., 2015; Guo et al., 2019) but exhibited limited performance. Subsequent advancements in OCR (Baek et al., 2019a,b; Jung et al., 2021) led to the development of various OCR-dependent VDU models by combining with language models (Hwang et al., 2019; Bhushan and Lee, 2022). These methods typically address the task in two stages: 1) extracting text from document images using deep learning-based OCR; 2) comprehending the document as a whole. However, OCRdependent techniques face challenges as OCR errors can negatively impact downstream processes (Hwang et al., 2019, 2021). To address this, Kim et al. (2022) introduced a Transformer-based OCRfree architecture (Devlin et al., 2019; Dosovitskiy et al., 2020; Kim et al., 2021) but still it faces problems particularly with non-English words. Considering these points, we integrate both OCR techniques (local information) and OCR-free models (global information) to address the respective limitations of each.
Image to Data Generation Earlier, Montalvo (1990) explored understanding symbolic descriptions in diagrams and suggested a method that links symbolic descriptions with visual attributes to enhance comprehension and analysis. JulcaAguilar and Hirata (2018) introduced a handwritten block diagram detection system using Faster R-CNN. However, their method’s inability to recognize edges prevents it from identifying the symbols connected by the arrows, thus limiting its effectiveness in block diagram understanding tasks. Arrow R-CNN addresses this by employing an arrow keypoint predictor to learn and predict arrow positions (Schäfer and Stuckenschmidt, 2019). Furthermore, Schäfer and Stuckenschmidt (2021) improves arrow detection by modeling arrows as relationships between shapes rather than as independent objects with bounding boxes. However, these systems only detect symbol categories without capturing key information conveyed by the block diagram. Bhushan and Lee (2022) introduced the BloSum

heading: Instruction

heading: Mermaid Code
text: framework, which employs a language model to generate textual descriptions for identified block diagram images, thereby improving tasks such as document summarization. However, language models face challenges such as the illusion problem, where the generated content may lack meaning or reliability compared to the original content. Moreover, the dataset that they used for training the model was very small. In contrast, our approach is to integrate the local information (summary of each sub-block) from the diagram as well as the global information (overall summary). Moreover, our model is trained on a significantly larger dataset (around 76K) compared to the BloSum framework (around 500).

heading: 3 Method
text: In this section, we divide into four subsections. The first explains our unique data generation method. The second discusses the large-scale dataset “BDEnKo”. The third analyzes the dataset, and fourth presents “BlockNet”, our fusion-based framework.

heading: 3.1 Data Generation
text: In this subsection, we detail our unique approach of data generation. It mainly comprises three parts as shown in Figure 1. These parts are detailed in the following paragraphs.
Mermaid Code Generation The first part is to collect different varieties of mermaid code. Mermaid code is a markup language for creating diagrams, charts, and flowcharts. It is concise and intuitive, allowing for the easy creation of complex
visual representations. It supports a wide range of diagrams, including flowcharts, sequence diagrams, Gantt charts, etc. We instruct GPT-3.5 (Ouyang et al., 2022) to generate mermaid codes encompassing various types and random themes of diagrams. This method guarantees the creation of diverse mermaid data with appropriate structure, essential for image generation. Additionally, by controlling these characteristics, we intentionally reduce bias, leading to a more balanced dataset.
Block Diagram Image Generation The next part is to transform the generated mermaid code into visual block diagrams using the popular diagram generation tool, Mermaid2. We provide further details on enhancing aspects such as color and theme schemes. To broaden the variety, we also applied random noise, blur effects, and grayscale effects to the images. This yielded a diverse collection of block diagram images, each accurately depicting connections and symbols while being visually appealing, thereby demonstrating the effectiveness of our approach.
Block Diagram Text Generation The final part involves generating text descriptions for each block diagram image. Leveraging the structure of mermaid code, which GPT-3.5 turbo understands well, makes it particularly straightforward to generate descriptions based on this code. To accomplish this, we utilize the previously generated mermaid code from mermaid code generator. We then instruct GPT-3.5 turbo once more to generate coherent and meaningful descriptions based on this mermaid code structure. The prompts utilized at each part and the pipeline explanation with examples are detailed in Appendix A.1.
2https://github.com/mermaid-js/mermaid

heading: 3.2 BD-EnKo Dataset
text: Our method for generating block diagram data resulted in a large-scale dataset comprising 83,394 samples. These samples consist of both English (46,599 samples) and Korean (36,795 samples) languages, featuring a diverse range of block diagram images with their high-quality summaries. As depicted in Table 1, our method generated eight distinct types of block diagrams, each possessing its unique style and significance. We supplemented our dataset by collecting a few real block diagram images from publicly available sources on the Google search engine, consisting of 91 English samples and 354 Korean samples. The decision to collect more samples for the Korean language stemmed from the availability of the public dataset “CBD” for English cases, as collected by Bhushan and Lee (2022) from various search engines. For the real-world samples, we engaged Korean native speakers to provide meaningful summaries for the given diagrams in Korean, while English native speakers performed the same task for the English samples. Dataset samples of various categories can be found in the Appendix A.2.

heading: 3.3 Dataset Analysis
text: Figure 2 presents the statistical insights from the training data of the BD-EnKo dataset. The top-left histogram, colored in blue, shows the distribution of the number of nodes, with a high frequency of
occurrences around 0 to 20 nodes, peaking at approximately 25,000 occurrences, and tapering off significantly beyond 50 nodes. The top-right histogram, colored in red, illustrates the distribution of the number of arrows, similarly showing a high frequency around 0 to 10 arrows, peaking at around 29,000 occurrences, and diminishing beyond 30 arrows. The bottom scatter plot, with green dots, depicts the relationship between the number of nodes and the number of arrows, showing a positive correlation. The linear regression line fitted to the scatter plot indicates a strong positive correlation with a correlation coefficient of 0.91 and a p-value less than 0.001, suggesting a statistically significant relationship between the number of nodes and the number of arrows in the dataset. Descriptive statistics of the dataset are in Appendix A.3.

heading: 3.4 BlockNet
text: This subsection details BlockNet’s architecture, shown in Figure 3, with three main blocks: local extractor, global extractor, and integrator. Local information alone misses overall structure, while global information alone overlooks details. BlockNet integrates both for better diagram summarization. Each block of BlockNet is elaborated in the following paragraphs.
Local Information Extractor We construct this module following the divide-and-conquer princi-
ple, emphasizing the division of block diagrams into smaller pairs to improve efficiency and prevent oversight of detailed information. In this block, the very first step is to detect all symbols and arrows within the diagram, which we classify as an object detection task. Unlike previous work (Bhushan and Lee, 2022), we train the object detection model based on YOLOv5 (Jocher et al., 2020) that provides fundamental abilities crucial for detecting different objects in an image. We train this model with CBD, FC_A, and FC_B dataset which provides the annotations of different shapes and arrows in a diagram. We train the model for six labels: arrow, terminator, process, decision, data, and text. Additionally, we utilize the Pororo OCR3 that supports both English and Korean to extract text from the images. Subsequently, we developed an algorithm “BlockSplit” to divide the block diagram image into sub-diagrams, each containing two nodes connected by an arrow, detailed in Algorithm 1.
This algorithm takes three inputs: shapes detected by the object detection module, texts recognized by the OCR, and the head and tail keypoints of arrows. To identify the head and tail keypoints of arrows, each arrow is divided into two parts, and the number of white pixels is counted. A higher count indicates the arrow’s head, while a lower count in-
3https://github.com/yunwoong7/korean_ocr_ using_pororo
dicates its tail. For each head and tail keypoint, the algorithm first identifies the closest shapes, then determines the associated shape’s name. Subsequently, it locates text within the shape; if text is present, it is labeled as either Head or Tail, otherwise, the shape’s name is used. Regarding relations, the algorithm calculates the distance between the arrow and all text bounding boxes written outside shapes. Texts within a threshold distance (set at 5) are labeled as Relations. In the absence of texts within this threshold, the relation is denoted as ‘connected with’. Each sub-diagram is represented as a triplet (<H> Head, <R> Relation, <T> Tail). The output of this block consists of a collection of all triplets found within the diagrams.
Global Information Extractor We train the transformer encoder and decoder model based on the configuration specified in Donut (Kim et al., 2022) to extract the global information of the block diagram. The straightforward architecture comprises a visual encoder module and a text decoder module based on the Transformer architecture (Vaswani et al., 2017). This model does not rely on any OCR-related modules but instead utilizes the visual encoder to extract features from the given image. We utilize Swin Transformer (Liu et al., 2021) as the visual encoder. Initially, the visual encoder divides the input image into non-overlapping patches, applying a window-based
multi-head self-attention module and a two-layer multi-Layer Perceptron (MLP) to these patches. Subsequently, patch merging layers are applied to the patch tokens at each stage. The resulting output is then fed into the text decoder.
For the text decoder architecture, we employ BART (Lewis et al., 2020). The text decoder takes the outputs of the visual encoder and generates a sequence of tokens. Specifically, we initialize the decoder model weights using a publicly available pre-trained multilingual BART model (Parikh et al., 2020). We train the end-to-end model using our BD-EnKo dataset, that contains both English and Korean diagrams, as well as the CBD dataset collected by Bhushan and Lee (2022), FC_A handwritten block diagram dataset collected by Awal et al. (2011) and FC_B handwritten block diagram datasets collected by Bresler et al. (2016). These datasets comprise block diagram images and their corresponding text summaries, which aid in the end-to-end training process. The pre-trained Donut model is capable of reading the texts in the image according to the reading order, typically from topleft to bottom-right. However, applying the model directly to our task is suboptimal due to the complexity of block diagram layouts and relationships.
Integrator After obtaining both the local and global information from the diagram, the final step involves integrating them to mitigate the errors or limitations of each. To address these limitations, we utilize an LLM GPT-4V to seamlessly integrate both local and global information based on the provided block diagram image. This module receives inputs from two sources: the local information extractor, which produces triplets as outputs, and the global information extractor, which provides an overall summary. Additionally, it takes the input block diagram image. The output is a final summary that integrates information from all these sources. The prompt utilized for integration is outlined in the Appendix A.4.

heading: 4 Experiments
text: We first describe the experimental setup in Section 4.1, followed by a discussion of the metrics employed to measure performance in Section 4.2, and at last present the main results in Section 4.3.

heading: 4.1 Experimental Setup
text: Datasets We utilize three datasets to evaluate and compare the performances of various models. Our
Algorithm 1 BlockSplit Algorithm Input: Shape s ∈ S, Texts t ∈ T, Head & Tail
Arrow Keypoints a ∈ A (ahead, atail) Output: <H> head, <R> relation, <T> tail
1: for ai in A: 2: <H>i = argmins(dist(a h i ,s)), s ∈ S 3: <T>i = argmins(dist(a t i ,s)), s ∈ S 4: if any t ∈ T in <H>i then 5: <H>i = t, t ∈ T 6: end if 7: if any t ∈ T in <T>i then 8: <T>i = t, t ∈ T 9: end if
10: if min(dist(ai,t)), t ∈ T < 5 then 11: <R> = argmint(dist(ai,t)), t ∈ T 12: else 13: <R> = ‘connected with’ 14: end if
dataset, BD-EnKo, is partitioned into two subsets: BD-En (4,666 samples) for English diagrams and BD-Ko (3,694 samples) for Korean diagrams. This division is necessary as some models do not support the Korean language, ensuring a fair comparison across different languages. Additionally, we employ CBD (106 samples) (Bhushan and Lee, 2022), an English real-world dataset, and FC_B (196 samples) (Bresler et al., 2016), consisting of English handwritten diagrams.
BlockNet For symbol detection within the local information extractor task, we employ a pretrained YOLOv5 model (Jocher et al., 2020) as the backbone network. Although the diagram images differ greatly from real-world images in the COCO dataset (Chen et al., 2015), we fine-tune the model using the CBD dataset due to its provision of annotations for symbol detection tasks. Within the global information extractor, we train the transformer encoder and decoder model using a combined dataset including BD-EnKo (75,034 samples), CBD (396 samples), FC_A (357 samples), and FC_B (476 samples).
Additional details regarding the experimental setup for BlockNet and various other models used for comparison are provided in the Appendix A.5.

heading: 4.2 Evaluation Metrics
text: Text quality can be assessed through both human evaluation and automatic evaluation metrics. Automatic evaluation metrics fall into two categories:
those not requiring training and those learned via neural networks. Metrics that do not require training, such as BLEU (Post, 2018) and ROUGE (Lin, 2004), are commonly used due to their convenience. The overall BLEU score is calculated by averaging BLEU n-grams (n=1 to 4) while considering the brevity penalty. Additionally, we use the F1 score of ROUGE-1. However, experiments show that these metrics, based only on n-gram overlap and lacking semantic understanding, often inadequately reflect text quality compared to human evaluation. Hence, for a more accurate alignment with human evaluation, we employ learned metrics like BLEURT (Sellam et al., 2020) (specifically BLEURT-large-512). Moreover, we utilize Perplexity (PPL) (Radford et al., 2019) using a pre-trained GPT-2 Large to evaluate the fluency and readability of the generated text descriptions derived from block diagram images.

heading: 4.3 Experimatal Results
text: Automatic Evalution Table 2 presents the results of automatic evaluation conducted on our dataset separately for English and Korean, providing clearer insights. In the English dataset (BDEn), our proposed framework, BlockNet, achieves the highest scores for BLEU (58.61), ROUGE-1
(74.25), and BLEURT (0.38), indicating its effectiveness. However, mPLUG (Ye et al., 2023) obtains the best PPL score (5.25), indicating its ability to generate fluent sentences. Nonetheless, its BLEU (6.2), ROUGE-1 (20.7), and BLEURT (- 0.34) scores are significantly lower compared to BlockNet, suggesting that while mPLUG generates fluent sentences, it may not accurately capture facts. Similar trends are observed in the Korean dataset (BD-Ko), where BlockNet achieves the highest scores for BLEU (62.41), ROUGE-1 (78.61), and BLEURT (0.27). Notably, there is a substantial performance gap between BlockNet and GPT-4V in the Korean dataset, underscoring the superiority of our model over GPT-4V. Figure 4 illustrates the comparison of outputs between BlockNet and GPT-4V. It’s evident that BlockNet better captures the structure and flow of the diagram compared to
GPT-4V. While GPT-4V performs poorly on Korean diagrams and its performance on English is also lacking, BlockNet overcomes these limitations and captures finer details more effectively.
Table 3 displays the results of automatic evaluation conducted on computerized (CBD) and handwritten (FC_B) datasets, both comprising English diagrams. In the CBD dataset, BlockNet achieves higher BLEU (58.61) and ROUGE1 (74.25) scores, while GPT-4V attains a better BLEURT (0.48) score, indicating its proficiency in generating grammatically correct sentences. However, the difference in BLEURT scores between BlockNet (0.41) and GPT-4V is minimal. Similar to the results for BD-EnKo, the Perplexity (PPL) scores for both CBD (5.46) and FC_B (4.68) datasets are higher for mPLUG, as PPL assesses sentence fluency rather than factual accuracy. In the FC_B dataset, BlockNet again outperforms all models, achieving the highest BLEU (67.74), ROUGE1 (90.63), and BLEURT (0.59) scores. Additional experimental results and sample outputs are detailed under Appendix A.6. Human Evalution There are various ways to describe block diagrams, and no single automatic evaluation metric is perfect. To assess the quality of text descriptions generated from these diagrams, we enlisted the input of 30 individuals, comprising 15 native English speakers and 15 native Korean speakers. They were tasked with rating the descriptions based on four criteria: (1) Accuracy, which evaluates if the converted text faithfully represents the original diagram’s content; (2) Completeness, assessing if all aspects of the diagrams were adequately translated; (3) Readability, gauging the ease of understanding and clarity of the converted text; and (4) Semantic correctness, determining if the converted text effectively conveys the intended meaning of the original diagram. Each criterion was rated on a scale of 0 to 100, where 0 signified “strongly disagree” and 100 indicated “strongly agree”. We randomly selected 60 English block diagrams from the BD-En dataset and 60 Korean diagrams from the BD-Ko dataset. Each examiner received corresponding textual descriptions generated by the proposed BlockNet model and GPT-4V.
The average scores provided by the reviewers for the generated text descriptions are illustrated in Table 4. Based on the table results, it is evident that BlockNet outperforms GPT-4V across all four evaluation criteria for English and Korean diagrams.

heading: 5 Ablation Study
text: As shown in Table 5, we further investigate the effectiveness of our method across various modules, including the substitution of the GPT-4V integrator module with LLaVA-1.6 “liuhaotian/llava-v1.634b” (Liu et al., 2023). Notably, integrating both local and global information with GPT-4V yields superior performance (60.68), highlighting the significance of each module: local and global information extractors. Remarkably, when one module, either local or global information, is omitted, there is a significant decrease in accuracy, underscoring the indispensability of each component. However, it appears that the performance degradation is more pronounced without Local Information. Therefore, we can infer that GPT-4V and LLaVa models tend to exhibit weaker performance in capturing details. These findings underscore the holistic approach required for optimal performance in our framework, emphasizing the interplay between local and global information for enhanced results.
Furthermore, we investigated the feasibility of automating the generation of Mermaid code directly from block diagrams extracted from images, with subsequent summarization using GPT-3.5. Initially, we trained a transformer encoder-decoder model for this task but encountered considerable challenges. The strict syntax requirements of Mermaid code, particularly regarding arrow and text arrangement, posed significant hurdles, as even minor deviations could drastically alter the code’s meaning. Despite our efforts, the model frequently produced inaccurate outputs due to difficulties in capturing diagram structure accurately. Additionally, generating coherent summaries using GPT-3.5 from inaccurately formatted Mermaid code proved problematic, leading to errors.

heading: 6 Conclusion
text: In conclusion, this paper introduces the “BlockNet” framework, which effectively integrates local and global information from block diagrams to enhance summary generation. Our innovative approach to multi-lingual data collection, designed for block diagram understanding and generation, not only
provides flexibility and scalability but also facilitates the creation of the diverse and high-quality benchmark dataset “BD-EnKo” in English and Korean languages. Through rigorous automatic and human evaluations, our framework showcases stateof-the-art performance on both English and Korean diagrams. These findings underscore the significance of emphasizing the challenges that even advanced models like GPT4V face in comprehending block diagrams. Moreover, integrating GPT4V as a component in BlockNet underscores the crucial role of “local and global” fusion in achieving the superior performances observed. Overall, these contributions drive advancements in various fields, including block diagram understanding, Document AI, and QA, laying a solid foundation for future research across diverse domains.

heading: Limitations
text: After conducting several automatic and human evaluations, two primary limitations of this work have been identified. Firstly, despite proposing a multilingual data generation approach, we only generated data in English and Korean. Secondly, not many LLM options were explored for the Integrator module within our framework. Future efforts will concentrate on creating a smaller and faster LLM to enhance efficiency and accessibility.
Additionally, a limitation was discovered with the BD-EnKo dataset: its diagrams lack additional graphics akin to real-world images, although their structure remains consistent. In the future, our attention will be directed towards curating a more diverse dataset, inclusive of richer graphical content and encompassing multiple languages. This initiative aims to advance the development of a genuinely multilingual model.

heading: Acknowledgment
text: This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R1A5A7026673)(50%) and by the NRF grant funded by the Korea government (MSIT) (No. NRF-2021R1A2C3011169)(50%).

heading: A Appendices

heading: A.1 Dataset Generator
text: Pipeline of our data generation approach involves three steps: first, generating mermaid codes using GPT-3.5 turbo; next, utilizing the generated code to draw diagrams using the Mermaid tool; and finally, employing GPT-3.5 turbo once more to generate text based on the provided mermaid code.
Figure 5 showcases the prompt utilized for generating mermaid code, which constitutes the first step in our data generation pipeline. Within this prompt, there are three variables highlighted in red. The first variable pertains to the block diagram type, which is initially set as “Graph TD”. There exist a total of seven types of block diagrams, and this variable can be adjusted to any one of them as needed. The second variable is the language, which defaults to “English” but can be modified to any desired language. We utilized both English and Korean languages for generating the dataset. The third variable serves as an example of the given block diagram type, enabling GPT-3.5 turbo to comprehend the code’s structure and produce output in that specific format.
Figure 6 illustrates the transformation process conducted by the Mermaid tool, which seamlessly converts the provided mermaid code into an image representation. Notably, the Mermaid tool offers users the flexibility to customize the appearance of the generated diagrams by incorporating various colors and themes. By incorporating diverse colors and themes, the Mermaid tool ensures the creation of a rich and varied dataset. This variability in visual presentation enhances the dataset’s versatility, accommodating different aesthetic preferences and catering to a wide range of visual styles.
Figure 7 illustrates the prompt utilized for generating text from mermaid code, constituting step three in our data generation pipeline. There are two variables involved in this prompt. The first variable is the language, initially set as “English”, yet it can be altered to accommodate any desired language. The second variable is the mermaid code generated from step one. GPT-3.5 turbo is capable of generating descriptive text based on the provided mermaid code.

heading: A.2 BD-EnKo
text: The BD-EnKo dataset comprises nine distinct categories: Flowchart, GraphLR, GraphTD, Journey, Sequence Diagram, Simple Graph, State Diagram,
C4, and Real world. Each category represents a specific type of diagram or visualization commonly encountered in various domains. Flowchart is typically used to represent processes or workflows, where each step is connected by arrows indicating the flow of control or information. GraphLR is a type of directed graph where nodes are arranged from left to right, commonly used to represent hierarchical structures or dependency graphs. GraphTD is similar to GraphLR, but nodes are arranged from top to bottom, often used to represent hierarchical structures or decision trees. Journey represents a series of steps or stages in a process, commonly used in customer journey mapping or user experience design. Sequence Diagram illustrates the interactions between objects or components in a system over time, commonly used in software engineering to visualize system behaviors. C4 represents software architecture at different levels of abstraction, including Context, Container, Component, and Code views. Real world represents real-world scenarios or situations, such as floor plans, network diagrams, or organizational charts. Each category serves a distinct purpose and is characterized by specific structural and visual elements tailored to its intended use case. Figure 8 showcases samples of each category from our dataset BD-EnKo, illustrating the diverse range of diagrams encompassed by this dataset.

heading: A.3 Additional Dataset Analysis
text: We have analyzed the descriptive statistics of our BD-EnKo dataset. The number of nodes in the training dataset has a mean of 12.15 and a standard deviation of 7.53, indicating moderate variability around the mean. The distribution of nodes shows that 25% of the data points have 7 or fewer nodes, 50% have 9 or fewer nodes, and 75% have 16 or fewer nodes. Similarly, the number of arrows has a mean of 11.07 and a standard deviation of 6.03, suggesting slightly less variability compared to nodes. The minimum number of arrows is 0, and the maximum is also 196. The distribution of arrows reveals that 25% of the data points have 6 or fewer arrows, 50% have 10 or fewer arrows, and 75% have 15 or fewer arrows. For the validation set, the number of nodes in the dataset has a mean of 12.11 and a standard deviation of 7.23, indicating moderate variability around the mean. The distribution of nodes shows that 25% of the data points have 7 or fewer nodes, 50% have 10 or fewer nodes, and 75%
have 16 or fewer nodes. Similarly, the number of arrows has a mean of 11.06 and a standard deviation of 5.63, suggesting slightly less variability compared to nodes. The minimum number of arrows is 0, and the maximum is 70. The distribution of arrows reveals that 25% of the data points have 7 or fewer arrows, 50% have 10 or fewer arrows, and 75% have 15 or fewer arrows.
We have produced block diagrams encompassing a diverse array of topics. The graph presented in Figure 9 provides a glimpse into the top twenty topics extracted from our dataset. This visualization offers a concise overview of the most prevalent themes identified through our analysis, serving as a condensed representation of the breadth and scope of the data under examination.

heading: A.4 BlockNet: Integrator
text: Figure 10 illustrates the prompt utilized to integrate outcomes from both the local information extractor and the global information extractor. This prompt comprises three variables that can be adjusted based on the specific use case. The first variable is “lang”, which can be set to any language since GPT-4V is proficient in handling various languages. The second variable represents the output from the global information extractor, denoted as “global”, providing an overall summary of the diagram. Lastly, the third variable represents the output from the local information extractor, referred
to as “local”, which consists of a collection of all potential triplets depicting the connections within the diagram.

heading: A.5 Experimental Setup Details
text: BlockNet We train the YOLOv5 model utilizing the SGD optimizer with parameters lr=0.01 and weight decay=5e-5, maintaining a batch size of 128 and 300 fixed training steps. The training of the transformer encoder and decoder model is conducted according to the configuration outlined in Donut (Kim et al., 2022). For the visual encoder, we employed Swin-B with 10 layers and window sizes of 2, 2, 14, and 2 respectively. Additionally, we utilized the initial four layers of BART as the decoder. The model was optimized using the Adam optimizer with a learning rate of 3e-05. The input resolution of the model was set to 1280×960, with a maximum decoder length of 768. Splitting of dataset is shown in Table 6.
mPLUG We compare our model with mPLUG (Ye et al., 2023), specifically using the “MAGAer13/mplug-owl-llama-7b” model with a maximum token limit of 512 and retaining all settings unchanged. mPLUG, trained on a large dataset of image-text pairs with discriminative and generative objectives, achieves state-of-the-art performance across various vision-language tasks. Its efficient architecture incorporates cross-modal skip-connections for accelerated information processing. As this model lacks support for the Korean language, our testing was conducted exclusively with English diagrams.
Donut We employ the pre-trained Donut-base model from Hugging Face and fine-tune it with a combination of datasets used in training BlockNet to ensure a fair comparison. We configure the max epochs to 100 and warm-up steps to 20, with a maximum sequence length of 768. All other hyper-
parameters remain consistent with those utilized in the pre-training of Donut (Kim et al., 2022).
BloSum/Faster R-CNN + T5/BART We adopt the methodology outlined in the BloSum/Faster R-CNN and T5/BART papers (Bhushan and Lee, 2022), maintaining consistency by adhering to their specified settings and procedures. We conduct experiments exclusively with English data, as training T5 or BART models for the Korean language is not feasible due to the unavailability of appropriate open-source Korean triplet datasets for fine-tuning. Our focus on English datasets ensures comparable evaluations and comparisons as outlined in the original papers.

heading: A.6 Additional Experimetal Results
text: We conduct symbol detection evaluation specifically on the CBD dataset, which possesses labeled data for symbol detection. Table 7 illustrates the precision, recall, and mAP50 (mean Average Precision at 50 IoU threshold) score values of the object detection module used in the local information extractor of BlockNet. Precision refers to the ratio of correctly predicted positive observations to the total predicted positives, indicating the model’s accuracy in identifying relevant symbols. Recall, on the other hand, represents the ratio of correctly predicted positive observations to the actual positives in the dataset, indicating the model’s ability to correctly identify all relevant instances. mAP50 measures the average precision across different IoU (In-
tersection over Union) thresholds up to 50%, providing a comprehensive evaluation of the model’s detection performance. The experimental findings indicate that following fine-tuning with Yolov5m, achieves an impressive average precision of 0.979 for all categories on the CBD dataset.
Figure 11 depicts a side-by-side comparison of sample outputs generated by our model, BlockNet, and GPT-4V. A clear observation emerges: while GPT-4V faces challenges in accurately generating relations for block diagrams, BlockNet demonstrates a superior ability to adeptly capture all the intricate details and relationships within the diagrams. This stark contrast underscores the effectiveness and precision of our BlockNet model in generating coherent and contextually relevant outputs compared to GPT-4V. In Figure 12, we provide a glimpse of a sample output generated by our BlockNet model using the intuitive interface of the Gradio app. This visualization underscores the effectiveness and user-friendliness of our model within an accessible environment. Additionally, we have expanded the functionality of our method to include QA capabilities, as illustrated in Figure 13. This exciting enhancement showcases the adaptability and versatility of our approach, seamlessly transitioning between different tasks with minimal adjustments. While we have made significant strides, further research and experimentation are needed to fully solidify its robustness, which
will be our focus for future work.

heading: Future Work
text: While we have made significant strides, further research and experimentation are needed to fully solidify its robustness, which
will be our focus for future work