FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts (2024)

Abstract: Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results underscore the benchmark’s potential as a vital tool for advancing the field of multimodal modeling, providing a focused and challenging environment for enhancing model performance in visual and logical reasoning tasks.

heading: 1 Introduction
text: Since the inception of Vision Language Models (VLMs), tasks and benchmarks for visual question answering (VQA) and reasoning have received significant attention. Most benchmarks evaluate pretrained model extraction capabilities, neglecting their ability to comprehend complex spatial relationships and visual logical reasoning. Research on spacial path following or visual sequential reasoning in VLMs is limited. Current benchmarks for assessing VLM reasoning abilities mainly fall under VQA, a task formalized by Goyal et al. (2017), which involves generating responses to questions based on a given image. These works evaluate the spatial reasoning and visual information extraction abilities of VLMs.
∗, † contributed equally, ‡ primary mentor & corresponding author
Visual Grounding (VG) of a visual questionanswering system evaluates models’ abilities to attribute their generations to different image regions referenced in the query (Reich et al., 2023). The absence of VG has been a frequent issue among the current VQA systems, manifesting in an overreliance on irrelevant parts of images or a complete disregard for the visual modality. Existing benchmarks (Yue et al., 2023) require models to rely on pre-trained knowledge to answer queries posed on image contexts. In this work, we aim to test the capabilities of VLMs in following visual information
1330
without any pre-existing knowledge. To accomplish this, we delve into the realm of flowcharts, as depicted in figure 1, which entail intricate structural configurations and path reconstruction, a considerably more challenging task compared to mere image comprehension.
Flowcharts emphasize sequential and logical reasoning, as they necessitate traversal of steps or decisions in a specific sequence. Flowcharts are inherently visual, and provide a clear and structured method for representing processes, decision paths, and flows. Unlike traditional text, which flows linearly, flowcharts require an understanding of directional logic; their flow is often multi-directional, representing various paths that can be taken based on certain conditions or decisions. Despite being long and complex, flowcharts have compact, systematic representations and provide insights regarding information in a step-by-step manner.
In this paper, we set out to answer a crucial question: "Can modern Vision Language Models effectively handle challenges that demand understanding both structural and semantic aspects, along with capturing macroscopic and granular context within visually complex yet straightforward flowcharts?" To tackle this question, we introduce FlowVQA, a novel benchmark comprising intricate structural and path-based questions posed on lengthy flowchart images. We propose a novel approach to Visual Question Answering (VQA) on Flowchart tailored for VLM, with a focus on harnessing flowcharts as the primary contextual framework for visual logic and spatial reasoning.
FlowVQA consists of 2,272 Mermaid.js flowchart scripts generated with human input, sourced from process workflow articles like Instructables and WikiHow, as well as Code. Accompanying these are 22,413 Q/A pairs covering various reasoning skills like information localization, fact retrieval, scenario deductions, flow reasoning, and topological understanding. The creation process involves a meticulous multi-step machine generation and human verification to discard up to 51% of samples, ensuring they meet high standards of challenge, coherence, and insightfulness. This rigorous process grounds the flowchart reasoning in textual domain, enriching the visual task complexity. Extensive experimentation revealed that both closed and open-source Vision Language Models (VLMs), equipped with a range of prompting strategies and fine-tuning techniques, struggled
to execute visual and spatial reasoning tasks within the FlowVQA dataset. Moreover, our findings highlighted a directional bias and non-uniform performance pattern across flowcharts of varying lengths exhibited by these VLMs. Our contributions are the following:
• Introduction of VQA for FlowCharts, focusing on visual logic and spatial reasoning, filling a gap in previous benchmarks.
• Development of a detailed framework for generating intricate VQA samples transitioning from text to visual domains, ensuring quality, complexity, and accuracy via rigorous verification.
• Introducing the novel benchmark FlowVQA, consisting of 2,272 high-quality Flowchart Images and 22,413 Q/A samples spanning four distinct question types, created using the framework.
• Thorough evaluation of closed and open-source VLMs, employing various prompting strategies and fine-tuning methods. An analysis of directional bias and non-uniform performance across different flowchart lengths.
The FlowVQA dataset, along with modeling and evaluation scripts, generation pipeline and prompts, and the human verification platform, can be accessed at https://flowvqa.github.io/.

heading: 2 Proposed FlowVQA Resource
text: In this section, we will see the details of the construction of FlowVQA. We outline the process of collection of raw data from wild sources, multistep generation of mermaid scripts and flowchart images and complex Q/A creation.

heading: 2.1 Flowchart Sources
text: We draw input texts from three primary sources: WikiHow articles, Instructables DIY blogs, and FloCo (Shukla et al., 2023a) code snippets. WikiHow and Instructables provide step-by-step instructions for everyday tasks, while the FloCo dataset, a flowchart-to-code resource, features lowcomplexity code samples. We categorize all the WikiHow articles, Instructables DIY based on the domains of these articles. FloCo code snippets are categorized into code category. The distribution across categories is outlined in Appendix B.
We manually select high-quality code snippets from FloCo to ensure uniformity in our pipeline across all text sources. FloCo image samples enable us to iteratively compare the generated
flowcharts with the original samples. This step was crucial as it helped perfect our prompts and allow applicability to the WikiHow and Instructables set. We sample 1,268 WikiHow articles, 789 Instructables blogs, and 475 FloCo examples as an input to our human verification pipeline.
Generation and Filteration. GPT-4 based data generation of data and benchmarks is prevalent (Han et al., 2023) in prior works. Machine generation method for flowcharts and Q/A has several advantages to crowdsourcing: (i) The complex and intricate process of creating flowcharts and Q/A pairs constitutes a laborious, efficient and a timeintensive task for human workers, (ii) Using GPT4 for the generation of structured representations and subsequent conversion into flowcharts and Q/A pairs enables rapid scaling, (iii) The Stochastic nature of LLMs helps in the creation of an unbiased and diverse Q/A dataset. To produce Flowchart and Q/A Samples, we employ an automated ’generateand-test’ approach, where we exhaustively gener-
ate questions of multiple reasoning types and apply rigorous filtration to maintain the quality, hardness, and correctness of samples through effective prompting with GPT-4. Our meticulous verification through experts and rubrics, along with our custom-built annotation platform, ensures a thorough and impartial evaluation of both flowcharts and Q/A pairs.

heading: 2.2 Flowchart Generation
text: Our primary supposition for flowchart creation is that any process-based workflow, regardless of domain, can be converted to a flowchart which highlights key aspects of the process in a detailed stepby-step fashion. We treat the conversion of source article to flowchart Mermaid Scripts as a two-step soft-syntax summarization task. Ideally, we would use real-world flowcharts from external sources such as books and documents, but the availability of such structured data is extremely sparse. Initially, we aimed to use real-life flowcharts, but the scarcity of standardized flowcharts and the lack of sufficient open-source examples made it unfeasible to create a dataset as comprehensive as ours. We decouple the structured summarization into a flowchart script to implement this two-step process.
First Step. We query GPT-4 with the source text
to generate a step-by-step structured representation of the text annotated with functional control tags (e.g., “START,” “PROCESS,” “DECISION”). This step converts the source text into a tagged textual representation suitable for converting into mermaid flowchart scripts. For FloCo-sourced texts, we generate pseudocode for the code scripts as the input to the next step.
Second Step. In this step, we generate the Mermaid.js flowchart script(top-down) using the output of the first step by querying GPT-4 with a template Mermaid.js script. The control tags facilitate mapping the steps to the node types used in the script. Constraining points are provided alongside both prompts for improved normalization. The Mermaid.js scripts are then compiled to create highresolution PNG images.
Table 1 represents the number of samples after the two-step conversion process. We exclude the scripts and representations with minor syntactical and rendering errors. Figure 2 showcases the generation pipeline of the flowcharts in our dataset. Appendix D.1 lists the prompts used in first step and second step.

heading: 2.3 Question Answer Creation
text: We curate four question types designed to analyze and test different aspects: Fact Retrieval, Applied Scenario, Flow Referential and Topological Question and Answer. First three can be broadly categorized under granular flowchart comprehension while topological tests structural information.
T1. Fact Retrieval: These simple questions involve the localization and retrieval of direct factual information from flowchart’s nodes. Despite being simple, they still necessitate image analysis and retrieving relevant cues that localize the final answer.
T2. Applied Scenario: These questions describe a real-life scenario and test the models’ application of the flowchart to a practical problem. These questions capture reasoning skills used by humans parsing flowcharts in day-to-day life. It leads to
interesting puzzle-like word problems that test the understanding of decision steps, content, and reasoning in the presence of distractor context, which needs to be filtered to understand the question.
T3. Flow Referential: In these questions, A random sub-graph/section of the flowchart, usually involving a decision node, is considered, and a question is formulated on backward-forward flow with decision-based logic. It assesses granular path dynamics in a flowchart.
T4. Topological: This question type addresses the larger topology of a flowchart, requiring analysis of the flowchart at a more macroscopic level to give an answer related to the structural topology of the graph. These questions are created by parsing Mermaid.js scripts to convert them into an adjacency matrix representing the flowchart in the form of a graph. It generates template-based questions that usually have quantitative correct answers.

heading: Statistics Train Test Total
text: Q/A Generation. We construct a prompt to query GPT-4 using the tagged textual representation, Mermaid.js script and text-only few-shot examples to generate high quality Q/A pairs of types, T1, T2 and T3 (listed in Appendix D.2). For each question, we generate three paraphrased gold answers, which allows us to evaluate models irrespective of their generation syntactics and semantics. As part of text-only few-shot examples we pass a variety of creative high-quality examples. Topological Q/A pairs (T4) are generated by parsing the Mermaid script, converting the graph into an adjacency matrix, and creating templatebased questions. Answers are usually quantitative. After formulating the template-based answers, we
obtain two additional paraphrased answers for each template answer to achieve three gold-standard answers, thus maintaining the standard with the other question type for three gold short answers.

heading: 2.4 Human Verification Pipeline and Platform
text: To ensure strong validity of our work, we establish a robust human verification pipeline for our models and flowcharts. All generated outputs for flowcharts and subsequent Q/A pairs undergo a rigorous quality check by a team of five expert annotators. As we adhere to a "Generate-and-test" paradigm (section 2), we provide detailed rubrics for both flowchart and Q/A pair verification and annotation, with parameters such as logical flow, complexity, context alignment and more, for flowcharts and Q/A pairs which allow the annotators be strict and thorough. To assist with their work and eliminate any bias and stress, we also provide them with a detailed, custom-built annotation platform to provide scores, filter out, etc. This custom platform enables parallel viewing.
.
Annotation Platform. Our custom-built annotation platform consists of UI, where we pass the flowchart and Q/A pairs together so they can be
viewed simultaneously. The annotators provide quality scores1 for all components of the dataset and a final holistic score2. We filter out flowcharts below a fixed quality threshold and Q/A pairs which rate below average. Topological questions are not passed into the platform as they are hardtemplate-based and obtained via scripting. All verification products are verified with two separate supervising experts who ensure the quality of annotations is consistent and scores remain unbiased. The verification lasts for ten days from start to end.
The final samples, see Appendix A, ensure appropriate complexity and correctness of flowcharts, questions and corresponding answers. Figure 3 showcases the complete question-answer generation pipeline used to create the dataset.

heading: 3 Experimental Evaluation
text: We address the following research questions through our experiments:
RQ1. Does the introduced visual multimodal dataset present a significant challenge to current multimodal language learning models (VLMs), and can it provide valuable insights that could contribute to their future advancement?
RQ2. Is the efficacy of VLMs influenced by factors such as (a) the source of flowcharts, (b) the type of questions posed, and (c) the level of complexity inherent in the flowcharts?
RQ3. Are there ways to enhance the perfor1captures the consistency, correctness and complexity. 2captures the relevancy between the components.
mance of visual question answering tasks related to flowcharts through the use of specific directives tailored to flowcharts? Moreover, does the process of fine-tuning these models with the train split of FlowVQA dataset improve their proficiency in handling questions tied to flowchart-based data?
RQ4. Is there an observable directional bias in existing VLMs when they are applied to flowchart analysis?
Limitations of Smaller Models. FlowVQA represents a complex multimodal challenge that requires visual logic and reasoning across largescale high-resolution images. In our assessment of several widely utilized open-source multimodal language learning models (VLMs) – including LLaVA (Liu et al., 2023b), OpenFlamingo (Awadalla et al., 2023), BLiPv2 (Li et al., 2023a), mPLUG-OWL (Ye et al., 2023b), Sphinx (Lin et al., 2023) — we observe that their performance on our test dataset is notably subpar (<10%). These multimodal language learning models (VLMs) lack a sizable vision encoder, leading to the internal distortion of flowchart images with high aspect ratios when passed into the vision encoder. Furthermore, even if they can interpret the image a bit, their inadequate reasoning abilities render them extremely ineffective for any further analysis utilizing this resource.
Models for Comparison. We perform evaluations on FlowVQA with five different VLMs. We employ GPT-4V (OpenAI, 2023) and Gemini Pro (Anil et al., 2023)3 to test the visual understanding capabilities of best proprietary (closed) models available. We also employ three opensource models. CogAgent-VQA (Hong et al., 2023) is an 18- billion-parameter visual language model (VLM) specializing in GUI understanding and navigation (fine tuned on smaller VQA Tasks). This model supports inputs at the resolutions of 1120x1120, enabling it to recognize tiny page elements and text in the flowcharts.
InternLM-X-Composer2 (Dong et al., 2024) uses a novel approach (PLORA) that applies additional LoRA parameters exclusively to image tokens to ensure that linguistic abilities are not affected, striking a balance between precise vision understanding and text composition. QwenVL-chat (Bai et al., 2023) is the instruction tuned
3We use the preview version for Gemini Pro at Vertex API (Vertex). Gemini Ultra is/was not made public yet.

heading: Open Model LM VM Norm. Res.
text: model in the Qwen-VL series. Its position-aware vision language adapter ensures that, even though the images are resized to a fixed resolution long image feature contexts are captured effectively by the model. We summarize the base language models and visual models used in our baselines in Table 5.

heading: 3.1 Baseline Evaluation
text: We evaluate the baseline models under multiple settings: 1. Zero-Shot: Given a flowchart, we prompt the
VLM to answer the question with a small instruction and provide a short concise answer. 2. Zero-Shot CoT: Given a flowchart, we prompt the VLM with the question to first elicit a rationale and then deduce the final answer (Wei et al., 2023).
3. Text Only Few-Shot CoT with Reasoning Directives: We create a custom prompt outlining the reasoning steps involved in answering questions specific to flowcharts. We scrutinize the areas where improved prompting is necessary for the models and draw inspiration from (Zhang et al., 2023), (Li et al., 2023b), and (Kojima et al., 2023) to devise a text-only few-shot CoT approach with directional stimulus and step-bystep reasoning. The central objective is to deconstruct complex questions, identify which elements to map, and determine the answer. Each example, or "shot," encompasses four key components: The Question, Directional Stimulus Tags, Step-by-Step Rationale, and the Answer. These distinct parts aid in breaking down the question into relevant segments, offering a logical, step-by-step analysis, and concluding with an answer. We develop this strategy based on its potential effectiveness for flowcharts, with its actual efficacy demonstrated ahead. The few-shot samples we give are dynamic in nature, i.e the each question type gets more similar samples from our train set annotated samples samples for the method.
4. Fine-Tuning: We fine-tune the VLM on the
train split of FlowVQA, and then prompt the VLM to answer the question.4

heading: 3.2 Evaluation Method
text: Our methodology adopts an "AI as an Evaluator" approach similar to Fu et al. (2023); Lin and Chen (2023); Chiang and Lee (2023). We employ three evaluator models—GPT-3.5 (Ye et al., 2023a), Llama-2 70B (Touvron et al., 2023), and Mixtral 8*7B (Mixtral-of-Experts) (Jiang et al., 2024) —to assess the model-generated responses, which are compared against three gold standard short answers and the question (context excluded). The evaluators’ task is to dissect and align the responses, eliciting a detailed rationale that demonstrates Chain of Thought behavior, and then assigning a binary label to indicate whether the response is correct or incorrect. This process essentially boils down the evaluation into a "length-invariant" paraphrase detection task for short text responses, surpassing traditional similarity metrics and rule-based matching in effectiveness. We determine the final label via a majority vote among the evaluator models.
Fine-tuning Settings. We fine-tune Qwen-VLchat FT using LORA (Hu et al., 2022) strategy on 2xNVIDIA A100 40GB GPUs. We train with an effective batch size of 8 using a cosine-based learning scheduler with a warmup. We set a higher warmup to ensure no loss of pretraining knowledge in the base model.

heading: 3.3 Baseline Results and Discussion
text: Table 6 tabulates the results of model evaluations across multiple strategies, with the scores split across various question types and text sources. Figure 5 in Appendix C provides a horizontal bar chart that compiles the results from the table.
FlowVQA is sufficiently hard. The dataset resource presents a challenging task, with all the models. The evaluations highlight a scope for improvement for all the models. Our Best performing model with the top performing strategy, i.e. GPT-4 prompted with Few-shot directive-based prompting achieves 68.42% Majority voting across all the evaluators.
Few-Shot Directives are helpful. In the evaluation of most of our models, we observe that text-only few-shot CoT with reasoning directives
4Due to resource constraints we only Fine-Tune on QwenVL-Chat through LoRA Finetuning
outperforms other prompting strategies. We observe 7% improvement in GPT-4 evaluation and 12% improvement in Gemini-Pro with this strategy. CogAgent-VQA , however does not show an improvement with few-shot directives. We observe in our initial experiments that it was unable to generate directives and hence it could not make use of reasoning directives.
Proprietary models perform better than opensource models. We observe that proprietary models heavily outperform the open-source models. GPT-4 with few-shot directives outperforms QwenVL-chat by a significant 30%.
Fine-tuning helps. We fine-tune Qwen-VLchat and evaluate by prompting with Zero-Shot and Zero-Shot CoT strategies. We see an improvement of 3% from Zero-Shot prompting and 11% improvement from Zero-Shot COT. This improvement emphasises the lack of flowchart understanding in original pretraining mixtures of these VLMs. The improvement in T2, T3 and T4 ( 10%) being more significant than T1 (5%), can be attributed to the fact that fact-retrieval is a simpler task and does not need in-depth understanding of the flowchart structure. The fine-tuned model outperforms all other existing open-source models, which highlights the fact that FlowVQA can be effectively used to introduce visual logic and reasoning in existing VLMs.
Question Types. We present the question-wise metrics in Table 5. It is evident from the table that all models consistently perform better on Fact Retrieval (T1) and Applied Scenario (T2) based based questions than on Flow-Referential (T3) and Topological (T4). Outlined in Sec. 2.3, T3 and T4 question types require thorough understanding of the flowchart and complex reasoning over the visual modality.
Number of Nodes. Using the Mermaid.js scripts, we obtain the count of nodes in each flowchart. We categorize the flowchart by binning the number of nodes present in them. A Large number of nodes implies a more complex representation of visual information, and hence the flowchart is harder to reason upon. The results in the Table 7 confirms this fact. Figure 7 in Appendix C shows the decline of performance of models with increase in number of nodes.

heading: Model Strategy MVTotal MVT1 MVT2 MVT3 MVT4 MVWiki MVInstruct MVCode

heading: 3.4 Directional Bias
text: To study RQ4, we parse the mermaid scripts of the FlowVQA flowcharts and systematically invert them to produce a inverted flowchart "Bottom Top" set. Bottom Top analysis helps further evaluate the Visual and Sequential nature of our resource. The Bottom Top Flowcharts look directionally counter-intuitive with the start nodes at the bottom and end at the top. We perform this inversion on 1,500 flowchart-question pairs on which all evaluators evaluate to "True" (correct response for all). We evaluate a the top-performing models and strategies obtained in Section 3.1 on the inverted flowchart set to detect any presence of directional bias in the VLMs.
Table 8 highlights the fact that our best performing models do suffer from a directional bias in understanding and reasoning over flowcharts. We see a significant 15% drop in majority voting accuracy thorough with GPT-4.

heading: Model (Strategy) Top-Down Bottom-Up
text: Analysis. The directional bias evaluation underlines an important lacking of existing VLMs. They suffer from biases introduced in pretraining mixture and do not ground their inferences in the context images which leads to a significant drop in their evaluation performances. Strategies like augmenting pretraining mixtures with counterfactual examples might help alleviating these issues, which we leave for future study.

heading: 4 Related Work
text: Vision language models have made large progress in diverse vision-language applications (Bai et al., 2023; Liu et al., 2023a; Xia et al., 2024) with multiple benchmarks being proposed to aid effective evaluation of visual and textual grounding capabilities of these models. The MMMU benchmark (Yue et al., 2023) is designed to assess the model’s inherent "subject-specific" knowledge and reasoning abilities across various subjects (such as Technology, Humanities, Health, and more).
Benchmarks like TextVQA and DocVQA (Singh et al., 2019; Mathew et al., 2021b; Zellers et al.,
2019; Park et al., 2020; Lu et al., 2022; Hudson and Manning, 2019) evaluate the models’ fine-grained transcription abilities on low-resolution images. More complex multimodal reasoning tasks, such as MathVista (Lu et al., 2024), examine the models’ abilities to integrate visual and mathematical logic. Benchmarks focusing on spatial multimodal reasoning include ChartQA (Masry et al., 2022; Xu et al., 2023; Methani et al., 2020) and InfographicVQA (Mathew et al., 2021a). ChartQA is aimed at evaluating straightforward chart understanding and analysis, while InfographicQA poses direct logical questions about data visualizations and charts.
Prior Flowchart Works. To our knowledge, there exists a study on Flowchart QA (Tannert et al.), that suffers from major limitations. (i) Synthetically generated flowcharts with randomized scripts, (ii) Primarily poses structural questions and (iii) Uses multiple choice-based questions to evaluate weaker existing models. Other research in this domain addresses issues like Flowchart Object Recognition and Flowchart to Code/Script conversion, where a modest parallel flowchart resource is paired with corresponding code or script (Liu et al., 2022; Shukla et al., 2023b; Thean et al., 2012; Sun et al., 2022). However, notable limitations here include poor flowchart image quality, niche or overly complex context, structural imbalance (only linear or excessively complex), lack of ground truth scripts for flowcharts, and insufficient context for effective Q/A or practical tasks. In contrast to these works, we construct a complex benchmark suitable to test practical applicability of existing VLMs. The complex and diverse QA types ensure an effective and just evaluation over multi-modal visual and textual understanding.

heading: 5 Conclusion and Future Work
text: In conclusion, this study evaluates the effectiveness of existing Multimodal Large Language Models (VLMs) in reasoning upon a complex visual, sequential logical reasoning based task, FlowVQA. We introduce the novel dataset resource, FlowVQA, consisting of 2,272 Flowchart images, Mermaid.js scripts, 22,413 Q/A pairs with gold standard answers. Our extensive evaluation on these models with multiple strategies and scenarios highlights the need for advancements in architecture and prompting strategies in existing VLMs. We also study the presence of any directional bias in the
flowcharts by re-evaluating the test sets with an inverted flowchart subset. We find that both proprietary and open-source models suffer from directional bias due to lack of visual grounding and complex structural reasoning required for flowchart reasoning.
Future Work. Our work and resources give rise to many research avenues in (a) Flowchart Reasoning: FlowVQA can be used to enhance the visual logic and reasoning capabilities of the models. Constructing VLMs that are flowchart specific is also a encouraging research direction. (b) Graph-Encoder Models: In this study, we consider the graph nature of flowcharts solely to generate topological questions. This consideration can also be taken into account while designing model architectures and inference strategies to enhance structural reasoning in the base models. (c) Adversarial and Counterfactual probes: We provide questions of four different types which can be augmented with multiple probe sets like negative path following, counter-intuitive questions and noisy-graph based questions. (d) Complex Subtasks: The parallel nature of FlowVQA allows us to formulate multiple subtasks using the resource. Primary task of FlowVQA is the Flowchart→Q/A. We can create multitude of tasks: article→Q/A, Mermaid.js→Q/A, Flowchart→Mermaid.js. The tasks can then act as an additional resource for training LLMs and VLMs. (e) NeuroSymbolic AI Approaches like in Trinh et al. (2024) can also be considered to enhance performance and training on our resource as flowcharts are inherently symbolic and sequential structures.

heading: Limitations
text: There are a few notable limitations to our work. Primarily, the inability to fine-tune all models under consideration due to financial and computational resource constraints has led to a potential underrepresentation of the capabilities of various NLP models beyond our primary focus. Moreover, the language limitations encountered in this research, particularly the focus on English for generating Visual Question Answering (VQA) methods, underscore the need for linguistic diversity in NLP applications to ensure broader applicability and inclusivity. Given the novelty of the task at hand, it is also important to acknowledge that the insights provided may not be exhaustive, highlighting the potential for future research.

heading: Ethics Statement
text: We, the authors of this work, confirm that our research adheres to the highest ethical standards in both research and publication. Throughout this study, we have diligently considered and addressed various ethical issues to ensure the responsible and equitable use of computational linguistics methodologies. To promote the reproducibility of our results, we provide comprehensive information, including sharing code, datasets (we use publicly available datasets and comply with the ethical standards set by their authors), and other relevant resources. This enables the research community to validate and extend our work. The claims presented in this paper are consistent with our experimental results. However, given the inherent stochasticity of black-box large language models, we have minimized variability by maintaining a fixed temperature. We thoroughly detail the annotations, dataset splits, models used, and prompting methods employed, ensuring the reproducibility of our work.

heading: Acknowledgements
text: Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-20-1-0080. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was partially funded by ONR Contract N00014-19-1-2620. We extend our gratitude to the annotators who verified our flowcharts and corresponding question answer pairs. Lastly, we extend our appreciation to the reviewing team for their insightful comments.

heading: B Dataset Distribution
text: Figure 4 illustrates how our data is distributed among various sources and types of questions.

heading: C Additonal Results
text: Figure 5 shows the performance of FlowVQA dataset on various modelling strategies as outlined in Section 3. Table 9 shows VLMs across the three evaluator models - GPT, Llama and Mixtral over the various categories in the FlowVQA dataset. Figure 6 show category wise distribution of majority score for GPT-4V model. We also measure the average performance vs number of nodes in the flowcharts . The average is across all models and strategies and the graph is created after smoothening with an exponential weighted moving average (α = 0.4), as shown in figure 7.

heading: D Prompts for Generation
text: In this section we lists the prompts we use to query GPT-4 in various steps outlined in Section 2

heading: D.1 Flowchart Creation
text: First Step: Breaking down source text to structured summaries
Please provide a comprehensive structured summary, detailed step-by-step representation of the blog post below. Each step in the representation summary should be labeled with specific control codes that define its nature in the system. These codes include: START: Marks the first step. There must be only one start step and the whole summary representation must follow a single step-by-step structure. PROCESS: Indicates an ongoing process step. DECISION [IF] [ELSE]: Denotes a conditional decision-making step, with outcomes being either ’Yes’ or ’No’. For steps with multiple outcomes, break them down into smaller decision steps. INPUT: Introduces new variables or elements, like ingredients in a recipe. OUTPUT: Highlights the results, outputs or products of a step END: Marks all terminal points where the process ends or cannot go any further. ! Treat the blog instructions as a system. The system has some inputs and some output. Describe the entire detailed summary in that particular format. Be it the working of an ATM machine or the steps to create pizza from raw ingredients everything can be looked at like a system or pseudocode. Make sure not to miss any critical points in processes. ! Try to retain context and structure it well. ! Important. Design the decision/conditional steps to have only ’Yes’ or ’No’ outcomes and treat their text like questions. ! Start from a single start point, do not have multiple parallel starts, make sure things remain step-wise with conditionals, loops etc. Make the steps comprehensive and detailed, final output in markdown.
Second step: Converting structured summaries to Mermaid Scripts
Here is a detailed step-by-step summary tagged with detailed control codes for a blog post. Treat the step-wise summary as a system or a detailed pipeline. For this create a Mermaid Live Flowchart Script (flowchart TD) that is detailed, does not miss any key points, and captures all integral nodes perfectly. Treat the blog instructions and the flowchart as a system representation. Be it the working of an ATM machine or the steps to create pizza from raw ingredients everything can be looked at like a system. Objective: Convert Passed Structured Summary to detailed Mermaid Live Flowchart (flowchart TD) Control Codes for Assistance: START: Marks the first step. There must be only one start step and the whole summary representation must follow a single step-by-step structure. PROCESS: Indicates an ongoing procedure or action. Rectangle Shape. DECISION [IF] [ELSE]: Denotes a conditional decision-making step, with outcomes being either ’Yes’ or ’No’. For multiple outcomes, decompose into smaller decisions. Diamond Shape. INPUT: Introduces new elements or variables, akin to ingredients in a recipe. Parallelogram Shape. OUTPUT: Results, Outputs or end-products of a step. Parallelogram Shape. END: All points of no further go terminal. Oval Shape. Important Points 1. Treat the blog post instructions as a single system workflow or pipeline. 2. The system should include I/O, processes, decisions and terminals. 3. Ensure that the flowchart accurately depicts a real-life system flowchart, it should be contextually rich and practical for reference. 4. Maintain an optimal length for the flowchart not too long not too short, if there are multiple process steps in sequence you may consider combining them if the flowchart is too long. 5. Important! Design the decision steps to have only ’Yes’ or ’No’ outcomes. For steps with multiple outcomes, break them down into smaller decision steps. 6. Ensure a singular flow for the system, with all subroutines being direct components of the main system. 7. Ensure use of all flowchart symbols like rectangles, ovals, diamonds, circle, arrows etc. 8. Ensure the actual control codes are not mentioned in the flowchart nodes. 9. Verify flowchart syntax carefully
Sample of a small mermaid flowchart TD for reference: flowchart TD A(["Start"]) –> B["Process 1"] B –> C"Decision?" C –>|"Yes"| D["Process 2"] D –> E["Process 3"] E –> C C –>|"No"| F[/"Output or Input"/] F –> G(["End") Make sure to verify each point above before your output.

heading: D.2 Question Generation

heading: Fact Retrieval
text: Task: You will analyze a step-by-step structured summary and Mermaid Flowchart Representation of a blog post or code script. The blog post includes specific steps for handling tasks.
Your Role: As a fact-extractor and question creator, your objective is to locate factual content within the summary. Your goal is to construct several question-answer pairs that each relate to distinct and critical facts presented in the summary.

heading: Guidelines for Question Development:
text: 1. Begin by determining the presence and quantity of direct facts in the summary. If there are multiple concrete facts, especially quantitative ones, generate questions for each. If fewer facts are present, create fewer questions. The ideal question range is 2-4 questions. 2-3 for fewer facts and 3-4 for ones with more facts. 2. Focus on specific and relevant facts, asking questions like Who? What? Why? How much? How many? Emphasize quantitative facts over qualitative ones. 3. Questions should be straightforward, with answers in the summary. Avoid direct references to the summary or the blog post in your questions. 4. Ensure each question highlights a different fact from the summary.

heading: Answer Guidelines:
text: 1. Provide brief and clear answers. 2. Answers must be definitive, avoiding open-endedness. 3. Offer several paraphrased answers for each question. (A1, A2, A3)
Output Format: Present your questions and answers in a structured JSON format, following the provided example. Example Structure: - Output JSON: { "1": { "Q": "First Fact-based Question here", "A1": "", "A2": "", "A3": "", }, "2": { "Q": "", "A1": "", "A2": "", "A3": "", }, }

heading: Sample Question-Answer Pairs:
text: 1. What is the correct temperature for preheating the oven? A1. 80 Degrees Celsius A2. Preheat the oven to 80 degrees Celsius A3. ...
2. How long should crayons be left in the oven to melt? A1. 20 Minutes A2. Leave the crayons in the oven for about 20 minutes
3. What might tempt someone to peek? A1. Gifts A2. The temptation to peek at Christmas gifts
4. At what angle should the target be struck for full extension? A1. A 90-degree Angle
5. How long should the cork be left to cure? A1. Overnight A2. Cure the cork overnight
PS: Your Answers should be BRIEF, definitive and must offer three paraphrased versions A1, A2, A3. Make sure the questions are not too open ended and concrete.
Also DO NOT MENTION THE BLOG/STRUCTURED SUMMARY/SCRIPT IN THE QUESTION.

heading: Applied Scenario
text: Task: You will analyze a step-by-step structured summary and Mermaid Flowchart Representation of a blog post or code script. The blog post includes specific steps for handling tasks.
Your Role: As a complex situational question-answer generator, your task is to focus on the most interesting parts of the blog post’s structured summary. Create 2-4 Complex Question-Answer Pairs. Each pair should correspond to a different, interesting area of the structured summary of the blog post.

heading: Guidelines for Question Development:
text: - Focus on specific, relevant / crucial steps of the structured summary such as decisions, loops and other critical steps. - Craft situational questions that are creative, practical, and likely to occur in real life. - Ensure each question is directly related to a specific step mentioned in the blog post summary. - Important: The question must be created in a way that the answer to the question can be directly obtained or inferred from the structured summary but no logical thinking should be done to further process the information in steps. The blog post should only be used to construct the context of the situation, not to generate the question itself. - Important: Don’t explicitly mention the structured summary or blog post in the question. Assume the person answering can reference it. Create long complex situations and questions. - Provide suitable distractors in the question, complex stories, unique names, etc. Anything that makes the question more interesting, yet, answerable. - Make sure all questions attend separate parts of the structured summary.

heading: Answer Guidelines:
text: - Provide short, concise answers. - Answers should be definitive and not open-ended. - Offer several paraphrased answers for each question. (A1, A2, A3)
Output Format: Present your questions and answers in a structured JSON format, following the provided example. Example Structure: - Output JSON: { "1": { "Q": "First Applied Scenario Based Question", "A1": "Concise Answer 1", "A2": "", "A3": "", }, "2": { "Q": "", "A1": "", "A2": "", "A3": "", }, ... More Q/A Pairs here }

heading: Sample Questions:
text: 1. Ram, aged 45 years old, was going home from the office in his Minivan and his Minivan broke down on the way. He now wants to find a Minivan mechanic to get it repaired. He was trying to follow the given article, but being a little forgetful, he could not remember the age of his Minivan. He thought his warranty documents could help, Where should he try to find them? 2. Alice has decided to make custom fabric paint for a set of cotton t-shirts. She mixed equal parts of acrylic paint and a transparent gloss medium, but after testing on a swatch of cotton, the paint soaked through. What adjustment should she make to the paint mixture? 3. Selena has recurrent tonsil stones and her doctor has prescribed a course of antibiotics to address the issue. Unfortunately, the antibiotics weren’t successful and Selena hasn’t experienced any side effects or a relapse. What would her doctor’s advice likely be at this stage? 4. Mark, an aspiring VFX artist, is enthusiastic about networking to enhance his opportunities in the field. He wants to join an industry group like the Visual Effects Society (VES). However, he is uncertain about the number of VES members and their global distribution. How can Mark find this information to ensure the group’s relevance to his networking goals?
PS: Your Answers should be BRIEF, definitive and must offer three paraphrased versions A1, A2, A3. Make sure the questions are not too open ended and concrete.
Also DO NOT MENTION THE BLOG/STRUCTURED SUMMARY/SCRIPT IN THE QUESTION.

heading: Flow Referential
text: Task: You will analyze a step-by-step structured summary and Mermaid Flowchart Representation of a blog post or code script. This post details specific steps to handle certain tasks.
Your Role: As a capable flowchart path and flow analyzer your task is to focus on critical sub-areas of the processes and flowchart and create path-based questions from that subflowchart.

heading: Guidelines for Question Development:
text: - The first step is to decide on how many questions to create: If the flowchart is long and complex, break it down into smaller areas and create more questions (3). If the flowchart is short create fewer (2-3) but still good quality questions that would not be easy to answer directly. Focus on specific, relevant / crucial paths of the structured flowchart script and summary. - Create questions based on node information looking FORWARD, BACKWARD, IN THE MIDDLE, etc. Questions about crucial decisions taken in a possible path. - Craft questions about paths that are creative and hard but MUST HAVE A SINGLE DEFINITIVE TRUE ANSWER. - Important: Don’t explicitly mention the structured summary or flowchart in the question. Assume the person answering can reference it. Create long complex situations and questions. - Create questions about backtracking, future paths, conditionals, nodes or steps in the middle, etc. Anything that is interesting in a flowchart path. - IMPORTANT! It is very important that the current node/step or the node/path in question later is mentioned clearly. The rules for counting must be clearly mentioned. Look at the sample questions below to create questions.

heading: Answer Guidelines:
text: - Provide concise direct answers that are relevant to the question asked. - Answers should be definitive. - Offer several paraphrased answers for each question. (A1, A2, A3)
Output Format: Present your questions and answers in a structured JSON format, following the provided example. Example Structure: - Output JSON: { "1": { "Q": "First Path Based Question", "A1": "Concise Answer 1", "A2": "", "A3": "" }, "2": { "Q": "", "A1": "", "A2": "", "A3": "" }, } Sample Questions: 1. What is the second step, given my zeroeth step is taking a negative decision at "Bostik Spritzkork 3070 Available?"? 2. If I currently have to fill the mold with plaster, what decision must have I taken a few steps back and what is the condition present at that node? 3. What is the minimum number of steps required to reach ’Final Inspection’ from the "change job?" conditional? 4. Given the current zeroeth step is to close the top of the lid, what is the fifth step that I will be completing if I take the affirmative decision at any conditional present in between? 5. If at the current step the bathtub is not yet full and requires more water, what are the labels or descriptions of the fifth and seventh steps encountered when following the affirmative path from the current decision node? 6. How many steps are there from the initial "Start" node up to, but not including, the first decision point? In this count, the "Start" node is to be considered as the initial node or the ’zeroeth’ step. 7. Alice is preparing for a rock-themed party and recalls Scarlet’s unique style. She decides to start with a band T-shirt but is unsure whether to buy it online or at a concert. Given her limited budget, what should Alice’s decision be based on? 8. If a patient’s eligibility for tonsillectomy is currently being evaluated and they proceed with tonsillectomy following a positive recommendation, what would be the immediate next step, and what decision must have been made directly prior to this step?

heading: Answers:
text: 9. If I am currently at the ’Choose Show Audio Animation or press Control-A’ step, what was the decision made at the first decision point, and what is the immediate next step? A1: "The decision made was ’Yes’ at the ’Decision to edit audio effects?’ node, and the immediate next step is ’Audio effects editing mode activated’. A2: "At the ’Decision to edit audio effects?’ node, a positive decision was taken, leading to the next step of activating the audio effects editing mode. A3: "The first decision point led to a ’Yes’ outcome, and the following step is to activate the audio effects editing mode.
PS: Your Answers should be BRIEF, definitive and must offer three paraphrased versions A1, A2, A3. Make sure the questions are not too open ended and concrete. Also DO NOT MENTION THE BLOG/STRUCTURED SUMMARY/ FLOWCHART SCRIPT IN THE QUESTION. 1349

heading: E Prompts for Question-Answering
text: In this section we lists the prompts we use to query models

heading: E.1 Few-Shot-COT-with-Directives

heading: Few-Shot COTD
text: Examine the provided flowchart to answer the given question below. Here are some illustrative examples accompanied by a sequence of reasoning directives intended to stimulate analytical thought and elicit a rationale. These guidelines should facilitate the development of a rationale. Once a rationale has been formulated, proceed to present a conclusive final answer as in the examples. Question - Answer pairs with Tags. The exemplar questions given depend upon the type of question we are asking (Fact Retrieval/Applied Scenario/Flow Referential/Topological) Example: ... Q1. What temperature should the oven be preheated to for making the cake? Tags: Temperature, Oven, Preheating, Cake Reasoning: Take it step-by-step. Look for a node or cluster of nodes, in the flowchart that mention preheating the oven. Identify node / nodes that mention a ’preheating’. After locating relevant nodes extract final answer if already present or reason further to deduce the correct answer. A. 325 degrees Fahrenheit. ...

heading: Concerned Question to Ask
text: Example: What action is taken when the ’file’ is found to be not empty?

heading: Future Work
text: - Create questions about backtracking, future paths, conditionals, nodes or steps in the middle, etc. Anything that is interesting in a flowchart path. - IMPORTANT! It is very important that the current node/step or the node/path in question later is mentioned clearly. The rules for counting must be clearly mentioned. Look at the sample questions below to create questions