Flowchart-Based Decision Making with Large Language Models
Authors: Yuuki Yamanaka, Hiroshi Takahashi, Tomoya Yamashita
Accepted at: acl25

Flowchart-Based Decision Making with Large Language Models

Yuuki Yamanaka, Hiroshi Takahashi, Tomoya Yamashita
NTT
{yuuki.yamanaka, hiroshibm.tamahashi, tomoya.yamashita}@ntt.com

Abstract
Large language models (LLMs) are widely used for conversational systems, but they face significant challenges in interpretability of dialogue flow and reproducibility of expert knowledge. To address this, we propose a novel method that extracts flowcharts from dialogue data and incorporates them into LLMs. This approach not only makes the decision-making process more interpretable through visual representation, but also ensures the reproducibility of expert knowledge by explicitly modeling structured reasoning flows. By evaluating on dialogue datasets, we demonstrate that our method effectively reconstructs expert decision-making paths with high precision and recall scores. These findings underscore the potential of flowchart-based decision making to bridge the gap between flexibility and structured reasoning, making chatbot systems more interpretable for developers and end-users.

1 Introduction
The rapid advancement of large language models (LLMs) has enabled their widespread adoption in real-world applications, including customer support, medical diagnosis, and security incident response (Sun et al., 2024; Hu et al., 2024; Wang et al., 2024; Liu, 2024; Hays and White, 2024; Ouyang et al., 2022). In particular, conversational agent systems use LLMs for more flexible and adaptive interactions compared to traditional rule-based approaches (Xi et al., 2025; Adamopoulou and Moussiades, 2020; Hussain et al., 2019). However, two fundamental challenges remain: interpretability of dialogue flows and reproducibility of expert knowledge.

First, LLM-driven dialogues lack interpretability, making it difficult for system administrators to control and manage interactions. While retrieval-augmented generation (RAG) and fine-tuning can improve LLM knowledge, the decision-making process remains opaque (Lewis et al., 2020; Gao et al., 2023; Barnett et al., 2024; Ding et al., 2023; Rafailov et al., 2024; Ovadia et al., 2023).

Second, the reproducibility of expert knowledge is critical in high-stakes domains such as medical diagnosis and troubleshooting. Experts rely on structured decision-making processes, but existing LLM-based conversational agent systems lack explicit modelling of these reasoning flows, leading to inconsistencies. Moreover, expert knowledge is constantly evolving, making it challenging to keep dialogue systems up-to-date with the latest domain expertise (Hudeček and Dušek, 2023; Sekulić et al., 2024; Ulmer et al., 2024).

To address these challenges, we propose a novel method that extracts flowcharts from dialogue data and incorporates them into LLMs. This approach not only makes the decision-making process more interpretable through visual representation, but also ensures the reproducibility of expert knowledge by explicitly modeling structured reasoning flows. The flowchart-based representation allows system administrators to easily monitor, update, and maintain the dialogue system while preserving the expert decision-making patterns embedded in the training data.

Our method consists of three main steps: extracting key decision points from dialogues by identifying expert questions and suggestions, structuring them into logical decision flows, and constructing a refined flowchart by aggregating these flows.

Contributions. The key contributions of this paper are:
• We propose a novel method for extracting and visualizing expert decision flows from dialogue data using flowcharts.
• We evaluate the reproducibility of expert decision flows using the Flo-dial dataset, measuring precision, recall, and F1-score.
2 Proposed Method

Flowcharts are effective tools for visualizing complex processes (Gilbreth and Gilbreth, 2023). Our method extract decision-making flows as flowcharts from dialogue data between expert operators and users, as illustrated in Figure 1. This approach is particularly effective in high-stakes applications such as disease diagnosis and troubleshooting, where operators identify and resolve users’ issues. Our method consists of the following three steps: (1) Question and Suggestion Extraction, (2) Dialogue Flow Extraction, and (3) Flowchart Construction.

Step 1: Question and Suggestion Extraction
The first step involves identifying key decision points within dialogues, specifically questions posed by experts and suggestions provided in response to user inputs.

Each dialogue is processed using an LLM to categorize utterances into two types: (1) Questions: Information-seeking utterances that guide the user (e.g., “Do you have a fever?”). (2) Suggestions: Recommendations, diagnoses, or conclusions provided by the expert (e.g., “You might have a cold.”). The classification process also filters out irrelevant statements such as greetings and acknowledgments.

Since this classification is conducted independently for each dialogue, the extracted lists of questions and suggestions often contain significant semantic redundancy. To address this, similar questions (e.g., “Are you experiencing dizziness?” and “Do you feel lightheaded?”) are grouped using an LLM-based merging process. Suggestions with equivalent meanings but different expressions are also normalized to ensure consistency in the flowchart representation. This allows us to extract questions and suggestions from all dialogues based on a unified perspective.

The prompts used for extracting and merging questions, as well as their equivalents for suggestions, are summarized in Table 1.

Step 2: Dialogue Flow Extraction
After identifying questions and suggestions, the next step is to structure their connections within dialogues. Using the extracted elements from Step 1, unstructured dialogues are converted into ordered sequences of questions, user responses, and operator suggestions.

First, each question posed by the operator in a dialogue is mapped to the corresponding question ID from the extracted list. Next, for each identified question, the user’s response is extracted and normalized into short categorical answers such as "Yes", "No", or "I don’t know". This normalization ensures that decision branches in the flowchart are clearly defined based on user input. Finally, we identify the operator’s suggestion following each sequence of questions and responses, using the extracted list of suggestions. By structuring dialogues in this manner, we convert conversational interactions into an ordered series of questions, user responses, and expert suggestions, creating a structured representation suitable for flowchart construction.

The prompts used for mapping questions, user responses, and suggestions are summarized in Table 2.

Step 3: Flowchart Construction
In this step, the structured sequences of questions, user responses, and expert suggestions from Step 2 are aggregated to form the final flowchart. The goal is to create a decision tree that accurately represents expert decision-making while removing infrequent transitions for clarity.

The flowchart is represented as a directed graph, where nodes correspond to extracted questions or suggestions, and edges represent transitions based on user responses. The construction process begins by sorting questions based on their frequency as parent nodes, ensuring that frequently asked questions are prioritized. The most common question is linked to a root node ("ROOT"), establishing the starting point of the flowchart.

For each parent node, child nodes are determined using recorded transitions from the dialogue data. These transitions are sorted by frequency, and low-frequency transitions are discarded to reduce noise. User response types (e.g., "Yes", "No", or "I don’t know") are explicitly recorded as edge attributes to clarify decision paths. If a child node represents a suggestion rather than a follow-up question, it is registered as a terminal node, ensuring a clear distinction between inquiry and decision points.

Throughout this process, cyclic dependencies are checked and removed to maintain a well-structured hierarchy. Additionally, if multiple edges exist between the same nodes with different response labels, they are merged into a single edge while preserving all response variations. This ensures that the resulting flowchart remains compact and interpretable while effectively reconstructing expert reasoning patterns. The complete flowchart construction process is outlined in Algorithm 1.
(a) Question and suggestion extraction        (b) Dialogue flow extraction        (c) Flowchart construction

Figure 1: Overview of the proposed method: Our approach (a) first extracts questions and suggestions from dialogue data, (b) then structures these elements into sequential flows, and (c) finally aggregates them into a comprehensive flowchart, enabling systematic transformation of expert dialogues into structured decision flowchart.

Table 1: Prompts used for question extraction. Similar templates are used for suggestion extraction as well.

<table>
  <tr>
    <th>Task</th>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td><b>Question Extraction</b></td>
    <td>Refer to the following conversation between USER and OPERATOR to identify questions that the operator should ask the user, together with sample questions. The questions should be as closed-ended as possible, eliciting information useful for assisting the user. Split multi-topic questions into separate simple ones.<br>## conversation {text}</td>
  </tr>
  <tr>
    <td><b>Question Merging</b></td>
    <td>Merge the two provided sets of questions into a single, consolidated set.<br>- Include all unique questions from both sets.<br>- Merge semantically similar questions into a single entry.<br>- Choose the simpler and clearer version for merged questions.<br>## questions 1 {questions1}<br>## questions 2 {questions2}</td>
  </tr>
</table>

3 Experiments

We conducted experiments using GPT-4o-2024-08-06 as the LLM to evaluate the effectiveness of our proposed method (Achiam et al., 2023).

3.1 Dataset

The FloDial dataset consists of 2,738 troubleshooting dialogues between users and customer support operators, collected via Amazon Mechanical Turk (Raghu et al., 2021). These dialogues are based on 12 distinct troubleshooting flowcharts covering technical issues with cars and laptops. The dataset is released under the CDLA-Sharing-1.0 license.

On average, each flowchart corresponds to 228.2 dialogues and contains 15.25 unique decision paths, representing different troubleshooting scenarios. Dialogues have an average of 7.19 turns, where users describe initial fault symptoms, and operators diagnose issues through targeted questions.

Our experiments focused on reconstructing flowcharts from these dialogues.

3.2 Evaluation Metrics

To evaluate our method, we compare the decision-making paths in the generated and ground truth FloDial flowcharts, measuring their overlap to assess structural similarity.

We extract all possible paths from both the ground truth and generated flowcharts, where each path represents a sequence of questions, responses, and suggestions. These paths are then compared using an LLM-based similarity assessment to determine whether they represent the same problem-solving flow (Zheng et al., 2023; Gu et al., 2024). We used GPT-4o-2024-08-06 for this similarity assessment; the prompt is shown in Appendix A.

To quantify the accuracy of the generated flowcharts, we compute precision (\( P = M/G \)), recall (\( R = M/T \)), and F1 score (\( F_1 = 2PR/(P + R) \)), where \( M \) is the number of matched paths, \( G \) is the total number of generated paths, and \( T \) is the total number of ground truth paths. Matched paths are those generated paths that were successfully matched to a ground truth path.
Table 2: Prompts used for dialogue flow extraction.

<table>
  <tr>
    <th>Task</th>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>Question Mapping</td>
    <td>Please select the IDs of the question contained in the following utterance from the list of questions. A single utterance can contain multiple questions with different meanings, or none at all. If no questions are included, respond with an empty list.<br>## Utterance {utterance}<br>## Questions {question_list}</td>
  </tr>
  <tr>
    <td>User Response Mapping</td>
    <td>Please extract the user’s response to the following question below based on the user’s most recent utterances in the following conversation. The answer should be as concise as possible (Yes, No, I don’t know, etc.).<br>## Conversation {conversation}<br>## Question {question}</td>
  </tr>
  <tr>
    <td>Suggestion Mapping</td>
    <td>Refer to the following conversation and select the solution IDs proposed by the OPERATOR to the USER from the list of solutions.<br>## Conversation {conversation}<br>## Solutions {solution_list}</td>
  </tr>
</table>

Algorithm 1 Flowchart Construction

Require: Sequences \((Q, A, C)\) where
\(Q\): question,
\(A\): user response,
\(C\): next question or suggestion
Ensure: Directed Graph \(G\) representing the flowchart
\(G \leftarrow\) initialize graph
\(cnt \leftarrow\) count occurrences of \((Q, A, C)\)
\(pcent \leftarrow\) count occurrences of each \(Q\) as parent
\(plist \leftarrow\) sort \(pcent\) by frequency (desc)
add_edge(G, "ROOT", plist[0], "START")
for each \(p\) in \(plist\) do
  \(ccnt \leftarrow\) filter \(cnt\) where key = \((p, *, *)\)
  \(clist \leftarrow\) sort \(ccnt\) by frequency (desc)
  for each \((p, r, c)\) in \(clist\) do
    if \(c \notin G\) or no cycle\((G, c, p)\) then
      add_node(G, c)
      if edge\((p, c)\) exists then
        update_edge(G, p, c, r)
      else
        add_edge(G, p, c, r)
      end if
    end if
  end for
end for
return \(G\)

Table 3: Evaluation results of flowchart reconstruction.

<table>
  <tr>
    <th>Flowchart</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1-score</th>
  </tr>
  <tr>
    <td>Brake Problem</td>
    <td>0.750</td>
    <td>0.947</td>
    <td>0.837</td>
  </tr>
  <tr>
    <td>Car Electrical Failure</td>
    <td>0.722</td>
    <td>0.929</td>
    <td>0.813</td>
  </tr>
  <tr>
    <td>Car Steering</td>
    <td>0.900</td>
    <td>0.947</td>
    <td>0.923</td>
  </tr>
  <tr>
    <td>Car Won’t Start</td>
    <td>0.708</td>
    <td>1.000</td>
    <td>0.829</td>
  </tr>
  <tr>
    <td>Engine Overheats</td>
    <td>0.809</td>
    <td>1.000</td>
    <td>0.894</td>
  </tr>
  <tr>
    <td>Laptop Battery</td>
    <td>0.666</td>
    <td>1.000</td>
    <td>0.799</td>
  </tr>
  <tr>
    <td>Laptop Drive</td>
    <td>0.800</td>
    <td>0.800</td>
    <td>0.800</td>
  </tr>
  <tr>
    <td>Laptop Overheating</td>
    <td>0.846</td>
    <td>0.846</td>
    <td>0.846</td>
  </tr>
  <tr>
    <td>LCD Problem</td>
    <td>0.714</td>
    <td>1.000</td>
    <td>0.833</td>
  </tr>
  <tr>
    <td>Power</td>
    <td>0.722</td>
    <td>0.867</td>
    <td>0.788</td>
  </tr>
  <tr>
    <td>Ticking</td>
    <td>0.900</td>
    <td>0.750</td>
    <td>0.818</td>
  </tr>
  <tr>
    <td>Wireless</td>
    <td>0.666</td>
    <td>0.933</td>
    <td>0.777</td>
  </tr>
  <tr>
    <th>Average</th>
    <th>0.767</th>
    <th>0.918</th>
    <th>0.830</th>
  </tr>
</table>

3.3 Results

The evaluation results of our flowchart reconstruction method are summarized in Table 3. We assessed the precision, recall, and F1-score for each of the 12 flowcharts in the FloDial dataset.

Overall, our method demonstrated a strong ability to capture the complete decision-making paths present in the ground truth flowcharts, as evidenced by the high recall scores. In several cases, such as "Engine Overheats", the recall reached a perfect 1.000, indicating that all relevant decision paths were successfully captured. This highlights our method’s effectiveness in ensuring comprehensive coverage of expert reasoning.

However, the precision scores were relatively lower, suggesting that the generated flowcharts included some extraneous paths. This is likely due to challenges in merging similar questions and suggestions, leading to redundant nodes. For example, the "Laptop Battery" flowchart exhibited a precision of 0.666, indicating room for improvement in distinguishing between similar decision paths.

The F1-scores, consistently above 0.80 across various flowcharts, demonstrate the effectiveness of the proposed method in reconstructing expert decision-making processes. While there is room for improving precision by reducing redundant paths, the high recall scores validate our approach’s ability to capture comprehensive troubleshooting knowledge. This structured representation of dialogue flows enables the development of more explainable and reliable chatbot systems. An example of a generated flowchart, along with the corresponding gold-standard flowchart for comparison, is provided in Appendix B.
3.4 Human Evaluation
To validate the reliability of our LLM-based similarity assessment used for evaluating the match between generated and reference flowchart paths, we conducted a limited human evaluation.

Specifically, we sampled 3 representative decision paths for each of the 12 flowcharts in the FloDial dataset (totaling 36 pairs), and manually judged whether each generated path semantically matched the corresponding reference path. The human annotation was performed by one of the authors with expertise in dialogue systems and task-oriented modeling.

The human judgments were compared with the LLM-based similarity assessment. We found that GPT-4o’s judgments aligned with the human annotations in 33 out of 36 cases, resulting in an agreement rate of 91.7%. Qualitatively, discrepancies arose mostly in borderline cases where synonymous but structurally different sequences were present. This result supports the reliability of using LLMs for semantic path matching, consistent with prior studies (Zheng et al., 2023) demonstrating high agreement between LLM-based evaluators and human annotators.

Despite its limited scale and single annotator, the evaluation suggests that our automatic method aligns well with human judgment.

4 Conclusion
We introduced a method for extracting and visualizing expert decision flows from dialogue data using flowcharts, addressing interpretability and reproducibility challenges in conversational agent systems. Utilizing the FloDial dataset, our approach effectively captures comprehensive decision paths, as evidenced by high recall scores.

These findings underscore the potential of flowchart-based decision making to enhance the transparency and reliability of chatbot systems, making them more interpretable for developers and end-users.

5 Limitations
While our method for extracting and visualizing expert decision flows using flowcharts offers significant advancements in interpretability and reproducibility, several limitations must be acknowledged.

Firstly, our approach does not fully account for scenarios where questions can be asked in any order, such as asking for a name and gender. Although Step 3 prioritizes statistically frequent patterns, it does not explicitly incorporate the concept of "order independence," which may lead to inefficiencies. Introducing a flow alignment step between Steps 2 and 3 could address this limitation.

Secondly, the method focuses on extracting structured decision-making processes, which may not fully capture the nuances of complex dialogues. This could limit its effectiveness in domains where context dependency is high, potentially restricting its applicability in such areas.

Thirdly, our experiments were conducted exclusively with GPT-4o, which may not generalize to other models. The task of merging questions is particularly challenging for LLMs, and achieving high accuracy may require models with capabilities similar to GPT-4o. Additionally, our evaluation was limited to the FloDial dataset, as it is the only dataset with corresponding ground truth flowcharts. This dependency on the quality and comprehensiveness of dialogue data means that incomplete datasets may result in flowcharts that do not fully reflect expert reasoning processes.

Finally, we have not yet evaluated the performance of chatbots incorporating these flowcharts. While we assume that accurate flowcharts facilitate the creation of guided chatbots, future work should include performance metrics such as Winrate to validate this assumption across other datasets like MultiWOZ (Budzianowski et al., 2018).

Despite these limitations, our method provides a valuable framework for enhancing the transparency and reliability of chatbot systems, offering a structured approach to decision-making that can be further refined and expanded upon in future research.

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Eleni Adamopoulou and Lefteris Moussiades. 2020. Chatbots: History, technology, and applications. Machine Learning with Applications, 2:100006.

Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. Preprint, arXiv:2401.05856.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadhan, and Milica Gašić. 2018. Multiwoz—a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220–235.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Frank B. Gilbreth and L. M. Gilbreth. 2023. Process charts: First steps in finding the one best way to do work. Transactions of the American Society of Mechanical Engineers, 43:1029–1043.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.

Sam Hays and Jules White. 2024. Employing llms for incident response planning and review. Preprint, arXiv:2403.01271.

Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. 2024. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Vojtěch Hudeček and Ondřej Dušek. 2023. Are llms all you need for task-oriented dialogue? arXiv preprint arXiv:2304.06556.

Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents/chatbots classification and design techniques. In Web, Artificial Intelligence and Network Applications: Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications (WAINA-2019) 33, pages 946–956. Springer.

Patrick Lewis, Ethan Perez, Aleksandra Pikuts, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Zefang Liu. 2024. Multi-agent collaboration in incident response with large language models. Preprint, arXiv:2412.00652.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Dinesh Raghu, Shantanu Agarwal, Sachindra Joshi, and Mausam. 2021. End-to-end learning of flowchart grounded task-oriented dialogs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.

Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavičius, André Ferreira Manso, and Roland Mathis. 2024. Reliable llm-based user simulator for task-oriented dialogue systems. arXiv preprint arXiv:2402.13374.

Yiyou Sun, Junjie Hu, Wei Cheng, and Haifeng Chen. 2024. Chatbot meets pipeline: Augment large language model with definite finite automaton. arXiv preprint arXiv:2402.04411.

Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, and Yi Zhang. 2024. Bootstrapping llm-based task-oriented dialogue agents via self-talk. arXiv preprint arXiv:2401.05033.

Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, and Dinggang Shen. 2024. Interactive computer-aided diagnosis on medical image using large language models. Communications Engineering, 3(1):133.

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2025. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
Figure 2: Example of a generated flowchart for car steering troubleshooting. The flowchart captures the main decision points and solutions.

Figure 3: Ground truth flowchart for car steering troubleshooting.

A  Prompts used for evaluation

Table 4: Prompt used for LLM-based path similarity evaluation.

<table>
  <tr>
    <th>Prompt Template</th>
  </tr>
  <tr>
    <td>Determine if two paths represent the same problem-solving flow. If similar questions and answers are exchanged in the same order and the final solution is similar, regard them as equivalent.<br>Compare the following two troubleshooting flows and determine if they represent the same problem-solving process.<br>## Path 1: {path1}<br>## Path 2: {path2}</td>
  </tr>
</table>

To assess the similarity between generated and reference decision paths, we used GPT-4o to perform LLM-based matching. The prompt used for this evaluation is shown in Table 4. It was consistently applied across all similarity assessments, including the subset used for comparison with human judgments.

B  Example of a generated flowchart

To demonstrate the effectiveness of our method, Figure 2 presents an example flowchart generated for car steering troubleshooting, which achieved the highest F1 score in our evaluation experiments. As a comparison, Figure 3 shows the ground truth flowchart for car steering troubleshooting.

Reviewer paper summaries: The paper describes experiments into recovering decision-making flowcharts from dialogues which are governed by them, using LLM-based methods to identify question/answer/suggestion elements and sequences, and group them together to form flowcharts. Results show good recall, although less impressive precision. The method is tested with one LLM on one dataset (FloDial).
Summary of strengths: The task seems like a novel one, taking what seems essentially a reverse perspective on the more normal task from previous literature of using flowcharts to learn decision-making dialogue systems (here, the dialogue is used to estimate the original flowchart). This could potentially allow system developers a greater level of interpretability from such dialogues, and allow more reliable troubleshooting / development.
The evaluation method seems sound, and gives quite good F-scores, with particularly high precision, for recovering flowcharts.
Summary of weaknesses: The approach is only tested on a dialogue dataset in which the dialogue data was explicitly generated from a flowchart (by giving Amazon Mechanical Turk workers the task to paraphrase question sequences in the flowcharts). While it is therefore good to see that we can recover these flowcharts from the dialogue transcripts, it raises the question of how effective the approach would be on more general dialogue that is less directly related to (directly generated from) the original flowcharts. Many decision-making processes rely on knowledge that is not directly encoded this way, or where such knowledge is implicit in the human agent's understanding rather than explicitly encoded as a flowchart; and many dialogues are not generated by such explicit flow-chart-to-dialogue processes as the FloDial dataset was. It's quite possible that this approach would be helpful there too, but it might also introduce many errors.

The performance has lower precision than recall; while the high recall shows the effectiveness of finding correct reasoning paths, the low precision suggests that many invalid reasoning paths may be being identified. This could be a real problem, if these paths can lead to invalid conclusions, and bad suggestions to the user. Some evaluation (or at least discussion) of these seems important.

I also had a minor worry over the evaluation method, which relies on a LLM to match the extracted flowchart paths against the gold standard. Do we know how reliable this is?
Comments, suggestions and typos: I would be very interested to see deeper analysis of the low-precision cases: e.g. an evaluation of how often non-matching paths are associated with outcomes that are invalid or undesirable, as opposed to valid ones that just didn't get exhibited in the dataset.

I wasn't sure what "reproducibility of expert knowledge" really means here - having read the paper, I guess the degree to which a system / its dialogue behaviour actually embodies/follows an expert-generated gold-standard flowchart. But I'm not sure the word "reproducibility" really gets that across. Also not sure that the paper has shown that this method really "ensures the reproducibility", given the low precision in many cases.

084 extract[s]
023, 272 advance[ment]
Overall assessment scores: 3