Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic
  Agents
Authors: name: Manan Suri; name: Puneet Mathur; name: Nedim Lipka; name: Franck Dernoncourt; name: Ryan A. Rossi; name: Vivek Gupta; name: Dinesh Manocha
Published: 2025-06-02 06:02:41

Abstract: Flowcharts are a critical tool for visualizing decision-making processes.
However, their non-linear structure and complex visual-textual relationships
make it challenging to interpret them using LLMs, as vision-language models
frequently hallucinate nonexistent connections and decision paths when
analyzing these diagrams. This leads to compromised reliability for automated
flowchart processing in critical domains such as logistics, health, and
engineering. We introduce the task of Fine-grained Flowchart Attribution, which
traces specific components grounding a flowchart referring LLM response.
Flowchart Attribution ensures the verifiability of LLM predictions and improves
explainability by linking generated responses to the flowchart's structure. We
propose FlowPathAgent, a neurosymbolic agent that performs fine-grained post
hoc attribution through graph-based reasoning. It first segments the flowchart,
then converts it into a structured symbolic graph, and then employs an agentic
approach to dynamically interact with the graph, to generate attribution paths.
Additionally, we present FlowExplainBench, a novel benchmark for evaluating
flowchart attributions across diverse styles, domains, and question types.
Experimental results show that FlowPathAgent mitigates visual hallucinations in
LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our
proposed FlowExplainBench dataset.