ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual
  Attribution
Authors: name: Kanika Goswami; name: Puneet Mathur; name: Ryan Rossi; name: Franck Dernoncourt
Published: 2025-02-03 02:00:51

Abstract: Large Language Models (LLMs) can perform chart question-answering tasks but
often generate unverified hallucinated responses. Existing answer attribution
methods struggle to ground responses in source charts due to limited
visual-semantic context, complex visual-text alignment requirements, and
difficulties in bounding box prediction across complex layouts. We present
ChartCitor, a multi-agent framework that provides fine-grained bounding box
citations by identifying supporting evidence within chart images. The system
orchestrates LLM agents to perform chart-to-table extraction, answer
reformulation, table augmentation, evidence retrieval through pre-filtering and
re-ranking, and table-to-chart mapping. ChartCitor outperforms existing
baselines across different chart types. Qualitative user studies show that
ChartCitor helps increase user trust in Generative AI by providing enhanced
explainability for LLM-assisted chart QA and enables professionals to be more
productive.