ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in
  Complex Chart Question Answering
Authors: name: Rachneet Kaur, name: Nishan Srishankar, name: Zhen Zeng, name: Sumitra Ganesh, name: Manuela Veloso

Recent multimodal LLMs have shown promise in chart-based visual question
answering, but their performance declines sharply on unannotated charts, those
requiring precise visual interpretation rather than relying on textual
shortcuts. To address this, we introduce ChartAgent, a novel agentic framework
that explicitly performs visual reasoning directly within the chart's spatial
domain. Unlike textual chain-of-thought reasoning, ChartAgent iteratively
decomposes queries into visual subtasks and actively manipulates and interacts
with chart images through specialized actions such as drawing annotations,
cropping regions (e.g., segmenting pie slices, isolating bars), and localizing
axes, using a library of chart-specific vision tools to fulfill each subtask.
This iterative reasoning process closely mirrors human cognitive strategies for
chart comprehension. ChartAgent achieves state-of-the-art accuracy on the
ChartBench and ChartX benchmarks, surpassing prior methods by up to 16.07%
absolute gain overall and 17.31% on unannotated, numerically intensive queries.
Furthermore, our analyses show that ChartAgent is (a) effective across diverse
chart types, (b) achieve the highest scores across varying visual and reasoning
complexity levels, and (c) serves as a plug-and-play framework that boosts
performance across diverse underlying LLMs. Our work is among the first to
demonstrate visually grounded reasoning for chart understanding using
tool-augmented multimodal agents.