ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
Authors: name: Ahmed Masry; name: Megh Thakkar; name: Aayush Bajaj; name: Aaryaman Kartha; name: Enamul Hoque; name: Shafiq Joty
Published: 2024-07-04 22:16:40

Abstract: Given the ubiquity of charts as a data analysis, visualization, and
decision-making tool across industries and sciences, there has been a growing
interest in developing pre-trained foundation models as well as general purpose
instruction-tuned models for chart understanding and reasoning. However,
existing methods suffer crucial drawbacks across two critical axes affecting
the performance of chart representation models: they are trained on data
generated from underlying data tables of the charts, ignoring the visual trends
and patterns in chart images, and use weakly aligned vision-language backbone
models for domain-specific training, limiting their generalizability when
encountering charts in the wild. We address these important drawbacks and
introduce ChartGemma, a novel chart understanding and reasoning model developed
over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
trained on instruction-tuning data generated directly from chart images, thus
capturing both high-level trends and low-level visual information from a
diverse set of charts. Our simple approach achieves state-of-the-art results
across 5 benchmarks spanning chart summarization, question answering, and
fact-checking, and our elaborate qualitative studies on real-world charts show
that ChartGemma generates more realistic and factually correct summaries
compared to its contemporaries. We release the code, model checkpoints,
dataset, and demos at https://github.com/vis-nlp/ChartGemma.