ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
Authors: name: Ahmed Masry, name: Megh Thakkar, name: Aayush Bajaj, name: Aaryaman Kartha, name: Enamul Hoque, name: Shafiq Joty

Given the ubiquity of charts as a data analysis, visualization, and
decision-making tool across industries and sciences, there has been a growing
interest in developing pre-trained foundation models as well as general purpose
instruction-tuned models for chart understanding and reasoning. However,
existing methods suffer crucial drawbacks across two critical axes affecting
the performance of chart representation models: they are trained on data
generated from underlying data tables of the charts, ignoring the visual trends
and patterns in chart images, and use weakly aligned vision-language backbone
models for domain-specific training, limiting their generalizability when
encountering charts in the wild. We address these important drawbacks and
introduce ChartGemma, a novel chart understanding and reasoning model developed
over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
trained on instruction-tuning data generated directly from chart images, thus
capturing both high-level trends and low-level visual information from a
diverse set of charts. Our simple approach achieves state-of-the-art results
across 5 benchmarks spanning chart summarization, question answering, and
fact-checking, and our elaborate qualitative studies on real-world charts show
that ChartGemma generates more realistic and factually correct summaries
compared to its contemporaries. We release the code, model checkpoints,
dataset, and demos at https://github.com/vis-nlp/ChartGemma.