Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
Authors: name: Victor Carbune, name: Hassan Mansoor, name: Fangyu Liu, name: Rahul Aralikatte, name: Gilles Baechler, name: Jindong Chen, name: Abhanshu Sharma

Vision-language models (VLMs) are achieving increasingly strong performance
on multimodal tasks. However, reasoning capabilities remain limited
particularly for smaller VLMs, while those of large-language models (LLMs) have
seen numerous improvements. We propose a technique to transfer capabilities
from LLMs to VLMs. On the recently introduced ChartQA, our method obtains
state-of-the-art performance when applied on the PaLI3-5B VLM by
chen2023pali3, while also enabling much better performance on PlotQA
and FigureQA.
  We first improve the chart representation by continuing the pre-training
stage using an improved version of the chart-to-table translation task by
liu2023deplot. We then propose constructing a 20x larger dataset than
the original training set. To improve general reasoning capabilities and
improve numerical operations, we synthesize reasoning traces using the table
representation of charts. Lastly, our model is fine-tuned using the multitask
loss introduced by hsieh2023distilling.
  Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B
without using an upstream OCR system, while keeping inference time constant
compared to the PaLI3-5B baseline. When rationales are further refined with a
simple program-of-thought prompt chen2023program, our model outperforms
the recently introduced Gemini Ultra and GPT-4V.