Enhancing Question Answering on Charts Through Effective Pre-training Tasks

Authors: ashim  gupta
vivek  gupta
shuo  zhang
yujie  he
ning  zhang
shalin  shah
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-15 15:37:16

Abstract: To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings indicate that existing models particularly underperform in answering questions related to the chart's structural and visual context, as well as numerical information. To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions. We evaluate our pre-trained model (called MatCha-v2) on three chart datasets - both extractive and abstractive question datasets - and observe that it achieves an average improvement of 1.7 % over the baseline model.

Main paper: 
 introduction: 
 Understanding and extracting insights from charts and plots is a fundamental aspect of data analysis that is critical for various domains, including finance, healthcare, and scientific research. To bridge the gap between raw data and actionable knowledge, question answering (QA) systems tailored for charts and plots have gained increasing attention in recent years. These systems aim to enable users to pose natural language questions about the content of visual data representations, such as bar graphs, line charts, and scatterplots, and receive informative answers. However, despite * Work done during summer internship at Bloomberg. the remarkable progress made in developing such QA systems, there remains a significant challenge: their performance often falls short when subjected to human-generated questions. This discrepancy between machine performance and human expectations underscores the need for a comprehensive investigation into the limitations of existing models and the development of strategies to address their shortcomings.To shed light on the shortcomings of current chart-based QA systems (Masry et al., 2022) , this paper first undertakes a detailed checklist-based behavioral analysis (Ribeiro et al., 2020; Bhatt et al., 2021; Rogers et al., 2021) . Choosing the models trained on the ChartQA dataset (Masry et al., 2022) for the representative case study, we systematically evaluate model responses against examples constructed from the checklist of expected behaviors, allowing us to pinpoint the areas in which these models falter. The checklist is designed to assess various dimensions of chart-based question answering, including the ability to interpret the structural and visual context of the chart, handle questions requiring numerical reasoning, and offer meaningful insights that align with human expectations. Concretely, our analysis reveals that current stateof-the-art models perform poorly on two types of questions. The first type of questions are those that pertain to the visual aspects of a chart (e.g., color), while the second type are questions that require application of numerical operators to numerical items present in the chart (e.g., average, etc.).To address these two shortcomings, we propose a set of three pre-training tasks: Visual-Structure prediction, Summary Statistics prediction, and Numerical Operator prediction. Through evaluation on three chart question answering datasets, we find that models fine-tuned after using this pre-training outperform the baseline model by more than 1.7 percentage points in an absolute sense.In summary, our contributions are: 1) We per-form a checklist-based behavioral analysis of the current state-of-the-art chart question answering systems to identify issues and challenges faced by such systems. 2) We propose three simple, yet effective, pre-training tasks to address these issues.The new pre-trained model outperforms the baseline systems significantly. 
behavioral analysis via checklist: 
 We first describe the procedure for constructing the checklist for chart-based question answering systems and then discuss the results and observations.To perform the perturbation analysis for the chartbased models, we choose the popular ChartQA dataset, where the objective is to answer questions based on the information provided in the accompanying charts. Numerical QA Last, the Numerical QA templates are tailored to assess the model's proficiency in answering both simple and complex numerical questions related to the data points present in the charts.1 https://github.com/owid/owid-grapher-pyIn total, we have 33 distinct QA templates belonging to these three classes. A few examples are shown in Table 1 and detailed examples are presented in the Appendix in Table ??.As mentioned earlier, our focus is to perform behavioral analysis using the checklist for the chartbased models on the ChartQA dataset to first understand their extent of chart understanding and then pinpoint areas where these models require improvement.Models Evaluated Our focus in this work entails the two large, recently introduced chart pretrained models in MatCha, and DePlot + LLM. While MatCha is an end-to-end chart-to-text pretrained model, DePlot + LLM is a pipelined approach where DePlot first converts an input chart into its textual representation in the form of a table and then performs few-shot question answering via a Large Language Model (LLM). In our experiments, we use the FLAN-UL2 (Tay et al., 2022) with 20 billion parameters as the LLM for DePlot + LLM evaluation. We do not evaluate models such as VisionTapas (Masry et al., 2022) , Vl-T5 (Cho et al., 2021) since they are harder to work with than the two models we use.Evaluation Metric For each of the templates we use, we measure the model's Failure Rate, i.e., the number of examples where model prediction does not match the expected/gold output. We present examples of some failure cases along with failure rates in Table 1 .The results are shown in Table 1 . Notably, the Structural & Visual templates exhibit alarmingly high failure rates, prompting an indepth investigation into the underlying causes. One plausible explanation for these high failure rates is the absence of explicit enforcement of structural and visual information in the pre-training tasks for models like MatCha and DePlot. This absence underscores a critical gap in the models' understanding of fundamental chart structures and visual elements, posing a significant challenge in their interpretation of complex data visualizations.In stark contrast, the models demonstrate a significantly higher proficiency in handling templates focused on data extraction. The lower failure rates observed in this category highlight the models' capability to accurately extract data points from diverse charts, indicating a relatively robust performance in tasks requiring precise information retrieval.We also notice intriguing disparities in the models' performance concerning numerical QA templates. While the failure rates soar for complex mathematical operations, such as intricate calculations involving multiple operators, a marked improvement is observed in tasks involving simpler operations like finding maximum or minimum values. This nuanced discrepancy suggests that while the models struggle with advanced numerical reasoning, they exhibit a more stable grasp on elementary mathematical concepts. This observation not only sheds light on the specific challenges faced by these models in handling complex mathematical operations but also underscores their potential strengths in addressing simpler, more straightforward numerical queries. 
proposed pre-training tasks: 
 As discussed earlier, we proposed a comprehensive approach to address the challenges identified through checklist-based analysis. In this section, we introduce three distinct pre-training tasks designed to enhance the MatCha model's chart understanding capabilities.Visual Structure Prediction The first task, termed Visual Structure Prediction, demands the model predict intricate details of input charts, encompassing chart types (such as bar, line, etc.), colors associated with chart entries, and even chart titles. We pre-train both models on the charts extracted from the training data of the ChartQA dataset. We continue pre-training from the initial pretrained variants and pre-train each model for only one epoch as we saw significant reduction in accuracy on the validation set. In addition, we use batch size of six ( 6) and use all the other hyperparameters as suggested by (Liu Implementation Details and Computing Infrastructure Used We use batch size of six (6) to train the MatCha models due to computational constraints. Additionally, all the models that require training (e.g., MatCha) were trained up to five epochs. All of our experiments required access to GPU accelerators. We ran our experiments on two machines: NVIDIA Tesla V100 (16 GB VRAM) and Tesla V100 (32 GB VRAM). We did not experiment with VisionTapas (Masry et al. ( 2022)) as we could not run the publicly released implementation due to a missing dependency. 2 We train all our models using the Transformers library from Hugging Face (Wolf et al., 2019) with the PyTorch back-end (Paszke et al., 2019) .Improved QA on Checklist We first measure the effectiveness of these two pre-training tasks on the proposed checklist. As shown in Table 1 , the proposed variations of the two models (called v2 versions) significantly decrease failure rates across all template and question types.To assess the effectiveness of our proposed pretraining tasks on actual question answering tasks, we conduct experiments on the ChartQA dataset. Furthermore, we extend our evaluation to a different question-answering dataset named PlotQA, wherein charts and associated questions are derived from disparate sources. This cross-dataset evaluation enables us to gauge the model's adaptability and generalizability beyond its original training data.Both ChartQA and PlotQA are extractive question-answering datasets, where an answer is retrieved by combining entries from the chart. In addition, we explore the model's performance in abstractive question-answering, a more complex task necessitating detailed descriptive responses, using the OpenCQA dataset. Notably, both ChartQA and PlotQA focus on extractive question-answering, demanding precise extraction of information from the source data, while OpenCQA necessitates the generation of more extensive, contextually rich answers. Through these evaluations, we gain a holistic understanding of the MatCha model's capabilities, from basic chart comprehension to nuanced and elaborate question answering across diverse datasets and question types.Datasets We evaluate the effectiveness of our proposed pre-training methods on the three chartrelated datasets. Here is a brief summary of each dataset, along with comments on how we use it to evaluate a model's chart understanding capabilities: Chart QA The Chart Question Answering (QA) (Masry et al., 2022) dataset, as the name suggests, is a natural language query and answer generation dataset with one key difference from most NLP datasets -it also provides the visual representation of data or the chart image, which contain richer information such as layout, colors, etc. It contains approximately 28,000 training examples and comes with two evaluations sets: Augmented and Human. The Augmented set was constructed using a question generation system, while the Human set was made entirely from human annotations. Plot QA The Plot Question Answering (QA) dataset (Methani et al., 2020) is also a visual question answering dataset based on real-world scientific charts and question-answer pairs collection from crowd-sourced templates. This dataset has 80% QA pairs whose answer is either not present in the chart or not in vocabulary, which means it con- Evaluation Metrics For ChartQA and PlotQA, we follow the original authors and use relaxed accuracy (correct answer within a tolerance of 5%), while we use ROUGE metrics for OpenCQA, as it is a generative text-heavy dataset.As can be observed from Table 2 , we see consistent improvements for the MatCha model across all three evaluation datasets. Perhaps surprisingly, we observe a much larger improvement for PlotQA than for the ChartQA dataset from which pretraining data was used. We also gain improvements for the OpenCQA dataset, which requires longform question answering. The improvement for DePlot-v2 is smaller than that for MatCha-v2, as the component responsible for question answering is the LLM that remains unchanged. Since our pre-training procedure uses charts sourced from the ChartQA dataset, the evaluation on two other datasets forms an out-of-domain evaluation. Particularly on PlotQA, where charts are from different sources than those used in ChartQA, we see more significant improvements than ChartQA. Finally, although performance improvements are less significant on the OpenCQA dataset, this is not entirely unexpected, as the checklist we devised was for extractive QA. As such, the pre-training tasks motivated from those results are also more suited for extractive QA. 
related work: 
 Numerous studies have highlighted various robustness challenges in NLP models, including their over-sensitivity to minor perturbations (Ebrahimi et al., 2018; Wallace et al., 2019) as well as undersensitivity to large changes (Gupta et al., 2021; Feng et al., 2018) . A prominent method for identifying these issues is through behavioral analysis using specifically designed checklist examples (Ribeiro et al., 2020) . In this work, we build upon this approach by applying it to multimodal chart-reasoning models, developing a checklist that aids in identifying similar robustness concerns.Tackling these robustness challenges presents a distinct set of difficulties. Although increasing model size has resolved some of these issues (Gupta et al., 2024) , this approach is not always ideal. In this study, we demonstrate that certain robustness problems can be mitigated by designing targeted pre-training tasks. For instance, we introduce a pre-training task where the model predicts the visual structure of the input chart, which improves its ability to answer questions related to colors and plot types. Similar to our work, UniChart (Masry et al., 2023) explores pre-training a multimodal model to perform both low-level tasks (e.g., extracting visual elements) as well as highlevel tasks (e.g., chart summarization). 
conclusion: 
 In this work, we present a detailed study to evaluate the chart understanding capabilities of two current state-of-the-art QA models. We propose a detailed checklist that can be used to high-light the current shortcomings of these models. Broadly, we evaluate end-to-end QA models like MatCha and pipeline based models like DePlot + LLM. These help us identify avenues of improvement. Using them, we show that adding relevant pre-training tasks improves the performance of the model to achieve performance improvements across three datasets. Across the three tasks and datasets we consider, our pre-training methods help MatCha achieve an average improvement of 1.7% points. 
limitations: 
 We foresee one main limitation of this work. We do not conduct checklist analyses and evaluation with the latest proprietary models like GPT-4o (Achiam et al., 2023) , or Claude 3.5 Sonnet. The main reason for this is the cost and budget restrictions of our project due to the large number of checklistbased evaluation examples. Additionally, we do not experiment with the latest open-source Large Multimodal Models (LMMs) such as LLaVA-1.5 (Liu et al., 2023b) or LLaVA-NeXT (Liu et al., 2024) as we found them to underperform the task-specific chart models like MatCha. 3 
 
 Appendix: An issue on the github repository of the code base: https: //github.com/vis-nlp/ChartQA/issues/9Please refer 


Decision: The reviewers found the checklist analysis interested and appreciated that it led to a concrete solution that was implemented. One reviewer commented about the relatively short analysis and expected for more details, including raising many questions that the checklist can answer. The authors would do well to significantly expand their analysis for the final version.

Reviews: review: paper_summary: paper_summary: This paper first creates a checklist evaluation suite for ChartQA, visual question answering dataset over chart images. This reveals some weaknesses of models on simple questions, such as looking for presence of certain colors or summing up values. Based on this, the paper then proposes additional continual pretraining tasks to improve the performance of two tested models--MatCha (a fully end-to-end model) and DePlot (a chart-to-table generation model whose output can be fed to an LLM). This yields improvements on the checklist failure cases.

Overall this is a solid paper that blends interesting analysis of error modes with actionable solutions.
strength_weakness: strength_weakness: * Does not evaluate larger models, including GPT-4
questions: questions: Are remaining DePlot + LLM errors caused mainly by DePlot issues or LLM issues?
paper_summary: paper_summary: This paper focuses on question-answering from charts, and includes both an evaluation component and a pretraining experiment. It explores the limitations of two models, MatCha and DePlot + LLM, finding two common error areas: failures to understand visual properties of the charts and failures related to numerical reasoning. Based on these findings, the authors train MatCha and DePlot with three additional tasks to try to augment their abilities in these areas. These versions of the models demonstrate modest performance gains over the originals.
strength_weakness: strength_weakness: My general sense is that there is not enough in this paper. The paper is short and lacks detail. Since I found the evaluation component most compelling, I wish there was a more in-depth discussion of the findings. The checklist evaluation uses 33 different templates, but the authors only discuss the three broad categories of structural and visual templates, data extraction templates, and numerical QA templates. I wanted to know more about the properties tested by the different templates. I also would have liked to know more about the kinds of errors that models make. Particularly for the numerical tasks, I felt that key evaluation details were lacking. What counts as a failure? If the model rounds differently than the gold answer, is that incorrect? How far are models from the correct answer, typically?

The training section was also lacking key details. Most importantly, it wasn't clear to me whether the models were *pretrained* on the new tasks-- that is, trained from scratch on their original data and these new tasks-- or *finetuned* on the tasks-- that is, the original models were further trained on the new tasks. I know the paper says pretraining, but it is vague about how this was done and doesn't mention reusing the original datasets the models were trained on, which would be necessary in pretraining. The details provided sounded more like finetuning.

I also found the results section in 3.4 very brief. It is puzzling, because the authors take care in selecting the datasets for evaluation, but then provide so little discussion of the actual findings.
questions: questions: My main question is about finetuning versus pretraining-- see discussion above.