Faithful Chart Summarization with ChaTS-Pi (2024)

Abstract: Chart-to-summary generation can help explore data, communicate insights, and help the visually impaired people. Multi-modal generative models have been used to produce fluent summaries, but they can suffer from factual and perceptual errors. In this work we present CHATS-CRITIC , a reference-free chart summarization metric for scoring faithfulness. CHATS-CRITIC is composed of an image-to-text model to recover the table from a chart, and a tabular entailment model applied to score the summary sentence by sentence. We find that CHATS-CRITIC evaluates the summary quality according to human ratings better than reference-based metrics, either learned or n-gram based, and can be further used to fix candidate summaries by removing not supported sentences. We then introduce CHATS-PI , a chart-to-summary pipeline that leverages CHATS-CRITIC during inference to fix and rank sampled candidates from any chart-summarization model. We evaluate CHATS-PI and CHATS-CRITIC using human raters, establishing state-of-the-art results on two popular chart-to-summary datasets.1

heading: 1 Introduction
text: Chart summarization requires faithfully extracting quantitative data and describing them using natural language. Recent natural language generation (NLG) studies have explored different flavors of chart-to-summary generation tasks including caption generation for scientific figures (Hsu et al., 2021), chart summary generation (Kantharaj et al., 2022), or analytical textual descriptions for charts (Zhu et al., 2021). These tasks can be advantageous for the visually impaired (Benji Andrews, 2023) as well as for automating interpreting complex domains such as finance
1Code and demo at hf.co/spaces/chats-pi/chats-pi.
Input: Chart Image
Stage 1: Sample candidate summaries from model of choice.
Stage 3: Score summaries with the ratio of kept sentences.
Summary 1: This data is about the impact of the coronavirus pandemic on the level of revenue. 19.8% of companies saw an increase in fixed costs …. Summary 2: In 2020, the coronavirus pandemic had a significant impact on the revenues of companies in Poland. More than 36% of companies reported a decrease… Summary 3: …
Stage 2: Using CHATS-CRITIC, drop sentences not supported by chart.
This data is about the impact of the coronavirus pandemic on the level of revenue.
19.8% of companies saw an increase in fixed costs, while 43.57% saw a … In 2020, the coronavirus pandemic had a significant impact on the revenues of companies in Poland. More than 36% of companies reported a decrease in revenues of more than 50%…
…
X
✓ ✓
✓
Stage 4: Re-rank and output the best summary
In 2020, the coronavirus pandemic had a significant impact on the revenues of companies in Poland. More than 36% of companies reported a decrease in revenues of more than…
This data is about the impact of the coronavirus pandemic on the level of revenue.
19.8% of companies saw an increase in fixed costs, while 43.57% saw a…
In 2020, the coronavirus pandemic had a significant impact on the revenues of companies in Poland. More than 36% of companies reported a decrease in revenues of more than 50%…
0
1 1
1
…
mean(1, 0) = 0.5
mean(1, 1) = 1
Impact of the coronavirus pandemic on the level of revenue and fixed costs of companies in Poland in 2020
data-analysis, news reporting, and scientific domains (Siegel et al., 2016).
While a wide range of models and techniques have been applied for chart summarization, faithfulness remains a major challenge for the task. Specifically, the models often misread details in the charts (due to perceptual mistakes) or miscalculate the aggregations
8705
(due to reasoning flaws). To overcome some of these limitations, optical character recognition (OCR) models and object detection systems are usually employed to extract meta-data such as axis, values, titles, legend (Luo et al., 2021; Masry et al., 2022). These data are then used as auxiliary inputs to finetune NLG models. Nonetheless, these modeling efforts are still limited by two fundamental issues (i) training & evaluation dataset quality and (ii) the reference-based metrics being used for evaluation. As examples, two widely used datasets, Chart-to-Text (Kantharaj et al., 2022) and SciCap (Hsu et al., 2021), are automatically extracted from web articles and academic journals. As a result, the summary references are prone to hallucination, i.e. the reference might contain context that cannot be entailed solely by the chart content. Training on this data can encourage the NLG models to improvise/hallucinate. Besides, the auto-extracted summaries sometimes emphasize only certain aspects of the chart, missing out critical insights.
On the other hand, n-gram based metrics such as BLEU (Papineni et al., 2002), or learned metrics such as BLEURT (Sellam et al., 2020) rely only on gold references. They are not capable of recognizing unreferenced but correct insights since they solely rely on the reference for scoring the summaries, as shown in Figure 3. This issue is especially pronounced when the gold references are noisy, which is the case for Chart-to-Text and SciCap. Last but not least, reference-based metrics also heavily penalize summary style mismatches, giving an artificial disadvantage to LLMs which are not tuned on the task data (Maynez et al., 2023).
This motivates building a reference-free critic CHATS-CRITIC (Figure 2) that can be used as a metric to score and re-rank summaries. We additionally introduce CHATSPI (Figure 1) that leverage CHATS-CRITIC scores to generate a high quality summaries. We summarize our contributions as follows:
1. We present CHATS-CRITIC, a referencefree metric composed of a model that extracts the underlying table data from the
chart and a table-entailment model acting on each sentence within a chart summary. 2. We design CHATS-PI, a pipeline that (i) generates multiple candidate summaries using a generative model, either finetuned or with in-context learning; (ii) then leverages CHATS-CRITIC to refine the summaries by dropping unsupported sentences; (iii) computes a summary score to rank the summaries by penalizing summaries with dropped sentences to increase the fluency, and (iv) outputs the best one. 3. To assess the efficacy of CHATS-CRITIC, we juxtapose human preferences against both CHATS-CRITIC and other prevailing metrics. Our results indicate that CHATS-CRITIC aligns more consistently with human evaluations. Furthermore, when contrasting CHATS-PI with other leading models that serve as baselines, CHATS-PI establishes state-of-the-art on two popular English benchmarks.
BLEU 100 9.3 27.55 BLEURT-20 0.94 0.43 0.4
PARENT 0.84 0.25 0.32
ChaTS-Critic
s1 0.1375 1 0.0125 s2 1 1 0 s3 1 1 0.025 s4 0.125 1 s5 0.0125 1

heading: 2 The CHATS-CRITIC metric
text: As shown in Figure 2, CHATS-CRITIC is composed of a chart de-rendering model that generates the table content of the input chart image, and a table entailment model applied on a sentence level. This motivation stems from the observation that fine-grained evaluations are simpler than full-summary evaluations, mirroring the ease observed in human assessments (Krishna et al., 2023).
Chart de-rendering. To utilize the information in chart, previous works have incorporated
a step to transcribe the image across modalities to a data table (Luo et al., 2021; Kantharaj et al., 2022; Liu et al., 2023a). This process of de-rendering enables leveraging downstream text model capabilities to process the information, rather than relying on an image model, which is typically only pre-trained on natural images. Similarly, in our work we start with a de-rendering step to extract the table t from an image of a chart C (Liu et al., 2023a).2
Sentence level faithfulness score f(s) (interchangeably referred to as CHATS-CRITIC) is a sentence-level score defined as the probability of entailment of a sentence s conditioned on the de-rendered table t. This can be accomplished using a fine-tuned table-specialized model such as TAPEX (Liu et al., 2022) and TAPAS-CS (Eisenschlos et al., 2020), or by prompting an LLM such as PALM-2 (Anil et al., 2023). For the latter case, we can use few-shot examples with chain-of-thought as well as ensemble across K model runs by averaging the binary scores produced in each run, to improve the entailment accuracy, as shown in the example in Figure 2. The sampling produces a similar effect as Monte-Carlo dropout used by Steen et al. (2023).

heading: 3 The CHATS-PI pipeline
text: CHATS-PI , presented in in Figure 1, shorthand for Chart-To-Summary Pipeline , generates a set of candidate summaries in stage 1, then uses CHATS-CRITIC’s per sentence scores to repair (stage 2) score the summaries (stage 3) and re-rank them (stage 4). This is done by removing sentences with low entailment scores and picking the candidate summary with the highest Summary-level faithfulness score.
Summary-level faithfulness score F (S) is a per summary score defined as the ratio of kept sentences:
F (S) = 1
|S|
|S|∑
i=1
1[T,1] (f(si))
2We also tested end-to-end models using chart images as direct input, but the current de-rendering-based pipeline yielded the best performance.
where 1[T,1](f(si)) is the indicator function with T as the threshold, which is equal to 1 if f(si) > T and 0 otherwise.

heading: 4 Experimental Setup
text: We assess our methods on diverse datasets to prove their broad applicability.

heading: 4.1 Datasets
text: Chart-to-Text (Kantharaj et al., 2022) is a large-scale benchmark for chart summarization including bar, line, area, scatter and pie charts, composed of two data sources: Statista (35k examples) and Pew Research (9k).3
SciCap (Hsu et al., 2021) is a large-scale benchmark for figure-captioning. It is extracted from science arXiv papers published between 2010 and 2020 and contains more than 2 million figures. We use 3 subsets of SciCap: First Sentence collection (133k), Singlesentence Caption (94k data points), and Caption with No More than 100 Words (131k).
TabFact (Chen et al., 2020) is a large-scale dataset for table-based fact verification. It contains 16k Wikipedia tables as evidence for 118k human-annotated statements. This dataset allows us to study fact verification with semi-structured inputs. We use it to evaluate the entailment accuracy of CHATS-CRITIC.
All our models are developed on the dev sets of the mentioned benchmarks and performances are reported on their test sets. We include more detailed descriptions and processing details of the benchmarks in Appendix A.1.

heading: 4.2 Setups for evaluation & comparison
text: Evaluating CHATS-CRITIC. We evaluate the quality of CHATS-CRITIC by comparing the model output entailment to human annotated examples randomly extracted from the Chart-To-Text (Statista). We also evaluate the metric’s correlation with human judgments on summary level. We compare CHATSCRITIC to reference-based metrics, including
3statista.com and pewresearch.org
BLEU (Papineni et al., 2002), PARENT (Dhingra et al., 2019) that takes the table into account to compute n-gram similarity and as well as BLEURT-20 (Sellam et al., 2020; Pu et al., 2021), a learned metric.
Evaluating CHATS-PI. We report a wide range of metrics’ scores across the three benchmarks. We compare CHATS-PI applied on different base models, as well as state-of-theart baselines in the literature which do not rely on CHATS-PI where applicable. The SOTA baselines include PaLI (Chen et al., 2023) and MATCHA (Liu et al., 2023b) MATCHA on Chart-To-Text; M4C-Captioner (Horawalavithana et al., 2023) on SciCap. We additionally train and evaluate PaLI (Chen et al., 2023) ourselves to report more comprehensive results across different benchmarks and metrics. All metrics are reported on the sentence level except for the correlation study.

heading: 4.3 Our models
text: Plot-to-table model. As described, our approach relies on a plot-to-table translation model. For all our models, we make use of DePlot (Liu et al., 2023a), a state-of-the-art model for extracting table contents from chart images (i.e. chart de-rendering).4 The de-rendered table is passed to a generative text-to-text model for further processing.
Generative models. We use two models for summary generation with the de-derendered table from last step as input. We adapt a FLANT5 (Suresh et al., 2023) base model with table embeddings to enhance table structure understanding, following the scheme of TabT5 (Andrejczuk et al., 2022). We fine-tune this model for each datasets for 220k training steps with a batch size of 128. We denote this setup as DePlot+FLAN-T5 (see Appendix A.4.1). The second approach is PALM-2 (L) (Anil et al., 2023) with in-context learning. The full prompt is described in Appendix A.4.3. We experiment with other models including end-to-end models in Appendix B.4.
4More details about DePlot can be found in Appendix A.2.
CHATS-CRITIC is used for the CHATS-PI pipeline and as an additional metric in our experiments. We experiment with different model sizes and families for CHATS-CRITIC’s entailment component. When not specified, CHATS-CRITIC uses DePlot and PaLM-2 (L) with Chain-of-thought (Wei et al., 2022) for the entailment model (shown in Figure 2). The full prompt is reported in Appendix A.4.4.

heading: 5 Results

heading: 5.1 Meta evaluation of CHATS-CRITIC
text: CHATS-CRITIC is evaluated by assessing its correlation with human ratings and the overall quality of the generated summaries. We randomly sampled 60 different charts from Chart-To-Text (Statista) test set and surveyed the Entailment, Relevance, and Grammaticality (see Appendix B.1) on the sentence and summary level when appropriate, making a multidimensional quality metric (Huang et al., 2020). The provenance of the summaries is hidden to prevent biasing the raters. The raters are 10 volunteering researchers from our institution (not including the authors). We refined the guidelines with a small sample of examples and raters before formally starting the survey. The Cohen’s Kappa in this sample between pairs of raters is 0.61, which suggests substantial agreement (Landis and Koch, 1977). In the formal survey, the raters annotated the full set, one rater per example. As shown in Figure 7 in Appendix B.1, we display the chart alongside the title, then for each sentence we ask the rater if is (1) entailed, (2) relevant, and (3) grammatically correct. The full annotation guidelines are reported in Appendix B.2. We collect annotations for four data collections presented in table 1. In the two collections using CHATS-PI,
Human annotation set (60 charts) Avg # sentences Avg # not entailed Reference 2.5 1.0 PALM-2 4.3 1.1
(PALM-2, (PALM-2)) 5.4 1.0 (PALM-2, (DePlot, PALM-2)) 5.0 0.5
Table 1: Human annotation data collections sizes. : CHATS-PI generates a set of 10 candidates using
PALM-2, and : CHATS-CRITIC considers either the original tables or the generated ones using DePLot.
the predictions are generated without dropping the unsupported sentences, to allow a thorough analysis of CHATS-CRITIC quality.
Entailment performance. We compare CHATS-CRITIC against a no-op baseline f(x) = 1, where no sentences are filtered, as binary classifiers acting on each sentence. As such, we report Accuracy, F1 and AUC in Table 2. AUC measures the ability of a binary classifier to distinguish between the classes {0, 1}. When no sentences are dropped, no classifier is used, the AUC is equal to 50%. In this use case we can focus on F1 results to evaluate the quality of the input sentences. Precision and recall are reported to showcase the trade-off of CHATS-CRITIC at a threshold of 0.75. We show that CHATS-CRITIC significantly improves upon all the metrics reaching better Precision-Recall trade-off.
The reference summaries in SciCap are extracted automatically, implying that extra information might be present that cannot directly be deduced from the provided chart and metadata alone. As expected, the F1 score is low when considering all sentences entailed (i.e. baseline f(x) = 1). Our proposed metric improves F1 by 11 points and increases AUC by 31.5 points. For the three other datasets, the summaries’ quality is already better than the reference. Thus, the gain is less significant: by 1 to 2 points for F1 and 20 to 22 for AUC.
We report the Pearson coefficient and the pvalue in Table 2. For all the sets, the p-value is significantly small, indicating a high probability of observing a correlation to human ratings. The Pearson coefficient indicates that CHATSCRITIC has a human rating correlation from moderate (> 30) to strong (> 50).
Impact of critic model size. We compare in Table 3 different LLMs to implement the entailment component of CHATS-CRITIC. We evaluate the performance of the models using the SciCap reference human annotation set and DePlot as a de-rendering model. We additionally study the entailment quality factoring out the de-rendering step by providing the original gold tables in SciCap and TabFact datasets.
Annotation set Sentence Selection Acc. Balanced Acc. Recall Precision F1 AUC Pearson (p-value) Reference f(x) = 1 60.0 50.0 100.0 60.0 75.0 50.0 −− (−−) 82.0 79.45 92.22 80.58 86.01 81.56 62.2(1.9e− 17) PALM-2 f(x) = 1 75.48 50.0 100.0 75.48 86.03 50.0 −− (−−) 81.23 71.24 90.86 85.24 87.96 70.54 45.15(1.6e− 14)
(PALM-2, (PALM-2)) f(x) = 1 83.33 50.0 100.0 83.33 90.91 50.0 −− (−−) 84.49 68.93 93.9 87.83 91.81 70.6 43.6(1.7e− 15) (PALM-2, (DePlot, PALM-2)) f(x) = 1 89.4 50.0 100.0 89.4 94.41 50.0 −− (−−) 92.38 66.76 99.26 92.73 95.89 72.53 51.01(2.1e− 21)
Table 2: Evaluating CHATS-CRITIC ( ) with a 0.75 threshold against human labels on Chart-To-Text, we contrast
with a no-op baseline (f(x) = 1) and report sentence binary classifier metrics. For CHATS-PI ( ), we generate 10 candidates at temperature T = 0.7, and CHATS-CRITIC ( ) is computed with T = 0.3 over 8 samples.
As shown in the table, model size is a critical factor to improve CHATS-CRITIC overall quality. In SciCap using DePlot respectively gold tables, we see a 10.6 respectively 12.6 points increase on accuracy by using the small model compared to selecting all sentences (f(x)=1) and 11.3 respectively 8.6 increase when switching from small to large models. We observe the same behavior in TabFact with 4.2 increase from small to large.

heading: 5.2 Metrics correlation to human ratings
text: We investigate the correlation of the referencebased metrics to human ratings and compare it to CHATS-CRITIC. Since these metrics are applied on the summary level, we extract the human entailment rating per summary: if any sentence is not entailed, the entire summary is refuted. PARENT (Dhingra et al., 2019) uses also the table information on top of the summary. To thoroughly assess CHATS-CRITIC, we report the correlation on summary level.
Additionally, we study the p-value and Pearson coefficient in Table 4. To observe a possible correlation, reference-based metrics require optimizing for the entailment threshold (reported in the Appendix B.3.1). Even accounting for that aspect, most of the referencebased metrics fail at providing a p-value that is statistically significant to identify a correlation (less than 0.05). The majority of the metrics have a Pearson coefficient lower than 0.30, indicating a small correlation. However, these metrics are less reliable than CHATS-CRITIC, as these values are obtained by optimizing the threshold and the curve is not smooth; a deviation of 0.1 in the threshold reduces the Pearson coefficient dramatically and increases the p-value. The results reported in the table, further confirm that our metric is more reliable and has a higher correlation with respect to the reference-based metrics. We additionally report the precision and recall curves for all metrics in Appendix B.3.2.

heading: 5.3 Evaluation of the CHATS-PI pipeline
text: In the second experimental setup, we compare in Table 5 different models to solve the chartto-summary task on three data collections. We show that adding CHATS-PI improves any of the presented generative models on CHATSCRITIC. Additionally, it increases BLEURT20 by around 1 point for all the data collections. The best generative model is PALM-2. CHATS-PI (PALM-2) consistently reaches between 93% and 96% of CHATS-CRITIC. For more details, models and metrics results see Appendix B.4.
DePlot for chart de-rendering. CHATS-PI (i.e. rows with ) uses CHATS-CRITIC configured in the same way. We report SciCap (First sentence) split for brevity. Full results and additional evaluations are in Table 10.
mance (i.e. column), instantiated with PALM-2 and DePlot for chart de-rendering. CHATS-PI (i.e. rows with ) uses CHATS-CRITIC configured in the same way. We report SciCap (First sentence) split for brevity. Full results and additional evaluations are in Table 10.

heading: 6 Analysis
text: Ablation study (CHATS-PI 4 stages) We report a performance study of the four stages of CHATS-PI, as depicted in Figure 1, in Table 6. Droppings sentences in Stage 2 increases F1 by 1.9 points compared to Stage 1. Ranking with CHATS-CRITIC without repair shows 8.3 points compared to Stage 1 and 6.4 to Stage 2. Dropping the sentences of the top ranked summary increase F1 by 1.3 reaching 95.69% compared to using the top ranked summary.
We ablated the impact of DePlot on CHATSCRITIC, using the original tables as a baseline.
The findings are detailed in Table 7. Given that DePlot’s extracted tables may include missing or inaccurate data, we anticipated a greater sentence drop in CHATS-CRITIC with DePlot. Contrarily, the F1 remains consistent for the reference and even sees an increase in CHATSPI sets. Upon examining specific instances, we discerned the primary reason as following: Some numbers in gold tables are “overly precise” (sometimes several digits after the decimal, making it hard for humans to distinguish). In contrast, DePlot always outputs a “rounded”/lossy value, which is preferred by human raters over those using the ultraprecise numbers from the gold table. Despite these observations, the overall difference remains marginal (less than 1 percentage point). This suggests that DePlot’s performance is commendably accurate, even when juxtaposed with gold tables.
Grammaticality defined as the human ratings on grammatical errors (see Section 5.1) on non dropped sentences and summaries is reported in table 8. When applying CHATSCRITIC, we see a constant sentence-level Grammaticality for the CHATS-PI last stage –The quality is already at 98.6%, leaving little room for improvement– and a consistent im-
provement over all other sets. As for summarylevel Grammaticality (S), the story is more nuanced. On the Reference set (i.e. ∼ 3 sentences per summary), the impact on Grammaticality (S) is less prominent. On the PALM-2 annotation set, which features longer and more complex highlights (i.e. ∼ 5 sentences per summary), we can see a small drop of−1.97%. CHATS-PI last stage remains constant, showing the importance of ranking.
Relevance defined as the percentage of relevant sentences among the selected ones is reported in Table 8. We see a performance drop on this metric, mainly due to the design of CHATS-CRITIC. The relevant sentences usually feature a more complex structure. CHATSCRITIC tends to prioritize less complex sentences during the entailment verification stage, thus producing an overall drop in Relevance. One such example occurs when multiple statistics and computations are included in a single sentence: “The nonstore retailers increased by 22.8% in April 2020 compared to April 2019, whereas store retailers jumped 12% points in the same period.”. This is the case for the Reference and the CHATS-PI ranking stage.

heading: 6.1 Multilingual generalization
text: To evaluate the feasibility of our approach on internationalization (i18n) datasets, we investigated a controlled setting using the TATA dataset (Gehrmann et al., 2022), a multilingual
table-to-text dataset. We selected a localized image per language from the test set, including Portuguese, Arabic, French, Russian, and Swahili. Regrettably, the test set did not include images in Hausa, Yoruba, or Igbo. Figures 4 and 9 include the images alongside the generated summaries.
Generally, the approach can provide an accurate summary if the deplotting component satisfies the following conditions: (a) no errors are introduced, or (b) a reasonable level of detail is provided. For Portuguese, the approach encounters failure case (b), as the deplotted table contains limited information (e.g., the subtext is not considered), resulting in a generic and factually deficient summary. Conversely, the error case (a) is observed for Arabic, where the information is extracted in an incorrect order, leading to factual inaccuracies.
When the table extraction is executed without errors, the summary generated is also accurate and reliable.However, it is acknowledged that there are specific instances where potential stylistic errors may manifest within the summary. Such errors may include the reiteration of phrasal structures at the initiation of sentences or the potentially inappropriate integration of code-mixed information fragments.

heading: 7 Related work
text: Limits of reference-based metrics have been explored for text summarization tasks and different solutions have been proposed to address them. Scialom et al. (2019, 2021); Fabbri et al. (2022) use question generation pipelines to evaluate the faithfulness of summaries without relying on references. Anderson et al. (2016) study the use of intermediate dependency parse trees to rate natural image captions. For semi-structured data such as table-to-summary generation, PARENT (Dhingra et al., 2019) demonstrates the limitation of BLEU as it do not highlight the key knowledge from the table, whereas Opitz and Frank (2021) focuses on generating text from abstract meaning representations. Gehrmann et al. (2022) observed the poor correlation of BLEURT-20 to human ratings and proposed STATA, a learned
metric using human annotation. In this work we explore building a referencefree metric for chart summary that does not require human-annotated references. We show that our metric CHATS-CRITIC has much higher correlation with human judgment than reference-based metrics such as BLEU.
Chart-to-summary generation has recently gained relevance within multimodal NLP. Obeid and Hoque (2020) created the Chart-to-Text dataset, using charts extracted from Statista. Kantharaj et al. (2022) extended
it with more examples from Statista and Pew Research. Besides efforts on evaluation, multiple modeling methods have been proposed to reduce hallucinations. The approaches can be roughly divided into (1) pipeline-based methods which first extract chart components (e.g. data, title, axis, etc.) using OCR then leverage text-based models to further summarize the extracted information Kantharaj et al. (2022); Choi et al. (2019); (2) end-to-end models which directly input chart-attribute embeddings to Transformer-based models for enabling structured understanding of charts (Obeid and Hoque, 2020).
In this work we explored both (1) and (2). The best approach CHATS-PI generally follows the idea of (1). Instead of relying on OCR we use a de-rendering model for extracting structured information in charts and we explore a self-critiquing pipeline with LLMs for the best quality chart summarization.

heading: 8 Conclusion
text: In this paper, we tackle the chart-to-summary multimodal task, which has traditionally been challenging since it requires factual extraction and summarization of the insights presented in the image. To measure the quality of a summary (especially faithfulness which has been overlooked by previous metrics), we present a reference-free metric called CHATS-CRITIC
for accurately and factually scoring chart-tosummary generation. CHATS-CRITIC obtains substantially higher correlations with human ratings compared to prior reference-based metrics. We additionally present CHATS-PI , a self-critiquing pipeline to improve chart-tosummary generation. CHATS-PI leverages CHATS-CRITIC scores to refine the output of any model by dropping unsupported sentences from the generated summaries and selecting the summary that maximizes fluency and CHATS-CRITIC’scores. Compared with state-of-the-art baselines, CHATS-PI demonstrates stronger summarization quality across the board, achieving better scores for both the CHATS-CRITIC which stresses faithfulness and also traditional metrics such as BLEURT.

heading: Limitations
text: In the following, we outline the limitations of our work to ensure transparency and inspire future research. First, the chart domains we experimented with is limited to a few popular websites (e.g. Statista and Pew). This is due to the fact that existing academic chart-tosummary datasets only cover limited domains. However, to comprehensively evaluate the effectiveness of CHATS-CRITIC and CHATS-PI, it is desirable to also evaluate our approaches in other chart domains such as infographics and scientific/financial charts. Second, the CHATS-CRITIC depends on a deplotter (image-to-text) model, specifically DePlot (Liu et al., 2023a). DePlot has been trained on similar domains as the chart-to-summary datasets used in this work (e.g. Statista), and its performance may not generalize to other domains. The SciCap dataset evaluated in Table 5 provides some evidence of generalization. In future work, we plan to build out-of-domain evaluations to understand the impact of the deplotter’s robustness better. Third, we focused mostly on English chart summary in this work. We plan to also expand multilingual chart summary in future works and expanding our evaluation on the TaTa dataset (Gehrmann et al., 2022) as a test bed.
Another potential limitation is the use of CHATS-CRITIC to evaluate CHATS-PI, although it is important to note that in order to mitigate the possibility for bias, we have cross checked with a variety of evaluation metrics and conducted ablation studies to analyze the individual components of our proposed solution. CHATS-PI has proven to also provide significant benefits when employing standard reference-based metrics. Comprehensive human studies have corroborated the usefulness of our results and we can confidently say that the improvement is consistent across the board.
We would also like to highlight the underlying risk of blindly trusting models to summarize content from an image accurately. Special care should be taken to verify outputs in accuracy-sensitive applications.
Despite its limitations, our work serves as
an initial step in constructing reliable chart summarization evaluations and models. We hope future research can greatly benefit from this starting point.

heading: Acknowledgments
text: We would like to thank Massimo Nicosia, Srini Narayanan, Yasemin Altun, Averi Nowak, Chenxi Pang, Jonas Pfeiffer, Mubashara Akhtar, Parag Jain, and the anonymous reviewers for their time, constructive feedback, useful comments and suggestions about this work.

heading: Appendix

heading: A Experimental Setup

heading: A.1 Datasets
text: We use two popular chat-to-summary datasets for our experiments. The first one is Chart-to-Text (Kantharaj et al., 2022), which can be found in https://github.com/ JasonObeid/Chart2Text. The second one is SciCap (Hsu et al., 2021), which is available at https://github.com/tingyaohsu/ SciCap. More details about the two datasets are introduced below.
Chart-To-Text has mainly two sources: (i) Statista and (ii) Pew Research. (i) Statista is automatically extracted from an online platform that publishes charts in different topics including economics, market, and opinion; it is composed of 34,811 table, charts and summary triplets. (ii) Pew is automatically extracted then manually annotated from data-driven articles about social issues, public opinion and demographic trends; it is composed of 9,285 chart summary pairs.
SciCap (Hsu et al., 2021) is a large-scale benchmark for figure-captioning. It is extracted from science arXiv papers published between 2010 and 2020 and contains more than 2 million figures. The figure-caption pairs are extracted using PDFFigures 2.0 (Clark and Divvala, 2016), then an automatic figure type classifier is used to select graph plots. To be comparable to the work of Hsu et al. (2021), we evaluate our model on the three subsets containing no sub-figures: First Sentence collection including 133, 543 figures, SingleSentence Caption collection containing 94, 110 figures and Caption with No More than 100 Words composed of 131, 319 figures.

heading: A.2 De-rendering
text: We use DePlot (Liu et al., 2023a) model in all our experiments. The model code and checkpoint are available at https://github.com/google-research/ google-research/tree/master/deplot. We use the GCS path to the base model
gs://deplot/models/base/deplot/v1 fine-tuned to solve the chart-to-table task. We do not perform any additional training, and use the model as a pre-processing step to extract the tables from the chart.

heading: A.3 Baselines
text: We report the state-of-the-art models BLEU scores as presented in their papers. To be able to compare their models to ours and compute our new metric, we fine-tune a PaLI (Chen et al., 2023) model that gives a comparable results in BLEU as the other models. We select PaLI (Pathways Language and Image model) as our method of choice, because it takes the image as input directly, without the need for pre-processing or any OCR model to extract metadata, which can be difficult to reproduce. In our experiments, we use the larger 17B variant and fine-tune for 5k iterations with an image resolution of 588× 588. The PaLI model is fine-tuned with 128 GCP-TPUv4. We use a batch size of 256 and max sequence length of 128.

heading: A.4 Our models

heading: A.4.1 DePlot+T5 and DePlot+Flan-T5
text: We adapt T5 (Raffel et al., 2020) and FLANT5 (Suresh et al., 2023) models: T5 is available at https://huggingface.co/t5-base and FLAN-T5 is available at https: //huggingface.co/google/flan-t5-base. We adapt both base models to the chartto-summary task. We add a de-rendering model to extract the table form the chart and use it as input of the models. Additionally, table embeddings are added to enhance table structure understanding. We fine-tune both models for 220k with 16 GCP-TPUv3 cores using a batch size of 128 and a max sequence length of 128.

heading: A.4.2 MatCha-DePLot+FLAN-T5
text: We use in our experiments MatChaDePlot+FLAN-T5, which is composed of a MatCha (Liu et al., 2023b) image understanding module coupled to a DePlot+FLAN-T5 model, both of which
are base size. MatCha base is available at https://github.com/google-research/ google-research/tree/master/deplot. This model takes in input both the a chart image and its table content (i.e. obtained by invoking DePlot). This setup should allow capturing visual aspects that DePlot ignores in its de-rendering process. MatChaDePlot+FLAN-T5 is fine-tuned for 220k training steps with 32 GCP-TPUv3, 128 batch size, 1024 image length and a max sequence length of 128.
A.4.3 PALM-2 In our experiments for summary generation we use PALM-2(L) (Anil et al., 2023) with in-context learning. The prompt is displayed in Figure 5.

heading: A.4.4 Critic model for CHATS-CRITIC
text: We use PALM-2 (Anil et al., 2023) as a critic model for CHATS-CRITIC. Prompting is crucial for the interpretability of the entailment results. PaLM-2 outputs a text to refute or entail the claim. Following Wei et al. (2022), we use Chain-of-thought prompting to emphasize the reasoning before making the decision on the claim. More precisely we use 2 shots prompting for the critic models as shown in Figure 6. We use the same prompting for the large and small PALM-2 models. The small model is available at https://cloud.google.com/vertex-ai/ docs/generative-ai/model-reference/ text?hl=en.

heading: B Results

heading: B.1 Annotation framework
text: Figure 7 contains a screenshot of the annotation framework used to collect human ratings.

heading: B.2 Annotation guidelines
text: We provided to the raters the following annotation guidelines:
1. is_interesting highlights an important insight from the chart such as min / max / avg value or comparison.
The title copy is not interesting.
If the sentence is not entailed or grammatically not correct but highlights important info please select is_interesting.

heading: 2. cleaned_summary_is_grammatically
text: _correct = grammar and fluency. Here the critic model drops some sentences. Please focus on the fluency of the paragraph.
3. Entailed = you do not need additional info: using the chart only, be able to extract the text. (look at the chart the table can help you but not considered as ground truth.)
If the prediction is equal to the title it is entailed you can consider it as not interesting.
Please make sure that the meaning of the sentence does not add additional info about the chart.
Examples:
(a) The chart is about kids enrolled in kindergarten and nursery. The sentence contains: kids aged from 3 to 5. The title or the chart dose not refer to the age. This adds a condition on the conducted study not referred in the title or the chart. We considered it not entailed.
(b) If the sentence contains a general knowledge such as definitions: • if you know that the definition is
correct select is_entailed • if you know that it is wrong or
do not know please select not entailed.
4. Approximate numbers is allowed up to 2 digits after the decimal.
Example: exact number in the chart between 2000 and 3000.
• Text_1: "... around 2.51k" is entailed.
• Text_2: "... around 2.5123k" is not entailed.
5. grammatically_correct = look at grammar errors / fluency / repetition. Punctuation only if it changes the meaning of the sentence. Small errors are acceptable.
Example: forget a letter/ invert letters / forget punctuation.

heading: B.3 Correlation to human ratings

heading: B.3.1 Pearson’s coefficient and p-value
text: We extract the summary level human annotation as following; when a summary contains at least one unsupported sentence it is considered as unfaithful. As a result, the human summary annotation is binary while all the summary metrics BLEU, BLEURT-20, ChaTS-Critic are continuous. Without the use of a threshold to binarize them, the results of the p-value and Pearson coefficients are extremely uninformative. We choose the best possible threshold
for each of the metrics in order to compare them in a fair way. In other words, we evaluate the classification task by selecting a threshold. In the case of giving 2 binary vectors to compute the Pearson coefficient we have Pearson = Spearman = Phi (standardized Chi-square). Table 9 reports the different thresholds used to measure the p-value and Pearson’s coefficient in Table 4.

heading: B.3.2 Precision and Recall curves
text: Figure 8 shows the correlation of different metrics with human ratings by reporting Precision and Recall on the predicted summaries generated by PALM-2 compared to the original reference. A good correlation would display a continuously decreasing step function allowing to trade-off between Precision and Recall at a given threshold level. The CHATS-CRITIC summary scores curve shows that it is a better classifier compared to all other metrics.

heading: B.4 CHATS-PI pipeline evaluation
text: We report supplementary experiments and baselines in Table 10, alongside additional metrics. We report CHATS-CRITIC and CHATS-PI using DePlot as a de-rendering model and if the original table is provided we add an extra row to ablate the effect of DePlot. We use the following model checkpoint for BLEURT computation: https://storage.googleapis.com/ bleurt-oss-21/BLEURT-20.zip.

heading: B.5 Multilingual generalization
text: Figure 9 contains additional i18n examples from the TATA dataset (Gehrmann et al., 2022). Aside from a few stylistic quirks and redundancies, both summaries are accurate and faitfhful.
stantiated as (Gemini, (Gemini)).
PALM-2). Additionally, CHATS-PI uses CHATS-CRITIC following the same logic. The reported numbers for Kantharaj et al. (2022) uses BLEURT-128 base. For the our experiment we use BLEURT-20

heading: Future Work
text: In the following, we outline the limitations of our work to ensure transparency and inspire future research. First, the chart domains we experimented with is limited to a few popular websites (e. g. Statista and Pew). This is due to the fact that existing academic chart-tosummary datasets only cover limited domains. However, to comprehensively evaluate the effectiveness of CHATS-CRITIC and CHATS-PI, it is desirable to also evaluate our approaches in other chart domains such as infographics and scientific/financial charts. Second, the CHATS-CRITIC depends on a deplotter (image-to-text) model, specifically DePlot (Liu et al. , 2023a). DePlot has been trained on similar domains as the chart-to-summary datasets used in this work (e. g. Statista), and its performance may not generalize to other domains. The SciCap dataset evaluated in Table 5 provides some evidence of generalization. In future work, we plan to build out-of-domain evaluations to understand the impact of the deplotter’s robustness better. Third, we focused mostly on English chart summary in this work. We plan to also expand multilingual chart summary in future works and expanding our evaluation on the TaTa dataset (Gehrmann et al. , 2022) as a test bed. Another potential limitation is the use of CHATS-CRITIC to evaluate CHATS-PI, although it is important to note that in order to mitigate the possibility for bias, we have cross checked with a variety of evaluation metrics and conducted ablation studies to analyze the individual components of our proposed solution. CHATS-PI has proven to also provide significant benefits when employing standard reference-based metrics. Comprehensive human studies have corroborated the usefulness of our results and we can confidently say that the improvement is consistent across the board. We would also like to highlight the underlying risk of blindly trusting models to summarize content from an image accurately. Special care should be taken to verify outputs in accuracy-sensitive applications. Despite its limitations, our work serves as
an initial step in constructing reliable chart summarization evaluations and models. We hope future research can greatly benefit from this starting point