TaTA: A Multilingual Table-to-Text Dataset for African Languages
Authors: Sebastian Gehrmann
Sebastian Ruder
Vitaly Nikolaev
Jan A. Botha
Michael Chavinda
Ankur P Parikh
Clara E. Rivera
Venue: EMNLP 2023 Findings (2023)
Keywords: table-to-text
multilingual
NLG
African languages
TL;DR: TaTA: The first large multilingual table-to-text dataset with a focus on African languages.

Abstract: Existing data-to-text generation datasets are mostly limited to English. To address this lack of data, we create Table-to-Text in African languages (TaTA), the first large multilingual table-to-text dataset with a focus on African languages. We created TaTA by transcribing figures and accompanying text in bilingual reports by the Demographic and Health Surveys Program, followed by professional translation to make the dataset fully parallel. TaTA includes 8,700 examples in nine languages including four African languages (Hausa, Igbo, Swahili, and Yorùbá) and a zero-shot test language (Russian). We additionally release screenshots of the original figures for future research on multilingual multi-modal approaches. Through an in-depth human evaluation, we show that TaTA is challenging for current models and that less than half the outputs from an mT5-XXL-based model are understandable and attributable to the source data. Our results highlight a) the need for validating metrics;  and b) the importance of domain-specific metrics.

TATA: A Multilingual Table-to-Text Dataset for African Languages
Sebastian Gehrmann
Sebastian Ruder
Vitaly Nikolaev
Jan A. Botha
Michael Chavinda
Ankur Parikh
Clara Rivera
Google !Bloomberg
Abstract
Trends Prevalence
Existing data-to-text generation datasets are mostly limited to English. To address this lack of data, we create Table-to-Text in African languages (TATA), the first large multilingual table-to-text dataset with focus on African languages. We created TATA by transcrih- ing figures and accompanying text bilin- gual reports by the Demographic and Health Surveys Program, followed by professional translation to make the dataset fully paral- lel. TATA includes 8,700 examples in nine languages including four African languages (Hausa, Igbo, Swahili, and Yorùbá) and zero- shot test language (Russian). We additionally release screenshots of the original figures future research on multilingual multi-modal approaches. Through an in-depth human eval- uation, show that TATA challenging for current models and that less than half the outputs from an XTT-XXX-basee model are understandable and attributable the source data. We further demonstrate that existing met- perform poorly for TATA and introduce learned metrics that achieve high correlation with human judgments. Our results highlight a) the need for validating metries; and b) the importance of domain-specific metrics.!
Percent of children 6959 months who tested positive for malaria microscopy
2010 NMIS
2015 NMIS
48
27
12
More than one quarter of children under 5 tested positive for malaria by microscopy.
Prevalence higher among rural households than urban households.
The malaria prevalence among children has decreased by 15% between 2010 and 2015.
An example from TxTA, which demonstrates many the reasoning challenges poses.
Figure I:
as Wikipedia to under-represenneed languages (Le- bret et al., 2016). Data-to-text tasks are also an effective testbed to assess reasoning capabilities models (Suadaa et al., 2021).
However, creating challenging, high-quality datasets for NLG is difficult. Datasets frequently suffer from outputs that are not attributable the inputs or are unnatural, and overly simple tasks fail to identify model limitations (Parikh et al., 2020; Thomson et al., 2020; Yuan et al., 2021). To pro- vide a high-quality dataset for multilingual data- to-text generation, African languages (TATA). TATA contains mul- tiple references for each example, which require selecting important content, reasoning over multi- ple cells, and realizing (see Fig. 1). The dataset is parallel and covers nine languages, eight of which are spoken in Africa: Arabic, English, French, Hausa, Igbo, Portuguese, Swahili, Yorùbá, and Russian.2 We create TATA
1 Introduction
Generating text based on structured data is a classic natural language generation (NLG) problem that still poses significant challenges current models. Despite the recent increase in work focusing on creating multilingual and cross-lingual resources for NLP (Nekoto et al., 2020; Ponti et al., 2020; Ruder et al., 2021), tata-to-text datasets are mostly limited to English and small number of other lan- guages. Data-to-text generation presents important opportunities in multilingual settings, e.g., the ex- pansion of widely used knowledge sources, such
introduce Table-to-Text in
the respective language
Work done while Google.
We release all data at https://githhn thtp:///gtttt..... com/ goggle-reseerchuur1-mm
TThe languages were selected based on the availability of

---

by transcribing' and translating charts and their descriptions in informational reports by the De- mographic and Health Surveys (DHS) Program, which publishes population, health, and nutrition data through more than 400 surveys in over 90 countries in PDF format. In an analysis of TATA using professional an- notators, we find that over 75% of collected sen- tences require reasoning over and comparing mul- tiple cells, which makes the dataset challenging for current models. Even our best baseline model generates attributable language less than half of the time, i.e., over half of model outputs are not faith- ful to the source. Moreover, we demonstrate that popular automatic metrics achieve very low corre- lations with human judgments and are thus unreli- able. To mitigate this issue, train own met- ries on human annotations, which we call STATA, and we use them to investigate the tross-lingual transfer properties of monolingually trained mod- els. This setup identifies Swahili as the best transfer language whereas traditional metrics would have falsely indicated other languages.
(2) generate a description of only (but not all) the information in the imput. Corpora targeting the first typically have short inputs, for example key-value attributes describing a restaurant (Novikova et al., 2017) or seteect-verb-predicce triples (Gardent et al., 201 7a,b). Datasets in the second category include ones with the goal to generate Wikipedia texts (Lebret et al., 2016; Parikh et al., 2020) and sport commentary (Wiseman et al., 2017; Thomson et al., 2020; Puduppully and Lapata, 2021)
TATA deals with generating text based on infor- mation in charts, following the second task setup. This task has long history, starting with work by Fasciano and Lapalme (1996), Mittal et al. (1998) and Demir et al. (2012), among others, who built modular chart captioning systems. But there are only a limited number of mainly English datasets to evaluate current models. These include Chart- to-Text (Obeid and Hoque, 2020; Kantharaj et al.. 2022) consisting of charts from Statista paired with crowdsourced summaries and SciCap (Hsu et al., 2021), which contains figures and captions auto- matically extracted from scientific papers.
Overall, our experiments highlight that a) met- rics need to always be validated in how they are used; and b) that domain-specific metrics may be necessary. Consequently, as our field increasingly relies on trained metrics, e.g., reward models for RLHF (Christiano et al., 2017), we must ensure that metrics are appropriate for the chosen task.
Dealing with Noisy Data While creating more challenging datasets is necessary to keep up with modeling advances, their creation process can in- troduce noise. Noise in simpler datasets can of- ten be detected and filtered out through regular expressions (Reed et al., 2018), as done by Dušek et al. (2019) for the E2E dataset (Novikova et al., 2017). However, the larger output space in com- plex datasets requires more involved approaches and researchers have thus devised strategies to en- sure that references are of sufficient quality. For example, ToTTo (Parikh et al.. 2020) used an an- notation scheme in which annotators were asked to remove non-attributed information from crawled text. SynthBio (Yuan et al., 2021) followed a sim- ilar strategy but started with text generated by a large language model (Thoppilan et al., 2022). The downside of involving crowdworkers in the lan- guage generation steps is that outputs can be un- naturally phrased compared to naturally occurring descriptions; studies translationese in machine translation (Tirkkonen-Condit 2002; Bizzoni al., 2020) highlight potential negative effects on the fi- nal model and its evaluation (Graham et al., 2020). Our approach aims to mitigate these issues by tran- scribing naturally occurring descriptions.
2 Background and Related Work
Data-to-Text Generation Generating natural language grounded in structured (tabular) data is an NLG problem with a long history (Reiter and Dale, 1997). The setup has many applica- tions ranging from virtual assistants (Arun al., 2020; Mehri et al., 2022) to the generation of news articles (Washington Post, 2020) or weather re- ports (Sripada et al., 2004). To study the problem in academic settings, there are two commonly in- vestigated tasks: (1) generate a (short) text that uses all and only the information provided in the input;
parallel data published by the Demographic and Health Sur- com/). Arabic, Hausa, Igho, Swahili, and Yorubá are spoken in the coun- tries where the DHS conducts its surveys. These surveys are published alongside the colonial language spoken these countries: English, French, and Portuguese. Russian was selected as an unseen test language.
veys (DHS) Program ptttps://dssrroraat
'We use the terms trunscribe and transcription as a short- hand for the process where the images of charts and diagrams (info-gree grapp-ggaphis) and their deseriptions are manually converted by human annotators into spreadshect tabular representations.

---

Multilingual Generation At present, there exist only few data-to-text datasets that cover languages beyond English (Gardent et al., 2017b; Kanerva et al., 2019; Dušek and Jurčfček, 2019). None of these datasets offers parallel data two or more languages." In contrast, TATA supports cight fully parallel languages focusing on African languages, and Russian as an additional zero-shot language. Each source info-graphic covers least two of these languages; we provide data for the remain- ing languages using professional translations, lead- ing to a fully parallel corpus while minimizing the drawbacks only relying on translated texts.
Ownership of House and Land
Percent of women and men age 15-49 who:
Women
40
34
18
Own a home alone or jointly
Own land alone or jointly
Title Ownership of House and Land Unit of Measure Percent of women and age 15-49 who:
Existing datasets available in African languages mainly focus on classic NLP applications such as machine translation (Nekoto al., 2020) and sum- marization (Varab and Schluter, 2021). among oth- ers. TATA enables data-to-text generation as new task for these languages and-due to its parallel nature-also supports the development of MT mod- els that generalize to new domains and multi-modal settings (Adelani et al., 2022). In addition, due the foeus surveys in African countries, the topics and entities in TATA distinctly Africa- centrie. This in contrast other African lan- guage datasets where Western-centric entities are over-represented (Faisal et al., 2022).
Women
Men
Own home alone or jointly Own land alone or jointly
40
15
34
Linearized Form: Ownership of House and Land Percent of and age 15-49 who: (Women, Own home alone jointly, 18) (Men, Own home alone jointly, 40) (Women, Own land alone jointly, 15) (Men, Own land alone jointly, 34)
References
Only 18% women own house, either alone or jointly. only own land.
comparison, men are more than twice as likely to own home alone or jointly (40%).
Men are also more than twice as likely to own land alone or jointly (34%).
Figure An example of the process from infographic linearized input. Each table value is encoded into triple of (Column, Row, Value). The goal of the model to generate text similar to the references below.
3 TATA
3.1
Desiderata and Overview
Our goal was to construct challenging data-to-tex, dataset based on naturally occurring data that is not included in pretraining datasets and ii) which contains (preferably multiple) references that can be attributed to the input.
3.2 Data Collection
We extract tables from charts in 71 PDF reports published between 1990 and 2021 by the Demo- graphic and Health Surveys Program², a USAID- funded program to collect and disseminate nation- ally representative data on fertility, family planning, maternal and child health, gender, and nutrition. The reports and included info-graphics are pub- lished in English and commonly second language (Portuguese, French, Arabic, Yorùbá, Igbo, Hausa, and Swahili). 22 of the selected documents were only available in English, while 49 were bilingual, and the number of charts per document ranged from two to 97. A team of paid annotators tran- scribed the info-graphics into tables, in addition to extracting sentences referencing each chart in the
To fulfill these desiderata, TATA bridges the two communicative goals of data-to-text generation: It has medium-length inputs with a large output space, which allows studying content selection; and ii) it restricts generation to a single sentence at a time. The data is created through transcriptions of info-graphics and their descriptions found in PDF files, which ensures high quality references while avoiding training data overlap issues. Since each example can be described in multiple sentences, we select examples with the most sentences test examples, assuming that they cover larger share the potential output space.
'n exception is the Anglish-Germal RotoWire subset created for a shared task (Hayashi et al., 2019).
9https://dhsprogram.ccm

---

Transcribed/ Translated
Input Lengths / Output Lengths
tion to filtering examples without transcribed table values, we ensure that every example of the de- velopment and test splits has at least 3 references. 0.16
Language
F1
Arabic
157 / 711
952 / 15
0.23
English
903 /
369-16 / 134+4
From the examples that fulfilled these criteria, we sampled 100 tables for both development and test for a total of 800 examples cach. A manual review process excluded few tables in each set, resulting in training set of 6,962 tables, a development set of 752 tables, and test set of 763 tables.
French
88 778
470 / 167
0.18
Hausa
62 / 804
424-20 / 160+s
0.20
Igbo
32 834
455 / 172#5
0.24
Portuguese
833
453-20 174++
0.18
Swahili
68/ / 800
438 / 154
0.19
Yorūbá
/ 841
662
180
0.31
Table 1: An overview of the TATA training data, in- cluding the number of transcribedttrnnslated examples, input/output lengths based on the mT5 tokenizer (with 95% confidence interval), and Table Fl metric ($4.3). Since each language has the same tables, that the tokenizer favors English since it has the short- est input lengths, while being the least compatible with Arabic and Yorùbá.
Zero-Shot Russian We further transcribed En- glish/Russian bilingual documents (210 tables) fol- lowing the same procedure described above. In- stead of translating, we treat Russian zero-shot language with a separate test set, selecting 100 ta- bles with least one reference.
3.4 Linearization
surrounding t 6 Due to this process, the data in TATA is virtually unseen by pre-trained models, as confirmed in Section 4.
To apply neural text-to-text models to data-to-text tasks, the input data needs to be represented string. Nan et al. (2021) demonstrated in their DART dataset that sequence of triplet representa- tions (columu name, row name, value) for each cell is effective in representing tabular data." 9 However, tables in TATA have between zero and two column and row headers while DART assumes homoge- neously formatted tables with exactly one header. We thus additionally adopt the strategy taken by ToTTo (Parikh et al., 2020) and concatenate all rel- evant headers within the triplet entry, introducing special formats for examples without one of these headers. Similar to ToTTo, we append the table representation to the title and the unit of measure to arrive at the final representation, an example of which we show in Figure 2.
During the annotation, we also collected the fol- lowing metadata: table ID (order of appearance in the report), page number (in the PDF), table ti- tle, unit of measure, and chart type (vertical bar chart, horizontal bar chart, map chart, pie chart, table, line chart, other). Each example additionally includes a screenshot of the original info- refo-graphic. The extracted tables were then translated from En- glish by professional translators to all eight lan- guages, maintaining the table format. Each exam- ple includes marker that indicates whether it was transcribed or translated. The final dataset com- prises 8,479 tables across all languages.
To maximize the amount of usable data, we did not filter examples that lack associated sentences, and we also included examples in which the tran- scriptions do not include any values in the table (e.g., from bar charts without value labels). These examples were assigned to the training set and we explore in Section 4 how to use them during train- ing. Figure provides an example of the data.' ?
To identify the headers across the different table formats, we rely on a heuristic approach informed by our transcription instructions: we assume that the first rows in the left-most column are empty if the first rows are column headers (i.c., one in Figure 2). We apply the same process to identify row headers. the top-left corner is not empty, we assume that the table has one row header but no col- umn header; this frequently happens when the unit of measure already provides sufficient information. Our released data includes both the unmodified and the linearized representation, which we use
3.3 Data Splits
The same table across languages is always in the same split to prevent train-test leakage. In addi-
'Each annotator went through 2-3 rounds training dur- ing which authors provided feedback on 558 transcribed tables for correctness. Annotators relied on document paral- lelism to extract content for the languages they do not speak (and were familiar with the relevant orthographic system). We release all annotation instructions alongside the data. "More examples shown in Appendix A.
We selected Russian due the number of available PDFs the same format. We additionally considered Turkish, but found only four usable tables.
"While prior work (c.g. Wiseman et al., 2017) used simi- lar representation, DART was the first larger study on how to represent tables.

---

throughout our experiments.
empty string. For examples with references, we append "1" We then append "1" all dev and test inputs (TAGGED).
3.5 Dataset Analysis
Table provides an analysis of the training split of our data. The data cach language is comprised of transcriptions of 435 vertical bar charts, 173 map charts, 137 horizontal bar charts, 97 line charts, 48 pie charts, 5 tables, and 9 charts marked as "other". On average, table has 11=16 cells (not count- ing column and row headers), with minimum of one and maximum of 378 cells. Due to varying tokenizer support across languages, the lineariza- tion lengths of inputs and even the output lengths have a very high variance. To ensure that targets in the dataset are unseen, we measured the frac- tion of TATA reference sentences that overlap with mC4 (Xue et al., 2021) at the level of 15-grams as 1.5/1.7% (dev/test)." 10 This validates that our data collection approach produced novel evaluation data that is unlikely have been memorized by large language models (Carlini al., 2022).
Missing Table Values Since inputs from tables with missing values will necessarily have non- attributable outputs, we investigate two mitigation strategies. To remain compatible with the results from the above experiment, we base both cases on the TAGGED setup. First, we filter all examples where tables have no values (SKIP NO VALUES). We also take a stricter approach and filter refer- ences whose content has no overlap with the con- tent of the table based on our Table Fl metric (see Section 4.3), denoted as SKIP NO OVERLAP.
4.2 Models
We evaluate the following multilingual models. Multilingual T5 (mT5; Xue al., 2021) is a multilingual encoder-decoder text-to-text model trained on Common Crawl data in 101 languages. To assess the impact of model scale, we evaluate it in both its small (m'T5 300M parameters) and XXL (mT5XXL: 13B parameters) configurations. S5 As African languages are under- represented in mT5's pre-training data, we addition- ally evaluate a model that was pre-trained on more data in Sub-Saharan African (SSA) languages. The model was trained using the same hyper-parameter as mT5, using mC4 (Xue et al., 2021) and addi- tional automatically mined data in around 250 SSA languages (Caswell et al., 2020). We only use the small configuration (mT5ssA: 300M parameters) for this model.
4 Experiments
4.1 Setup
We study models trained on TATA in three settings: monolingual, cross- lingual and multilingual. We train monolingual models on the subset of the data for each language (8 models) and evaluate each model on the test set of the same language. The cross-lingual setup uses these models and evaluates them also all other languages. The multilingual setup trains single model on the full training data, If a training example has multiple references, we treat each reference as a separate training exam- ple.!! 1I In the multilingual setup, we compare differ- ent strategies for dealing with incomplete data: 12
4.3 Evaluation
Human Evaluation Automatic metries are untested for many of our languages and the setting of TATA, and outputs may still be correct with- out matching references (Gehrmann et al., 2022). Our main evaluation is thus through having expert human annotators judge model outputs in direct- assessment setup where they have access to the input table. We evaluate one (randomly sampled) reference and three model outputs for every devel- opment and test example. Model outputs are from multilingually trained models. We report results on the test set, while we use the annotations of the de- velopment set to create an automatic metric.! 13 All evaluation annotators are fluent in the respective
Missing References To handle examples with missing references, i.e., examples for which no verbalizations were available transcribe, we com- pare two strategics. First, we simply do not train on these examples (SKIP NO REFERENCES). Second, we use tagging approach suggested by Filippova (2020) where we append a "0" to the input for ex- amples without a reference and learn to predict an
For comparison, the estimate for relevant languages the widely used Universal Dependencies 2. trecaanks (De Mam- effe al., 2021) with mC4 is 45% (averaged over Arabie, English, French. Portuguese, Russian and Yorùbá).
" For hyperparameters. see Appendix B.
"These strategics can also be applied to the monolingual settings, but we omit such experiments for brevity and focus on the highest- perforing (multilingual) setting.
L3 Both sets will be publicly released.

---

languages and were instructed in English.
Setting
nU - U+A
Reasoning
Cells
To maximize annotation coverage, outputs were only evaluated one-way and we are thus unable to provide inter-annotator agreement numbers. How- ever, all instructions were refined through multiple piloting and resolution rounds together with the annotators ensure high quality. Moreover, in a comparison with internal gold ratings created by the dataset authors, the annotators achieve an Fl- Score of 0.9 and 0.91 for references and model outputs, thus closely tracking the "correct" ratings. Each sentence is annotated for a series of up four questions that implement task-specific variant of Attribution to Identifiable Sources (AIS; Rashkin et al., 2021) and enquire about understand- ability, attributability, required reasoning, and cell coverage (see Appendix D for details).
Reference
0.40 / 0.75
8.06.7
mTS
0.03 / 0.78
6.95.9
STSSSA
0.03 0.77
6.85.1
mT5XXL
0.34 / 0.77
7.96.0
Table 2: Results from the human evaluation aggregated over all languages. Left is the distribution of not under- standable (nU; red), understandable (U; grey), and un- derstandable and attributable (U+A; green). Right, we show the fraction of examples marked demonstrat- ing reasoning compared to all examples and fraction of U+A examples. The rightmost column shows how many cells were reasoned over (with standard devia- tion). The references and the XXL model both achieve high U+A rates of 0.53 and 0.44 respectively. Note that only one reference was evaluated per example. Sur- prisingly, the reasoning extent is very similar across all models if focus on only good outputs.
Automatic Evaluation We investigate multiple automatic metrics and use the human annotations to assess whether they are trustworthy indicators of model quality to make a final recommenda- tion which metric to use. (1) Reference- Base:: We assess (P.R.F)-score variants of ROUGE- (1,2,L) (Lin, 2004) n-gram based metric, CHRF (Popović, 2015) with maximum n-gram order of 6 and B 2 as character- based metric, and BLEURT-20 (Sellam et al., 2020; Pu et al., 2021) as learned metric. We compute the score between the candidate and each reference and take the maximum score for each table. (2) Reference- less (Quality Estimation): We define TABLE F1, which compares candidate and the input table. We create set of tokens contained in the union of all cells in the table, including headers (R), and the set of tokens in a candidate output (C). From there, we can calculate the token-level preci- sion (R (RCC/C) recall (RNC/ RR), and Fl -corc. This can be seen as very simple, but language- agnostic variant of the English information match- ing system by Wiseman et al. (2017). (3) Source and Reference-Based: We use PARENT (Dhin- gra al., 2019), which considers references and an input table. As assumes only a single level of hier- archy for column and row headers, we concatenate all available headers, and collect PARENT-R/PIP
STATA As additional metrics, we fine-tune mT55xLL on the human assessments of model out- puts and references in the development set. To do so, we construct the metric training data by treating all understandable + attributable examples as posi- tives and all others as negative examples. We adapt mT5 into regression-mmtric by applying RMSE loss between the logits of special classification token and the label which is either 1. During inference, we force-decode the classification token and extract its probability.5 15
We denote this metric Statistical Assessment of Table-to-Text in African languages, STATA. We train three STATA variants that follow the setups of the previously introduced automatic metrics: as quality estimation model that predicts score based on the table input and the model output without references (QE), with references (QE-Ref), and as a traditional reference-based metric (Ref).
5
Results
5.1 Human Evaluation
The human evaluation results Table show that only 44% of annotated samples from mT5xxI. were rated as both understandable and attributable to the input. This means that TATA still poses large challenges to models, especially small mod- els, since even the best model fails 56% of the time and the smaller models most of the time not
For all metrics that require tokenization (ROUGE, TABLE F1, PARENT), we tokenize references, model outputs, and table contents using the mT5 tokenizer and vocabulary.
14 Instrructions are available in Appendix
More details are in Appendix C.

---

Correlation with U+A
This finding is intuitive as these metrics were not designed to evaluate the correctness of reasoning. Nevertheless, they are used to assess outputs recent data-to-text approaches (e.g.. Mehta et al., 2022; Yin and Wan, 2022; Anders et al., 2022), although most point out the limitations of such au- tomatic assessments.
BLEURT-20
0.12
ROUGE-1 P/R/F
0.07 / 0.09 /0.11
ROUGE-2 P/R/F
0.12 / 0. /0.13
ROUGE-L P/R/F
0.08 / 0.11 0.13
TABLE P/R/F
0.02 / 0.06 0.05
CHRF
0.16
STATA QE
0.66
STATA QE+REF
0.61
Performing the correlation analysis using only understandability as target, which is much easier task for metric, leads to only slightly improved results, with BLEURT-20 having a correlation of 0.22, while all remaining metrics are at or below 0.13. The results for the reasoning and cell count questions are similarly poor, with maximum corre- lations of 0.09 and 0.18 respectively.
STATA REF
0.53
Table 3: Pearson Correlations between metrics and the U+A human ratings.
understandable. This finding is consistent across all languages (see Appendix E).
Our annotated references perform better across these quality categories, mostly failing the attribu- tion test when the transcribed sentences include unclear referential expressions or additional infor- mation not found in the infographic (sce references 2 and 3 in Figure 2). However, since only one of the 3+ references was annotated, the probability an example having least one high-quality ref- erence is high. Interestingly, of the examples that were rated as attributable, over 75% of sentences from all models require reasoning over multiple cells, and the number of cells sentence describes closely follows the number from the references.
The dataset-specific metric STATA fares much better. Similarly, comparing outputs to references does not contribute to an improved correlation and Quality Estimation is the best setup, achieving a correlation of 0.66, which in this case is equivalent to AUC of 0.91. This correlation is par with the agreement between the raters and our internal condtannotstions. Our experiments further showed that it was necessary to start with a large pre-trained model: Training in the QE setup starting from an m'55 only achieved a correlation of 0.21, only slightly outperforming existing metrics.
As result of these findings, we mainly report results with STATA in the next section (see Ap- pendix for results with other metrics). We report chrF as the best performing existing metric, but plead caution in interpreting its numbers.
We further performed qualitative analysis of 50 English samples to identify whether sentence requires looking the title or unit of measure of table. While 1/3 of references follow or make use the phrasing in the title, and 43% of the unit of measure, for mT55XL, the numbers are 54% and 25% 25%-stayinn closer to the title while relying less on the unit of measure.
5.3 Automatic Evaluation
We show the automatic evaluation results in Table 4. Similar to the findings from the human evalu- ation, the model pre-trained on additional data in African languages slightly outperforms the stan- dard mT5 model, but neither get close to the performance of mT5xXL demonstrating the im- pact of scale even for under-represented languages. Similarly, all multilingually trained models outper- form monolingual training. The multilingual set- tings perform similar to each other, with SKIP No REFERENCES and SKIP No OVERLAP leading to the highest scores. While chrF correctly ranks the XXL models above the others, but only with a very minor margin, and it fails to distinguish between monolingual and multilingual training setups. 16 "Since PARENT reports aggregate scores over the entire test corpus, we cannot compute the segment-lovel correlation,
5.2 Existing Metrics are Insufficient
Following our initial hypothesis that existing met- ries are untested and may not be suitable for TATA, we conduct a correlation analysis between the hu- man evaluation ratings and metric scores. Table shows the result of comparing to the main desired outcome: whether a sentence is understandable and attributable. Existing metrics perform very poorly at this task, with maximum correlation of 0.16 for chrF. This confirms that comparing set of references fails to detect non- understandable and non-attributable outputs, but even the TABLE Fl metric, which is reference-agnosti falls short." 16
hut we found similariy poor performance when assessing it on the system-level.

---

Setup
Model
CHRF
STATA QE
STATA REF
STATA QE+REF
SSA
0.37
0.04
0.50
0.25
MONOLINGUAL
Small
0.36
0.03
0.49
0.25
XXL
0.39
0.28
0.62
0.44
SSA
0.35
0.05
0.51
0.26
SKIP NO REFERENCES
Small
0.33
0.01
0.47
0.23
XXL
0.40
0.61
0.76
0.74
SSA
0.35
0.09
0.54
0.29
TAGGED
Small
0.34
0.07
0.52
0.27
XXL
0.41
0.57
0.76
0.69
SSA
0.37
0.11
0.57
0.32
+ SKIP NO VALUES
Small
0.34
0.10
0.54
0.30
XXL
0.40
0.55
0.76
0.71
SSA
0.32
0.01
0.47
0.22
SKIP No OVERLAP
Small
0.28
0.00
0.42
0.19
XXL
0.39
0.59
0.77
0.75
Table 4: Evaluation Results. MONOLINGUAL represents the average score the in-language performances of separately trained monolingual models. All others are multilingually trained models and we average over their per- language scores. SKIP No REFERENCES omits training examples without references while TAGGED uses a binary indicator in the input whether output empty. The final two variants build TAGGED additionally filter out training examples where table values are missing or where a reference has no overlap with any value in the table. For each column we bold-face the highest results (including those that are not sigmificantly different from them). According STATA, the largest gains come from scaling larger models and both SKIP No REFERENCES and SKIP No OVERLAP outperform the other modalities.
Cross-lingual We show and discuss the monolin- gual and cross-lingual performance of models us- ing STATA QE in Appendix F. Our main observa- tions are: i) English is far from the best source lan- guage; ii) Swahili is the best source language, likely due to its strong linguistic connections to other lan- guages in our data; and iii) models perform on par for tonal languages like Hausa and Yorùbá, which may be due to the government report data using manized orthography for Hausa, which omits tone and vowel length information (Schuh and Yalwa, 1993) and may thus facilitate cross- cross-lingual transfer. contrast, we had relied on the cross-lingual results using standard metrics in Appendix and H (for zero-shot Russian), we would have been led to very different conclusions. For instance, the standard metrics fail to identify the weak results of smaller models and mistakenly present Hausa and Yorùbá as the strongest languages. This discrep- ancy highlights the need for good metries.
as likely" in the same context, showing that it has learned the correct associations required for rea- soning. Moreover, many the un-uurrstttadlblee cxamples suffer from repetitions and grammatical mistakes as "However, the majority of women age 30-49 have a typical of births and the lowest percentage of women who have a twice as high typical of births."
In contrast, the large models rarely fail such fundamental level and instead sometimes generate dataset artifacts that include generalizations like "The results show that the earlier start of family formation is very similar to the typical pattern." Another issue that arises in large models are outputs in which the reasoning is correct, but stated in a very clumsy way, in "Women own home alone jointly with partner, compared with 34% of men". More examples can be found our released human annotations.
least likely
Conclusion
Failure Cases We observe a qualitative differ- ence between the smaller and the large model out- puts. The smaller models very commonly fail at parsing the table and generate nonsensical output like "Hoowverr, the majority of women age 30-49 are twice as likely be fed the first birth." in the context of ages at first birth. In addition to complete failure, the model generates "majority" and "twice
In this paper, we introduce TATA, table-to-text dataset covering nine different languages with a fo- cus African languages and languages spoken in Africa. TATA is the first multilingual table-to-text dataset among many existing English-only datasets and it is also fully parallel, enabling research into cross lrngs-lingual transfer. We experiment with differ-

---

ent monolingual and filtered and augmented multi- lingual training strategies for various models. Our extensive automatic and human evaluation identi- fies multiple avenues for future improvements in terms of understandability, attribution, and faith- fulness of neural generation models and their met- rics. We develop the metric STATA based on ad- ditional data collected on the development set and demonstrate that it has much better agreement with human ratings than existing metrics, which we consider unreliable and which we show lead to mis- leading results when analyzing transfer between languages.
Dietrich Klakow, Peter Nabende. Ernie Chang Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Milli- cent Ochieng, Anuoluwapo Aremu, Perez Ogayo. Jonathan Mukiibi, Fatoumata Ouoba Kabore, God- son Kalipe, Derguene Mbaye, Allahsera Au- guste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabene Valencia Wagner, Idris Abdul- mumin, Ayodele Awokoya, Happy Buzaaba, Bless- ing Sibanda, Andiswa Bukula, and Sam Manthalu. 2022. A few thousand translations go long way! leveraging pre-trained models for African news translation. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 3053-3070, Seattle, United States. Association for Computational Lin- guistics.
Limitations
Firstly, while we transcribed all available tables in their language. the majority of the tables were published in English as the first language. We use professional translators translate the data, which makes plausible that some translationese exists in the data. Moreover, was unavoidable to collect reference sentences that are only partially entailed by the source tables, as shown in Table 2. Since our experiments show that additional filtering does not lead to improved performance, we are releasing the dataset as-is and encourage other researchers to investigate better filtering strategies. Moreover, treat STATA QE as the main metric for the dataset, which is agnostic to references and should thus be more robust to the noise.
Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022. Multi task learning for zero shot performance prediction of multilingual models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 5454-5467, Dublin, Ireland. Association for Computational Linguistics.
Jesujoba Alabi, Kwabena Amponsah-Kaake David Adelani, and Cristina España-Bonee 2020. Mas- sive vs. curated embeddings for low-resourced lan- guages: the case of Yorùbá and Twi. In Proceed- ings of the 12th anguage Resources and Evaluation Conference, pages 2754-2762, Marseille, France. European Language Resources Association.
We finally note that the domain of health re- ports includes potentially sensitive topics relating to reproduction, violence, sickness, and death. Per- ceived negative values could be used to amplify stereotypes about people from the respective re- gions or countries. We thus highlight that the in- tended academic use of this dataset is to develop and evaluate models that neutrally report the con- tent of these tables but not use the outputs to make value judgments.
Tisha Anders, Alexandru Coca, and Bill Byree 2022. uFACT: Unfaithful alien-corpora training for seman- tically consistent data-to-text generation. In Find- ings of the Association for Computational Linguis- tics: ACL 2022, pages 2836-2841, Dublin, Ireland. Association for Computational Linguisties.
Ankit Arun, Soumya Batra, Vikas Bhardwaj, Ashwini Challa, Pinar Donmez, Peyman Heidari, Hakan Inan, Shashank Jain, Anuj Kumar, Shawn Mei, Karthik Mohan, and Michael White. 2020. Best practices for data-efficient modeling in NLG:how to train production-ready neural models with less data. In Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, pages 64 77, Online. International Committee on Compu- tational Linguistics.
References
Iie Adebara and Muhammad Abdul-Mageed, 2022. Towards afrocentric NLP for African languages: Where we are and where we can go. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pages 3814-3841, Dublin, Ireland. Associa- tion for Computational Linguistics.
Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury. Josef van Genabith, and Eike Teich. 2020. How human is machine transla- tionese? comparing human and machine translations of text and speech. In Proceedings of the 17th Inter- national Conference on Spoken Language Trunsla-
David Adelani, Jesujoba Alabi, Angela Fan. Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter,

---

tion, pages 280-290, Online. Association for Com- putational Linguistics.
Katja Filiippoa.. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 864-870, Online. As- sociation for Computational Linguistics.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. CoRR, abs/2202.07646.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Breez-Beltacchin.. 2017a. Creating train- ing corpora for NL.G micro-planners. In Proceed- ings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 179-188, Vancouver, Canada, Associa- tion for Computational Linguistics.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unexpected challenges the path to thousand- language web text corpus. In Proceedings of the 28th International Conference on Computa- tional Linguistics, pages 6588-6608, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Claire Gardent, Anastasia Shimorina. Shashi Narayan, and Laura Beitr-belt... 2017b. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, pages 124-133, San- tiago de Compostela, Spain. Association for Compu- tational Linguistics.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic. Shane Legg. and Dario Amodei. 2017. Deep inforcement learning from human preferences. Ad- vances in neural information processing systems, 30.
Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the cracked foundation: A survey of obstacles in evaluation practices for gener- ated text. CoRR, abs/2202.00635.
Marie-Catherine De Marneffe, Christopher D Man- ning, Joakim Nivre, and Daniel Zeman. 2021. Uni- versal dependencies. Computational linguisties. 47(2):255-308.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in ma- chine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (FMNLP), pages 72-81, On- line. Association for Computational Linguisties.
Seniz Demir, Sandra Carberry, and Kathleen F. McCoy. 2012. Summarizing information graphics textually, Computational Linguisties, 3853):527-574.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-We Wei Chang, Dipanjan Das, and William Co- hen, 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884-4895, Flo- rence, Italy. Association for Computational Linguis- tics.
Hiroaki Hayashi, Yusuke Oda, Alexandra Birch, Ioan- nis Konstas, Andrew Finch, Minh-Thang Luong, Graham Neubig, and Katsuhito Sudoh. 2019. Find- ings of the third workshop on neural generation and translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 1-14. Hong Kong. Association for Computational Linguis- tics.
Ondiej Dušck, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural lan- guage generation. In Proceedingy of the 12th Inter- national Conference on Natural Language Genera- tion, pages 421-426, Tokyo, Japan. Association for Computational Linguistics.
Ting-Yar Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021. SciCap: Generating captions for scientific fig- ures. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 3258-3264, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ondřej Dušek and Filip Jurāiek.. 2019. Neural gener- ation for ezech: Data and basclines. arXiv preprint arXiv:/910.05298.
Jenna Kanerva, Samuel Rönnqvist, Riina Kekki, Tapio Salakoski, and Filip Ginter. 2019. Template-free data-to-text generation of Finnish sports news. In Proceedings of the 22nd Nordic Conference on Com- putational Linguisties, pages 242-252 Turku, Fin- land. Linköping University Electronic Press.
Fahim Faisal, Yinkai Wang, and Antonios Anasta- sopoulos. 2022. Dataset geography: Mapping lan- guage data to language users. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3381-3411, Dublin, Ireland. Association for Com- putational Linguistics.
Shankar Kantharaj, Rixie Tiffany Leong. Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 4005 4023, Dublin, Ireland. Association for Computational Linguistics.
Massimo Fasciano and Guy Lapalme. 1996. Post- Graphe: A system for the generation of statistical graphics and text. Eighth International Natural Language Generation Workshop.

---

Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Lin- guistics.
Dangana, Herman Kamper. Hady Elsahar, Good- ness Duru, Ghollah Kioko, Murhabazi Espoir, Flan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. 2020. Participatory research for low-resourced machine translation: A case study in African languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2144-2166 Online. Association for Computational Linguistics.
Cew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Shikib Mehri, Jinho Choi, Luis Fernando D' D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, and Chen Zhang. 2022. Re- port from the nsf future directions workshop auto- matic evaluation of dialog: Research directions and challenges.
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end- to-end generation. In Proceedings of the 18th An- nual SIGdial Meeting Discourse and Dialogue, pages 201-206, Saarbrücken, Germany, Association for Computational Linguistics.
Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the 13th International Conference on Natural Lun- guage Generation, pages 138-147, Dublin, Ireland. Association for Computational Linguistics.
Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mi- hir Kale, Ankur Parikh, and Emma Strubell. 2022. Improving compositional generalization with self- trining for data-to-text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics /Volume I: Long Papers), pages 4205-4219, Dublin, Ireland. Association for Computational Linguistics.
Ankur Parikh. Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to- text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1173-1186, On- line. Association for Computational Linguistics.
Vibhu O. Mittll, Johanna D. Moore, Giuseppe Carenini, and Steven Roth. 1998. Describing com- plex charts in natural language: A caption genera- tion system. Computational Linguisties, 24(3):431- 467.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, Online. As- sociation for Computational Linguistics.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xian- gru Tang, Aadit Vyas. Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mu- tuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open-domain structured data record to text genera- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies, pages 432-447, Online. Association for Com- putational I. Linguistics.
Maja Popović. 2015. chri:: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for MT. In Proceedings the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 751 Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohunghe, Solomon Oluwole Akinola, Shamsuddeen Muham- mad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Music Mcressa Berhe. Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi. Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Troro Orife, Ignatius Ezeani, Idris Abdulkadir
Ratish Puduppully and Mirella Lapata. 2021. Data- to-text generation with macro planning. Transac- tions of the Association for Computational Linguis- tics, 9:510-527.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gau- rav Singh Tomar, Iulia Ture, and David Reitter. 2021.

---

Measuring attriibution in natural language generation models. CoRR, abs/2112.12870.
Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kath- leen S. Meier-Hellsterm Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenhergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Fd H. Chi, and Quoe Le. 2022. Lamda: Lan- guage models for dialog applications. CoRR. abs/2201 1.82399.
Lena Reed, Shereen Oraby, and Marilyn Walker. 2018. Can neural generators for dialogue learn sentence planning and discourse structuring? In Proceed- ings of the 11th International Conference Natu- ral Language Generation, pages 284-295, Tilburg University, The Netherlands. Association for Com- putational Linguistics.
Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Lan- guage Engineering, 3(1):57-87.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Jun- jie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more chal- lenging and nuanced multilingual evaluation. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 10215- 10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sonja Tirkkonen- Condii.. 2002. Translationese- myth or an empirical fact?: A study into the lin- guistic identifiability of translated language. Tar- get. International Journal of Translation Studies, 14(2):207-220.
Daniel Varab and Natalic Schluter. 2021. Mas- siveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 10150-10161, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Russell G Schuh and Lawan D Yalwa. 1993. Hausa. Journal of the International Phonetic Association, 23(2):77-88 82.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computa- tional Linguistics.
The Washington Post. 2020. The washington post to debut ai-powered audio updates for 2020 election re- sults. Washington Post.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253 2263, Copenhagen, Denmark. Association for Computational Linguistics.
Somayajulu G Sripada, Ehud Reiter, Ian Davy, and Kristian Nilssen. 2004. Lessons from deploying ulg technology for marine weather forecast text genera- tion. WEATHER, 5:7.
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Taka- mura. 2021. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence Natural Language Processing (Volume 1: Long Papers). pages 1451-1465, Online. Associa- tion for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre- vre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 483-498, Online. Association for Computa- tional Linguistics.
Craig Thomson, Ehud Reiter, and Somayajulu Sripada. 2020. SportSettbaskethall- a robust and maintain- able data-set for natural language generation. In Proceedings of the Workshop on Intelligent Informa- tion Processing and Natural Language Generation, pages 32-40, Santiago de Compostela, Spain. Asso- ciation for Computational Lingustics.
Xunjian Yin and Xiaojun Wan. 2022. How do Seq2Seq models perform on end-to-end data-to-text genera- tion? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 7701-7710, Dublin, Ire- land. Association for Computational Linguistics.
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch. Andy Coenen, and Sebastian Gehrmann. 2021. Synthbio: A case study in human- ai collaborative curation of text datasets. CoRR, abs/211 106:2111.066 1.06667.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao

---

(A)
B
Malaria Prevalence in Children
Figure Trends crude birth and death Egypt 2000-2013
Percent children 6 6-59 monthis testing positive for malaria by rapid diasgnostic test (RDT)
theusand population
3033
27.4
25.5
25.7
27.3
9
10
11
12
9
6.
64
65 64 044 6.3
62
2000
2001
2002
2003
2004
2005
2006
2007 2008
2009
2010
2011
2012
2010
9-11
12-17 18-23
24-35 36-47
48-59
Total 6.59
Vee
Clube douth
Age months
Malaria prevalence increases with age.
The downward trend reversed in 2006,
and the rate of natural increase rose to a peak of 25.5 per thousand in 20T2,] beeliin slightly in 2013.
9% of children under the age of tested positive for malaria according to rapid diagnostic tests.
Figure 3: Two example info-graphics and their associated descriptions. Colored rectangles indicate where informa- tion from the text can be found in the figure. (A) The first sentence compares all numbers except the aggregate and infers that the numbers are increasing. The second sentence does not require any reasoning, but requires the infer- ence that 6-59 months can be stated as "under the age of 5". (B) This sentence requires identifying the overall trend and calculating the peak population increase as the difference between birth and death rate (31.9 6.5 25.5). In addition to the values, sentences across both examples require accessing the title, unit of measure, axis labels.
Additional Examples
D Human Evaluation Details
Figure provides two additional examples of rea- soning challenges in TATA.
Each sentence is annotated for series of up four questions. The first two questions ask whether notators agree with binary statements, implement- ing a variant of "Attribution Identifiable Sources" (AIS; Rashkin al., 2021). The first asks whether a sentence is overall understandable by an anno- tator, allowing minor grammatical mistakes." 17 If the answer is no, the annotation of the example is complete. Otherwise, the task proceeds to the next question, which asks whether all of the information in the sentence is attributable to the table its meta information. i.e., whether every part of the model output is grounded in the input. single mistake (e.g., number or label) means that the sentence is not attributable, the only exception being minor rounding deviations (e.g., "two thirds" instead of "65%") "). If the answer to the second question is no, the task terminates; otherwise, we ask two final questions.
B Details on Model Training
We train all models With a constant learning rate of 0.001 and dropout rate of 0.1 for all tasks, follow- ing the suggestions by Xue et al. (2021). During training, we monitor the validation loss every 25 steps for maximum of 5,000 steps and pick the checkpoint with minimum loss. While the XXL. model commonly converges within 100-200 steps. the smaller models often require 2,000+ steps to converge.
Details on STATA
We use mT5-XXL (Xue et al., 2021) as base model which we finetume for 2,500 steps with a batch size of 32 using constant learning rate of le-4. Inputs are truncated to a maximum length of 2048. We add the following inputs depending on the metric type: If the metric uses the input, use the lin- earized representation of the example following tag [source]. If the metric uses the references, we sample three of them for consistency, and add them after [reference] tags. The output always follows [candidate] tag.
The third question asks whether the generated text requires reasoning or comparison of two or more cells ("X has the highest Y", or "X has more Y than Z"), and the last question asks annotators to count the number of cells one has to look at to generate the information in given sentence.
The exact definition we use is "A non-understondable de- scription is not comprehensible due to significantly malformed phrasing."

---

mT5 small
mT5 small (SSA)
mT5 XXL
8.22 0.07
5:10
0.11
0.036 0.11
035
0.13 0.12
0.276
0.21
0.072
0.53
0.11 0.
0.31
5.15 0.13 0.32
000
0.24
0.1
0.2
0.31
5,32
0:13
5:34
5.14
0.14
62
9.35
6.16
8.37
8.34
0.16
6.19
0.16
0.15
6.15
0.18
0:15
0.14
0.16
0.34
0:15
0.19
0.17
0.15 0.15 0.18
0.14
0.18
0.18
5.37
0.16
5.19
0.19
5.17
8.14
0.19
001
1g
avg SW pc
A.00 0.0730.085 092
A:17
0.11
3.24
0.08
0.14
0.007
0.12
8.278 0.070.07.088
0:18
0.11
0.26
5.095
816
5 001 0.12
2.043 0.13
6.33
0.2
0.15
3.24
0.12
0.14
0.690
0:15
0:38
0:33
0.99
9:21
5:33
0.32
0:17
9.11 8.11 9.13
9.12
0.34
9.12 0
0.4
0.11
0.15
0.11
0.11
0.13
0.13 0.13 0.13 0.091
9:095 0.15
0.12 0.13 0.34
8.14
0.15 7:14
0:13
ar en ha ig pt SW yo
ru avg
ar en ha pt SW yo ru avg
ar
en
fr
ha
1g pt SW yo ru avg
Figure 4: Cross-lingual zero-shot transfer performance of different monolingual models across all language pairs using standard metrics. Each value represents an average over the traditional metries for a model trained on one language (rows) and evaluated on another one (columns). The final row/column represent average. As expected, the highest values are along the within-language diagonal, but we also observe some curious behavior for Hausa and Yorùbá and in general large disagreements with the numbers presented in Figure
mT5 small
mT5 small (SSA)
mT5
XXL
GORTS C0060 GOLL ECEGT O 25676 100100.01
L00020 0-02 6696 4.2 3ELL 3819
00066 6012
0.32
0.28
8.26
3.29
0.27
0.21
0.26
0.28
0.27
03
-02
0.211 6.5
6.52% EVETS 6:2 BRON 0.54 0.11 4018
0.355 0:011 0/25 CEEN 0007
SES DOES 0.2 0001
0.15
0.14
8.6G
0.2
0.16
0.2
0:22
0:18
0.18
61
0.27
0:25
0.31
0.31
0.22
0:24
0.27
SW pt
8435 0.229
2112 0.221 E0073 0000
4.200 2.22
0.315
4.323
4
1853
4623
4.03
0.200
EL3
60084 2013
0.22
0:23
9:27
9.25
0.15
0.23
0:28
0.2
0.23
0.55 633
333
611
93
635
335
011
0.34
9:42
81012 E151 0314 0355
SET 0.267 0458 7002 2223
0:12
631
333
5873
1.1
0:28
9:23
0.29
0.28
0.22
0.23
avg
022
0:25
0.20
0.21
0.24
0.23
0.24
ar
en
ha
1g pt SW yo avg
ar en fr ha 1g pt
SW
yo
avg
ar
en
fr
ha
1g
pt
SW
yo avg
Figure 5: Cross-lingual zero-shot transfer performance of different monolingual models across all language pairs. Each value represents the STATA QE metric for a model trained on one language (rows) and evaluated on another one (columns). The final row/column represent average.
E Detailed Human Evaluation Results
and attributable, which no model achieves.
While the references are not perfect either. STATA does not use them in the QF setting, and TATA is thus a fitting testbed for learning from somewhat noisy labels. We further note that the results for reference represent only one reference out of the 3+ available and there is thus a high probablity of the set of references to paint more accurate picture of the output space for a table.
Table 8 presents the detailed human evaluation re- sults by language. We can observe that there is some variation between languages (e.g., All ex- amples in Yorùbá were judged as using reason- ing), which we attribute to different understand- ing of the instructions for annotators different languages. As a result, we suggest not comparing results across languages but instead focusing on the between-model comparison for a given language. Focusing on this, the results are surprisingly con- sistent. The two smaller models have extremely low scores for the first two questions while the XXL-sized model follows the references with small margin between the two. There is significant room for improvements on the task since success would mean being close to 1.0 for understandable
F Cross-lingual Results with StATA
We present the monolingual and cross lins-lingual per- formance of models trained on every language indi- vidually in Figure 6. The figure shows the STATA QE score for each training language (rows) evalu- ated on each target language (columns). along with averages.

---

ar
Setup
Model
CHRF
en
0.32
0:28
0.26
0.99
0:27
0.21
0.25
0.28
0:27
-0.40
SKIP NO REFERENCES
SSA
0.17
Small
0.18
fr
0.11
03
0.32
23
0.2
0:31
26
08
XXL
0.34
ha
TAGGED
SSA
0.18
Small
0.18
avg yo SW pt lg
0:27
0:25
0.3
0.31
0.31
0.22
0.24
24
0:27
XXL
0.23
-0.25
SKIP NO VALUES
SSA
0.17
02
0:23
0.
0.25
0.
0:23
0.26
0.2
23
Small
0.15
XXL.
0.27
SKIP No OVERLAP
SSA
0.17
0.26
0:21
0.28
0.3
0.28
0.22
0.24
0.2
0:25
Small
0.18
066
0:26
0.27
0.20
0.24
0.23
0.27
0.24
XXL
0.24
ha
ig
yo
avg
Table 3: Noisy chrF test results on zero-shot transfer to
Figure 6: Cross-lingual zero-shot transfer performance different monolingual models across all language pairs. Each value represents the STATA QE metric for an XXL model trained on one language (rows) and eval- uated on another one (columns), The final row/column represent average.
Russian for multilingually trained models.
par as for other languages with a slight edge for Hausa. This edge could be explained by the fact that TATA is based on government reports that use the romanized orthography for Hausa, which omits tone and vowel length information (Schuh and Yalwa, 1993), and which may thus facilitate cross cross-lingual transfer.
We make three key observations. First, it is evident that English is far from the best source language despite its prevalence in the pretraining corpus. This validates our design choice to avoid English-centric data during the collection of TATA, and points future work collect data specific non-Western locales. Second, Swahili achieves the highest performance across almost all transfer sce- narios. Swahili has substantial shared vocabulary with English, Arabic, and Portuguese, and it is dis- tantly related to Igbo and Yorûbá. These strong lin- guistic connections likely explain the observation. However, Swahili even transfers well to Hausa, which is both geographically and genetically dis- tant from Swahili (Chadic vs. Volta Conto-oong//igger for the others).
The cross-lingual peformance numbers also fur- ther emphasize the need for good metrics. We present extended version of the figure using average of all existing metrics and with STATA in the next section in Figures 4 and 5. If had relied on the existing metrics, we would have been led to very different conclusions.
G Cross-Lingual Results with Different Metrics
Figures and show the detailed cross-lingual results when relying on standard metrics (4) com- pared to STATA (5). For the standard metrics, we present an average of our baseline metrics for each (model, language) pair. The standard metrics fail to identify the weak results of the smaller models and mistakenly present Hausa and Yorùbá the strongest languages.
Finally, cross-lingual transfer between the other African languages, we observe some weak effects of their geographic distribution or genetic relationship, in line with previous findings that the geographic location of speakers and linguistic simi- larity between two languages are indicative of posi- tive transfer (Ahuja al., 2022). We would expect positive transfer effects between Hausa, Igbo, and Yorubá, due to their proximal geographic distribu- tion in Western Africa, and even stronger effects between Igho and Yorúbá, which are closer related to each other than Hausa. But, we do not see strong evidence for this relationship.
H Zero-shot Evaluation to Russian
Since there is no training data for STATA in Rus- sian, we leave the in-depth zero-shot evaluation in distant language for future work and focus noisy chrF numbers in Table 5. While differences between setups were small using chrii in Table 4, for zero-shot transfer to a new language SKIP No REFERENCES seems to perform best with a signifi- cant margin.
Additionally, while tonal languages like Hausa and Yorùbá are not well represented in pretraining data and tokenizers (Alabi et al., 2020; Adebura and Mageed, 2022), the models perform on

---

I Transcription Instructions
Check the spelling, especially when transcrib- ing a language you do speak.
The following are the instructions that were pre- sented to annotators during transcription. Note that there are multiple references to spreadsheets which record links to the external documents and to loca- tions where annotators can enter the transcriptions. Every step of instructions was additionally accom- panied by screenshots we are unable share.
Check that the number of the table ID corre- sponds with the name of the tab.
1.3 Creating your first Table in English
The documents you will extract the data from are in PDF format. You will be working on one document at a time. Claim the docu- ment you will be working from this the Docu- ment Index by adding your user name the "Claimed by" cell of one of the rows
1.1
Overview
You will be accessing two PDF documents with similar content and structure. One will be in En- glish and the other one is version of the document in different language. You don't need to know the other language for this task, English enough. Starting in English, you will work from the start of the document and find the charts, convert them into a table using Google Sheets, extract any text that refers to the chart, and then repeat the process in the other language. You will also take a screenshot of both charts and save them to Google Drive.
2. Now open the PDF documents listed in columns C and E of the same row. There can be two different PDFs, one per language, just one, where the content in the second language comes after the content in the first language. If this is the case, you will see the same text in columns C and E.
3. Now, open the spreadshects linked from columns D and F. This is where you'll cre- ate the tables. The two spreadsheets have the same alphanumerical name with differ- ent two letter code at the end to differenti- ate between English ("en") and the second language. The spreadsheets have been pre- populated with templated tabs for you to trans- form your chats into. The tabs are named with numbers starting 1. You'll 'Il'll transform the first chart into tab the second chart into tab the third chart into tab 3, and on. To cess tabs with higher numbers, click the right arrow at the bottom right of Spreadsheets (and then the left one to return to the original view)
We're interested in the outcome as well as in the process: please let us know if anything in this docu- ment unclear and what challenges you encounter following the steps below. you would like to leave specific comments on a particular table (eg. doubts, questions, something you were unsure of), kindly add them in column S "Comments" in the Table index.
1.2 Cheat Sheet
Claim your document assignment from this list
Open the PDF files and tables.
Find the first chart in the English document and transpose into table in the spreadsheet by filling the template in Tab 1.
4. Templates include the following fields for you fill in:
Table ID: refers the order in which the chart appears in the document: the first chart will be 1, the second chart will be and so on. This should match with the tab number in the spreadsheet into which you extract the chart's information.
Take screenshot of the chart, upload the folder and rename it to match the name of table.
Add the text from the Table.
Page Number: this is the page number where the chart appears. Do not trust the page numbers written inside the docu- ment. Use the page number shown in the PDF viewer (see example)
Repeat steps 2-5 for the same chart in the second language.
Find the next chart in English and repeat steps 2-6.
Title: This is the title of the chart as the document
Always remember to:

---

Unit of measure: include the chart / ta- ble's unit of measure here
6. Once all the chart text has been added to row and column headers, fill in the table with val- ues from the chart.
Screenshot link: You will take a screen capture of the chart and enter the link
7. Now look in the text surrounding the chart to find text that refers to the chart. It can be anything explicitly mentioning the data in the table. In the document, you will normally find it before or after the chart. We are interested in capturing only those sentences that describe or compare the chart's data, so watch out for irrelevant sentences appearing mid- mid-paragraph, as illustrated here. [This a common pitfall] When you find the text you't 'll add it to the table below the line "START OF TEXT <En- ter text below. Move this row further down if the table needs more space=". You'll Ill add the text sentence by sentence, per cell in the first column only, shown in the following example. Make sure to include full sentence casing and punctuation, as in the original text.
here (instructions below).
This is how you capture your screen:
[Omitted for brevity]
5. Now it's time to look for charts. Start skim- ming the PDI's from the beginning of your English text looking for charts and tables. We are interested in all types of charts: bar charts, pie charts, map charts, etc. (see chart exam- ples below). The only exception to this rule HIV related information: please skip any HIV- related charts, these will not be transcribed. Non-exclusive list of HIV-related examples:
sentence
HIV Prevalence by Marital Status
Trends in Recent HIV Testing
HIV Prevalence by Province
Trends in HIV Prevalence
8. Once you have added all the text, add your new table to the Table Index next to your screenshot (step 4): [details omitted for brevityl
The next steps to transcribe the chart require some more thinking: please interpret at a ba- sic level what the English chart is about and what information it conveys. It won't work well to pastee it blindly.
CONGRATULATIONS! You have completed all the steps for the first English chart. Now it's time to move to the second language.
You can then start populating your table. En- ter the Table ID, the Page Number, the chart Title, the Unit of measure and the screenshot link as explained above.
1.4 CREATING YOUR FIRST TABLE IN THE SECOND LANGUAGE
Please enter the table below the line START OF TABLE &Enter table contents below>, leaving this text unchanged. Note you may need to add rows to fit in all the information from the chart.
the PDF for the second language (or the sec- tion of the bilingual document you're working from that corresponds to the second language), find the same chart you just transformed into a table. The documents are aligned in both languages, they should be in the same page in both documents.
Then identify the axes in the chart and start populating the table with row and column headers: copy the text exactly it appears in the PDF. As best you can, try to include the item with the largest number of items rows, so that the table taller (vertical) rather than wider (horizontal). You do not have to dupli- cate the unit of measure in the table if there are other headers already (see example to the right, and 4-5 below). If the unit of measure is the same the column header (see examples 1-3 below), include it as column header. One example is shown here (image below, ta- ble to the right) and for other charts on the final page of this document.
When you find the chart, repeat steps 10 using the second language table the docu- ment index and replacing where says En- glish (EN) by the two letter code of your sec- ond table (e.g. "SW" for Kiswahili). Tips to working in a second language:
Documents are aligned by language. This means that the formatting of the text in one language mirrors the second language. Therefore, when looking for the same chart, go to the same page

---

the second document if you have two documents. Or find the start of the sec- ond language if you're re working with one bilingual document, and use images and charts to guide you until you find what you're re looking for.
1.7 COMMON PITFALLS
NNote: Every pitfall is accompanied by a screenshot or an example.]
Ensure that all copied text is relevant to the ta- ble. Irrelevant Text: Stunting also varies by gov- ernorate. Stunting is below 25% in Aden, Abyan, and Al Mhrah, and is highest in Reimah, at 63%. Relevant Text: According to the survey, 47% of children under five are stunted, or too short for their age.
Check the number of sentences in the paragraph by identifying the full stops. This will help you find the start of the sentence you'd d like to copy-paste.
Geographical names may be similar or related: you can use them to confirm you're re copying the sentence you wanted copy.
Please only copy sentences that are directly supported by the chart which means that numbers in the table or its column/row names or title are directly referenced or compared. Should a sentence partially be supported by a table, please still copy it in full.
Identify the numbers in the sentence: you use them to confirm you'l re copying the sentence you wanted to copy.
Arabie: Arabic is language with right-to-left writing system (in English, the words are written left-to-right). As you go through Arabic text, remember that sentences start on the right and con- tinue to the left. Also note the cursor of your mouse may behave strangely when selecting the text for copy pasting. You may need to start selecting on the right- hand side, and drag towards the left.
In Arabic, ensure that entire sentences are copied, and that only the relevant sentences are copied. Periods are hard to spot, but if you are unsure about where the sentence ends you can dou- ble check by sending a copied sentence through Google Translate:
Make sure to also transeribe map charts In map charts, every region should be a separate row.
Do not leave empty rows and columns for spac- ing Here we have two empty rows under the table before the start of the text, highlighted in yellow, These rows should be deleted (right-click on your mouse to quickly access this option). Empty spaces make difficult to read the tables automatically.
WELL DONE! You have two aligned tables, one in English and one in different language. Now continue adding more tables.
1.5
ADDING MORE THAN ONE TABLE TO YOUR SPREADSHEETS
Ensure number formatting matches the orig- inal values When entering percentages (e.g. 10%) or ranges (e.g. 10-20) into Google Sheets, they may get reformatted automatically and no longer match the original values. Watch out for this. Fix by adjusting the number formatting inside the Sheet until your table matches the original doc- ument's 's values. If ccopy-pasting from the document, try paste without formatting (Ctrl+Shifttvv.
After completing your first table in English and your second language, it's time continue looking for the second chart back in the English document. When you find the next chart in the document, fill out the next tab (e.g. 2) by selecting it at the bottom of your screen.
Now repeat steps 4-10 for English, adding new line to the table index, and again for the second language until you reach the end of the PDF. At this point, your table should have as many tabs as the number of tables in column F of the Document index.
Representation of additional graph elements Some charts are designed in such a way that they contain additional elements. When representing them, keep the below in mind:
If anything in these guidelines is unclear, please let us know. Thanks!
Secondary scale annotations Add a column to the left of the main categories and fill out the values as they correspond in the graph.
L6 CHART AND TABLE EXAMPLES
Supercategories Identify supercategories in a separate row with the value cell left empty. Add all
[5 examples here omitted for brevity]

---

subcategories of given supercategory below.
for this stage in case that context required, but you should ignore all other sentences when mak- ing your judgment. Carefully read the sentences one-by-one and decide whether you agree with the following statement: "The text is understandable".
Representation of Footnotes When a table or its text includes footnotes, you do not need to add the footnotes to the spreadsheet. We additionally ask you to try to remove the footnote symbol from the text itself.
Definition: A non-understandable description is not comprehensible due to significantly malformed phrasing.
Use the PDF page numbers, not the ones writ- ten in the document Please use the page num- bers indicated by the PDF reader and not those appearing inside the document.
The purpose of Q1 is to filter out descriptions that you cannot rate along the other dimensions because you cannot understand their meaning. If the description is unclear, select "No". In other words, you cannot understand what sentence is trying to say to the extent that you will not be able to rate it along the other dimensions, mark it as "No."
J Human Evaluation Instructions
The following are the instructions that were pre- sented to annotators in our human evaluation.
J.1 Overview
In the case a sentence has conjunctions that don't make sense for alone-standing sentences (c.g., start- ing with "However", you can still mark it as under- standable if the rest of the sentence makes sense.
In this task you will evaluate the quality of one- sentence descriptions of table. You will evaluate multiple sentences for the same table that are all in- dependent of each other. Each descriptive sentence should make sense and be grounded in information provided in the source table. For each sentence, we ask you to rate along four dimensions:
Please do not mark anything as "No" that is factu- ally incorrect, making value judgments. The ques- tion is only meant to filter out sentences like "The proportion of women declined by 30% or "Women proportion by 30%" where it is completely unclear what the text refers to or that are so ungrammatical that it becomes nonsensical.
1. The text is understandable
you select "No", you do not have to answer the remaining questions.
2. All of the provided information is fully at- tributable to the table, its title, and its unit of measure.
(Q2) All of the provided information fully attributable the table, its title, and its unit of measure. If a sentence understandable, we next ask you to read the associated table and its meta- data above it. Then, mark whether you agree with the above statement for each individual sentence. You should write "Yes" only if all of the informa- tion provided in the sentence is in accordance with the data in the table and its meta information. Lvven single error should lead to you answering "No". When making the judgment you may lenient if a number is off by a bit (e.g.. reporting 3% instead of 3.5% or "two thirds" instead of 70%). but you should select "no" if any number significantly de- viates from the corresponding number in the table.
3. How many cells does the text cover?
4. Generating the text requires reasoning or com- parison of multiple cells.
The first three dimensions are binary statements where ask you to answer with "No" (false) or "Yes" (correct) and the last one asks you count the number of cells that description covers. The sections below describe each of the dimensions in detail.
J.2 Additional notes
The descriptions may appear very fluent and well- formed, but can contain inaccuracies that are not casy to discern at first glance. Pay close attention to the table. If you unsure about any particular answer, please enter "-1" in the relevant cell.
Inferences based on numbers are okay here, as long as they are not attributed to anyone. For exam- plc, "the result merits further study" is acceptable. but "Two men claimed that the results merit further study" is not.
(Q1) The text is understandable. In this step you will evaluate whether you can understand the sentences on their own. You may consult the table
An exception from this question are references to figures If everything about a sentence is okay,

---

but it includes reference to a figure (e.g.. "Figure 2 shows X"), you may still mark this question as "yes".
Title
Total Fertility Rates by Background Characteris- ties
If you select "No", you do not have to answer
Unit of Measure
Total fertility rate
Q3 and Q4.
Kenya
4.6
(Q3) How many cells does the text cover? This question asks you count how many cells table talks about. You should count as a mention every time a description uses a value from the row or column header, or mentions a table entry. Only cells of the table (anything under START OF TA- BLE) should be counted. If a description compares multiple values, count all of them even if they are not explicitly mentioned. For example, "X has the highest Y" should count all of the cells that men- tion "Y" (a statement does not have to be true for the cells to count). The unit of measure and title should not count toward this number.
Residence
Urban
2.9
Rural
5.2
Province
Nairobi
2.8
Central
3,4
Coast
4.8
Eastern
4.4
Nyanza
5.4
Rift Valley
4.7
Western
5.6
North Eastern
5.9
Education
(Q4) Generating the text requires reasoning comparison of multiple cells. The next ques- tion asks whether sentence requires reasoning. A positive example could compare values a col- umn or row, e.g.. "X has the highest Y", or "X has more Y than Z" If a sentence contains any statement that requires such comparison or implicit reasoning, answer "Yes". In all other cases, answer "No".
No education
6.7
Primary incomplete
5.5
Primary complete
4.9
Secondary+
3.1
Table 6: The example provided in the annotation in- structions.
compactness of the typical family formation pro- cess in each country.
J.3 Example
Q1: It is completely unclear what "This" in the sentence refers to and the answer should thus be "No"
Below, show one exemplary table (Tab. 6 with descriptions and corresponding answers.
Description Fertility lowest in Nairobi province (2.8 children per woman), followed by Central province at 3.4 children per woman, and highest in North Eastern province (5.9 children per woman).
Q2: Because Q1 is "No", we leave this blank.
Q3: Because Q1 is "No", we leave this blank. Q4: Because Q1 is "No", we leave this blank.
Description 3 These differentials in fertility closely associated with disparities in educational levels and knowledge and use of family planning methods
Q1: This text is understandable, and the answer is thus "Yes".
Q2: All of the details (province names and fertil-
Q1: This sentence understandable and the an- swer is thus "Yes".
ity rates) can be found in the table. The answer is "Yes".
Q2: The mentions of disparities as reasons for the differentials is not attributable to information in the table and the answer should thus be "No". Q3: Because Q2 is "No",, we leave this blank.
Q3: Writing this sentence requires looking at all 8 values of the entire "Province" section in the table, in addition to its section title and the 3 province names. The answer is thus 12.
Q4: Because Q2 is "No", leave this blank.
Q4: The text describes "highest" and "lowest" which requires comparison. The answer thus "Yes" again.
This is particularly clear in the median ages at which events take place and the
Description

---

Train Sctup
Model
STATA QE
STATA REF
STATA QE+REF
SSA
-0.00
0.45
0.22
Small
0.36
0.16
XXL
0.20
0,56
0.39
en
SSA
-0.01
0.48
0.24
Small
0.01
0.53
0.29
XXL.
0.28
0.68
0.54
fr
SSA
0.03
0.51
0.27
Small
0.05
0.53
0.30
XXL
0.33
0.63
0,52
ha
SSA
-0.01
0.50
0.22
Small
0.01
0.53
0.23
XXL
0.20
0.55
0.32
ig
SSA
0.09
0.53
0.25
Small
0.03
0.44
0.21
XXL.
0.31
0.61
0.42
SSA
-0.01
0,48
0,23
Small
0.01
0.50
0,26
XXL
0.23
0.62
0,44
SSA
0.11
0.55
0.29
Small
0.12
0.60
0.31
XXL
0.43
0.72
0,55
SSA
0.11
0.49
0.26
Small
0.08
0.44
0.24
XXL
0.23
0.56
0.35
MONOLINGUAL
SSA
0.04
0.50
0.25
Small
0.03
0.49
0.25
XXL.
0.28
0.62
0.44
SKIP NO REFERENCES
SSA
0.05
0.51
0,26
Small
0.01
0.47
0,23
XXL.
0.61
0.76
0.74
TAGGED
SSA
0.09
0.54
0,29
Small
0.07
0.52
0,27
XXL
0.57
0.76
0.69
+ SKIP NO VALUES
SSA
0.11
0.57
0.32
Small
0.10
0.54
0.30
XXL
0.55
0.76
0.71
+ SKIP No OVERLAP
SSA
0.01
0.47
0.22
Small
0.00
0.42
0.19
XXL
0.59
0.77
0.75
Table 7: Full evaluation results using STATA. On top, the individual monolingual results, and on the bottom the aggregated results. We highlight notable numbers in the separate sections. In the monolingual setups, Swahili and English lead to the highest performance. The aggregated setups lead on average to a much better performance, with the SKIP No OVERLAP and SKIP No REFERENCES setups outperforming the others.

---

Language
Model
Understandable
Attributable
Reasoning
Cells
ar
small
0.11
0.05
1.00
mA5SSAA
0.27
0.10
1.00
4.40
xXLXXL
0.78
0.50
0,86
reference
0.86
0.56
0.84
6.11
mT5 small
0.28
0.05
1.00
4.67
mT5SSA
0.30
0.05
1.00
4.67
mT5XXL
0.87
0.47
0.74
6.79
reference
0.95
0.56
0.76
7.14
fr
mT5 small
0.14
0.22
0.33
8.17
mTSsSA
0.21
0.07
0.33
4.67
mT5XXL
0.84
0.66
0.30
5.97
reference
0.95
0.60
0.27
7.88
ha
mT5 small
0.20
0.11
0,50
11.00
mT5sSA
0.24
0.04
0.50
8.00
mT5XXL
0.66
0.42
0.46
8.23
reference
0.78
0.52
0.58
12.30
ig
mT5 mm5lllll small
0.10
0.59
0.79
6.07
m T5SSA
0.13
0.50
0.93
10.07
T55XL
0.60
0.91
0.97
8.77
reference
0.65
0.95
0.93
9.22
pt
mT5 nTlllall
0.14
0.08
1.00
9.50
T5SAA
0.20
0.03
1,00
10.00
x
0.58
0.53
0.88
8.93
reference
0.70
0.54
0.79
7.53
mTS small
0.23
0.48
0.86
TT5SS
0.40
0.38
0.68
x
0.83
reference
0.91
0.76
0.78
8.42
yo
mT5 mT5small small
0.21
0.10
1.00
8.25
mTšSSA
0.32
0.02
1.00
13.00
mT5xXL reference
0.76
0.47
8.13
0.97
0.55
1.00
Table 8: Full human evaluation results. Note that the attributable fraction is only of those examples marked understandable and realls is only answered if example attributable.


Reviews:
[{"review_id": "t5YyM7KYKP", "reviewer": "Reviewer_RmMM", "summary": "", "questions": "1. I believe the \"Reasons To Reject\" 2 is a consequence of 1\nIt would be helpful if the authors can clarify the issues mentioned in \"Reasons To Reject\" 2 during the rebuttal. \n\na)  it is mentioned that the test splits have at least three references (line:234), but are these references per language or is it cumulative?\n\nb) Why is it that a manual evaluation of all the references (of test and dev splits) is not considered, with an inter-annotator agreement (on the references alone)? \n\nc) What aspects are different in sTATA which bring more insights to the task? It is not entirely clear how can the same be generalised when preparing future datasets (assuming the required dataset specific model building is possible). In other words, why sTATA is required is not well motivated in the writing.\n\n2. Have you considered using a token free pretrained model, such as byT5, as the african languages may not be sufficiently represented in several of the tokenizer baed pretained models. byT5 has shown to generalie well for several languaegs such as Tamil, Sanskrit etc. as compared to mT5.  ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reading and posing the list of questions. We will restructure the paper to clarify the dataset configuration and evaluation setups and answer your questions below:\n\n1a) Since the dataset is fully parallel, the minimum three test reference refer to the number of examples per language. So, for example, if you sample a random English data point, it will have at least three references. Every other language for the same example will have the same number of references.\n\n1b) We agree that a full human evaluation of every test reference would have been more complete, especially with multiple annotators. Due to budgetary constraints, we had to balance between the following parameters: \n\n- number of evaluated examples\n- number of evaluated languages\n- annotation of dev (yes/no)\n- number of replicated examples\n- number of annotated system outputs\n\nMaximizing all of these would have led to costs higher than constructing the full dataset. For that reason, we chose to maximize the number of evaluated examples, languages, and systems, which came at the cost of not being able to report inter-annotator agreement on the evaluations. To combat this limitation, we compared annotations to a small set of gold-annotations produced by the dataset authors as a qualification test, as reported in lines 367-376. \n\n1c) Why is StATA required: As shown in Table 3 and discussed in 5.2, no automatic metric has a high correlation with attribution, which means that it is impossible to detect whether a model faithfully describes the underlying table -- the core of the machine learning task. Moreover, in Appendix G we show that standard metrics actually lead to misleading and wrong results when conducting cross-lingual experiments. Thus, for TaTA to be a useful artifact, better metrics are needed. \n\nIs StATA generalizable? While there is some evidence that trained metrics generalize (see, e.g., BLEURT/BERTScore’s ubiquitous application beyond MT), we do not want to make such claim without empirical evidence. The main lesson we learned from our experiments is that (1) metrics need to always be validated in how they are used, and that (2), domain-specific metrics may be necessary. Especially as the field more and more relies on trained metrics (reward models for RLHF, for example), we consider this an important data point to contribute to the discourse and will make this more clear in our paper. \n\n2) Thank you for the suggestion of using token-free models. We considered ByT5 and for a similar reason experimented with the SSA-T5 model. Since that model did not yield promising results, we did not add additional experiments with ByT5, especially considering that the core contribution of our work is the dataset itself and not the models. Another consideration against ByT5 is that using it for generation is often prohibitively slow, since multiple inference steps are required for every single character. \n\nWe will further add more information about the translators and annotators to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The resource will be a valuable addition to the community - It consists of table-to-text generation data for four African languages, in addition to 5 other \n\n- The evaluation is thorough\n", "reasons_to_reject": "1. The draft in its current form is not easy to follow, as the information seems to be scattered all over the place. I believe the writing can be substantially simplified which would only improve the overall quality of the work.\n\n\n2. While the dataset is thoroughly evaluated for the problem-setting, it is not entirely elaborated on the sanity checks for the human evaluation for the dataset preparation. For instance, it is mentioned that the test splits have at least three references (line:234), but it is not clear if these references are per language or cumulative. Further, it is mentioned that only one reference is manually evaluated (line:359). However, it is not mentioned why that was the case. Especially, since evaluation (and metrics) is a central aspect of the work, I believe more details or justification needs to be provided here. Further, It is not entirely clear to me why the inter-annotator agreement (IAA) wasn't provided for the human evaluation of the test set (or for that matter line 367 to 369 was not entirely clear to me regarding output evaluation and IAA). Similarly, it is stated (and established) that the dataset-specific metric correlated better with human judgment. But it is not entirely clear what aspect is unique to the dataset, where the specific metric is better suited for. In other words, there is nothing stated on how the metric can be generalized in other table-to-text problem settings ( say for future datasets)\n\n3. No details of translators and transcribers are present, including the number of translators.", "questions_to_authors": "1. I believe the \"Reasons To Reject\" 2 is a consequence of 1\nIt would be helpful if the authors can clarify the issues mentioned in \"Reasons To Reject\" 2 during the rebuttal. \n\na)  it is mentioned that the test splits have at least three references (line:234), but are these references per language or is it cumulative?\n\nb) Why is it that a manual evaluation of all the references (of test and dev splits) is not considered, with an inter-annotator agreement (on the references alone)? \n\nc) What aspects are different in sTATA which bring more insights to the task? It is not entirely clear how can the same be generalised when preparing future datasets (assuming the required dataset specific model building is possible). In other words, why sTATA is required is not well motivated in the writing.\n\n2. Have you considered using a token free pretrained model, such as byT5, as the african languages may not be sufficiently represented in several of the tokenizer baed pretained models. byT5 has shown to generalie well for several languaegs such as Tamil, Sanskrit etc. as compared to mT5.  ", "paper_topic_and_main_contributions": "The authors release a table-to-text dataset consisting of a total of 8,700 instances spread across nine languages, including four African languages. One of the languages, Russian, is added as a test split for zero-shot evaluation. Based on PDF reports spanning a couple of decades on the Demographic and Health Surveys Program by USAID, the authors crowdsourced the transcription and translation for eight languages. Based on the linearized forms of these tables, they perform table-to-text in monolingual, cross-lingual, and multilingual setups (using a common multilingual model). They evaluate the models using a number of evaluation metrics, which consider reference free as well as with reference metrics. Further, propose their own dataset-specific metric, which they demonstrate to correlate better with human ratings than other metrics. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KR0rRjVijX", "reviewer": "Reviewer_qzSY", "summary": "", "questions": "Please explain earlier in the text what \"attributable language\" means.\n\nFootnote 3 “We use the terms transcribe and transcription as a shorthand for the process where the images of charts and diagrams (info-graphics) and their descriptions are manually converted by human annotators into spreadsheet tabular representations.” Seems to be in contradiction with the statement (page 2) that “Our approach aims to mitigate these issues by transcribing naturally occurring descriptions.” This needs clarification.", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind words. We are excited about contributing to making under-documented and under-resourced languages a higher priority in the *CL community. \n\nIn our edits we will move the definition of attribution earlier. We will also clarify that the “issues” in the statement on page 2 refer to crowd workers actively coming up with textual descriptions, rather than just crowd sourcing the task of extracting existing naturally occurring text from PDFs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Under-documented and under-resourced languages should be a priority of the field, and this paper is a valuable contribution.", "reasons_to_reject": "None.", "questions_to_authors": "Please explain earlier in the text what \"attributable language\" means.\n\nFootnote 3 “We use the terms transcribe and transcription as a shorthand for the process where the images of charts and diagrams (info-graphics) and their descriptions are manually converted by human annotators into spreadsheet tabular representations.” Seems to be in contradiction with the statement (page 2) that “Our approach aims to mitigate these issues by transcribing naturally occurring descriptions.” This needs clarification.", "paper_topic_and_main_contributions": "This paper provides a new dataset for generating text from tabular data in 8 languages, including 7 languages used in Africa (plus Russian). The authors also provide an evaluation of the dataset and of the texts generated. This evaluation shows that half of the texts are not understandable or attributable. They also provide new metrics for human evaluation.\n\nI am unable to evaluate the contribution made by the paper regarding the evaluation metrics, but am highly confident that the dataset and the process of text generation are of great value to the field.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "gLZF0kvik6", "reviewer": "Reviewer_KJRS", "summary": "", "questions": "Question A: how the task input look like? Is it an image of the graphic or text?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and improvement suggestions. We will clarify our task definition and add information about the annotators. Regarding the agreement numbers, we refer to our answer to Reviewer RmMM and lines 367-376 in the paper which state that we worked with the annotators until they reached F1 scores of about 0.9 on an internal test set. \n\n> Question / Suggestion 2: how does the task input look like? Is it an image of the graphic or text? \n\nFor the text-based models, the input is provided in Figure 2 and the process of arriving at this representation is described in 3.4. We did not consider image-to-text models as part of this paper.\n\n> Suggestion 1: Clarify notion of “African” and “spoken in Africa” languages and justify Russian. \n\nThank you for this important point. Outside the main acronym, we tried to avoid referring to the European languages as African languages and only as “languages spoken in Africa” but will work on further disambiguating this confusion. Russian was specifically chosen as a language for which data was available and which is not directly related to any of the other languages to test “real” zero-shot performance. \n\n> Suggestion 3: Table 4 is never references in the main text. \n\nIt is referenced at the beginning of Section 5.3 which describes the results in detail.\n\n> Suggestion 4: The information about annotators, their spoken language, the annotators overlap and their agreement should be\nreported. \n\nWe will add information about annotators to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed dataset is novel;\n- The experiments are thoroughly documented.", "reasons_to_reject": "- The task definition is unclear;\n- There is no information about annotators that participated in data collection nor the annotators agreement on the data.\n- There are some flows in the presentation.\n\nSee presentation improvements section.", "questions_to_authors": "Question A: how the task input look like? Is it an image of the graphic or text?", "paper_topic_and_main_contributions": "The paper present a novel dataset for table-to-text generation for languages spoken in Africa. The authors provide dataset collection description as well as both automatic and human evaluations of models fine-tuned on their data. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The name of the papers mentions only African languages, however, the dataset includes several European languages. From the paper, my understanding is that those languages are spoken in Africa as well. So, my suggestion would be to change the name and description in the text to \"languages spoken in Africa\". However, the relation of Russian language is still unclear.\n\n2. There should be an example of input to the model. Right now, there are several examples but the input are images. It is unclear how it correlates with text-to-text generation models.\n\n3. Table 4 is never references in the main text.\n\n4. The information about annotators, their spoken language, the annotators overlap and their agreement should be reported."}}]

Author rebuttal:

Meta-review: All the reviewers agree that the paper contributes an extremely valuable multilingual dataset resource for the community as the dataset covers many low-resource languages and is curated carefully via transcription and translation with professional translators. They perform sufficient benchmark experiments and propose a better automatic metric system that better correlates with human ratings than the existing ones.

As mentioned by two other reviewers, and noticed by myself, the paper is slightly hard to read due to the authors' choice of presentation of information in the paper. For instance, the authors currently grouped their proposed metrics STATA under the experimental setup when their intention is to point out existing issues with existing automatic evaluation metrics and hence a need for better metrics (as discussed by the authors in their rebuttal). I believe the current paper structure may be one of the reasons why reviewers RmMM and qzSY have a difficult time evaluating the contributions of the proposed metric STATA, although it's much clearer now (to me) after the authors clarify the contributions of STATA in the rebuttal. 

Another weakness of the paper is the missing (important) information about the translators and annotators, which should be added to the revised version of the paper. The authors did not give much details about them in the rebuttal. The paper could have been also made stronger by clarifying the choice of Russian as the zero-shot test language (as the authors did in the rebuttal and should add to the revised section 3.3 of the paper).
Decision: Accept-Findings