Venue: NAACL 2024

# Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo

Barkavi Sundararajan and **Somayajulu Sripada** and **Ehud Reiter**

Department of Computing Science, University of Aberdeen

{b.sundararajan.21, yaji.sripada, e.reiter}@abdn.ac.uk

###### Abstract

Neural Table-to-Text models tend to hallucinate, producing texts that contain factual errors. We investigate whether such errors in the output can be traced back to problems with the input. We manually annotated 1,837 texts generated by multiple models in the _politics_ domain of the ToTTo dataset. We identify the input problems that are responsible for many output errors and show that fixing these inputs reduces factual errors by between 52% and 76% (depending on the model). In addition, we observe that models struggle in processing tabular inputs that are structured in a non-standard way, particularly when the input lacks distinct row and column values or when the column headers are not correctly mapped to corresponding values.

## 1 Introduction

Table-to-Text generation refers to the task of generating natural language descriptions from tabular data Parikh et al. (2020); Chen et al. (2020a, b) and is widely used in several application domains such as medical diagnosis Pauws et al. (2018), financial Zhu et al. (2023) and weather reporting Sripada et al. (2002); Gkatzia et al. (2017); Upadhyay and Massie (2022) and sports summaries Thomson et al. (2020). Neural language models are known to generate fluent texts Ji et al. (2023) but may generate outputs that are factually incorrect or unrelated to the provided input data. Such undesirable generation is called 'hallucination' Wang and Sennrich (2020); Raunak et al. (2021); Ji et al. (2023).

Previous studies on Table-to-Text tasks adopt traditional seq2seq methods to generate table descriptions Wiseman et al. (2017); Puduppully et al. (2019); Rebuffel et al. (2019). Recently, Transformer based models Devlin et al. (2019); Raffel et al. (2020); OpenAI (2023) have shown remarkable progress in language generation from textual input Badaro et al. (2023), however tabular data still needs more improvement to control hallucinations Rebuffel et al. (2021). Neural models struggle with tabular data, especially when the inputs do not have distinct cell values from rows and columns mapped along with their respective headers. These input problems lead the model to generate more factual errors Kasner and Dusek (2024).

Using the ToTTo tabular dataset Parikh et al. (2020), we identify and address input problems that are responsible for factual errors. Some common tabular input problems in the ToTTo are **i.** 'non-atomic' cell values, where a column contains multiple values such as leader name, party name and % of votes in one cell rather than a single indivisible value, **ii.** missing important cell values in the input (see Table 1) and **iii.** nested column headers and row headers in the Wikipedia tables1 that lead to incorrect mapping of the cell values.

Footnote 1: [https://en.wikipedia.org/wiki/Wikipedia:Manual](https://en.wikipedia.org/wiki/Wikipedia:Manual) ̇of ̇Style/Accessibility/Data ̇tudes ̇tutorial*Column ̇headers ̇bad ̇example

Table 1 presents a sample from ToTTo. Only the highlighted cells from Table 1a are passed to the model (as shown in Table 2). Passing Norton's % of votes and her party name (compare Table 1b to Table 1a) eliminates the hallucinated % of votes (see output for Table 1b); this correction is based on the input problem described in Sec. 5.2.

In this paper, we score the quality of output texts by manually annotating output errors instead of using automatic evaluation metric scores such as BLEU Papineni et al. (2002), ROUGE Lin (2004), PARENT Dhingra et al. (2019) and BLEURT Sellam et al. (2020). We conducted a pilot study, where we fine-tuned T5-base and T5-large models Raffel et al. (2020), analysing 1,677 _politics_ domain texts from the ToTTo dataset through manual error analysis adopted from Thomson and Reiter (2020). These manual error annotations allowed us to identify patterns of errors in the generated text which were then traced back to input problems.

Our approach is summarised as follows:

1. We systematically correct the tabular inputs for the _politics_ domain in ToTTo to adhere to a standard form to ensure the generation of factual texts from neural models. The correction procedure is elaborated in Sec. 5.2 and is supplemented by pseudocode in App. D. 1. We apply this correction to a larger subset of 210 samples, resulting in a 62% decrease in factual errors for T5-base and a 57% decrease for T5-large in the generated text (Sec. 6.1). 2. We conduct experiments on Llama 2-7B and Llama 2-13B models [48] with 40 challenging samples selected from the previous 210 samples. Tailoring zero-shot prompts for specific input and error annotation on 160 texts showed that correcting input reduces factual errors by 52% in Llama 2-7B and 76% in Llama 2-13B (Sec. 6.2).
2. The manual error annotation methodology adopted from Thomson and Reiter [9] is detailed in App. B; this builds on the work of Sundararajan et al. [10] for ToTTo _politics_ domain outputs2. The inter-annotation agreement on the error annotation was good, with a Fleiss' Kappa of 0.622 (Sec. 7).

Footnote 2: Error annotation guidelines and sample annotations from our human evaluation is available at [https://github.com/SarkaviSIJ/totto_politics_human_annotations](https://github.com/SarkaviSIJ/totto_politics_human_annotations)

### Table to Text dataset, ToTTo

ToNTo is an open-domain English language dataset [13], where the input \(X\) is taken from Wikipedia table \(T\), which includes the table's metadata (such as title) and a set of highlighted cells, along with their headers. This structured information is flattened to create a linearized text representation of the table, as mentioned in Table 2. This crowdsourced dataset is paired with a natural language description, denoted as output \(Y\), comprising a sequence of tokens \(y_{1},y_{2},\dots,y_{n}\), which provides a coherent summary of the content present in the input table \(X\).

These input-output pairs from the ToTTo dataset can be used for fine-tuning or prompting the neural language models. As shown in Table 0(a), the Input X is often observed to be problematic and fixing these problems is the main focus of this paper.

## 2 Related Work

**Prior work on ToTTo:**Wang et al. [10] proposed a framework, LATTICE, that preserves the structural information of the cell values (tokens) within the same rows or columns, and by removing the attention mechanism flow for other unrelated tokens. Hu et al. [11] incorporates content planning and generation from Su et al. [10] and synthetically added noisy cells in their fine-tuning regime. While these approaches are model agnostic and improved automatic metric scores such as

\begin{table}
\begin{tabular}{l|c|c|c|c|} \hline  & \multicolumn{4}{c}{**(a) Original cells highlighted in Yellow as Input**} \\ \multicolumn{4}{c}{**1996 United States House of Representatives election**} \\ \cline{2-5}
**Party** & **Candidate** & **Votes** & **\%** \\ \hline Democratic & Eleanor Holmes & 134,996 & 90.00 \\  & Norton (inc.) & & \\ \hline Republican & Sprague Simonds & 11,306 & 7.54 \\ \hline \end{tabular} \begin{tabular}{l l} \hline \multicolumn{4}{c}{**Llama 2-13B Output for tabular input a:**} \\ Eleanor Holmes Norton (inc.) won with **7.54\%**\({}^{\text{U}}\) of the vote. \\ Sprague Simonds was the Republican candidate and received **22.38\%**\({}^{\text{U}}\) of the vote. \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline \multicolumn{4}{c}{**Llama 2-13B Output for tabular input b:**} \\ Democratic Party candidate Eleanor Holmes Norton won with 90\% of the vote. Republican Party candidate Sprague Simonds received 7.54\%. \\ \hline \end{tabular}
\end{table}
Table 1: ToTTo example: Highlighted cells in yellow are passed as input to the model. Passing the appropriate cells (i.e., % votes and party name) as input, as shown in Table 0(b) fixes the factual errors. Compare the Table 0(b) output (with no errors) to the Table 0(a) output (with NUMBER errors denoted by a superscript U).

\begin{table}
\begin{tabular}{l} \hline <page_title> 1996 United States House of Representatives election </page_title> <table>> <cell> Eleanor Holmes Norton (inc.) \\ <col_header> Candidate </col_headep> </cell> <cell> Republican <col_header> Party </col_headep> </cell> <longue> Symons <col_header> Candidate </col_header> </cell> <cell> > <cell> > ?54 <col_header> \% \% \textless{}/col_header> \% \textless{}/col_header> \% \textless{}/col_header> \textless{}/col> \textless{}/table> 

BLEU (Papineni et al., 2002), PARENT (Dhingra et al., 2019) and BLEURT (Sellam et al., 2020) by few points in the leaderboard 3, the fundamental problem with the tabular input remains.

Footnote 3: [https://github.com/google-research-datasets/ToTfo](https://github.com/google-research-datasets/ToTfo)

Chen et al. (2022) acknowledged that the target cells in the ToTTo tabular input are not always highlighted in their work titled 'Table Structure Understanding and Text Deliberation Approach'. They used a template to extract all facts from the raw table for 1.2K training samples (only for the inputs with rows and columns fewer than 8) and employed hierarchical multi-head attention to capture the structural information in their fine-tuning process. Though this approach promises to retain the facts from raw tables, it only addresses simpler tables with fewer than 8 rows and columns, still has limitations for longer tables, complex tabular structures and non-atomic tabular cells.

Our focus on correcting inputs aiming to achieve factually correct outputs aligns with the work of Dusek et al. (2019) on the E2E dataset (Novikova et al., 2017). Their study also demonstrated that improving inputs in an NLG dataset helps in improving model outputs.

**Error Analysis:** While the automatic metric scores such as BLEU, PARENT and BLEURT help evaluate the model's performance at a high level, relying solely on these metrics will not address specific weaknesses i.e., lower metric scores do not provide insights into specific error types in the output. We follow guidelines from van Miltenburg et al. (2021) to perform error analysis in NLG systems and investigate errors in the output at a more granular level by adopting the manual error annotation approach from Thomson and Reiter (2020); Sundararajan et al. (2022); Thomson et al. (2023).

Maynez et al. (2020) also emphasized automatic metrics are not sufficient to study the hallucination problem and provided a detailed study on intrinsic and extrinsic hallucination in abstractive summarization. We studied hallucination in our evaluation scheme by annotating different categories of errors in the output tokens (single token or group of tokens). Mapping our adopted methodology to Maynez et al. (2020)'s work, intrinsic is the main error category occurring when generated outputs are not faithful to the given input. It includes WORDW, NAMEN, DATEDIMENSIOND, NUMBERUR 

the input (Gonzalez Corbelle et al., 2022) without making any factual errors. If the output has errors and omissions, we classify it as an error. The T5-b had 36% omissions and T5-l had 29%.

**Errors**: Our analysis revealed that T5-b made factual errors in 17% and T5-l made factual errors in 11% of the total samples.

**Hypothesis:** Based on the insights from this analysis, we hypothesize that when tabular input data is structured in non-standard ways, models struggle to interpret these ambiguous inputs leading to generate factually inaccurate output. We test this hypothesis by addressing input problems related to non-standard tabular structures.

## 4 Input Problems

Due to the practical challenges involved in improving the tabular input for the entire dataset, which includes unique headers and tabular structures for each input, our focus is on analyzing a subset of samples containing errors. We examined 124 error samples from the T5-b and 86 error samples from T5-l models within the ToTo_politics_ domain, as identified in Table 3. We aim to segregate the errors originating from non-standard or illogical nested table structures. We categorize these input problems into two broad categories, as briefly elaborated upon in Sec. 4.1 and Sec. 4.2.

### Generic Input Problems

**Non-atomic tabular cell values:** When a table cell contains multiple atomic values (see Table 4). Examples of such non-atomic forms include multiple leaders' names, votes, term dates, or election years all in a single cell. We further categorize these problems into 'single record lacking atomicity' and 'multiple records lacking atomicity' (shown in Fig. 2 in App. F) to demonstrate how the models struggle when records of multiple leaders lack atomicity.

**Complex table type:** When a table contains election results in sentence form, models struggle to interpret and generate meaningful texts because the sentence form data lacks the needed context (see Table 11 in App. F).

**Insufficient input:** In some cases, the necessary cells are not highlighted in the tabular input, resulting in incorrect outputs. Our analyses in Table 1 and Table 5 demonstrate that outputs become factually correct when relevant cells are included.

**Longer table input:** Models often struggle to generate accurate texts for lengthy table inputs, especially when the data is not in a standard form and lacks clear cell relationships.

### Politics Domain-Specific Input Problems

**Politics specific headers:** In ToTTo, the use of symbols, for example, '+' or '-' instead of having clear semantic terms like _'swing percentage' or '% change compared to previous election'_ as column headers caused 5% of errors. This lack of semantic guidance in the input made it difficult for models to accurately generate the correct output text (see Table 10 in App. F).

**List of leader names in the input:** In the _polities_ domain, we observed a specific issue when input data contains a list of leader names. Models tend to favour the leader whose name appears first in the list (for example, either from the table title or as the first leader in highlighted cells), even if they have lost the election (see Table 13, Table 14, Table 15 in App. F). This becomes even more challenging when leader names are associated either with missing values (opponent leader's name or vote count) in other columns in the input or when the tabular cell values are non-atomic.

The manual fixes we applied to each of these input problems with examples are detailed in Sec. 5.2.

## 5 Main Study

### Llama 2 Models

In our main study, in addition to T5 models, we included Llama 2-7B (L-7B) and Llama 2-13B (L-13B) models to test our hypothesis on non-standard tabular inputs. We received official model weights from Meta4 and then quantized our

\begin{table}
\begin{tabular}{c|c c|c c} \hline \multirow{2}{*}{**Category**} & \multicolumn{2}{c}{**T5-base (T5-b)**} & \multicolumn{2}{c}{**T5-large (T5-l)**} \\ \cline{2-5}  & **Count** & **\%** & **Count** & **\%** \\ \hline
**No Error** & 358 & 47 & 450 & 60 \\
**Omissions** & 272 & 36 & 218 & 29 \\
**Errors** & 124 & 17 & 86 & 11 \\ \hline
**Total Count** & **754** & \multicolumn{2}{c}{**754**} \\ \hline \end{tabular}
\end{table}
Table 3: Pilot Study analysis: T5-b and T5-l Models in ToTTo_Politics Domain. ‘Errors’ category is the main focus of our work; ‘omissions’ are excluded.

 

Llama 2 models. We ran experiments using the full model weights and the model with 4-bit integer quantization for the ToTTo dataset (_politics_ domain). Both produced comparable output quality with no difference in errors for the quantized models. Therefore, we finalized our experiments with 4-bit quantized models to save computational resources and used them for local inference on a MacOS CPU. We set the temperature to 0.0 as it produced fewer errors. Further details on data corrections for these models are discussed in Sec. 5.4.

### Manual Fixes for Input Problems

We followed a systematic procedure to apply manual fixes to specific input problems, as discussed from Sec. 5.2.1 to Sec. 5.2.6. A supplementary pseudocode for these fixes is also provided in App. D. The algorithm 1 takes tabular data and title (metadata) from ToTTo as input parameters to execute six functions to gather insights and return with corrected tabular data. The first two functions, DataSizeManageableForLongTable and IdentifyLeaderNameOrder do not correct the tabular input but provide insights on the input problems leading to factual errors in the outputs. The other four functions correct the input problems as shown in algorithm 4, algorithm 5, algorithm 6 and algorithm 7.

#### 5.2.1 Non-Atomic Cell Values

**Correction:** We corrected the input to store individual leader data, including votes, party, and election details, as a separate column variable. In Table 3(a), when the 'candidates' column is not atomic, all models made errors for this tabular data. The corrections involved separating each leader's party and votes into separate columns (see Table 3(b)). We passed the leading two leaders' data, excluding the remaining leaders, in separate rows for the respective states (Alaska and Virginia). After input correction, T5-b and L-7B outputs in Table 4 were error-free. While some models omitted other candidate and election details, _all models corrected the previous factual errors after the correction_.

#### 5.2.2 Complex Table Type

**Correction:** Complex tabular structures were generally difficult to correct because the election results are in sentence form e.g., see 'Results' column in Table 11 in App. F. We could only make corrections to the 'non-atomic' cell values in 'candidates' column. Three models generated 'the incumbent senator Coe I. Crawford lost renomination to Edwin S. Johnson, a DemocratcandidateC'.' Losing renomination is when the current senator did not lose the seat to the opponent, but rather failed to be nominated in the primary to stand for reelection. This resulted in CONTEXT\({}^{\text{C}}\) error, specifically _when the models made unsupported assumptions_

\begin{table}
\begin{tabular}{l|c|c|c} \hline
**State** & **Candidates** & **Party** & **\% votes** \\ Alaska & Dan Sullivan & Republican & 48.0\% \\ \hline Alaska & Mark Begich & Democratic & 45.8\% \\ Virginia & Mark Warner & Democratic & 49.1\% \\ \hline Virginia & Ed Gillespie & Republican & 48.3\% \\ \hline \end{tabular}
\end{table}
Table 4: **Fixing Multiple leaders’ votes that are not atomic, as detailed in Sec. 5.2.1**. ‘Before fix’ shows the errors produced by models with Table 3(a) as input and ’After fix’ shows both models did not produce any factual error with Table 3(b) as input. The error tokens in outputs are annotated with respective error code in superscript (W for WORD, U for NUMBER and A for ADDITION errors).

 

about the given input data_, as defined in App. B. Even after correction, all models except L-13B made this \({\textsc{CONTEXT}^{C}}\) error (see Table 10(d)).

#### 5.2.3 Insufficient Input

**Correction:** This is a data annotation problem because it is related to how the ToTto dataset was created by pairing the sentence description and highlighting only a subset of cells from the Wikipedia table as Input (X). This problem is straightforward to fix by including the missing cells or headers from the tabular data.

In Table 5, when the choices for the Minnesota Amendment (i.e., 'Referendum failed No' and 'Yes' cells) were not included, all models incorrectly generated the favour of votes (_'approved'_, _'for'_) and made an unsupported assumption regarding the _turnout_ percentage. As shown in Table 5, all models corrected the errors after including the vote choices.

#### 5.2.4 Longer Table Input

**Correction:** When dealing with straightforward tabular data, it was easier to correct the input either by **i.**_correcting the cell values to ensure atomicity_ and/or **ii.**_including the missing cells_. However, this longer tabular input not only had over 100 rows but also included complex structures with nested column headers and row headers which posed difficulties in correcting the input.

Large models such as L-13B and T5-l could process tables with fewer than 20 rows for straightforward inputs. However, T5-b and L-7B struggled to produce factual information even for 20 rows. In our correction procedure, we chose an upper limit of 20 rows and 10 columns to simplify the longer tabular input. A simplified example of this problem with fewer rows is shown in Table 16 in App. F.

#### 5.2.5 ToTTo: Politics Specific

**Correction:** We included appropriate semantic cues in the headers to help the model differentiate between the generic percentage of votes and the swing percentage of votes. For other errors, we expanded the abbreviation of coalition party names to avoid errors. In Table 10 from App. F, the original input only had \(\pm\) in the header, resulting in errors from all models including L-13B. After explicitly including '\(\pm\) seats compared to the previous election', both Llama 2 models corrected the previous error. However, both fine-tuned T5 models committed mistakes even after this correction.

**Specific Semantic Generation Issue:** In some cases, the model cannot determine whether a minister was 'shortest-lived' or 'longest-lived' only based on their lifespan, birth, and death years from the input. It _may require additional context_ to produce accurate text. One such example is shown in Table 12. All models with the original data produced incorrect \({\textsc{WORD}^{W}}\). After we included the 'Total time in office' cell and customized the prompt for Llama 2, both Llama 2 models corrected the \({\textsc{WORD}^{W}}\) error. However, both T5 models did not show any improvement after including this detail. This could be due to the strong influence of the patterns learned from the fine-tuning process for similar tabular structures.

#### 5.2.6 List of Leader Names

**Correction:** We did not change the order of the leader's name in the title, but we addressed missing vote counts and leader names to map the relations for each record as shown in Table 13, Table 14, and Table 15 in the App. F.

In Table 14, the title leader 'Chuck DeVore' lost the election among four other party nominees. Both

\begin{table}
\begin{tabular}{c|c|c} \hline
**Choice** & **Votes** & **\% Votes** \\ \hline Referendum failed No & 1,588,752 & 53.84 \\ \hline Yes & 1,362,030 & 46.16 \\ \hline \end{tabular}
\end{table}
Table 5: **Fixing Insufficient input problems, as detailed in Sec. 5.2.3**. ‘Before’ shows WORD and CONTEXT errors due to missing choices of votes. ‘After’ shows these errors are resolved when these cell values are provided.

 

T5 models made errors by focusing on '_Chuck Device' as the main candidate_, resulting in \(\mathrm{WORD}^{\mathrm{W}}\) and \(\mathrm{CONTEXT}^{\mathrm{C}}\) errors. \(\mathbf{NUMBER}^{\mathrm{U}}\) errors are present in all models due to missing votes. After including the votes and party details for all candidates, both Llama 2 models corrected all errors. However, both T5 models still made errors stating the title leader either defeated or being defeated by all candidates, possibly due to the _learned patterns during fine-tuning on the ToTTo dataset_.

In Table 15, the title leader, 'Joseph Haslet' ran as a candidate for Governor office in 1804 and 1807 but lost both elections. All models incorrectly generated Haslet won in both years. The correction is only limited to passing the party name and modifying the header from 'Subject' to 'Candidate'. Despite this change, all models continued to generate errors. Both T5 models struggled to generate the right winning candidate in 5 out of 11 samples (Fig. 2a) for this input with the same set of headers.

### T5 Models on Corrected Data

Based on the insights gathered from different input problems in Sec. 4, we followed the procedure and manually corrected the original tabular input for 210 samples (taken from 124 and 86 'errors' category samples in Table 3). We ran both T5-b and T5-l models on the corrected data.

### Llama 2 Models on Corrected Data

From the corrected data, as described in Sec. 5.3, we selected 40 challenging samples from the previous 210 samples as inputs to the Llama 2-7B (L-7B) and Llama 2-13B (L-13B) models. These samples were chosen to cover each of the input problem types described in Sec. 4, guaranteeing a thorough analysis. Within these 40 samples, 21 are associated with general input problems and the remaining 19 are specific to the _politics_ input problems, as shown in Fig. 2c and Fig. 2d (in App. F).

We studied the _zero-shot_ capabilities of L-7B and L-13B models for the 40 challenging samples. For each sample, we employed different prompts tailoring to the content of the tabular input Kojima et al. (2022). Two common prompts we used are shown in App. C. For each sample and each of L-7B, and L-13B, we examined the outputs before and after correcting the tabular input.

## 6 Results and Discussion

We manually annotated the outputs from all models working on corrected input data, following the same procedure outlined in Sec. 3.1 and defined in App. B. In this section, we summarize the results of the total input data corrections and discuss them. The previous section also described and discussed error analysis locally while presenting individual corrections.

### Error Reductions in T5 Models

The input corrections significantly reduced factual errors, as evident in Table 6, which provides a comparison of error counts 'before' and 'after' input data fixes for each error category. It should be noted that _no prompt or instruction_ was provided to these two fine-tuned T5 models.

### Error Reductions in Llama 2 Models

In Table 7, we present the error analysis of 40 difficult samples, validated using L-7B and L-13B models with tailored prompts for each input. Outputs were manually error annotated, and the table compares the error counts before and after input correction. The _L-7B and L-13B_ models showed **reductions of 52% and 76% in errors**, after addressing tabular input issues. The _TS-b and T5-l models_ demonstrated **error reductions of 46% and 53%** respectively for the 40 difficult samples considered, which was shown for comparison purposes. The insights gathered from the individual error categories after input correction are mentioned below.

* \(\mathrm{WORD}^{\mathrm{W}}\) errors, predominantly incorrect verbs and prepositions, were most common among all models. Post-correction, the L-13B model reduced this error category by 77%, while the T5-b, T5-l, and L-7B models

\begin{table}
\begin{tabular}{l|r r|r r} \hline \multirow{2}{*}{**Category**} & \multicolumn{2}{c|}{**T5-base (T5-b)**} & \multicolumn{2}{c}{**T5-large (T5-t)**} \\  & \multicolumn{1}{c}{**Before**} & \multicolumn{1}{c|}{**After**} & \multicolumn{1}{c}{**Before**} & \multicolumn{1}{c}{**After**} \\ \hline WORD & 62 & 20 & 31 & 12 \\ NAME & 13 & 3 & 12 & 2 \\ DATE\_DIMENSION & 7 & 1 & 6 & 1 \\ NUMBER & 12 & 1 & 5 & 0 \\ OTHER & 5 & 2 & 2 & 0 \\ CONTEXT & 13 & 3 & 16 & 4 \\ ADITION & 2 & 1 & 2 & 1 \\ NON-ENGLISH & 20 & 20 & 21 & 21 \\ \hline Total errors & 134 & **51** & 95 & **41** \\ \hline
**Error reduction** & \multicolumn{1}{c|}{**62\%**} & \multicolumn{1}{c}{**57\%**} \\ \hline \end{tabular}
\end{table}
Table 6: Count of individual error annotations for 210 samples. The table compares the error counts between ‘before’ and ’after’ applying input correction. Each sample can contain multiple errors.

 

achieved reductions of 33%, 42%, and 34%, respectively.
* Input correction led to a reduction in \({\rm NAME}^{\rm N}\) and \({\sf CONTEXT}^{\rm C}\) errors across all models.
* The original input data exhibited more \({\bf NUMER}^{\rm U}\) errors in Llama 2 models compared to T5, which were significantly reduced after correction. Both Llama 2 models completely resolved \({\tt DATE_DIMENSION}^{\rm D}\) and \({\sf OTHER}^{\rm O}\) errors.
* ADDITION\({}^{\rm A}\) errors are more common in Llama 2 models than in T5. Despite including the prompt 'Use only the information mentioned in the input table data' and correcting the tabular data, both L-7B and L-13B models still produced five ADDITION errors each.

Table 8 shows two rows of data from our analysis of the 40 challenging samples. The first row shows that corrected data leads to an increased number of samples with 'no error' (Higher is better). However, the second row shows that omissions have increased after input corrections. We observed that the models omitted part of the information either when the corrected tabular data had multiple column variables for more than two records or when the tabular structure was complex. In the case of fine-tuned models, it learned to omit some information during the fine-tuning process. In our future work, we plan to study the reasons why this is happening.

### Model-Specific Results for Different Input Problems

In App. E, Fig. 2 presents how the four models are performing before (left bars) and after input corrections (right bars) for each input problem type.

**Non-atomic cell values:** T5-l and L-13B models corrected over 90% of the errors for this problem type, single record and multiple records lacking atomicity (see red and green bars in Fig. 2, App. F). T5-b and L-7B models corrected over 60% of the errors but still struggled with a few samples even after correction. For example, Fig. 2b shows that T5-large was making more errors for the input problem type labelled 'Multiple records lacking atomicity' before correction (in red) and made significant reductions after correction (in green).

**Complex table:** Due to the limitations in some tabular inputs, which require additional context, T5-b could not fix the errors. T5-l omitted the error for one sample, while L-7B and L-13B models partially fixed this input problem (see Fig. 2).

**Institicient input:** This data annotation problem fixed all factual errors in T5-b, T5-l and L-13B

\begin{table}
\begin{tabular}{p{56.9pt}|c c|c c|c c|c c} \hline \hline \multirow{2}{*}{**Category**} & \multicolumn{2}{c|}{**T5-base (T5-b) baseline Before After Before After Before After Before After Before After After Before 

after correction except for L-7B which added additional information for one sample.

**Longer table input:** Large models such as L-13B, L-7B and T5-l corrected factual errors for straightforward inputs, especially for tables with fewer than 20 rows. However, T5-b struggled the most to produce factual information.

**Politics specific semantic issue:** For the corrected input, L-13B fixed the factual mistakes for 5 out of 8 samples and L-7B fixed factual errors for 4 out of 8 samples. Both T5 models corrected 3 out of 8 samples (see Fig. 2).

**List of leader names:** T5 models struggled the most to correct factual errors for this problem. After correction, the L-7B model corrected three samples but produced factual errors for the remaining eight samples. L-13B made factual errors only for two samples (see Fig. 2).

While some models struggle with specific inputs, particularly regarding leader name order, tables requiring additional context for complex tabular structures, and semantic issues with symbols, our overall results indicate that correcting tabular inputs improves model outputs.

## 7 Inter-Annotation Experiment

One of the authors manually annotated errors in 1,508 outputs before input correction and 169 problematic outputs after correction (a total of 1,677 from both T5 models). Similarly, the annotation for both Llama 2 models covered 160 outputs before and after input correction.

Two additional annotators annotated 60 outputs each, generated by four different models both before and after input correction. We provided them with a detailed document that included definitions of error categories, guidelines, tabular inputs and output texts for error annotation5. Annotators followed the guidelines and marked the errors. Each annotator spent approximately 3 hours to complete this experiment.

Footnote 5: We release our error annotations from our human evaluation on [https://github.com/BarkaviSJ/totto_politics_human_annotations](https://github.com/BarkaviSJ/totto_politics_human_annotations).

The annotated errors are shown in a confusion matrix in Table 9, where the 'all agree' column shows all annotators agreed on the same error type and other columns show how often each annotator selected a different error type. Although the correct token was usually chosen, disagreements primarily occurred in choosing CONTEXT, ADDITION and WORD error types, as shown in red in Table 9. This might be due to the potential similarities in the definitions of CONTEXT errors and ADDITION errors (see App. B.3). While WORD error is comparatively straightforward, one annotator chose CONTEXT errors instead of WORD errors for a few outputs. Cases where an annotator did not mark any errors were in the minority. The inter-annotation agreement on error category classification for 60 outputs, as shown by a Fleiss' kappa of 0.622, indicates substantial agreement between the three annotators.

## 8 Conclusion

This paper presented a study that quantitatively demonstrates that fixing input problems such as insufficient data and data records containing nonatomic content improves the factual accuracy of output text by as high as 76% for one of the study models. Correcting inputs also seems to improve the number of entirely error-free output texts. However, we still need to investigate why errors categorised as 'omissions' increase after input corrections. In our future work, we aim to explore other tabular datasets for problems with input data and study the generalization of the fixes explored in the current work.

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|} \hline
**ERROR** & **ALL degree** & **WORD** & **NAME** & **DAT\_DIM** & **NUMBER** & **OTHER** & **CONTEXT** & **ADDITION** & **NO ERROR** & **TOTAL** \\ \hline
**WORD** & 17 & 0 & 0 & 0 & 2 & 0 & 9 & 3 & 0 & 31 \\
**NAME** & 3 & 0 & 0 & 0 & 0 & 0 & 2 & 0 & 0 & 5 \\
**DATE\_DIM** & 1 & 0 & 0 & 0 & 0 & 0 & 2 & 0 & 0 & 3 \\
**NUMER** & 2 & 2 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 4 \\
**OTHER** & 2 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 3 \\
**CONTEXT** & 7 & 9 & 2 & 2 & 0 & 0 & 0 & 6 & 0 & 26 \\
**ADDITION** & 12 & 1 & 0 & 0 & 0 & 0 & 5 & 0 & 0 & 18 \\
**NO ERROR** & 15 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 16 \\ \hline \end{tabular}
\end{table}
Table 9: Fleiss’ Kappa coefficient: overall agreement on 60 samples among three annotators. ‘All agree’ column signifies unanimous agreement on error types, while other columns show unique error selections.

 



### Limitations

We acknowledge some limitations in this work. First, we only looked at ToTTo and our scope of corrections to tabular input is limited to errors identified within the _politics_ domain of the ToTTo validation set. Second, we did not extend the correction of tabular input for the _politics_ domain to the training split of the ToTto dataset due to the time-consuming process of handling different table headers and metadata. Third, our experiment results are restricted to two specific models (T5 and Llama 2) and may not generalize to other models.

In our future work, we aim to simplify the definitions of the 'context' and 'addition' error categories, as the annotation experiment revealed disagreement in choosing these error types for some samples, despite annotators marking the same error token.

## Ethics Statement

This work seeks to address input problems in nonstandard tabular structures to reduce factual errors in the output text. We utilized the open-source dataset, ToTTo and maintained the same ground-truth generation as the original dataset. Our input correction did not introduce any further social bias to this dataset. We adopted an error annotation methodology to annotate factual errors and one of the authors performed manual error analysis for this complete study. We sought consent from two additional annotators, the annotators volunteered to participate and annotated errors for 60 output texts each. They had the right to withdraw from the study at any point without facing any consequences. We provided necessary guidelines, instructions and examples for them to annotate errors.

## Acknowledgements

We thank Craig Thomson and Adarsa Sivaprasad for their hard work in helping with the annotations in this paper. We thank the anonymous reviewers for their detailed feedback and suggestions which have significantly improved this work. We also thank the NLG (CLAN) reading group at the University of Aberdeen for their invaluable feedback.

## References

* Badaro et al. (2023) Gilbert Badaro, Mohammed Saeed, and Paolo Papotti. 2023. Transformers for tabular data representation: A survey of models and applications. _Transactions of the Association for Computational Linguistics_, 11:227-249.
* Chang and Fosler-Lussier (2023) Shuaichen Chang and Eric Fosler-Lussier. 2023. How to prompt llms for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings.
* Chen et al. (2022) Miao Chen, Xinjiang Lu, Tong Xu, Yanyan Li, Zhou Jingbo, Dejing Dou, and Hui Xiong. 2022. Towards table-to-text generation with pretrained language model: A table structure understanding and text deliberating approach. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8199-8210, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. _CoRR_, abs/2004.10404.
* Chen et al. (2020b) Zhiyu Chen, Wenhu Chen, Hanwen Zha, Kiyou Zhou, Yunkai Zhang, Saifam Sundaresan, and William Yang Wang. 2020b. Logic2Text: High-fidelity natural language generation from logical forms. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2096-2111, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _North American Chapter of the Association for Computational Linguistics_.
* Dhingra et al. (2019) Buwan Dingra, Manasal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4884-4895, Florence, Italy. Association for Computational Linguistics.
* Dusek et al. (2019) Ondrej Dusek, David M. Hovcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. In _Proceedings of the 12th International Conference on Natural Language Generation_, pages 421-426, Tokyo, Japan. Association for Computational Linguistics.
* Gkatzia et al. (2017) Dimitra Gkatzia, Oliver Lemon, and Verena Rieser. 2017. Data-to-text generation improves decision-making under uncertainty. _IEEE Computational Intelligence Magazine_, 12(3):10-17.
* Corbelle et al. (2022) Javier Gonzalez Corbelle, Alberto Bugarin-Diz, Jose Alonso-Moral, and Juan Taboada. 2022. Dealing with hallucination and omission in neural natural language generation: A use case on meteorology. In _Proceedings of the 15th International Conference on Natural Language Generation_, pages 121-130, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics

Hanxu Hu, Yunqing Liu, Zhongyi Yu, and Laura Perez-Beltrachini. 2023. Improving user controlled table-to-text generation robustness. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 2317-2324, Dubrovnik, Croatia. Association for Computational Linguistics.
* Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. _ACM Computing Surveys_, 55(12):1-38.
* Kale and Restogi (2020) Mihir Kale and Abhinav Rastogi. 2020. Text-to-text pre-training for data-to-text tasks. In _Proceedings of the 13th International Conference on Natural Language Generation_, pages 97-102, Dublin, Ireland. Association for Computational Linguistics.
* Kasner and Dusek (2024) Zdenek Kasner and Ondrej Dusek. 2024. Beyond reference-based metrics: Analyzing behaviors of open llms on data-to-text generation. _arXiv preprint arXiv:2401.10186_.
* Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In _Advances in Neural Information Processing Systems_, volume 35, pages 22199-22213.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1906-1919, Online. Association for Computational Linguistics.
* Novikova et al. (2017) Jekaterina Novikova, Ondrej Dusek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_, pages 201-206, Saarbrucken, Germany. Association for Computational Linguistics.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. _ArXiv_, abs/2303.08774.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manai Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1173-1186, Online. Association for Computational Linguistics.
* Pauws et al. (2018) Steffen C. Pauws, Albert Gatt, Emiel J. Krahmer, and Ehud Reiter. 2018. Making effective use of healthcare data using data-to-text technology. In _Data Science for Healthcare_.
* Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. _Proceedings of the AAAI Conference on Artificial Intelligence_, 33(01):6908-6915.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1172-1183, Online. Association for Computational Linguistics.

* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Sivarajkumar et al. (2023) Sonish Sivarajkumar, Mark Kelley, Alyssa Samolykarazzanti, Shyam Visweswaran, and Yanshan Wang. 2023. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing. _arXiv preprint arXiv:2309.08008_.
* Sripada et al. (2002) Somayajulu Sripada, Ehud Reiter, Jim Hunter, and Jin Yu. 2002. Sumtime-meteo: Parallel corpus of naturally occurring forecast texts and weather data. _Computing Science Department, University of Aberdeon, Aberdeon, Scotland, Tech. Rep. AUCS/TR0201_.
* Su et al. (2021) Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021. Plan-then-generate: Controlled data-to-text generation via planning. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 895-909, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Sundararajan et al. (2022) Barkavi Sundararajan, Somayajulu Sripada, and Ehud Reiter. 2022. Error analysis of ToTTo table-to-text 

neural NLG models. In _Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)_, pages 456-470, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Thomson and Reiter [2020] Craig Thomson and Ehud Reiter. 2020. A gold standard methodology for evaluating accuracy in data-to-text systems. In _Proceedings of the 13th International Conference on Natural Language Generation_, pages 158-168, Dublin, Ireland. Association for Computational Linguistics.
* a robust and maintainable data-set for natural language generation. In _Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation_, pages 32-40, Santiago de Compostela, Spain. Association for Computational Linguistics.
* Thomson et al. [2023] Craig Thomson, Ehud Reiter, and Barkavi Sundararajan. 2023. Evaluating factual accuracy in complex data-to-text. _Computer Speech & Language_, 80:101482.
* Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Upadhyay and Massie [2022] Ashish Upadhyay and Stewart Massie. 2022. Content type profiling of data-to-text generation datasets. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 5770-5782, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* van Miltenburg et al. [2021] Emiel van Miltenburg, Miruna Clinciu, Ondrej Dusek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppanen, Saad Mahamood, Emma Manning, Stephanic Schoch, Craig Thomson, and Luou Wen. 2021. Underreporting of errors in NLG output, and what to do about it. In _Proceedings of the 14th International Conference on Natural Language Generation_, pages 140-153, Aberdeen, Scotland, UK. Association for Computational Linguistics.
* Wang and Sonnrich [2020] Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3544-3552, Online. Association for Computational Linguistics.
* Wang et al. [2022] Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. 2022. Robust (controlled) table-to-text generation with structure-aware covariance learning. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5037-5048, Seattle, United States. Association for Computational Linguistics.
* Wiseman et al. [2017] Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
* Zhu et al. [2023] Fengbin Zhu, Moxin Li, Junbin Xiao, Fuli Feng, Chao Wang, and Tat Song Chua. 2023. Soargraph: Numerical reasoning over financial table-text data via semantic-oriented hierarchical graphs. In _Companion Proceedings of the ACM Web Conference 2023_, WWW '23 Companion, page 1236-1244, New York, NY, USA. Association for Computing Machinery.

 



## Appendix A Model fine-tuning specifications

The first model, T5-base (T5-b), was fine-tuned on the _full ToTTo training set of 120,761 samples_ on a commodity server with a GeForce RTX 2080 GPU. The training took around seven days. The second model, T5-large (T5-l), was fine-tuned on _a subset of 50,000 ToTTG samples_ on a secure cloud instance with an NVIDIA A100 GPU, completing in around 48 hours. Both models were fine-tuned using a constant learning rate of 0.0001, with the encoder's maximum length set to 512 tokens and the decoder's maximum length set to 128 tokens for ToTTo's generation task [39]. Single-precision floating-point format (FP32) was employed for training on their respective GPU servers. The batch size, beam size and training steps for each model are shown in Fig. 1.

## B Inter-Annotation Procedure for Participants

### Overview

The Input Data from the ToTTo dataset, includes the Page title, Section Title and a Table with highlighted cells in yellow. These key parameters are conditionally used for training the neural models to summarize a meaningful and factual Text (as Output) focusing on: (i). highlighted cells in the table (ii). their corresponding header values (iii) The main Title and (iv) The Section Title.

For each of these tabular input data, we provided outputs generated by different neural language models to annotators. Our goal is to evaluate whether the neural outputs remain faithful and produce factually accurate information based on the four parameters from the tabular input. The complete table, including the non-highlighted cells, is provided to offer a clearer understanding of the error annotation task.

### Domain

The inputs provided are specific to the domain of _politics_, sourced from Wikipedia tables (as part of the open-domain ToTTo dataset). The political data within these tables is not limited to a single demography. Instead, it encompasses various details from the election processes across multiple countries, including:

* Election specifics such as Presidential, state, by-elections, council, district, Legislative Assembly, and other elections unique to particular countries.
* Information about Governors, Mayors, Ministers, and Ambassadors (about Foreign Affairs).
* Details regarding the Speaker of the Assembly.

The first item related to election details is predominantly used in this error annotation task.

### Error Annotation guidelines

We are only interested if the highlighted cell values from the table were used to produce a factually correct sentence. Please also pay attention to the non-highlighted cells in the same row as the highlighted cells, as this might be required in some inputs to generate a meaningful sentence. Other non-highlighted values in the table are not expected to be used for your evaluation. Please read through the output texts and annotate cases for the error categories as mentioned below. Each error is denoted with a superscript for better readability.

\(\mathrm{NAME}^{N}\)

* When names of the Party, Leader, place (Electorate), Ambassador etc., are wrong (mostly nouns).
* Annotation includes both single tokens or multiple tokens to include the complete names.
* Example Output text: \(\mathrm{Urban~{}Ahlin^{N}}\) is the Deputy Speaker of the Riksdag. Remarks: NAME error because the correct deputy speaker was Tobias Billstrom as per the tabular data.
* Example Output text: Kansas was won by Mitt Romney, Paul Ryan, Barack ObamaN, and Joe BidenN, with Romney winning

Figure 1: Model Specifications

59.66% of the popular vote, six electoral votes and 38.05 percent. In this example, Barack Obama and Joe Biden are two NAME errors because they did not win the election.
* In general, Wednesday\({}^{\text{N}}\) instead of Tuesday is a NAME error but MayD instead of April is a DATE_DIMENSION error.

**NUMBERU**:

* When the number of seats and/or the number of votes and/or % of votes are incorrect. A single token is marked as an error.
* When the A-party won with a majority of 5.5%. But the correct one is 4.4%. **5.5%**U is a NUMBER error.
* Output: The voter turnout was 8,90%, with 10,052 votes. Remarks: The actual turnout was **81.90%**U****Please note: the error here is NOT with the comma used as decimal (as it is an acceptable decimal operator for international use); Error because the number 81.90 was incorrect.

**DATE_DIMENSIONSD**:

* When the Date and/or Month and/or Year are wrong in the generated text, it is annotated as one error.
* Example Output text: Cletus Avoka was the Minister for the Interior in the Mills government from 2009 to **2012**D. Remarks: 2010 is the right end term of the year.
* As a general note, if the Output text did not capture Month and/or Date, but has the correct year, then this is NOT an ERROR. It could go to OMISSION with remarks, Omission of Date and Month.

WORDW

* When incorrect words such as verbs, prepositions, adjectives, adverbs and conjunctions are found in the output.
* Single token is marked as a WORD error in most cases. Multiple tokens are annotated when the auxiliary verb (was), an extension of a prepositional phrase (along with) and others are incorrect.
* Example Output text: Carly Fiorina defeated Republican Tom Campbell with 56.4% of the vote to DeVore's 19.3%, along withM Al Ramirez and Tim Kalemkarian. Remarks: Fiorina independently defeated all the leaders, so 'along with' is wrong.
* Example Output text: Ling wonW the 2016 senate district against Democrat Josh Newman, with 49.6 percent of the vote. Remarks: Ling lost the election as per the tabular data.
* Some of the common WORD errors found in this data are _won, defeated, lost, succeeded_, adjectives such as _current_ governor other prepositions (_since, in_ and so on).

CONTEXTC

* When the model's output presents information that contradicts or makes _unsupported assumptions about the given input data_. Group of tokens/span of text are annotated as CONTEXT error.
* It can sometimes be tricky to check for this type of error. Please follow the below sequence before marking this error.
* In case of simple misrepresentation based on the information in the input data, it would be easier to mark the token as NAME, NUMBER, DATE_DIMENSION or WORD error.
* In the case of a complex table structure, the outputs are likely to mess up completely with the overall information in the provided input data. In this case, it is hard to mark individual errors. Please go ahead and mark the group of tokens/ span of text and annotate it as a CONTEXT error.
* For example, the output is: In the 2006 election for mayor of Florence Pendleton\({}^{\copyright C}\), Michael D. Brown received 62,415 votes while Philip Pannell received 21,552 votes and write-in candidates\({}^{\copyright C}\) received 1,363 votes. Annotation remarks:
* for mayor of Florence Pendleton\({}^{\copyright C}\) - the name of a person is misrepresented as electorate (jurisdiction) in the output
 

- this implies there was more than one write-in candidate.

ADDITION\({}^{\text{A}}\)

* When the model's outputs have _added words, phrases, or details_ that either diverge from the input's main topic or are unsupported by the given context.
* Single or group of tokens are annotated as ADDITION error.
* Example Output text: 2007 Algerian legislative election was held on May 17, 2007\({}^{\text{A}}\). The results were as follows: 24 political parties won a total of 389 seats in the National Assembly. **Remarks:** The date and month are additional information, that are not provided in the input. This is marked as an ADDITION error. Other details in the output are correct.

OTHERO

* When the output repeats the same input multiple times producing garbage data. In some cases, when the table data has the political party name in the abbreviation, it produces garbage output. For example, when the tabular data has a party name in abbreviation, it tries to produce a strange output. Output text: GSSSDULSVDHSSO gained 5.31% of the vote
* When the output is incomplete for longer table input or when the output repeats the same input multiple times without producing a complete text.
* When punctuation symbols are placed in inappropriate places, for example, an apostrophe is missed for the Name of the Leader or Place.

NON-ENGLISHNE

* when the Unicode characters in non-English names are either replaced with special characters or when these Unicode characters are omitted.
* For example, Pawe GraNE is a member of Sejm. **Remarks:** Pawel Gras is the correct name here.

## Appendix C Llama 2 prompts

The below prompt is for the corrected tabular data when the input table has party name, candidate and votes.

'Given the input table data, the task is to: (i). Identify the party name, candidate name, and the number of votes received by each candidate. (ii). Determine the winner based on the highest number of votes. Then, put together the gathered information from (i) and (ii) into a single coherent sentence. Input table data: <Linearized table data>'

Below is one of the prompts used for the original table data without mentioning any specific fields.

'The task is to summarize the information from the given input table data into a single coherent sentence. Use only the information mentioned in the input table data. Input table data is: <Linearized table data>'

## Appendix D Steps for Correcting ToTTo Tabular Input

In addition to the correction procedure detailed in Sec. 5.2, which provides examples of the corrections applied to each tabular input problems, we present a supplementary pseudocode in this section. In algorithm 1, we have a generic function that takes tabular data and title (metadata) from ToTTo as input parameters. This main algorithm executes six different functions.

``` 

```
functionMainCorrectionProcedure(\(tabularData,title\)) ifnotDataSizeManageableForLongTable(\(tabularData\))then return"Please simplify the tabular data with fewer records", null endif \(leaderName,recordedData\leftarrow\textsc{IdentifyLeaderNameOrder}(tabularData,title)\)\(correctionsMade\leftarrow\textbf{false}\) \(correctionsMade\leftarrow\textsc{CorrectNonAtomicCells}(tabularData)\textbf{or}~{} correctionsMade\)\(correctionsMade\leftarrow\textsc{UpdateHeaders}(tabularData)\textbf{or}~{}correctionsMade\)\(correctionsMade\leftarrow\textsc{AddressS MissingValues}(tabularData)\textbf{or}~{}correctionsMade\)\(correctionsMade\leftarrow\textsc{ReplaceSymbols}(tabularData)\textbf{or}~{}correctionsMade\) if\(correctionsMade\)then return(\(tabularData\),"Leader Data:"+\(recordData\))\(\triangleright\) Returns corrected input and leader data else return(\(tabularData\),"Leader Data:"+\(recordData\))\(\triangleright\) Returns original input (if no issues) and leader data endif endfunction
```

**Algorithm 1****Manual Correction Procedure for ToTTo Tabular Input** (elaborated in Sec. 5.2). This main function takes tabular data and title details as input parameters and returns the corrected tabular data. In all functions, we excluded row and column indices for simplicity and readability.

 



**Algorithm 3 Identify Leader Name Order in Title and Table Rows** (discussed in Sec. 5.2.6). This function verifies three main scenarios for the order of leader names in the input. It returns a tuple with title information for leader name and a list of leader names from tabular input depending on the scenario. Description for each scenario is briefly commented.

```
functionIdentifyLeaderNameOrder(\(tabularData,title\)) \(leaderNameFromTitle\leftarrow ExtractLeaderNameFromTitle(title)\) \(leaderNamesFromRows\leftarrow[~{}]\) for each rowin_tabularDatado for each cellin current rowdo ifleader_name is found in cellthen Add leader_name to \(leaderNamesFromRows\)\(\triangleright\) Record leader's names from rows endif endfor endfor \(\triangleright\) Scenario 1: Leader's name found in title ifleaderNameFromTitle is not None then \(recordedData\leftarrow[~{}]\)\(\triangleright\) To store sequential order of leader names Add \(leaderNameFromTitle\) to \(recordData\) for each leader_namein_leaderNamesFromRowsdo Add leader_name to \(recordedData\) endfor return(\(leaderNameFromTitle,recordedData\)) \(\triangleright\) for e.g., this could return ("b", ["a", "b", "c"]) else \(\triangleright\) Scenario 2: Leader's name not found in title if length of \(leaderNamesFromRows>0\)then \(\triangleright\) if leader's name is in at least one row \(recordedData\leftarrow\) "Leader name not in title" for each leader_name in_leaderNamesFromRowsdo Add leader_name to \(recordedData\) endfor return(\(leaderNameFromTitle,recordedData\)) \(\triangleright\) for e.g., this could return (None, ["Leader name not in title", "a", "b", "c"]) else \(\triangleright\) Scenario 3: Leader name neither in title nor in rows return(\(leaderNameFromTitle,\)"Leader name not in table") \(\triangleright\) for e.g., this could return (None, ["Leader name not in table"]) endif endif endfunction
```

```
functionCorrectNonatomicCells(\(tabularData\)) \(correctionsMade\leftarrow\textbf{false}\) for all\(row\)in\(tabularData\)do forall\(cell\)in\(row\)do ifCellIsNonAtomic(\(cell\))then \(correctedCell\leftarrow\textsc{CorrectCell}(cell)\triangleright\)Separate multiple values into individual columns \(tabularData[cell]\gets correctedCell\)\(\triangleright\)Update the corrected cell value \(correctionsMade\leftarrow\textbf{true}\) endif endfor endfor return(\(tabularData,correctionsMade\)) endfunction
```

**Algorithm 4 Correct Non-Atomic Cells** (discussed in Sec. 5.2.1). If a non-atomic cell is found, the CorrectCell function separates multiple atomic values into individual columns. We follow our manual correction procedure in CellIsNonAtomic and CorrectCell. The function returns the corrected tabular data along with a flag indicating if any corrections were made.

```
functionCorrectHeaders(\(tabularData\)) \(correctionsMade\leftarrow\textbf{false}\) forall\(row\)in\(tabularData\)do forall\(coll\)in\(row\)do ifCellIsNonAtomic(\(cell\))then \(correctedCell\leftarrow\textsc{CorrectCell}(cell)\triangleright\)Separate multiple values into individual columns \(tabularData[cell]\gets correctedCell\)\(\triangleright\)Update the corrected cell value \(correctionsMade\leftarrow\textbf{true}\) endif endfor endfor return(\(tabularData,correctionsMade\)) endfunction
```

**Algorithm 5 Update Column and Row Headers to Atomic cells** (discussed in Sec. 5.2.4). For each cell, the tabular input has header_value as true or false. When the header value is true, we manually verify the function HeaderIsIncorrect and correct the logic in UpdateHeader(cell). The function returns the corrected tabular data with corrections made flag.

 

```
functionAddressissingValues(\(tabularData\)) \(correctionsMade\leftarrow\textbf{false}\) for all\(row\)in\(tabularData\)do ifRowHasMissingValues(\(row\))then \(correctedRow\leftarrow\text{FillCellValues}(row)\)\(\triangleright\) Pass missing cells in the row \(tabularData[row]\gets correctedRow\) \(correctionsMade\leftarrow\textbf{true}\) endif endfor return\((tabularData,correctionsMade)\) endfunction
```

**Algorithm 6****Address Missing cell Values** (discussed in Sec. 5.2.3). For each row, we verify the missing cell values in RowHasMissingValues and pass the right cell values in FillCellValues through a manual process. The function returns the corrected tabular data with corrections made flag.

 

problem type represent output scores before data correction, while the right bars represent scores after data correction. Colour coding of output scores helps to identify and understand errors: green indicates no error, red indicates an error, and yellow indicates an omission.

The number of samples for each model differs because we focus on outputs only when the model had 'errors' or 'omissions' before correction. This approach emphasizes the actual improvement in corrections. For example, T5-large had errors in 37 samples before correction (left bar), the after-correction (right bar) also shows improvements for the same 37 samples.

## Appendix F Input Data Corrections and Output Text

In this section, the input tabular data with outputs supplements the examples elaborated in Sec. 5.2 for each of the input problems as presented from Table 10 to Table 16.

 



\begin{table}
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \multicolumn{3}{c}{**Page Title: 2012 Cardiff South and Penarth by-election Section Title: By-election result**} \\ \hline
**Party** & **\%** & \(\pm\) \\ \hline Conservative & 19.9 & 8.4 \\ \hline Liberal Democrat & 10.8 & -11.5 \\ \end{tabular} 
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \multicolumn{3}{c}{**Page Title: 2012 Cardiff South and Penarth by-election Section Title: By-election result**} \\ \hline
**Party** & **\%** & \(\pm\) **\% difference with previous election** \\ \hline Conservative & 19.9 & 8.4 \\ \hline Liberal Democrat & 10.8 & -11.5 \\ \hline \end{tabular} 
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \multicolumn{3}{c}{**Page Title: 2012 Cardiff South and Penarth by-election Section Title: By-election result**} \\ \hline
**Party** & **\%** & \(\pm\) **\% difference with previous election** \\ \hline Conservative & 19.9 & 8.4 \\ \hline Liberal Democrat & 10.8 & -11.5 \\ \hline \end{tabular} 
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \multicolumn{3}{c}{**Page Title: 2012 Cardiff South and Penarth by-election Section Title: By-election result**} \\ \hline
**Party** & **\%** & \(\pm\) **\% difference with previous election** \\ \hline Conservative & 19.9 & 8.4 \\ \hline Liberal Democrat & 10.8 & -11.5 \\ \hline \end{tabular} 
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \multicolumn{3}{c}{**Page Title: 2012 Cardiff South and Penarth by-election Section Title: By-election result**} \\ \hline
**Party** & **\%** & \(\pm\) **\% difference with previous election** \\ \hline Conservative & 19.9 & 8.4 \\ \hline Liberal Democrat & 10.8 & -11.5 \\ \hline \end{tabular} 
\begin{tabular}{l|p{142.3pt} p{56.9pt}} \multicolumn{3}{c}{**Page Title: 2012 Cardiff South and Penarth by-election Section Title: By-election result**} \\ \hline
**Party** & **\%** & \(\pm\) **\% difference with previous election** \\ \hline Conservative & 19.9 & 8.4 \\ \hline Liberal Democrat & 10.8 & -11.5 \\ \hline \end{tabular}
\end{table}
Table 10: ToTTo specific: Swing percentage difference with previous election, as detailed in Sec. 5.2.5. Before fix represents the output generated from input Table (a)a and after fix represents the output generated after explicitly changing the header, as shown in input Table (b)b

Figure 2: Comparison of different input problem types before (original data) and after data correction across four models. It shows the count of unique samples for three categories: (i) errors, (ii) omissions, and (iii) no errors.

 



\begin{table}
\begin{tabular}{c|c|c|c|c|c} \multicolumn{6}{l}{**Page Title:** 1914 United States Senate elections \\
**Section Title:** Races leading to the 64th Congress & \multicolumn{2}{c}{} \\ \hline \multirow{3}{*}{**State**} & \multicolumn{2}{c|}{**Incumbent**} & \multirow{3}{*}{**Results**} & \multirow{3}{*}{**Candidates**} \\  & **Senator** & & & & \\ \cline{2-5} South & Coe & Republican & Incumbent lost renomination. & Edwin S. Johnson (Democratic) 48.32\% \\ Dakota & I.Crawford & New senator elected. & Charles H. Burke (Republican) 44.47\% & Democratic gain. & \\ \end{tabular}
\end{table}
Table 11: Complex Table input problems, as detailed in Sec. 5.2.2. Difficult to fix this complex input type, yet Llama 2-13B produced most of the facts right except for ADDITION error. Lost renomination ‘to Democrat candidate’ is a CONTEXT error, because the model made unsupported assumptions about the given input. Losing renomination is when the current senator failed to be nominated in the primary to stand for reelection.





\begin{table}
\begin{tabular}{c c} \hline
**(a) Input Table with Original Cells highlighted in Yellow** \\ \multicolumn{3}{l}{**Page Title:** Ling Ling Chang \\
**Section Title:** 2016 Senate run \\ \hline
**California’s 29th State Senate district election** \\ \hline
**Party** & **Candidate** & **\% Votes** \\ \hline Democratic & Josh Newman & 50.4 \\ \hline Republican & Ling Ling Chang & 49.6 \\ \hline \end{tabular} 
\begin{tabular}{c|c} \hline
**(b) Input Table with Corrected Cells highlighted in Yellow** \\ \multicolumn{3}{l}{**Page Title:** Ling Ling Chang \\
**Section Title:** 2016 Senate run \\ \hline
**California’s 29th State Senate district election** \\ \hline
**Pary** & **Candidate** & **\% Votes** \\ \hline Democratic & Josh Newman & 50.4 \\ \hline Republican & Ling Ling Chang & 49.6 \\ \hline \end{tabular} 
\begin{tabular}{c|c} \hline
**Chairria’s 29th State Senate district election** \\ \hline
**Party** & **Candidate** & **\% Votes** \\ \hline Democratic & Josh Newman & 50.4 \\ \hline Republican & Ling Ling Chang & 49.6 \\ \hline \end{tabular}
\end{table} TABLE I3: Fixing ToTTo specific input problems: List of leader names for two leaders. ‘Before’ shows WORD errors due to missing vote percentages and party names for a leader. ’After’ shows these errors are resolved by all models when the missing values are provided. Key point is T5 model did not produce error even though the title leader ‘Ling Ling Chang’ lost the election.





**Llama 2-7B before fix:** Chuck DeVore is a Republican candidate for the U.S. Senate in California. He received 19.3% of the vote in the primary election, which was held on June 8th, 2010. The other candidates were Carly Fiorina (56.4%), Tom Campbell (**24.1%**)\({}^{\text{U}}\) and Al Ramirez (**10.2%**)\({}^{\text{U}}\). Tim Kalemkarian did not run for this office\({}^{\text{C}}\).

* 56.4% Tom Campbell
- 21.7% Chuck DeVore
- 19.3% Al Ramirez
- 1.8% Tim Kalemkarian
- 0.8%. Early Fiorina won the Republican primary for U.S. Senate in California with 56.4 percent of the vote, followed by Tom Campbell at 21.7 percent and Chuck DeVore at 19.3 percent.

**Llama 2-13B before fix:** Carly Fiorina is the leading candidate for U.S. Senate with 56.4% of the vote, followed by Tom Campbell with **20.3%**\({}^{\text{U}}\), Chuck DeVore with 19.3%, and Al Ramirez with **4.0%**\({}^{\text{U}}\).

**Llama 2-13B after fix:** The Republican Party candidate who won the most votes in the U.S. Senate election was Carly Fiorina with 56.4% of the total votes.





\begin{table} \end{tabular}
\end{table}
Table 15: ToTTo specific input problems: List of Leader names, as detailed in Sec. 5.2.6. Correction for this input is including the party name and modifying the header ‘Subject’ to ‘Candidate’. All models generated Joseph Haslet to be the winning candidate including the Llama 2 models that had specific instructions.





**T5-large before fix:** The 435th member of the United States House of Representatives in the 65th Congress was a member who joined the House after the start of CongressC.

**Llama2-13B before fix:** List of Representatives by seniority The table below lists the 100.Oo

**(c) Input Table with Corrected Cells highlighted in Yellow**

\begin{tabular}{l|l|l|l|l|l} \hline