Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation (2024)

Abstract: We analyze the behaviors of open large language models (LLMs) on the task of data-totext (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design QUINTD – a tool for collecting novel structured data records from public APIs. We find that open LLMs (Llama 2, Mistral, and Zephyr) can generate fluent and coherent texts in zero-shot settings from data in common formats collected with QUINTD. However, we show that the semantic accuracy of the outputs is a major issue: both according to human annotators and our reference-free metric based on GPT-4, more than 80% of the outputs of open LLMs contain at least one semantic error. We publicly release the code, data, and model outputs.1

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12045–12072 August 11-16, 2024 ©2024 Association for Computational Linguistics
We analyze the behaviors of open large language models (LLMs) on the task of data-totext (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design QUINTD – a tool for collecting novel structured data records from public APIs. We find that open LLMs (Llama 2, Mistral, and Zephyr) can generate fluent and coherent texts in zero-shot settings from data in common formats collected with QUINTD. However, we show that the semantic accuracy of the outputs is a major issue: both according to human annotators and our reference-free metric based on GPT-4, more than 80% of the outputs of open LLMs contain at least one semantic error. We publicly release the code, data, and model outputs.1

heading: 1 Introduction
text: Large language models (LLMs; Ouyang et al., 2022; Touvron et al., 2023a,b; Jiang et al., 2023; Tunstall et al., 2023) have already left a mark in many areas of natural language processing (NLP). Surprisingly, their applicability to the task of data-to-text (D2T) generation (Reiter and Dale, 1997; Gatt and Krahmer, 2018) remains underexplored, with limited evaluation on a handful of well-established benchmarks only (Axelsson and Skantze, 2023; Yuan and Färber, 2023). Generating text from structured data is arguably challenging for LLMs, given the specifics of D2T generation, such as long inputs, complex non-linear structure, and strict requirements on semantic accuracy. However, a more significant issue is the lack of testing grounds. The current D2T generation benchmarks are not only getting saturated (Van Miltenburg et al., 2023), but also promote optimization towards traditional reference-based evaluation metrics, which
1https://d2t-llm.github.io/
were shown to correlate poorly with human judgment (Gehrmann et al., 2023; van der Lee et al., 2021; Novikova et al., 2017). When it comes to the models, using closed LLMs (OpenAI, 2023a,b) is increasingly considered a bad research practice due to its non-reproducibility (Rogers, 2023; Chen et al., 2023). On top of that, contamination of LLM training data with standard benchmarks further restricts the space for experiments (Golchin and Surdeanu, 2023; Aiyappa et al., 2023; Balloccu et al., 2024).
In this paper, we propose an approach that allows us to analyze model behavior in D2T generation on novel, real-world structured data records with reference-free evaluation metrics. We begin by realizing that unlabeled data are plentiful. To leverage the data for our experiments, we introduce QUINTD2 – a tool for collecting structured data from five domains in standard formats: JSON,
2Quintet of Unlabeled Inputs for Natural Tasks in Datato-text, pronounced as “quintet”
12045
CSV, and Markdown. We choose the domains so that the data can be directly used as input for five distinct D2T generation tasks. Our tasks include generating weather forecasts, sports reports, product descriptions, chart captions, and entity descriptions (see Table 1). Next, we collect a set of 1,000 inputs with QUINTD and use the inputs as an adhoc benchmark (called QUINTD-1) for testing the abilities of LLMs for D2T generation. We assume that the data formats in QUINTD-1 are common in the LLMs’ pretraining corpora, so we specify the task using instructions instead of standard finetuning with human-written outputs, capitalizing on the zero-shot abilities of instruction-tuned LLMs (§2).
We push towards better reproducibility by focusing on open LLMs, which – apart from being more accessible – also achieve increasingly better results across tasks (Zheng et al., 2023; Beeching et al., 2023). For our experiments, we use three open LLMs with 7B parameters: Llama 2 (Touvron et al., 2023b; TogetherAI, 2023), Mistral (Jiang et al., 2023), and Zephyr (Tunstall et al., 2023). We also use GPT-3.5 (OpenAI, 2023b) as a closed model baseline for the final experiments. Given the behavioral nature of the experiments with LLMs (Holtzman et al., 2023), we put emphasis on reporting model behavior throughout the process (§3).
Another piece of the puzzle is reference-free evaluation: using the input data as a ground for comparison instead of reference outputs (§4). We focus on identifying semantic errors in the model outputs, i.e., the information that is not supported by the input data. We use two separate evaluation methods: manual annotations from human crowdworkers (van der Lee et al., 2021) and a custom automatic metric based on GPT-4 (Liu et al., 2023; Chiang and Lee, 2023; Kocmi and Federmann, 2023a). We annotate the errors on the level of individual words, getting fine-grained annotations of error spans in several categories (Thomson and Reiter, 2020; Thomson et al., 2023).
Based on our results, we provide general recommendations for D2T generation with open LLMs across tasks and formats (§5). Our main findings are as follows:
• Open LLMs can generate fluent outputs from structured data in common formats under zero-shot settings.
• Semantic accuracy is a major obstacle: both human annotators and GPT-4-based metric report that over 80% of outputs of open LLMs on our data contain a semantic error.
• Long data inputs cause practical issues, including the need for long-context models, increased GPU memory requirements, and unavailability of few-shot approaches.
• Outputs can be empirically improved by following several rules-of-thumb for preprocessing the model input, such as including units, removing unnecessary fields, or prefixing the model answer.

heading: 2 Reference-Free D2T Generation

heading: 2.1 Data Collection Tool
text: We introduce QUINTD,3 a tool for collecting adhoc test sets using public APIs in five different domains. Our main reasons for departing from the traditional scheme of benchmarking on wellestablished datasets are:
1. Any published test sets may be potentially included in the training data of LLMs.
2. Public sources of structured data offer enough resources for creating ad-hoc test sets.
3. Without human references, our data collection scheme is lightweight and replicable.
Given the available public sources of data, we settled on the five tasks which are described in Table 1 (see Appendix A for more details). The tasks
3https://github.com/kasnerz/quintd
are based on structured data in common formats: JSON, CSV, and Markdown.

heading: 2.2 QUINTD-1 Dataset
text: Using QUINTD, we collected the dataset for our experiments in this paper (QUINTD-1). The dataset contains 500 examples in the development set and 500 examples in the test set (100 examples per domain for each split). We downloaded the data between November 2023 and January 2024. Note that the dataset contains only unlabeled data without any reference outputs (e.g., weather data, but not a textual weather forecast), so the outputs need to be evaluated using reference-free metrics. New versions of the benchmark can be easily generated with our QUINTD tool.

heading: 2.3 Task Definition
text: Each example in QUINTD-1 consists of a structured data record x from a domain D P {openweather, gsmarena, ice_hockey, owid, wikidata}. Given x and a prompt P , the goal is to generate natural language output y faithful to the data x, according to the instructions in the prompt P (see Figure 2).

heading: 3 Experiments

heading: 3.1 Experimental Process
text: Our goal is to avoid extensive data preprocessing and prompt engineering since these steps could harm the reproducibility and generalizability of our
experiments. With this goal in mind, we decided to use the same prompt template P for all the domains and models.
For a set of preliminary experiments, we first wrote down the initial version of the prompt and used the data without further preprocessing. We then iteratively improved our experimental setup by observing outputs on the development set. In §3.3, we describe all the observations and modifications we made before generating the final outputs on the test set.

heading: 3.2 Models
text: For our experiments, we selected the following LLMs available under an open license:
• Llama 2 (Touvron et al., 2023a; TogetherAI, 2023), togethercomputer/Llama-2-7B-32K-Instruct
• Mistral (Jiang et al., 2023), mistralai/Mistral-7B-Instruct-v0.1
• Zephyr (Tunstall et al., 2023). HuggingFaceH4/zephyr-7b-beta
The models are instruction-tuned, operate with 32k context, and perform well on recent benchmarks (Beeching et al., 2023). All the models have 7B parameters and thus fit on a single NVIDIA A40 (48G VRAM) in 16-bit precision. The models are available through HuggingFace (Wolf et al., 2020).
We accessed the models via an API provided by the text-generation-webui framework4 running locally. For the final experiments, we also included GPT-3.5 (gpt-3.5-turbo-1106) accessed through the OpenAI API (OpenAI, 2023b).5

heading: 3.3 Observations from Preliminary Experiments
text: During development, we made several observations which we took into account for our final experimental setup:
Any input field may appear in the output. The models do not always select the most relevant fields for the given output. For example, we observed that
4https://github.com/oobabooga/ text-generation-webui
5We only included GPT-3.5 in our final experiments as proprietary models were not our focus. We did not include GPT-4 since we use the same model for evaluation (see §4.2) and LLMs tend to be biased towards their own outputs (Koo et al., 2023; Stureborg et al., 2024).
the models commonly mention identifiers, timestamps, files, and other metadata, leading to unnatural outputs. To mitigate these issues, we manually picked irrelevant fields and filtered them out from the input.
Units need to be specified explicitly. If the units are not specified in the data record, the models tend to resort to their best guess. This may go unnoticed if the unit is evident from the context (e.g., the model will usually not report the temperature in Fahrenheit instead of Celsius), but it may get problematic if the value is ambiguous (e.g., wind speed in km/h versus m/s). Therefore, we explicitly add units to all data records where appropriate.

heading: Understandable field names are enough. On
text: the flip side, we decided not to add extra descriptions to field names in the data if the field was understandable from its name (e.g., homeTeam or dimensions). As discussed by Kasner et al. (2023), pretrained models interpret field names correctly as long as they are human-readable. We only include chart metadata for the CSV files in the owid domain.
Long inputs can be troublesome. The inputs in some domains can easily get longer than 10-20k tokens. This issue is amplified by the fact that the evaluated LLMs tokenize numbers into individual digits. To accommodate for the long inputs, we picked models that accept up to 32k tokens.6 How-
6For this reason, we use Llama-2-7B-32k with 32k token context (TogetherAI, 2023) instead of the official Llama-27B-Instruct, which only supports 4k context (Touvron et al.,
ever, with long inputs, the GPU memory consumption also gets considerably higher, so we needed to downsample the data in owid and openweather to keep their length under ~8k tokens.
Few-shot experiments are infeasible. Due to the above context-length limitations, we were not able to run few-shot experiments since we could not robustly fit an additional (xexample, yexample) pair in the prompt. We attempted to include only yexample (making the setup “half-shot”), but we observed that the models then used entities from the example (unrelated to the actual input) in their outputs. Therefore, we decided to leave this line of experiments for future work (see §5.3 for discussion).
Deterministic decoding and sampling are on par. In our preliminary experiments, we observed a roughly similar output quality for both deterministic decoding and sampling.7 For the final experiments, we decided to use deterministic decoding, which is non-parametric and conceptually more suitable for D2T generation.
Prefixing the output makes parsing easier. Even with variations of a “generate only the output” instruction appended to the prompt, the models (especially Llama 2) tended to first confirm the request. For that reason, we decided to prefix the input for all the models with “Sure! Here is the {output_type}: "”. The opening quote at the end of the prefix allowed us to robustly parse the text sim-
2023b). 7We used the text-generation-webui default decoding parameters: temperature=0.7, top_p=0.9, and top_k=20.
ply by stripping the closing quote from the model output.
The outputs are fluent but inaccurate. We observed that the vast majority of model outputs were grammatically and stylistically correct, capturing the output type specified in the prompt. However, we also noticed that the outputs contained many semantic errors (even after emphasizing the focus on semantic accuracy in the prompt, see Figure 2). This observation led us to evaluate the model outputs using word-level annotations focused on semantic accuracy errors (see §4).

heading: 3.4 Final Experiments
text: Taking the observations in §3.3 into account, we proceeded to generate the outputs on the test set of QUINTD-1 for word-level error analysis. We first preprocessed the data as mentioned: we stripped out unnecessary fields, added units, and downsampled the data to fit the context. For all the models mentioned in §3.2, we used the prompt in Figure 2 and deterministic decoding with a maximum length of 512 tokens.
For comparison, we also generated outputs for the same inputs and identical prompts with GPT3.5. Note that even though we fixed the temperature and seed to 0, the rest of the decoding parameters are inaccessible to us and may differ from the parameters we used for the open models.

heading: 4 Evaluation
text: For evaluation, we focus on identifying semantic errors in model outputs. We compare the generated texts to the input data, looking for parts of texts that are not faithful to the input data. We annotate the errors on the word level, considering all the words in the output text as potential sources of errors.
We use two complementary referenceless evaluation methods:
• Ehum: human evaluation based on crowdsourcing (§4.1),
• Egpt: an automatic metric based on GPT-4 (§4.2).
The methods use similar instructions and produce outputs with equivalent format. The main idea is to compensate for the shortcomings of each approach: while human evaluation is costly (about ten times more expensive than automatic evaluation), using only an automatic metric based on a closed LLM would make the evaluation potentially
non-reproducible and biased (Kocmi and Federmann, 2023a; Wang et al., 2023b). Reporting the results of both methods should hopefully increase the robustness and replicability of our results.
Our error taxonomy and its notation are inspired by Thomson and Reiter (2020) and Thomson et al. (2023). After preliminary examination of the outputs, we settled on four error categories: INCORRECTI, NOT_CHECKABLENC, MISLEADINGM, and OTHERO. To set clear boundaries between the categories, simplify the annotation interface and reach better inter-annotator agreement, we decided to keep our taxonomy more high-level than Thomson and Reiter and not to distinguish between fine-grained categories (e.g., incorrect name vs. incorrect number). The descriptions of our error categories, as presented in the instructions for annotation, are included in Table 2.

heading: 4.1 Human-based Evaluation
text: For the human annotation metric, we prepared a custom web interface, where an annotator is instructed to annotate text spans with respective error categories. We created custom visualizations for each data format (see Figure 3 and Appendix E for examples).8
We hired annotators on the Prolific9 crowdsourcing platform. In total, we hired 100 annotators, each annotating 20 examples (four model outputs for each of the five domains). We selected annotators with at least 10 completed tasks, a 100% approval rate and English as their primary language. We paid the annotators £9 per hour, according to the platform’s recommendations. The median time for completing the annotations was 47 minutes. See Appendix B for the instructions for the annotators and the annotation interface.

heading: 4.2 GPT-4-based Evaluation
text: For automatic evaluation, we leverage the fact that LLM-based metrics can be customized for a particular task without the need for training data. In our experiments, we use a metric based on GPT4 (gpt-4-1106-preview, OpenAI, 2023a), which was shown to be superior to other LLMs in following fine-grained instructions, reaching high correlations with human judgment on evaluating generated
8We open-sourced our annotation framework as a stand-alone software package, see https://github.com/ kasnerz/factgenie.
9https://prolific.com
texts (Zhao et al., 2023; Sottana et al., 2023; Kocmi and Federmann, 2023a,b).10
We instantiate Egpt with a prompt and a system message describing the task. We instruct the model to produce a JSON output with sequentially ordered errors using the following format:
{ "errors": [{
"reason": [REASON], "text": [TEXT_SPAN], "type": [ERROR_CATEGORY] }, ...]
}.
Note that we require that the model first generates the free-form text reason for the error. Generating the reason comes at almost no extra cost and our cursory observations suggest that requiring it leads to more precise outputs.11
We align the model outputs with the original text by string matching on TEXT_SPAN, moving the current position forward after each match. We ensure that the model response is a valid JSON using OpenAI’s response_format parameter. See Appendix C for more details about the metric, including the prompt and the system message.

heading: 5 Results and Discussion
text: A summary of the word-level annotations is in Table 3 and 4, with detailed results per domain provided in Appendix F.
10We confirmed that GPT-3.5 and Llama 3 have lower correlations with human judgments also in our scenario, see Appendix D.
11We did not ask the crowdworkers for free-form reasoning about the errors since that would make the annotation notably more complex.

heading: 5.1 How Accurate Are the Model Outputs?
text: Depending on the model, between 76-86% of examples contain an error according to Ehum, suggesting that open LLMs make semantic errors very often. According to Egpt, the number is as high as 89-94%.
The most common error type is INCORRECTI. As shown in Table 3, all the open LLMs make more than two statements contradicting the data per output on average. The NOT_CHECKABLENC errors are also relatively common: more than one per output on average according to Ehum, and at least one being present in more than 25% of examples according to both metrics.
The results vary widely according to the domain (see Appendix F). For example, the outputs in wikidata contain much more NOT_CHECKABLENC errors on average (1.01 per output according to Ehum) than INCORRECTI errors (0.11 per output according to Ehum), suggesting that with simpler inputs, the models tend to introduce extra information. The openweather domain seems to be the most complex with the longest outputs (~164 tokens), more than eight errors in the output on average, and >90% of outputs containing an error.
The differences between the open LLMs are not major. Out of the open LLMs, Zephyr has the best results across categories and metrics, followed by Llama 2. However, the outputs of Mistral are longer on average, leaving more space for errors. GPT-3.5 (which we consider separately) does generally better according to both Egpt and Ehum, although it still makes an error in 60-75% of examples (2 errors per example on average). In general, the results show that LLMs make too many semantic errors to be usable in practice for D2T generation in a zero-shot setting.

heading: 5.2 Do Evaluation Methods Agree?
text: To quantify the agreement of Ehum and Egpt, we computed the Pearson correlation coefficient between the error counts on the level of words, examples, and domains as follows (note that each error category was considered separately):
• For rdomain, we used the average error counts per domain (see Table 13).
• For rexample, we used the count of errors per example.
• For rword, we used the binary indicators marking an error per word.
The correlation on the level of words is weak (rword “ 0.26) but gets better on the example-level (rexample “ 0.52) and even better on the domainlevel (rdomain “ 0.93). In Table 5, we show the percentage of words marked as errors by individual metrics. The metrics agree on the specific words in less than 6%, although they both mark around 21% of words as erroneous.
We also measure inter-annotator agreement between human annotators. For that, we obtained annotations from two annotators for 100 model outputs. The results are similar: the annotators agree weakly on the word level (rword “ 0.36), stronger on the example level (rexample “ 0.53), and even stronger on the domain level (rdomain “ 0.85). We conclude that while the details regarding error spans and categories may vary, the annotators as well as GPT-4 generally agree on the accuracy
of model outputs for a given set of examples. In the future, the agreement could be improved by measuring errors on the phrase level (Vamvas and Sennrich, 2022).

heading: 5.3 Recommendations for Future Work
text: Focus on semantic accuracy. The output of LLMs is satisfactory regarding the style, format, and purpose of the text. However, the amount of semantic errors remains very high. Improving the semantic accuracy of the models (Li et al., 2022), along with new model-based evaluation metrics (Liu et al., 2023; Xu et al., 2023), could thus help to bring improve LLM-based D2T generation systems where it is most needed.
Use efficient and long-context models. The memory issues with long context, making few-shot experiments infeasible, can potentially be solved by using more efficient models equipped with Flash Attention (Dao et al., 2022) and fast inference li-
braries such as llama.cpp12. The recent emergence of capable long-context models (Bai et al., 2023; Munkhdalai et al., 2024) also seems to play in favor of LLM-based D2T generation with long inputs.
Be careful about subtle bugs. During our preliminary experiments, we fixed several subtle bugs in our API calls such as incorrect instruction templates13 or involuntary input truncation. Therefore, we recommend careful checks of API calls, as with the apparent ease of API access and robustness of LLMs, such bugs could go unnoticed and artificially skew the model performance.
Test the models in the wild. Except for using an ad-hoc dataset of real-world data as we did in our work, the ecological validity of D2T evaluation can also be ensured by continuous evaluation with human users (Zheng et al., 2023) and evaluating the real-world impact of the systems (Reiter, 2023).
Multilinguality is an opportunity. With the recent efforts in extending D2T generation to lowresource languages (Cripwell et al., 2023), multilingual D2T generation with open LLMs seems a promising direction. Although we did not go beyond English, initial steps were already done by works such as Lorandi and Belz (2023).

heading: 6 Related Work

heading: 6.1 Evaluation of Generated Texts
text: Evaluation of generated texts is a complex task lacking a generally accepted solution (Celikyilmaz et al., 2020). While researchers are acknowledging the importance of combining multiple evaluation metrics (Gehrmann et al., 2021, 2023), most evaluation is still based on comparing the model outputs to human-written references, which tend to be noisy and expensive to collect (Dušek et al., 2019, 2020).
Many works recently investigated the potential of using LLMs for automatic reference-free evaluation of generated texts, generally achieving high correlations with human judgment (Zhao et al., 2023; Sottana et al., 2023; Kocmi and Federmann, 2023a,b; Chiang and Lee, 2023; Wang et al., 2023a; Fu et al., 2023). However, they also voice concerns about its non-reproducibility (Kocmi and Federmann, 2023a) and potential bias of these models (Wang et al., 2023b).
12https://github.com/ggerganov/llama.cpp 13https://huggingface.co/docs/transformers/
chat_templating
Human evaluation is an essential component of natural language generation experiments (van der Lee et al., 2019, 2021). The closest human evaluation protocol to our scenario is the reference-free word-level annotation of complex D2T generation output proposed in Thomson and Reiter (2020) and Thomson et al. (2023).

heading: 6.2 D2T Generation Tasks
text: Weather Forecasts First attempts for generating weather forecasts include template-based and statistical approaches (Belz, 2005, 2008; Angeli et al., 2010) for the Sumtime-meteo and WeatherGov datasets (Sripada et al., 2002; Liang et al., 2009). More recently, Balakrishnan et al. (2019) introduced a weather forecast dataset with treestructured meaning representations. Our weather forecasts are less structured and based on a 5-day weather outlook.
Product Descriptions Our phone specifications are closest to Wen et al. (2015, 2016), who introduced a dataset for generating descriptions of laptops and TVs. Their solution was based on recurrent neural networks, although templates remained a go-to approach for the task (Wang et al., 2017). Recently, Shao et al. (2021) and Koto et al. (2022) also proposed specialized architectures based on pretrained language models for the data from big e-commerce platforms.
Sport Reports All the D2T generation datasets from the Rotowire family (Wiseman et al., 2017; Wang, 2019), including SportSett:Basketball (Thomson et al., 2021), and ESPN-NBA (Nie et al., 2018) focus on generating basketball reports. Along with MLB (Puduppully et al., 2019b), these datasets belong among the most challenging D2T datasets, attracting various neural-based solutions (Puduppully et al., 2019a, 2022; Puduppully and Lapata, 2021; Rebuffel et al., 2020). We use instead simpler data covering ice hockey game summaries.
Chart Captions Following the early rule-based approaches (Demir et al., 2008, 2012), the approaches for chart captioning recently tackle largescale datasets from data analytic institutions (Obeid and Hoque, 2020; Kantharaj et al., 2022). We focus on one of the tasks from Sharma et al. (2021), which is generating descriptions of time series in the health domain.
Entity Descriptions The task of generating descriptions for a knowledge graph has been covered
extensively in D2T generation (Gardent et al., 2017; Castro Ferreira et al., 2020; Agarwal et al., 2021; Chen et al., 2020; Ribeiro et al., 2020, inter alia). Our task is to describe an entity provided a list of its properties, which is closely related to generating entity descriptions from Wikipedia infotables (Lebret et al., 2016).

heading: 6.3 D2T Generation with LLMs
text: Recent works have focused on exploring the capabilities of closed LLMs on existing D2T generation datasets. Axelsson and Skantze (2023) evaluated GPT-3.5 (OpenAI, 2023b) on WebNLG, along with Yuan and Färber (2023), who also tested the model on the AGENDA dataset (Koncel-Kedziorski et al., 2019). Both works found that regardless of potential data contamination, the LLMs rank behind state-of-the-art finetuned models on automatic metrics. Zhao et al. (2023) tested closed models on modified table-to-text generation datasets and found out that in terms of faithfulness, GPT-4 can outperform state-of-the-art models.

heading: 7 Conclusion
text: We provided an exploratory study into D2T generation with open LLMs. We proposed new directions for D2T generation, including using ad-hoc test sets, data in common formats, and referencefree evaluation. By a combination of GPT-4-based metric and human evaluation, we evaluated the performance of LLMs on five domains, providing word-level annotations of model outputs across five domains and recommendations for future directions in D2T generation.

heading: Acknowledgements
text: This work was funded by the European Union (ERC, NG-NLG, 101039303) and Charles University project SVV 260 698. It used resources of the LINDAT/CLARIAH-CZ Research Infrastructure (Czech Ministry of Education, Youth, and Sports project No. LM2018101).

heading: Limitations
text: In our work, we do not include a comparison to other D2T generation approaches. The main reason is that our benchmark is reference-free, while a large majority of prior approaches are based on models finetuned on reference outputs. However, we believe that our work still satisfies our main
goal of providing insights into behaviors of open LLM models on D2T generation.
We acknowledge that reference-free metrics currently have various shortcomings, including reliance on closed models and specific human annotation protocols, leading to limited replicability and a high price of execution. The approaches occasionally produce incorrect outcomes themselves and they have only moderate correlations with each other. We believe that these shortcomings will be addressed in the future with open model-based metrics.
Our choice of models is limited to 7B-parameter open LLMs due to our limited computational resources. Also, unlike some other LLMs such as GPT-Neo (Black et al., 2022) or BLOOM (BigScience Workshop et al., 2022), the models we used do not disclose the data they were trained on. For this reason, we find it ever more important to test the models on benchmarks whose labels could have not been included in their training data.
The approaches based on LLMs may produce factually and semantically incorrect information. Any text produced by the LLMs therefore needs to be carefully examined, and no decisions should be based on the generated text alone. Discovering the causes of LLMs’ hallucinations is out of scope of this paper, but is currently a major topic under investigation (Ji et al., 2023).

heading: Ethical Considerations
text: The human evaluation study was approved by the internal ethics committee of our institution. The annotators were hired over Prolific and paid the platform-recommended wage of 9 GBP/hour. The annotators were preselected based on their primary language (English) and their country of residence (US, UK, Ireland, Australia, New Zealand). All annotators were shown detailed instructions and explanation of the data types, data sources, and the purpose of the research (see Appendix B for details). The domains in QUINTD were selected so that they do not contain any sensitive or potentially offensive content. We do not collect any demographic data about the participants.

heading: A QUINTD Data
text: Here, we describe the data sources we include in the QUINTD collection tool and the procedure of collecting the QUINTD-1 benchmark. To replicate the data collection, please refer to the scripts we provide.14

heading: A.1 Selection of Data Sources
text: When selecting the data sources, we had the following desiderata:
• Data needs to be publicly available.
• Data needs to represent a common data-to-text task.
• Data needs to be in a common format (or straightforwardly transformable to one).
We settled on the data sources described in Appendix A.2. All the sources can be accessed using an API. Note that some of the APIs have access limits, either for the requests made from a single account per day or for a number of requests from an IP address within a time window. However, these limits do not severely limit the data collection process on the scale we use here.

heading: A.2 Data Collection
text: Table 6 summarizes the output types for each domain.

heading: QUINTD.
text: 14https://github.com/kasnerz/quintd

heading: A.2.1 OpenWeather
text: OpenWeather (OpenWeatherMap.org) is an online service that provides global weather data via web interface and API. The API responses are in the JSON format documented at the official website. For our experiments, we used the forecast5 API, which allows to download a 5-day forecast with 3-hour resolution for any location specified by its GPS coordinates.
The free tier is limited to 1,000 API calls per day, which is enough to download our whole test set in one bulk. However, at the time of experiments, the free API only allowed to download the data for the time when the request was made. At the time of writing, OpenWeather is pushing a new One Call API 3.0 which allows to download weather data for any timestamp, but only 4 days ahead (instead of 5). These restrictions somehow limit the replicability of our QUINTD-1 dataset (at least with the free API) but do not limit downloading a new batch of data with a similar format.
For the QUINTD-1 dataset, we randomly sampled 100 cities for each split from the list of cities with a population over 1000 and used their coordinates in the queries to OpenWeather API. All the data forecasts were downloaded on Jan 3, 2024.

heading: A.2.2 GSMArena
text: GSMArena is a website providing specifications and reviews for mobile devices. For downloading the data, we used the unofficial gsmarena-api tool, which returns the data in a JSON format. Note that GSMArena imposes limitations on the number of requests per IP address, which may induce delays when downloading a larger amount of data.
To create a balanced sample, we downloaded detailed specifications of 10 products from each available brand and randomly selected 100 products for each split from the downloaded set.

heading: A.2.3 RapidAPI Ice Hockey
text: RapidAPI is a service that provides API access to data from multiple domains, including sport, finance, entertainment, and others. Most APIs are provided in a freemium mode, i.e., with a limited number of daily API calls.
For QUINTD, we selected the IceHockeyAPI (popularity 9.1 / 10), which provides access to ice hockey games from world top leagues. Our choice was influenced by our own personal preferences, combined with the desire to cover a sport that has
not been covered previously in sports report generation.
We used the matches endpoint which returns high-level details about a game. Note that the API allows only 50 requests per day, but that does not limit the data collection since the endpoint returns all the games played on a particular day in a single request. We downloaded the games played on 27 November 2023 for the development set (184 games) and 29 November 2023 for the test set (216 games), taking a random sample of 100 for each split.

heading: A.2.4 OurWorldInData
text: OurWorldInData is a public database and web interface for data about world developments in various domains and sources. We used the official API (currently experimental), which is accessible through the Python package owid-catalog. The package allows accessing individual CSV tables as Pandas dataframes.
For our data collection, we decided to limit ourselves to time series, i.e., a single column with values changing over time. Besides the simplicity of visualizing such a chart (which is used by human annotators for checking the correctness of the output), there is also a clear goal for the target chart description: describing the developments of a value over time. We also limited ourselves to the health domain. In particular, we selected the tables COVID data (columns new_cases_smoothed_per_million, new_tests_smoothed_per_thousand, people_ vaccinated_per_hundred, reproduction_ rate, and positive_rate) and Life expectancy data (column life_expectancy_0).
We downloaded the data for all countries with non-empty entries in the table, taking a random sample of 100 examples for each split. On model input, we formatted the data for each time series as a two-column CSV, including the title, the description, and the unit for each example as a comment (#) at the beginning of the input.

heading: A.2.5 Wikidata
text: Wikidata is a large open-source knowledge graph containing factual information about entities and their properties. Wikidata provides access through an official API, but we instead decided to extract our data using the wikidatasets (Boschin, 2019) Python library, which provides access to preprocessed properties of entities from particular do-
mains. It allowed us to avoid crawling and filtering the knowledge graph, and its offline processing made the data collection faster.15
For our dataset, we selected the entities from the companies, countries, films, and humans domains. For each entity, we randomly extracted between 2 to 10 properties in the knowledge graph. We extracted up to 100 subgraphs for each domain and took a random sample of 100 subgraphs for each split. On model input, we formatted each subgraph as a simple Markdown-formatted text snippet, using the entity as a title and including a bullet point for each key-value pair.

heading: B Human Evaluation
text: As described in §4.1, we set up the human evaluation campaign on Prolific. To make the data more accessible to the annotators, we created custom data visualizations for each domain. For the data in openweather and owid, we used interactive graphs from Highcharts.com, and we manually created the tables for other domains. You can find the full instructions for human annotators in Figure 5 and the examples of data visualizations in Appendix E.

heading: C GPT-4 Evaluation
text: We used the prompt in Figure 4 for instantiating the GPT-4-based metric.16 We set the temperature to 0 to improve the replicability of the process. We ensured that the output is a valid JSON using the parameter response_format in the OpenAI API. At the price of $0.01 per 1k input tokens and $0.03 per 1k generated tokens, the evaluation process costs approximately $45 in total.

heading: C.1 Aligning the Errors
text: For aligning the errors with the original text, we perform string matching on the text span decoded by GPT-4 in the TEXT_SPAN field. In our preliminary experiments, this method proved to be more robust than either asking for start and end indices of the error span (which would rely on the model’s ability to count characters) or performing sequence tagging on the copy of the input (which would rely on the model’s ability to perfectly copy the input).
15All the entities and properties are linked with an identifier to the Wikidata database, making the process also replicable through the official API.
16Note that the example in the prompt differs from the example used for human annotators (see Figure 5). We revised the example to be more instructive, but we were not able to re-run the GPT-4 evaluation due to our limited budget.
We tried to respect the monotonic ordering of text spans but fell back to full-text search if the span is not found following the previous one. We consider this approach successful since matching completely failed only in a minority of cases (137 out of 6927). Based on our manual examination, these mostly include cases where GPT-4 tried to suggest a missing piece of text as an error or did not manage to copy the input text verbatim.

heading: D Experiments with Open LLMs as Evaluators
text: To select the most suitable LLM for automatic evaluation, we compared correlations with human judgment of the following models:
• GPT-4 (OpenAI, 2023a) used via OpenAI API (gpt-4-1106-preview),
• GPT-3.5 (OpenAI, 2023b) used via OpenAI API (gpt-3.5-turbo-1106),
• Llama-3-70B17 running locally via Ollama in 4-bit quantization (meta-llama/Meta-Llama-3-70B).
We used all the models with the same prompts, temperature 0, and force-decoded JSON outputs. In Table 7, we show Pearson correlation coefficients computed as described in §5.2. We can see that the strongest model is GPT-4, followed by Llama-370B and GPT-3.5. As the gap between the models is substantial, we opted for using GPT-4 which is the strongest model.

heading: E Examples
text: Here, we present an example of inputs and model outputs (along with annotations) for each domain:
• openweather: Figure 7 (in) and Table 8 (out),
• gsmarena: Figure 8 (in) and Table 8 (out), 17https://llama.meta.com/llama3/
• ice_hockey: Figure 9 (in) and Table 10 (out),
• owid: Figure 10 (in) and Table 11 (out),
• wikidata: Figure 11 (in) and Table 12 (out).
Note that the graphs for openweather and owid are interactive when accessed through the web interface.

heading: F Full Results
text: Here, we include the tables with results for individual domains:
• Table 13 presents the average numbers of errors per output separately for each domain (the aggregated results are in Table 3),
• Table 14 presents the ratio of outputs containing at least one error separately for each domain (the aggregated results are in Table 4).

heading: What is the purpose of the research?

heading: An example of the annotated output

heading: How should I decide on the errors?

heading: How to mark and submit the annotations?

heading: Future Work
text: We believe that these shortcomings will be addressed in the future with open model-based metrics. Our choice of models is limited to 7B-parameter open LLMs due to our limited computational resources. Also, unlike some other LLMs such as GPT-Neo (Black et al. , 2022) or BLOOM (BigScience Workshop et al. , 2022), the models we used do not disclose the data they were trained on. For this reason, we find it ever more important to test the models on benchmarks whose labels could have not been included in their training data. The approaches based on LLMs may produce factually and semantically incorrect information. Any text produced by the LLMs therefore needs to be carefully examined, and no decisions should be based on the generated text alone. Discovering the causes of LLMs’ hallucinations is out of scope of this paper, but is currently a major topic under investigation (Ji et al. , 2023)