# Large Language Models as Generalizable Text-to-Table Systems

Steven Coyne ${ }^{1 *} \quad$ Yuyang Dong ${ }^{2}$
${ }^{1}$ Tohoku University ${ }^{2}$ NEC Corporation
coyne.steven.charles.q2@dc.tohoku.ac.jp
dongyuyang@nec.com

## Abstract

Wu et al. [1] formalized the task of text-to-table generation, in which unstructured texts are summarized in table form. However, their experiments focus on fine-tuned, dataset-specific models that show limited performance outside of the data they were trained on. As the task assumes no user query, systems ideally generate valid tables across various input texts without favoring the layout or table count of a particular dataset. To address this, we apply prompted large language models (LLMs) to this task, assessing their suitability as generalizable text-to-table systems. To more accurately rate outputs, particularly in a zero-shot setting, we revise the evaluation script from [1] to align tables by column similarity before scoring.

## 1 Introduction

Text-to-table generation is a sequence-to-sequence task in which one or more structured tables are generated based on the content of an input text [1]. The input can be a sentence or a longer document. The output is a table that attempts to represent the information in the input text. An example using a basketball game report from the RotoWire dataset [2] can be seen in Figure 1.

In this task, there is no explicit user query. Tables are designed based only on the input text. This means that the task is at least partially open-ended, as there may be multiple valid tables or sets of tables that could accurately represent an input text. As a result, the "ground truth" reference tables in the datasets are not definitive.

As the goal of the text-to-table setting is to convert an input text into a "valid" table that makes sense with respect to the input, it is a high-level logical task that may depend on world knowledge and common sense reasoning. Examples of the sort of inferences that may be done as part of


Figure 1 Example of an input text and reference table in the text-to-table task. The table format is adapted from [1], with their <NEWLINE> separator replaced with line breaks for readability.

the table design process include the following:

Entity Linking: An entity may be referred to in multiple ways due to nicknames, metonymy, and other phenomena. A basketball team may be referred to by its name or by its home city. There may be a switch between these partway through the text. The model must resolve these references.

Implicit Reasoning: Some information in a text may be implied by other facts. If a basketball game is won 120-90, and the winning team is named, then the model must assign 90 to the losing team even if no sentence explicitly describes this.

Contextual Understanding: Depending on the input text, the "best" way to represent information may differ significantly. In a biographical description, information may be best summarized as a one-row table in which other entities are listed only in relation to the main entity the text is focused on.

Such principles may be learned as part of a fine-tuning process with respect to a particular dataset of relatively similar texts and tables. This can be seen in previous works such as [1] and [3]. However, these models are limited to the specific datasets that they were trained on and may
show poor performance on tables that do not resemble the training data's format or use similar header names.

Given their strong performance across a number of reasoning tasks [4], we hypothesize that LLMs such as GPT-3 [5] can perform this task in a generalizable manner without the need to fine-tune on hundreds or thousands of examples, taking advantage of principles such as chain-ofthought reasoning [6] or in-context learning [5] to address the aforementioned reasoning challenges. To this end, we experiment with LLM prompting strategies using several different task decomposition settings, which we apply uniformly to each of the text-to-table datasets designated in [1] to test the generalized performance of the selected models.

## 2 Related Work

Previous works have examined the use of LLMs to handle table-based tasks. Li et al. [7] apply base and fine-tuned GPT-3.5 models to diverse table tasks, showing strong performance and generalizability to new data and tasks. However, their study does not include the text-to-table task described in [1]. Tang et al. [8] analyze the text-to-table setting using a variety of prompted LLMs, generating tables in raw text, $\mathrm{LT}_{\mathrm{E}} \mathrm{X}$, and HTML. Their work focuses on the challenges of evaluation in this task, rather than the generalizability of LLMs compared to seq2seq baselines across datasets. Our work also differs in using a JSON format and in its approach to subtask decomposition.

## 3 Methodology

## 3.1 Baseline and Datasets

As a baseline, we use a system in [1], a seq2seq model based on BART [9]. We use the same datasets as their work: E2E [10], WikiTableText [11], WikiBio [12], and RotoWire [2]. As WikiBio is large, we use a $10 \%$ subset in our experiments to control costs. We reproduce the experiments in [1] based on their code, ${ }^{1)}$ utilizing the "table constraint" and table relation embedding features. However, to align with our other experiments, we modify the output format and scoring script as described below.

## 3.2 Output Format

The system described in [1] uses an idiosyncratic format that differs between the RotoWire dataset and the other


three used in their study. To unify handling of all datasets, and hypothesizing that a more common structured format may be easier for LLMs to generate consistently, we use a JSON format for our experiments. Furthermore, a JSON format permits experimentation with JSON-specific output parameters in our selected LLMs.

## 3.3 Evaluation

For evaluation, we use the scorer released as part of [1]. This scorer parses tables into relations and scores them with three metrics: exact match, chrF [13], and BERTScore [14]. We modify this scoring script to handle JSON inputs as part of the format change. We also create a JSON conversion script to allow the baseline system outputs reproduced from [1] to be evaluated by the adapted scorer.

In addition to format adaptations, we further modify the scoring script to allow tables in a multiple-table setting to be aligned by similarity. The original script can only match such tables by name exactly, but this is unlikely in cases such as a zero-shot table generation, as a table name difference as small as "Team" and "team" would result in a score of zero. We implement a table-matching algorithm in which the candidate tables are aligned by the similarity of their column names. We use the highest average chrF metric across column names to align tables. We experimented with using BERTScore for this purpose, but the resulting scores were slightly lower. We hypothesize that cases like "Team" vs. "Teams," in which there is some surface similarity, outnumber cases such as "Player" and "Athlete" in which there are two distinct semantically similar words.

## 3.4 Our Method

We selected the OpenAI LLMs gpt-3.5-turbo-1106 and gpt-4-1106-preview for our experiments. Since the outputs of our prompts are in a JSON format, we include experiments with the response_format parameter, performing the gpt-3.5-turbo-1106 experiments in both text mode and JSON mode to compare them.

We experiment with decomposing the task of table generation, resulting in the following subtasks. Examples of the outputs of each can be seen in the Appendix.

Grouping: Entities in the text are listed and sorted into categories. The input is the text, and the output is one or more lists of entities wrapped in JSON.

Schema Generation: Based on the input text, and

Figure 2 Prompt settings in our experiments. Experiments are performed with both few-shot and zero-shot prompts.

optionally the output of a grouping step, the model generates a list of one or more JSON table schemas. Each schema is associated with a planned output table. Table Generation: Based on the input text and one or more input schemas, potentially the result of a schema generation step, the model generates tabular JSON. Each table row is represented by a JSON object, and all rows in a given table use identical fields. Multiple tables are nested within a top-level JSON object.

We used a prompt chaining approach in which each of these subtasks was handled by a separate call to the LLM. We defined both zero-shot and few-shot prompts for each step. The zero-shot prompts incorporated chain-of-thought [6]. In the few-shot prompts, example inputs and outputs were drawn randomly from the validation set and appended before the inputs. We used 5 -shot prompting for all datasets in the GPT- 3.5 experiments, plus additional 1-shot prompting for RotoWire. We created two sets of prompts for the schema generation step: one using grouping step outputs and one using only the input text.

## 4 Experiments

Our first experiments focus on the table generation step. To test the performance of the LLMs in this subtask alone, we establish a setting where the ground-truth "gold schema" is extracted from the reference file and provided along with the input text. Theoretically, this establishes an upper limit of performance for other pipelines. Following this, we performed pipeline experiments using generated schemas, both with and without grouping. This resulted in six total settings: zero-shot and few-shot versions of each of the three prompt settings seen in Figure 2.

Due to budgetary constraints, our GPT-4 experiments were limited to the RotoWire dataset using the schema + table and group + schema + table settings, using only JSON mode. We additionally limit few-shot experiments to one example to control costs. We selected RotoWire due to its small size and high complexity, being the only dataset with two-dimensional tables and multiple tables per input. We selected JSON mode after it showed slightly improved performance on RotoWire in the GPT-3.5 experiments.

## 5 Results and Discussion

Results can be seen in Tables 1 and 2. In the "gold schema" few-shot setting, the scores of the LLMs were comparable or even superior to those of the respective baseline fine-tuned model. However, it must be stressed that this setting involves a leak of ground-truth information.

In settings with generated schemas, the LLM scores are lower than the baseline model scores. However, the LLMs achieve moderate performance across all datasets, suggesting their capabilities are generalizable.

The LLMs display higher performance in few-shot settings compared to zero-shot, and in the case of RotoWire, higher performance at a higher number of examples.

The results for GPT-3.5 show relatively similar scores between the JSON and text output formats. Which format scores higher varies by dataset, with E2E showing higher scores in text mode and RotoWire showing higher scores in JSON mode. All differences are under two points, except for the gold schema zero-shot E2E exact match score.

Overall, the experiments that included a grouping step produced lower scores than the experiments using only schema and table steps. In RotoWire, the group + schema + table prompt chain performed better than the no-group chain in a few-shot setting, and the no-group chain per-

} & E2E & 99.63 & 99.70 & 99.88 & 97.88 & 98.00 & 98.56 & 99.63 & 99.70 & 99.88 & 7.88 & 98.00 & 98.56 \\

Table 1 Scores for gpt-3.5-turbo-1106. E, Ch, and BS refer to exact match, chrF, and BERTScore, respectively. output_format does not apply to baseline scores, so scores are simply repeated. Scores in bold are higher than the baseline, but it should be noted that the gold schema setting involves a leak of reference table information. Scores for RotoWire are the average of Team and Player scores.

Table 2 Scores for gpt-4-1106-preview on the RotoWire dataset, using the average of Team and Player scores.

formed better in zero-shot. These trends were seen in both gpt-3.5-turbo-1106 and gpt-4-1106-preview.

Interestingly, while GPT-4 outperformed GPT-3.5 in the zero-shot setting, the reverse was seen in the one-shot RotoWire experiments, where GPT-3.5 scored higher.

Finally, the models were able to reliably generate valid JSON in all experiments, even without using JSON mode.

## 6 Conclusion

The experiments suggest that our prompt settings can permit LLMs such as GPT-3.5 and GPT-4 to serve as relatively generalizable text-to-table generation systems. While their performance lagged behind the baseline finetuned systems in general, the LLMs show promise in this task, particularly in few-shot settings in which some basic information about the target table format can be provided. Additionally, the models' high performance in the gold schema experiments suggests that they may be able to follow user-defined schemas reliably. There remain many unexplored possibilities for prompting settings in this task, and our decomposition experiments are by no means exhaustive. Future works may explore additional approaches to prompting, format decisions, and table evaluation.

## Acknowledgments

This work is mainly supported by NEC Corporation and partially supported by JSPS KAKENHI Grant Number JP22H00524.

## References

[1] Xueqing Wu, Jiacheng Zhang, and Hang Li. Text-to-table: A new way of information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2518-2533, Dublin, Ireland, May 2022. Association for Computational Linguistics.

[2] Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges in data-to-document generation. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253-2263, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.

[3] Tong Li, Zhihao Wang, Liangying Shao, Xuling Zheng, Xiaoli Wang, and Jinsong Su. A sequence-to-sequence\&set model for text-to-table generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pp. 5358-5370, Toronto, Canada, July 2023. Association for Computational Linguistics.

[4] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901. Curran Associates, Inc., 2020

[6] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.

[7] Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table-tuned gpt for diverse table tasks, 2023.
[8] Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Gerstein. Struc-bench: Are large language models really good at generating complex structured data?, 2023.

[9] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 78717880, Online, July 2020. Association for Computational Linguistics.

[10] Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis, editors, Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, Saarbrücken, Germany, August 2017. Association for Computational Linguistics.

[11] Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, and Tiejun Zhao. Table-to-text: Describing table region with natural language. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, No. 1, Apr. 2018.

[12] Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1203-1213, Austin, Texas, November 2016. Association for Computational Linguistics.

[13] Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina, editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.

[14] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020.

## A Appendix

Here, we present an example of the JSON format of our prompt chains, extending the example from Figure 1. This particular example is converted from the RotoWire development set, not generated by one of our models.

## Input Text

The Wizards launched yet another comeback on Tuesday, this time feasting on the relatively inexperienced Los Angeles Lakers. Washington not only overcame a 13 - point fourth quarter deficit, but won by double digits as well. The team shot over 51 percent from the field on the night and outscored LA 37 - 13 in the fourth. At the crux of the win was All-Star point guard John Wall, who scored 34 points to go along with 14 assists and four steals. Wall was an excellent 14 - of - 25 from the field. LA, meanwhile, saw promising play from its own point guard. Sophomore D'Angelo Russell filled the stat sheet respectably, shooting 10 of - 21 for 28 points to go along with nine assists and six rebounds. His backcourt partner, Jordan Clarkson, added another 22 points and shot 10 - of - 19.

## Grouping Output Format

\{

"Team " : [

"Lakers",

"Wizards"

],

"Player": [

"John Wall",

"D'Angelo Russell",

"Jordan Clarkson"

]

\}

## Schema Output Format

[

\{

"title": "Team",

"type" : "object",

"properties":\{

"Team " : \{

"type": "string"

\},

"Percentage of field goals":\{

"type ": "integer"

\},

"Points in 4 th quarter":\{

"type": "integer"

\}

\}

\},

"title": "Player",

"type": "object",

"properties":\{

"Player": \{

"type": "string"

\},

"Assists":\{

Table Output Format

\{