Venue: NAACL 2024

# Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data?

Xiangru Tang\({}^{\clubsuit*}\) Yiming Zong\({}^{\diamondsuit*}\) Jason Phang\({}^{\diamondsuit}\) Yilun Zhao\({}^{\spadesuit\spadesuit}\) Wangchunshu Zhou\({}^{\spadesuit\spadesuit}\) Arman Cohan\({}^{\spadesuit\spadesuit}\) Mark Gerstein\({}^{\spadesuit\spadesuit}\) Yale University Zhejiang University New York University xiangru.tangeyale.edu

Contributed equally. +
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers),_ pages 12–34 June 16-21, 2024 ©2024 Association for Computational Linguistics

###### Abstract

Despite the remarkable capabilities of Large Language Models (LLMs) like GPT-4, producing complex, structured tabular data remains challenging. Our study assesses LLMs' proficiency in structuring tables and introduces a novel fine-tuning method, cognizant of data structures, to bolster their performance. We unveil Struc-Bench, a comprehensive benchmark featuring prominent LLMs (GPT-NeoX-20B, GPT-3.5, GPT-4, and Vicuna), which spans text tables, HTML, and LaTeX formats. Our proposed FormatCoT aids in crafting format-specific instructions from the intended outputs to populate this benchmark. Addressing the gap in task-centered evaluation, we propose two innovative metrics, P-Score (**P**rompting Score) and H-Score (**H**euristical Score), to more accurately gauge LLM performance. Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains, outshining its LLM counterparts across most measures. In-depth error analysis and creating an ability map across six dimensions--coverage, formatting, reasoning, comprehension, pragmatics, and hallucination--highlight areas for future enhancements and suggest forthcoming research trajectories. Our code and models can be found at [https://github.com/gersteinlab/Struc-Bench](https://github.com/gersteinlab/Struc-Bench).

## 1 Introduction

Significant advancements have been made in various natural language processing tasks by Large Language Models (LLMs) Brown et al. (2020); Scao et al. (2022); Ouyang et al. (2022); Muennighoff et al. (2022); OpenAI (2023); Zhao et al. (2023), especially in text generation tasks Qin et al. (2023). The ability to output structured data, one of the key aspects of generative capability, has also attracted great interest in previous studies Wu et al. (2022); Zhao et al. (2023c, b); Zha et al. (2023).

Despite their advanced capabilities, LLMs have problems with generating complex structured tables, an indispensable skill for practical applications like coding copilot and automated report generation. This proficiency entails the organization of information from multifarious sources into coherent structures. Generating structured tables as outputs not only helps human understanding but also facilitates the automated data processing pipeline in autonomous language agents. Furthermore, generating structured tables can also serve as a critical preprocessing procedure for downstream tasks such as decision-making and knowledge extraction. However, the current landscape of LLM evaluation often neglects this aspect of table generation, which casts uncertainty on their full potential and utility in such scenarios. Our research seeks to thoroughly investigate these gaps.

_First, there is a lack of systematic analysis and comprehensive benchmarks_ of the ability of LLMs to output complex structured tabular data. Previous efforts on evaluating LLMs Qin et al. (2023); Ma et al. (2023) on structured data primarily centered around simple Information Extraction (IE) tasks: recognizing named entities, extracting relations, and detecting events. Here the goal of IE tasks is to gather the extracted data in a highly structured form Zhong and Chen (2020). Much earlier work was considerably more task-centric as opposed to LLM-centric. The focus was predominantly on generating structured data from text (text-to-data) tasks with pre-trained models He et al. (2023); Rossiello et al. (2022); Whitehouse et al. (2023); Pietruszka et al. (2022) like BART Lewis et al. (2019) and T5 Raffel et al. (2020).

_Second, there is a lack of evaluation metrics_ of structured tabular data generation. Existing benchmarks often rely on rudimentary objective metrics such as word overlap to measure the accuracy of

the content generated by the model Li et al. (2023); Wu et al. (2022); Pietruszka et al. (2022). This may be insufficient for evaluating whether LLMs can generate structured output, as an ideal evaluation metric ought to also consider the format of generated content.

Third, there is a lack of methods to enhance the performance of current LLMs to better _follow natural language inputs and generate tabular outputs with the correct format_.

Our contributions can be encapsulated as outlined in Figure 1: (1) We introduce Struc-Bench, a benchmark specifically constructed for generating structured tabular data. (2) We evaluate popular LLMs on Struc-Bench using two proposed metrics, providing a comprehensive insight into the prevailing limitations and common error types. (3) We propose FormatCoT to generate instruction tuning data, wherein we utilize GPT-3.5 to generate format instructions and then fine-tune LLaMA-7B model to follow these formats. The resulting impressive performance demonstrates that with FormatCoT small models can indeed surpass the performance of a larger model in this particular task.

## 2 Problem Analysis and Benchmark

### Problem Definition and Motivation

LLMs are tasked with generating complex structured tables, a process that involves understanding both the content and the specific format requirements, such as LaTeX syntax. This task extends beyond simple text generation as it demands precision not just in content creation but also in adhering to a detailed and precise structural format. Specially, we aim to convert unstructured textual data into structured tabular data, by extracting necessary contents from text and following the required structure or format.

### Problem Analysis

To assess LLMs' capability to convert textual descriptions to structured tables, we utilized the RotoWire dataset Wiseman et al. (2017), originally a table-to-text dataset, in reverse as a text-to-table task. After ensuring that the descriptions contained adequate information for table generation through a review of 20 samples, we found significant limitations in the performance of GPT-3.5 and GPT-4, especially when dealing with complex structures as detailed in Appendix A.

When put to the test of generating data in precise formats, such as tables, both GPT-3.5 and GPT-4, despite their advanced capabilities, frequently erred, as evidenced by a systematic MTurk human annotation study (refer to Appendix B). The types of errors, categorized into 'Element Errors', 'Element Format Errors', 'Structure Error', and 'Structure Naming Errors', are quantified in Figure 2.

A mere 3% of GPT-3.5's outputs were fully accurate, with GPT-4 only slightly better at 9%. These results suggest design limitations within the GPT architecture, which, although effective at mimicking language patterns, falter in tasks requiring sustained structural coherence over longer sequences.

### Benchmark Construction

We begin by selectively sourcing tables larger than 3x3 from the RotoWire Wiseman et al. (2017) dataset to present a baseline of complexity. Then, to broaden our dataset diversity across various domains, from The Stack Kocetkov et al. (2022), which includes GitHub code in 358 programming languages from the BigCode project, we first select LaTeX and HTML formats. Further refining our dataset, we extract elements relevant to table representations to ensure focused complexity and relevance to our structured data generation task. An example of our benchmark is shown in Figure 4.

Table 1 gives statistics for the Rotowire dataset and our constructed datasets. Then we evaluate 4 popular LLMs, including GPT-NeoX-20B Black et al. (2022), GPT-3.5, GPT-4, and Vicuna-13B Chiang et al. (2023). For LaTeX and HTML data without paired text, we harness GPT-3.5 to construct synthetic descriptions to be utilized as input. To guarantee the quality of our

Figure 1: Overview of our workflow: We commenced by creating datasets of raw text tables, HTML tables, and LaTeX tables. Subsequently, LLaMA-7B was trained using the training data constructed by FormatCoT. Finally, our benchmarks validate the effectiveness of the current LLMs to generate such tables.

 

benchmark, we sample 50 tables for each format to ensure the correctness of the descriptions. Initially, we achieved a satisfaction rate of 76%. However, upon incorporating a manual interpretation template (e.g. tab names for HTML) tailored to each format (Appendix D), our satisfaction rate improved significantly, reaching 96%. For example, HTML tables possess their unique tags and structure, conforming faithfully to the syntax rules of HTML language.

## 3 Methodology

### Data Generation

As shown in Figure 3, we propose FormatCoT with GPT-3.5, a self-instruct method to generate {data, instruction} pairs for fine-tuning purposes. Specifically, our prompt of FormatCoT involves guiding models to accurately describe and interpret the format elements presented in the output table, inspired by Wang et al. (2023b) in the summarization task. To verify the effectiveness of our proposedFormatCoT, we conduct an ablation study in Appendix F.

### Instruction Tuning

We introduce an instruction tuning approach designed specifically to enhance LLMs' abilities in generating structured text Touvron et al. (2023); Patil et al. (2023). Specifically, we combine GPT-3.5-generated format descriptions of output tables and the original text input as the new input of LLaMA fine-tuning. In other words, we start with GPT-3.5 processing table data and synthesizing comprehensive format instructions. The LLaMA model is then fine-tuned on these enriched instructions we generate. This approach simulates a user-agent interaction where GPT-3.5 effectively fetches and consolidates table information, conversationally instructing LLaMA for the final text generation, outlined in Figure 3.

### Evaluation Metrics

Assessing the accuracy of generated tables against ground truth is complex due to the variability in formatting, like HTML. An ideal evaluation metric needs to discern substantial data discrepancies

\begin{table}
\begin{tabular}{l c c l r} \hline \hline
**Dataset** & **\# Train** & **\# Test** & **Format** & **Rows \& Columns** \\ \hline Struc-Bench Table & 3.4k & 728 & Raw tcv & 7.26 \& 8.8 \%5 \\ Struc-Bench LifeX & 5.3k & 500 & IipEx & 2.75 \& 4.47 \\ Struc-Bench IFTML & 5.4k & 499 & HTML & 5.50 \& 3.54 \\ \hline \end{tabular}
\end{table}
Table 1: Struc-Bench data statistics. The number of Rows & Columns has been averaged.

Figure 3: The upper-left corner box represents the original input, which notably lacks a description of the format. To explicitly instruct the model on format understanding, we employ the FormatCoT located on the right, which produces the <FORMAT INSTRUCTION>. The lower-left box illustrates what the input for LLaMA fine-tuning looks like after passing through FormatCoT. <TEXT> provides a descriptive text for the expected table output (original input), ¡TABLE¿ serves as a reference table (output), and the ¡FORMAT INSTRUCTION> is a format guideline generated through FormatCoT (added into input). Detailed prompts are displayed in Appendix D.1.

Figure 2: Error analysis by human annotation. Some error types are explained in Appendix A.

 

while disregarding trivial formatting variations.

We propose to break down the similarity of two tables into two coarse components: _content_ and _format_. In scoring content similarity, we attempt to parse _content_ out the data within the table cells, and compute the similarity. This similarity is computed between the generated and ground-truth table cells by commonly used similarity metrics. In scoring format similarity, we place higher emphasis on components such as the number of columns and rows, cell alignment, and the table caption. We find that these two scores allow us to perform a more involved analysis of where predicted and ground-truth tables differ. The implementation of these two scores can be found in Appendix C.

#### 3.3.1 P-Score

We take two approaches to score each metric. First, we perform model-based evaluation, querying GPT-3.5 with both tables and having it score the similarity of content and format separately. Following Wang et al. (2023), we prompt the model to perform Chain-of-Thought Wei et al. (2023) reasoning before outputting its scores, and we query the model with the predicted and ground-truth tables in both orders and average the scores. We report these as the _P-Score_ (Prompting Score).

#### 3.3.2 H-Score

In addition, we also implement hand-crafted scoring functions to score the similarity of the tables. Since the tables can be presented in different formats, we implement several heuristics to normalize the tables and to compute their similarity. We use an average of Levenshtein distance and the Ratcliff/Obershelp similarity metric to compute the similarities between strings or data structures. These heuristically normalized metrics are reported as the _H-Score_ (Heuristical Score). The analysis can be found in Appendix A.3.

## 4 Experiments

### Basic Settings

For metrics, we use SacreBLEU, ROUGE-L, BERTScore, BARTScore, and BLEURT metrics as they are all classical metrics to evaluate text similarity, as well as two proposed metrics: P-Score and H-score. qWe evaluate the following models: GPT-NeoX-20B, GPT-3.5, GPT-4, Vicuna-13B, LLaMA-7B, and our finetuning LLaMa-7B. GPT-NeoX-20B, GPT-3.5 and GPT-4 represent the state-of-art performance of current LLMs and Vicuna-13B is another version finetuned on LLaMA, which can reach 90% of the capacity of GPT-3.5. We think these models are strong enough to be persuasive. For the first 4 models, we simply call their APIs from OpenAI or HuggingFace to generate results without further finetuning. In our dataset, each item consists of three parts: instruction, input, and output. When generating results, we put each item's instruction and input together as the final input to models. During inference, the user provides the prompt in natural language, this can be for a simple task (e.g., "please generate a table given by the following information and format"). During the inference process, we provide the model with a natural language prompt to describe the format

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline Model & SacreBLEU & ROUGE-L & BERTScore & BARTScore & BLEURT & Content P-Score & Format P-Score & Content H-Score & Format H-Score \\ \hline  & \multicolumn{8}{c}{_Tables from Raw Text_} \\ GPPT-NexX-20B & 35.24 & 55.78 & 68.91 & -2.34 & 33.51 & 3.86 & 6.10 & 0.50 & -1.32 \\ GTP-3.5 & 56.92 & 70.97 & 91.35 & -1.68 & 36.85 & 6.19 & 8.16 & 0.52 & -1.27 \\ GPT-4 & 68.13 & 75.44 & 94.89 & -0.99 & 55.24 & 6.88 & 8.30 & 0.85 & 0.53 \\ Vienna-13B & 40.12 & 50.77 & 75.21 & -2.05 & 40.02 & 4.07 & 6.33 & 0.55 & -1.38 \\ Ours-7B & **90.6** & **88.98** & **98.54** & **-0.69** & **66.07** & **7.69** & **8.00** & **1.65** & **3.61** \\ \(w.a.finette\) & 9.9 & 36.56 & 81.63 & -2.50 & 70.24 & 4.58 & 6.00 & 0.51 & -1.01 \\ \hline  & \multicolumn{8}{c}{_LaTeIX_} \\ GPT-NexX-20B & 45.92 & 65.10 & 76.09 & -2.05 & 40.87 & 7.23 & 7.02 & 0.56 & 0.72 \\ GPT-3.5 & 56.94 & 75.99 & 86.25 & -1.30 & 42.89 & 8.22 & 8.41 & 0.99 & 1.27 \\ GPT-4 & 78.15 & 85.34 & 88.07 & -1.09 & **67.11** & 8.78 & 8.81 & 1.10 & 1.35 \\ Vivana-13B & 50.80 & 69.48 & 80.44 & -1.07 & 36.74 & 7.70 & 8.10 & 0.78 & 1.06 \\ Ours-7B & **89.13** & **88.99** & **89.55** & **-0.69** & 66.07 & **8.94** & **9.05** & **1.14** & **1.52** \\ \(w.a.finette\) & 47.24 & 70.89 & 73.27 & -2.13 & 38.13 & 7.10 & 6.98 & 0.51 & 0.69 \\ \hline  & \multicolumn{8}{c}{_HTML_} \\ GPT-NexX-20B & 60.36 & 72.13 & 86.88 & -1.59 & 30.06 & 8.42 & 8.94 & 0.81 & 0.92 \\ GPT-3.5 & 73.80 & 85.19 & 96.76 & -1.46 & 34.81 & 9.11 & 9.35 & 1.10 & 2.15 \\ GPT+4 & **79.25** & 85.95 & **97.22** & -1.31 & 41.59 & 9.17 & 9.62 & 1.15 & 2.29 \\ Vacuum-13B & 58.75 & 70.37 & 88.65 & -1.58 & 31.11 & 8.55 & 8.88 & 0.79 & 0.93 \\ Ours-7B & 77.50 & **86.08** & 96.25 & **-1.30** & **42.89** & **9.20** & **9.70** & **1.18** & **2.49** \\ \(w.a.finette\) & 65.30 & 78.24 & 88.12 & -1.57 & 32.78 & 8.22 & 8.81 & 0.92 & 0.96 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Automated evaluation results on the test set, involving five types of previous metrics and four proposed ones. \(w.o.finette\) means that we also compared the performance of our model without finetuning as an ablation study. ‘Ours-7B’ is finetuned LLaMA.

 

and content of our task, as well as the expected response.

### Human Evaluation

Table 3 displays human evaluation results on two proposed metrics with instance-level Pearson correlation, reflecting a purposeful design that caters to the specific demands of structured output assessment. We engaged five undergraduate students to annotate 200 examples focusing on content and format quality. Equipped with the input description (with reference appended) and generated outputs, they scored each aspect on a 10-point scale. Both the P-score and H-score showcase a significant correlation with human judgment, indicating their relative robustness and effectiveness in this evaluation space. This level of correlation, which surpasses that of many prior meta-evaluation efforts Fabbri et al. (2020); Tang et al. (2021), reinforces the value of our metrics and addresses concerns about their ability to reliably reflect human evaluation. Additionally, we evaluated well-known metrics including ROUGE-L, BERTScore, BARTScore, and BLEURT. Limited space precluded a full discussion, yet our Content P-score showed the best instance-level correlation.

### Results

Table 2 provides a comparative analysis of different LLMs based on several metrics. For 'Tables from Raw Text', the Ours-7B outperforms the other models in every metric. Interestingly, without fine-tuning, the performance drops significantly, particularly in SacreBLEU, ROUGE-L, and BERTScore. The results for 'LaTeX' reveal a similar trend and in the 'HTML' category, GPT-4 scores the highest in SacreBLEU and BERTScore. However, these differences are slight and our 7B model comes out on top for the rest of the metrics. The results demonstrate that our approach exhibits superior performance, highlighting the efficacy of fine-tuning smaller models in surpassing much larger models. Moreover, we delve into an analysis based on our Murk annotation, attributing observed shortcomings to several error types. And we present an ability map in Figure 5 and Appendix E.

## 5 Conclusion

In summary, our study provides a thorough analysis of LLMs' challenges in structured table generation, introduces novel evaluation metrics, and assembles a specific benchmark covering a range of data types. We pinpoint key issues including content fidelity, format adherence, numerical reasoning, and management of extensive tables.

Figure 4: An exemplification of our benchmark. The input is made up of the instruction and the input text, whereas the output aims to present the target table. Notably, there are some inaccuracies in the predicted output; for instance, ‘Points in 4th quarter’ under ‘Hawks’ should be vacant, and correspondingly, ’Points in 4th quarter’ for ‘Magic’ should be 21.

\begin{table}
\begin{tabular}{l c c} \hline \hline
* \\ Format P-score & & 0.3812 \\ Content H-score & 0.1059 &
- \\ Format H-score &
- & 0.3021 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Human evaluation results.

Figure 5: Visualization of LLMs’ capability.

 



## 6 Limitations

Although we present a comprehensive analysis, the exploration of LLMs in structured text generation presented in this paper has several limitations:

Investigating Optimal Format for Tabular RepresentationIn this study, we did not investigate which table formats are most effective. Different presentations of the same information can be reasonable, and **table normalization** strategies, such as determining the best way to tabulate given facts or how to interconnect multiple tables, remain unexplored. Future research could engage in the study of table normalization to ascertain optimal strategies for tabular data structuring and representation.

Domain-Specific Benchmark DevelopmentWhile we've made strides in constructing benchmarks for structured text generation, it may be beneficial to develop benchmarks that cater to specific domains. Different fields might have unique structural requirements and understanding these nuances can significantly improve the models' applicability across diverse contexts.

Expand the Range of DatasetsThere are endless data types and sources that can be explored. Incorporating a broader variety of datasets could expose the models to an even wider range of structural formats, ultimately enhancing their overall performance.

Enhancing Numerical Reasoning CapabilitiesOur study identified inadequate numerical reasoning as one of the challenges faced by LLMs. Investigating techniques to bolster numerical reasoning in these models could lead to significant improvements in their performance.

Developing Advanced MethodsWhile our structure-aware instruction tuning method showed promising results, more sophisticated techniques could be developed. For instance, future work could explore ways of incorporating more explicit structural information into the model or developing methods that allow the model to learn structural patterns more effectively.

Exploring Multimodal LLMsAs LLMs continue to evolve, there are opportunities to explore multimodal models that can process and generate both text and other forms of data, such as sound or images (Kamigaito et al., 2023), in a structured manner.

## References

* S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, A. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. (2022)Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745. Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1573-649X, Link, Document Cited by: SS1.
* W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. (2023)Vicuna: an open-source chatbot impression gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023). Cited by: SS1.
* A. R Fabbri, W. Kryscinski, B. McCann, C. Xiong, R. Socher, and D. Radev (2020)Summeval: re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626. Cited by: SS1.
* Y. He, J. Hu, and B. Tang (2023)Revisiting event argument extraction: Can eae models learn better when being aware of event co-occurrences?. arXiv preprint arXiv:2306.00502. Cited by: SS1.
* H. Kamigaito, K. Hayashi, and T. Watanabe (2023)Table and image generation for investigating knowledge of entities in pretrained vision and language models. arXiv preprint arXiv:2306.02115. Cited by: SS1.
* D. Kocetkov, R. Li, L. Ben Allal, J. Li, C. Mou, C. Munoz Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries (2022)The stack: 3 tb of permissively licensed source code. Preprint. Cited by: SS1.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2019)Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. Cited by: SS1.
* T. Li, Z. Wang, L. Shao, X. Zheng, X. Wang, and J. Su (2023)A sequence-to-sequence&set model for text-to-table generation. arXiv preprint arXiv:2306.00137. Cited by: SS1.
* Y. Ma, Y. Cao, Y. Hong, and A. Sun (2023)Large language model is not a good few-shot information extractor, but a good reranker for hard samples!. arXiv preprint arXiv:2303.08559. Cited by: SS1.
* N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. Le Scao, M. Saiful Bari, S. Shen, Z. Yong, H.



Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. _arXiv preprint arXiv:2211.01786_.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis. _arXiv preprint arXiv:2305.15334_.
* Pietruszka et al. (2022) Michal Pietruszka, Michal Turski, Lukasz Borchmann, Tomasz Dwojak, Gabriela Palka, Karolina Szyndler, Dawid Jurkiewicz, and Lukasz Garraczek. 2022. Stable: Table generation framework for encoder-decoder models. _arXiv preprint arXiv:2206.04045_.
* Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatqgtr a general-purpose natural language processing task solver? _arXiv preprint arXiv:2302.06476_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Rossiello et al. (2022) Gaetano Rossiello, Faisal Chowdhury, Nandana Mihindukulasooriya, Owen Cornec, and Alfio Gliozzo. 2022. Knowgl: Knowledge generation and linking from text. _arXiv preprint arXiv:2210.13952_.
* Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Elie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, et al. 2022. Bloom: A 176bparameter open-access multilingual language model. _arXiv preprint arXiv:2211.05100_.
* Tang et al. (2021) Xiangru Tang, Alexander R Fabbri, Ziming Mao, Griffin Adams, Boriu Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. Investigating crowdsourcing protocols for evaluating the factual consistency of summaries. _arXiv preprint arXiv:2109.09195_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
* Wang et al. (2023) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023a. Large language models are not fair evaluators.
* Wang et al. (2023) Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023b. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. _arXiv preprint arXiv:2305.13412_.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Deny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models.
* Whitehouse et al. (2023) Chenxi Whitehouse, Clara Vania, Alham Fikri Aji, Christos Christodoulopoulos, and Andrea Pierleoni. 2023. Website: Faithful and robust information extraction on the web. _arXiv preprint arXiv:2305.14293_.
* Wiseman et al. (2017) Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. _arXiv preprint arXiv:1707.08052_.
* Wu et al. (2022) Xueqing Wu, Jiacheng Zhang, and Hang Li. 2022. Text-to-table: A new way of information extraction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2518-2533, Dublin, Ireland. Association for Computational Linguistics.
* Zha et al. (2023) Liangyu Zhao, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, et al. 2023. Tablegpt: Towards unifying tables, nature language and commands into one gpt. _arXiv preprint arXiv:2307.08674_.
* Zhao et al. (2023a) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023a. A survey of large language models. _arXiv preprint arXiv:2303.18223_.
* Zhao et al. (2023b) Yilun Zhao, Haowei Zhang, Shengyun Si, Lingong Nan, Xiangru Tang, and Arman Cohan. 2023b. Large language models are effective table-to-text generators, evaluators, and feedback providers. _arXiv preprint arXiv:2305.14987_.
* Zhao et al. (2023c) Yilun Zhao, Chen Zhao, Linyong Nan, Zheting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. 2023c. Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations. _arXiv preprint arXiv:2306.14321_.
* Zhong and Chen (2020) Zexuan Zhong and Danqi Chen. 2020. A frustratingly easy approach for entity and relation extraction. _arXiv preprint arXiv:2010.12812_.

 

Analysis with Examples

### Example Table A

The main difference between the reference tables and the tables generated by GPT-3.5 and GPT4, shown in figure 6, is in the completeness and precision of the data provided.

In the reference tables, all relevant data is fully represented: For the teams (Table 1), each team has a precise number or percentage for every statistic. Similarly, for the players (Table 2), each player has a definite number for every statistic, including minutes played in the format "mm:ss".

In contrast, the generated tables show data that is incomplete and imprecise. For GPT-3.5 generated one, the team statistics table has some statistics missing, as represented by empty cells, and some are not presented as percentages. The player statistics table also has missing data similarly, and it lacks the "minutes played" statistics entirely. For instance, in the 'team' table, the "Percentage of field goals" column for the Suns is missing. Similarly, in the 'player' table, many key statistics such as "3-pointers attempted", "3-pointers made", "Field goals attempted", "Field goals made", and "Minutes played" are missing for various players. Regarding the format, we observe a lot of format errors. For example, the 'Percentage of field goals' column for Grizzlies is represented as "50" instead of "50.0%". Moreover, the 'Wins' column for the Suns is represented as "3" instead of "0". This misrepresentation can lead to significant misunderstanding of the data. The 'Player' table also has format errors. For instance, the 'Minutes played' column is missing the time format (i.e., "00:00"). On the other hand, the reference tables adhere to a standard format. Percentage data is represented with a '%' sign, time data uses the '00:00' format, and numeric data correctly represents each statistic.

For Vicuna-13B results shown in figure 7, although it has the correct format for both tables, there are still many element errors. For instance, the 'team' table has wrong statistics such as "Losses" and "Win" for the Suns. Besides, in the 'player' table, many cells shouldn't have data. However, they have, which is a mistake. Some cells like Isaiah Thomas's and Eric Bledsoe's 'Assists' should be 2 and 4, but they are not in the Vicuna-13B 'player' table. Similarly, LLaMA2-7B results, have the same element errors in the 'team' table and worse errors in the 'player' table. It fills all cells, many of which should be none. As for some cells that should have data, their data are wrongly filled in like Eric Bledsoe's 'Assists' and 'Field goals made'.

 

The Geigerian (50) used a strong second half to evaluate the Sun's -2 102 - 9 in Phoenix on Wedensely night. Memphis sent itself behind six satellite hit sharing but collected Phoenix 30 - 19 in the third quarter and 26 - 20 in the final period. The Geigerian shot 50 percent from the field, led by strong performances from Courtney Lee and Malik Ccole Ley noted 22 points (9 - 14 FG, 4 & 5 JR), while Comley called all servers with 24 - 14 FG, 3 - 4 JRP) and 11 assists. Marc Gaison added 18 points, asiss, and five rounds. The Suns, who beat the Lakes 112 - 106 on Tuesday, were placed by 23 points (9 - 12 FG), five rounds for four assets from Erie Belkose. It was a quiet night for Gorn Drake, who scored just six points in 26 minutes. The third member of the backoutron, Irish Thomas, had 15 points and two assists of the bench, which Markief Morris add 20 points apart for they. The Gratier out - rebounded Phoenix 37 - 35 and outsored the Suns in the point 46 - 32. Memphis also registered 25 assists compared to only 13 - on 32 feild goals: for the Sans. Memphis now beads to Oklahoma City to take on the Thunder on Friday. Phonis, meanwhile, host the Kings on Friday.

Figure 6: Examples of generating tables with GPT-3.5 and GPT-4 based on the input text. The generated results contain a large number of errors, including format errors and content errors.

 

The Grizziess (50) used a strong second half to evaluate the Sums (3 - 2) 102 - 91 in Phoenix on Wednesday night. Memphis found itself behind six at halftime but outscured Phoenix 30 - 19 in the third quarter and 26 - 20 in the final period. The Grizzzzies shot 5 percent from the field, led by stop stop performances from Country Lee and Male Conley. Lee scored 22 points (9 - 14 FG, 4 - 3 yr), which Compley led as letters with 24 - 14 FG, 3 - 4 37b) and 11 assists. Marc Gaosd added 18 points, six assists, and five rebounds. The Suns, who beat theakers 112 - 106 - Ons Pasudios, were posed by 23 points (9 - 12 FG), five rebounds and four assistants form Erik Belodow. It was a quiet night for German Dragie, who scored just six points in 26 minutes. The third member of the backcourt tire, Isaiah Thomas, had 15 points and two assists of the bench, while Markiert Morris added 20 points and three rebounds. The Grizzies out - rebounded Prentisx 37 - 35 and outsored the Suns in the paint 46 - 32. Memphis also registered 25 assits compared to only 13 - on 32 field goals - for the Sams. Memphis now heads to Oklahoma City to take on the Thunder on Friday. Phoenix, meanwhile, hosts the Kings on Friday.

\begin{table} \end{table}
Table 1: Team Summary

Figure 7: Examples of generating tables with Vicuna-13B and LLaMA2-7B based on the input text, the generated results contain a large number of errors, including format errors and content errors.

 

The Grizzlies (50) used a strong second half to outlast the Suns (3 - 2) 102 - 91 in Phoenix on Wednesday night. Memphis found itself behind six at halftine but outscored Phoenix 30 - 19 in the third quarter and 26 - 20 in the final period. The Grizzlies shot 50 percent from the field, led by strong performances from Country Lee and Mike Conley. Leo scored 22 points (9 - 14 FG, 4 - 5 3Pt), while Conley led all scores with 24 (9 - 14 FG, 3 - 4 SPt) and 11 assists. Marc Gasol added 18 points, six assists, and five rebounds. The Suns, who beat the Lakers 112 - 106 on Tuesday, were paced by 23 points (9 - 12 FG), five rebounds and four assists from Eric Bledeso. It was a quiet night for Gorman Dragic, who scored just six points in 26 minutes. The third member of the backcourt trio, Isaiah Thomas, had 15 points and two assists off the bench, while Markieff Morris added 20 points and five rebounds. The Grizzlies out - rebounded Phoenix 37 - 35 and outscored the Suns in the paint 46 - 32. Memphis also registered 25 assists compared to only 13 - on 32 field goals - for the Suns. Memphis now heads to Oklahoma City to take on the Thunder on Friday. Phoenix, meanwhile, hosts the Kings on Friday.

\begin{table} \end{table}
Table 1: Team Reference

Figure 8: Using GPT-4 to generate a table based on the input text without FormatCoT, the generated results contain a large number of errors, including format errors and content errors.

 



### Error Type

Structure Errors:These errors pertain to the structural integrity of the generated tables. Specifically, they include instances where there are excess or missing rows or columns in comparison to the correct table structure. For instance, in figure 8 GPT4 generated result has missing columns like "Win" and "Losse" in the 'team' table.

Structure Naming Errors:This category captures errors related to the naming conventions used for rows or columns. Any discrepancies in a row or column names between the generated and correct table are flagged as structure naming errors. For instance, in figure 8, the GPT-4 generated result has wrong column names like "Half-Time Score" in the 'team' table.

Element Errors:These are inaccuracies observed at the element level within the generated table. Element errors encompass incorrect numbers, values, or inappropriately empty cells, reflecting discrepancies in individual table entries relative to the correct table. In figure 6 and figure 7, most errors are element errors.





### Metric Effectiveness

Table 4 crystallizes different results of examples in Appendix A.1 based on our H-score metric. Taking GPT-4 as an example, there exist slight drops from 2.0 to 1.86 in content H-score, and very accurately, this trend is followed by the results of GPT-4 revealing slight errors in its content. For the format H-score, GPT-4 also cannot do very well for its numeric performance, which is not close enough to the full score, matched by the poor performance of GPT-4 in this area. Other models' H-scores follow this trend as well. Therefore, the H-score can respond to differences in both content and format accurately.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **Content H-score** & **Format H-score** \\ \hline GPT-3.5 & 1.39 & 4.0 \\ GPT-4 & 1.86 & 3.44 \\ Vicuna-13B & 1.51 & 4.0 \\ LLaMA2-7B & 1.37 & 4.0 \\ Ours-7B & 2.0 & 4.0 \\ Reference & 2.0 & 4.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: H-scores for different results of Ours-7B, GPT-3.5, GPT-4, Vicuna-13B and LLaMA2-7B.



DTurk

To facilitate a comprehensive analysis of the LLM output, we designed a task on Amazon Mechanical Turk (MTurk) to gather detailed annotations about various error types encountered in the generated structured tables. The task was structured as follows:

From the RotoWire dataset, we randomly chose 100 instances of LLM-generated output, ensuring a representative mix of quality based on preliminary assessment.

Each Human Intelligence Task (HIT) presented the annotators with a side-by-side view of the LLM output and the expected structured table format. Annotators were instructed to identify and categorize errors according to predefined types: 'Element Errors', 'Element Format Errors', 'Structure Error', and 'Structure Naming Errors'.

We provided extensive guidelines, exemplified with step-by-step instructions, to clarify typical instances of each error type. These guidelines were reviewed and iteratively improved through a pilot study conducted with a small set of annotators.

About the qualifications of Amazon Mechanical Turk (MTurk) workers, we use the following qualifications to recruit in total of 10 MTurk workers with good track records: HIT approval rate greater than or equal to 98%, number of HITs approved greater than or equal to 500, and located in one of the following English native-speaking countries: Australia, Canada, New Zealand, United Kingdom, United States. Each annotator is limited to annotating 10 examples, including both the output of GPT-3.5 and GPT-4.

Annotators workers were compensated $7, calibrated to equal a $42/hour pay rate. We first annotated examples in-house to determine the required annotation speed. A summary block usually takes around 10 minutes.

To demonstrate our annotation template and facilitate future research, we show the interface for annotations.

Figure 10: Our Interface of Murk.

 

Scoring

### P-Score

Our approach involves prompting the model to engage in Chain-of-Thought reasoning before issuing its scores. Firstly, we instruct GPT on how to evaluate both "content similarity" and "structure similarity". Following this, the model is guided on the correct procedure to output its answer. To calculate the scores, the model is queried with both the predicted table and the ground truth table in varying sequences, after which the scores are averaged. We'll illustrate this process using the P-Scores prompt for raw text tables as an illustrative example:

_"Based on the above, we wanted to determine if the above tables are similar. Ideally, they should have identical content and structure. Score the "content similarity" and "structural similarity" between 0 and 10._

_- Content similarity: 10 if the contents of the table cells are identical, 0 if they are entirely different. If about 50% of the cells have the same data, the score should be 5._

_- Structural similarity: 10 if the tables have the same structure (e.g. same column and rows with identical ordering, same alignment, etc.) although text formatting differences can be ignored (e.g. colors, font)._

_Output a JSON object such as the following: """json [{  [  "contentsimilarity": ...  "structuralsimilarity": ...  ]} """ This carefully, and then output the scores."_

For instance, in figure 11, both tables have identical structures, so their structural similarity score is 10. The contents of the first table of Table1 and Table2 are the same, and the second table of Table1 and Table2 are almost 10% similar. Therefore their content similarity score is 5.

### H-Score

We attach our algorithm to calculate H-Score as Algorithm 1.

```
LaTeXWe use the pylatexnc library to parse a given LaTeX table, and walk through the parse-tree structure in the tabular environment to identify the table "cells". We score the content similarity based on strings within the cells, and score structural similarity based on having the matching number of rows and columns, the same caption, and the same cell alignment.

#### Html

We use the beautifulsoup4 library to parse a given LaTeX HTML snippet and walk through the parse-tree structure in <table>, <ul> or <ol> tags to identify data cells. We separately build a tree of white-listed HTML tags to score the structural similarity, traversing an HTML document tree structure, disregarding the actual content within the tags and simplifying it by focusing only on specific HTML tags (defined in RECOGNIZED_HTML_TAGS). We score the content similarity based on strings within the cells and score structural similarity based on the similarity of the structure tree and the total number of cells matching.

White-listed HTML tags:
``` RECONIZEQ.HTMLTAGS=[ "table", "tr", "th", "td", "u1", "o1", "div", "span", "p", "a", "img", "embed", "pre", "h1", "h2", "h3", "h4", "b5", "h6", "input", "button", ] ```

Raw Text TablesIn our evaluated dataset, each example consists of two tables (Team and Player). We do a string search for "Team" and "Player" headers to identify the two tables. We then parse the tables according to Markdown formatting, with newlines and pipes as row and column dividers respectively, to identify the table cells. We score the content similarity based on strings within the cells, and score structural similarity based on the similarity of column names and the number of rows and columns matching.

String Similarity Measurement:Our script includes methods to calculate the similarity between two strings. These methods can be used to compare the structure or content of HTML, latex documents, or any other pair of strings. The similarity is evaluated using well-established algorithms in text analysis: the Levenshtein distance and the SequenceMatcher from Python's difflib module 

Figure 11: An example of P-score calculation. The top two figures displays two tables, accompanied by a prompt below to calculate the content and structural similarity between them. The goal is to determine if the tables have identical content and structure. The content similarity will be scored based on the percentage of cells with identical data, ranging from 0 (no similarity) to 10 (complete similarity). The structural similarity will be scored 10 if the tables share the same structure, including columns, rows, and alignment. The output will be a JSON object with the calculated scores.

 

```
Data: predictions \(\alpha\), references \(\beta\) Result: Content H-score \(\gamma\), Format H-score \(\delta\)
1FunctionLevenshteinSimilarity(p, q): ifq is emptythen return0.0
2 end if
3\(s\gets 1\) - LevenshteinDistance(p, q) / (2*length(p)) returns FunctionDifflibSimilarity(p, q):
4\(s\leftarrow\) difflib.SequenceMatcher(None, p, q) returns
5\(\gamma\gets 0\)
6\(\delta\gets 0\) for\(\sigma,\phi\) in \(zip(\alpha,\beta)\)do Parse \(\sigma\) to get its number of columns \(\lambda_{c}\), number of rows \(\lambda_{r}\), column names \(\mu\) and data rows \(\kappa\) Parse \(\phi\) to get its number of columns \(\theta_{c}\), number of rows \(\theta_{r}\), column names \(\tau\) and data rows \(\omega\)\(\rho_{1}\leftarrow\textbf{if}\;\lambda_{c}==\theta_{c}\;\textbf{then}\;1.0\;\textbf{else}\;0.0\)// Whether their number of columns are same
7\(\rho_{2}\leftarrow\textbf{if}\;\lambda_{r}==\theta_{r}\;\textbf{then}\;1.0\; \textbf{else}\;0.0\)// Whether their number of rows are same
8\(\rho_{3}\leftarrow\textbf{LevenshteinSimilarity}(\mu,\tau)\)// Compute their columns Levenshtein Similarity
9\(\rho_{4}\leftarrow\textbf{DifflibSimilarity}(\mu,\tau)\)// Compute their columns Difflib Similarity
10\(\rho_{5}\leftarrow\textbf{LevenshteinSimilarity}(\kappa,\omega)\)// Compute their data Levenshtein Similarity
11\(\rho_{6}\leftarrow\textbf{DifflibSimilarity}(\kappa,\omega)\)// Compute their data Difflib Similarity
12\(\gamma\leftarrow\gamma+average(\rho_{5},\rho_{6})\)
13\(\delta\leftarrow\delta+average(\rho_{1},\rho_{2},\rho_{3},\rho_{4})\)
14 end while
15\(\gamma\leftarrow\gamma\) / length(\(\alpha\)) \(\delta\leftarrow\delta\) / length(\(\alpha\))
```

**Algorithm 1**The provided algorithm is the H-score algorithm with pseudocode. It aims to calculate the content similarity (Content H-score) and format similarity (Format H-score) between predictions and references. The algorithm begins by defining functions for Levenshtein similarity and Difflib similarity to compute string similarities. It then iterates over each pair of predictions and references, parsing their structures and data to determine the similarity in terms of the number of columns, rows, column names, and data rows. The final step involves averaging the similarity scores to obtain the Content H-score and Format H-score.

 

Prompt for FormatCoT and Inference

### Prompt for FormatCoT

Raw Text Table DescriptionTraditional data-to-text datasets only have raw text for each table. However, it is not enough for GPT-3.5 or other LLMs to generate correct tables. As a result, we added some format descriptions to help them generate the correct tables. We use GPT-3.5 to achieve this. We want to get detailed format information without concrete contents in cells, so we explicitly include these requirements in the prompt. Here is our prompt: "Describe details about the given text. First, give the number of tables, and then for each table, describe its format such as the number of columns and rows, column names, and row names."

HTML Table DescriptionUnlike data-to-text datasets, HTML datasets only have final outputs, so we are required to generate a detailed description of their format and content. For content descriptions, we can simply ask GPT-3.5 to output raw text without HTML tags. For format descriptions, however, we need to ask GPT-3.5 to describe each tag, otherwise, it will leave out some tags and describe the table in general rather than detailed information. Moreover, it is necessary to ask it to use specific numbers instead of 'several' or 'multiple'. Here is our prompt for HTML format descriptions: "Describe the format of this HTML in detail according to each HTML tag of the following HTML code. Be careful and make sure don't miss any HTML tags. Please use more than 300 words to explain the format. Use specific numbers rather than being vague about several."

LaTeX Table DescriptionSimilar to HTML prompt generation, it is necessary to ask GPT-3.5 to generate both format descriptions and content descriptions as latex datasets only have final outputs. For content descriptions, we can simply ask GPT-3.5 to describe the given latex table as detailed as it can and include all cells. For format description, since the latex format is too complex, we need to give it a small example to learn. Then we ask GPT-3.5 to describe the detailed format of a given latex table, including specific questions to help it generate format descriptions. Here is our prompt for latex format descriptions: "Describe the detailed format of a given latex table according to the commands and tags with more than 500 words. Include: Whether there is table border lines? How is text alignment? What are table attributes? Whether to bold? Whether to add vref? Please clearly explain whether there are horizontal and vertical lines bordering each row and column. Say anything about a special "\(\backslash\)" format token in latex if there is one. Don't display latex code directly. Use natural language. And provide enough format information for me to recreate this table based on your output description."

### Prompt for Inference

When inferencing raw text tables, LLMs tend to output tabular results rather than raw text tables. As a result, we need to give it an example output first, then tell the model that the input consists of two parts, text and format descriptions, and ask the model to generate the output based on them. For HTML and Latex inference, we can simply ask models to infer from the input and specify the format and content sections in the input, since models can generate correct syntax.

 



## 6 Conclusion

\begin{table}
\begin{tabular}{|c|p{142.3pt}|} \hline  & Describe details about the given text. First, give the number of tables, and then for each table, describe its format such as the number of columns and rows, column names, and row names. \\ \hline FormatCoT Prompt for HTML Table & Describe the format of this HTML in detail according to each HTML tag of the following HTML code. Be careful and make sure don’t miss any HTML tags. Please use more than 300 words to explain the format. Use specific numbers rather than being vague about several. \\ \hline FormatCoT Prompt for LaTeX Table & Describe the detailed format of a given latex table according to the commands and tags with more than 500 words. Include: Whether there are table border lines? How is text alignment? What are table attributes? Whether to bold? Whether to add vef? Please clearly explain whether there are horizontal and vertical lines bordering each row and column. Say anything about a special ”” format token in latex if there is one. Don’t display latex code directly. Use natural language. And provide enough format information for me to recreate this table based on your output description. \\ \hline Prompt for Inference & Based on the example output above, generate the raw text/HTML/LaTeX table according to the following description. \\ \hline  & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 5: Our prompts for FormatCoT and Inference. The prompt requests an overview of table formatting in raw text, HTML, and LaTeX formats, including descriptions of the number of tables, column and row structures, formatting elements, and specific instructions. The prompt for inference illustrates the request to generate the tables based on the provided details.



Ability Map

Based on our automated evaluation, we selected Vicuna, GPT-3.5, GPT-4, and Ours as representative models and conducted an in-depth analysis of the causes of model errors.

We identified content accuracy, formatting, numerical reasoning, and handling of long tables as the main sources of these errors.

At the fundamental level, we decompose the process of model-generated complex structured outputs into two parts: Content Selection and Format Planning. Initially, the model needs to identify key information from a given vast amount of unstructured input, extract this information, understand it, and organize it. Subsequently, it needs to plan how to summarize these extracted details, devise the format of the table to be generated, and then fill in the information.

Accordingly, we can break down the model's capabilities into Coverage, Formating Reasoning, Comprehension, Pragmatics, and Hallucination Control.

Coverage entails the model's ability to accurately cover the content in the input. Formatting Reasoning pertains to judgment about the output format, assessing if the model can find the most appropriate and reasonable structured format.

Comprehension reflects whether the model can understand the content of the input, as there are times when it is necessary to infer from a large amount of data (including performing addition or subtraction or comparing multiple elements).

Pragmatics involves the ability to utilize special formats, such as HTML tags and specific syntax in LaTeX.

Finally, Hallucination Control signifies the model's ability to refrain from generating content not present in the input.

We carried out manual annotations and obtained visualized results to demonstrate these aspects 

Ablation Study for FormatCoT

### Contrast between descriptions

In this section, we conduct an ablation study to examine the impact of our proposedFormatCoT. In the generation of table descriptions sansFormatCoT, we simply utilize the prompt: "Provide a description of the following tables." We display the results in figure 12. The primary differentiation between results pivots on the extent of details incorporated.

For instance, in theFormatCoT result, the description comprises an array of detailed format information - encompassing row names, column names, and table count. The precision in these details proves substantial enough for models to accurately recreate the tables in question.

Contrastingly, the outcome bereft ofFormatCoT conveys considerably less information - providing incomplete column names without the accompaniment of row names. This sparse degree of detail proves insufficient for models seeking to faithfully regenerate the corresponding tables.

### Contrast between results

In this section, we draw a comparison between two sets of description results, shown in figure 13. TheFormatCoT result showcases a table that stands remarkably close to the correct table, albeit with minor errors. It contains an extra row termed "Player" in the initial table, a discrepancy potentially attributable to the fact that the result comprises two tables, with "Player" denoting the header of the subsequent table. We posit that this error could potentially be circumvented with a different method of integrating table names.

Furthermore, an additional column surfaces in the second table, which in reality represents the final row of that table. Besides these minor inaccuracies, theFormatCoT result accurately replicates the content in each cell as well as maintaining the overall format.

Conversely, the alternative result contains multiple errors that span both content and format. Initially, an additional row is present in the first table, introducing an unrelated basketball team that bears no relevance to the game under consideration. Following this, the second table possesses an excessive number of player names, encompassing unnecessary players along with coaches who did not participate in the game.

Furthermore, its content is not entirely accurate, with discrepancies present in the statistics attributed to both Gordon Hayward and Gerald Green. These shortcomings underscore the efficiency and essentiality of implementing theFormatCoT to ensure accuracy and precision.

 

Figure 12: Using FormatCoT and normal instructions to ask GPT-3.5 to generate table descriptions based on the input text, FormatCoT results contain more detailed information about row names.

 

Figure 13: Using two descriptions to regenerate table descriptions based on the input text and descriptions, FormatCoT result is more correct in both format and content.